By Natascha van der Zwan & Andrei Poama
We’re continuing our conversation on research ethics with Dr. Andrei Poama (Institute of Public Administration, Leiden University). Dr. Poama is an expert on the ethics of criminal justice. He is also a member of the Ethics Committee at the Faculty of Governance and Global Affairs. In our exchange with Dr. Poama, we discussed the ethical dilemmas confronting researchers in the social sciences, possible solutions to these dilemmas, and how the codes of conduct for Dutch researchers apply to graduate students. This is Part Two of two blog posts, in which we present the highlights from our conversation.
Please note: The guest talk has been modified to a question-and-answer format for easier reading. The spoken words have been edited for length and readability.*
Q: So, let’s have a look at some of the codes of conduct for Dutch researchers. We have, for instance, the Netherlands Code of Conduct for Research Integrity (2018) and the Code of Ethics for Research in the Social and Behavioural Sciences Involving Human Participants (2018).
A: The Code of Conduct for Research Integrity is the main document, which I think is quite clear and well done. It draws on the European equivalent. But the process is quite weird. So, when you open these codes, you see “oh, there are five leading principles for conducting ethical research.” And these principles are honesty, scrupulousness, transparency, independence and responsibility. And it’s like like “oh!”. You don’t really know where they are coming from and so on.
I’m not going to go through them, but I want you to know what – based on those principles – would count as research misconduct. One of them you already know and you have known about it since you were an undergrad. That’s plagiarism. But then the other two are fabrication and falsification. Fabrication is simply frauding, so fraudulent science, making up data. Falsification is when you have the data but you keep tweaking it, until the data says what you wanted to say. I think many – I wouldn’t say most – but many scientists, especially in the quantitative tradition, engage not necessarily in full falsification, but they keep chasing that p value: rearranging the data and massaging it, until they get statistical relevance. There are very interesting discussions about dropping the statistical significance level. Because the reason why we have the p value today is that you want genuine findings to be published. Now, what has happened because you have this standard [the p value] is that people keep chasing the standard and modifying the data, until the data fits the standard. But by doing so, they modify the data so much, that actually they end up falsifying it, at least to some extent.
Q: So what kind of solutions to these problems are there?
A: There are two things that are very interesting happening today and that are addressing the falsification problem. One of these things is pre-registration. Pre-registration means that you have these online platforms, typically hosted by universities. So, for instance, my colleague Dr. Honorata Mazepus and I have been doing a survey experiment on how the socio-economic status of criminal offenders (whether they are poor or not) affects people’s judgments about whether to blame and punish the offenders. Before you even run the experiment, you go [to the platform] and you submit a document with your hypothesis and with your theory. Then you’re committed to those hypotheses, before you run the experiment. And it’s a requirement, when you submit the findings of the experiment, that you also submit the link with your pre-registered hypotheses […]. And the other thing is the discussion about dropping the p value.
There is novelty or positive findings bias in the way science, and in particular social science but also in medical science, as it is practiced today. Your stuff is only going to get published, if you find something. If you manage to some degree to confirm the hypothesis you’re after. If you found no relationship, so the null hypothesis holds, then no one is interested. One thing happening right now is that you have a few null hypothesis or negative findings journals […].
So that’s also interesting and it’s being debated in the replication crisis that you might have heard of, especially in social psychology. You also find it in management studies. Basically, only about 30-40% of social psychology studies are being replicated. So the remaining 60-70 % is not being replicated.
Q: I think most of our students would probably either do a non-experimental survey or qualitative interviews rather than an experiment. So how would falsification play into [those methods]?
A: You can falsify anything, really. You could also falsify the findings of a survey. There are different ways of falsification. You can say, for instance, my sample is the students in the Masters or the undergraduate [program], but then you also make your Qualtrics survey link available to family and friends. So you might have people who are not part of the sample or population that will be part of [it], but you just don’t say it. And there are ways of checking that, but I don’t think [anyone] is realistically going to do that.
Q: What I find a little bit tricky with interviews is, when you want to use quotes from the interview and you have to polish the language a little bit, because you’re not going to type in all the ‘uhs’ and the ‘ahs’. So you would have to tweak it a little bit. I’m wondering what the fine line is between this acceptable editing on the one hand and then falsification on the other hand.
A: I mean, it’s hard to say. I think if you change the meaning, then then you’re clearly in the wrong. The way it happens with research misconduct, for example, is that there are always clearly cases of wrongdoing. So, for instance, this case would be a clear case of falsification. And there are of course very clear cases, where you would just simply report the data and the data is of high quality. There is no interview or no open-ended question, so then you don’t have to engage too much with the interpretation. There are other cases in-between. My metric there is that if you see yourself interpreting the data in a way that tends to confirm your hypothesis – if you’re too friendly to your hypothesis, to put it that way – then that’s a red light. You should try to be as uncharitable to your hypothesis as possible. Your job is to try to falsify the hypothesis.
Q: We have a question from a student, who is doing research on co-production. This involves sitting in on meetings with citizens. So how do you prevent, that you falsify or misinterpret the observations you make?
A: You can have a reflection on your interpretation, so you get at this meta-level where you basically say “well, this is what I’ve been doing and these are the weaknesses of my interpretation and these are the things that I’m not sure about.” Another thing that you should be doing is that you write after you have your observation moment. If you postpone it and do it two days later, you’ll have all sorts of memory deception effects that will be kicking in. That will be a problem. So just do it afterwards.
Q: A lot of the examples of ethical breaches are really big breaches. I think for a lot of us, who are trying to do our research as ethically as possible, these big examples are not always that useful. Because, you know, we are not going to fake respondents. But sometimes those smaller dilemmas are actually the most difficult ones. You’re on the Ethics Committee in our faculty, so I was wondering what are some of the most common issues that you observe and that we could learn from?
A: Well, they often have this kind of structure (see slide below). So this is a made-up example, which is partly based on my supervision experiences. You know, students would do stuff similar to this. Imagine that one of your colleagues wants to test five hypotheses about the impact of socio-economic inequalities on educational opportunities in the Netherlands. To test these hypotheses, he plans to interview three teachers from a low-income neighborhood in The Hague. He comes to you to ask for some research advice about how to proceed with the study. The question is, what do you advise him? Is there an ethical problem with his research?
Q: In the discussion, our students quickly noted some issues with this research design. The researcher only selects a low-income neighborhood as his case study, instead of selecting multiple neighborhood that vary in terms of average income levels. This constitutes selection bias. The researcher also aims to test five hypotheses, based on only three interviews. This is known as the ‘degrees of freedom’ problem. But these seem issues of research design, not of research ethics per se. So why should we question this study in terms of research ethics?
A: Many of the cases that individual researchers submit with us [the FGGA Ethics Committee] are like this, because of our The Hague mission […]. Put yourself in the shoes of either the teacher or the students [in the low-income neighborhood], who have given these interviews. Then, you know, the researcher is sending you the article and says “oh look, here are the findings.” So what does that does that do to you as a teacher or as a student?
Many of the problems that we receive on the Ethics Committee have this kind of structure. You know, we want to draw very general conclusions based on a very small sample, because we don’t have a lot of data. But I think that one thing that you can do as individual researchers is to be very critical about the scope and range of your conclusions. Be very critical about the fact that this is not going to apply to the whole population. If you, as a part of this population, are reading about this research, especially as a teacher, I would imagine that basically in encounters with other teachers you will feel lower, less important, kind of responsible for this happening. And so I think one of the frequent problems that we do have is this stigmatization effect or potential for stigmatization.
Q: So what can you do?
A: I think there are two things that we could also do as individuals. One is to take charge of the science communication process […]. It’s often not the scientists themselves or the researchers themselves who are doing the communication about the findings (unless you’re talking about Twitter or Facebook), it’s someone else. And I think one thing that we could do is to take hold of the communication process. So to actually present that data ourselves, because we have more nuance in the way which we present. Of course, to do that, we would need more time, and time is very scarce in academia today.
And the second thing is… I will just give this example. I have my second Masters in criminology and I had this amazing teacher who was doing participant observation on offenders convicted for domestic violence charges. The way she was doing it was through interviews and just sitting there and observing things. And then she wrote her article thing and used, you know, fancy nice academic language. But then before actually sending the article for publication, she did one very interesting thing: she took the manuscript and sent it back to the prisoners. So when the article got published, the title was “being a nosy bloody cow”, because that was the reaction of one of the inmates.
So one thing that you can do, especially if you see there is a potential for stigmatization, is to promise to give voice to your participants. And if you’re real about giving voice, you can do that in the actual content of your research products.
Q: To sum up?
A: So two things. One, you are not a student, you are the actual author of the research that you’re going to produce. And the participants that you’re going to work with are in some sense co-authors of that work. So don’t be shy about what you did. And two, if you already have a draft, you can send it back to the participants, to the people who have generated knowledge for you. Do it, especially if you are doing qualitative research, because that is a way of giving people some control over what you’re going to say about them.
There is nothing fixed about those five principles. Principles don’t apply in an obvious way across cases. Even between the principle and the case, there is this thing called judgment. You have to exert your judgment about a) whether the principle applies at all and b) how and to what extent the principle is going to apply. So one obvious principle that would apply [in the earlier example] is scrupulousness, that you show care in the way that you produce knowledge, gather knowledge and disseminate it.
Q: Thank you, Andrei, for sharing your thoughts on research ethics with us. And thank you to our students for their insightful questions!
*With thanks to Brecht, Edo, Meike-Yang and Nev for their insightful comments and questions.
Natascha van der Zwan 