Opportunism in research

When I was in seventh grade I joint the youth-group of a political party. Approximately two years later we had the opportunity to meet with the leader of that party to “discuss”(1) the further directions said party could or should take.

The leader of that party had been of the view that the party should takes some positions of the green party, not of reasons related to the content of the positions, but because the green party had dropped these positions.

While I could understand that getting votes is important for a political party, I found that too opportunistic and left: I felt that you should argue for things bases on the arguments (and counterarguments) you have not based on popularity.

I changed a lot of my views, but I still believe that. However, maybe I understand better that it’s not always easy not to be opportunistic, because opportunisms seems to be what brings people forward. The questions is whether it brings you where you originally wanted to go.

I hope that research is primarily about learning new things about the world, but it’s also politics: Should you point your results just a little brighter than they really are? How much, if any, opposing literature should you cite? Change that effect size estimate a little to increase the chances of funding? Present only your positive sides in a CV?

Some people may say that none of these things are ok, ever, under any circumstances. I tend to agree, but then, how long do you search for counter evidence? For how long do you check if things aren’t worse than you think? Living, breathing, consuming is inherently opportunistic, isn’t it? Working is. If you can’t work in research, you’ve to work elsewhere, and I don’t know if selling burgers at McDonalds would be any less opportunistic?

So the question may be where to draw the line. That however is a difficult questions, especially when a lot of current problems in research may stem from people taking every chance (to get money, to publish, to stay in research) with no regard to the long term consequences… (2)

(1) Of course I know that such “discussions” are just for publicity/popularity as well.

(2) I don’t know if I could say what I want to say. Basically my question is: What is ok and what isn’t and how should we decide? It would be easy to say, all but perfect honesty is wrong, but a) perfect honesty is difficult to achieve, because you can never tell everything that *might possibly* be relevant and b) I think it would put most people out of research pretty quickly. Some people may say, the perfectionism shouldn’t be to goal, but then the question is where to draw the line? What is ok and what isn’t?

Having a story to tell...

In a TV-documentary I once saw there was a prosecutor who basically said (smiling): “Yeah, I know that he fitted the criteria for insanity, everyone knew, it was so obvious. I didn’t think we’d succeed with our strategy but we had to build this *narrative* that he was sane.” And she seemed to be totally fine with it, she did her job and won, so that’s something to celebrate, isn’t it? Well, I don’t know. While she succeeded in her job, shouldn’t trials be about evidence and justice instead?

In science, I think this is even more complicated because there isn’t necessarily a representative for every side of an argument. While there of course is peer-review (which serves that purpose), peer-reviewers (most of the time) can’t know what is not reported, what is not part of the narrative that the authors tell. They don’t go and dig failed preliminary studies or replications out of the authors’ file-drawers; they don’t search for evidence against the authors claims.

Therefore I think that in research it is even more important that the full story, not just the prettiest element of it is told. However, that is not what we are taught. In 2015 I attended a Summer School for PhD-students. It was very interesting and I learned a lot, however we also were told that we have to tell a story, report the aspects that are most convincing and interesting to tell. While I understand and agree with some parts of that (data can be shown in confusing or less confusing ways) I think that this advice is mostly understood (and probably meant) as “out of the many things you could tell about your study/your data, you’ve to pick the most interesting part and choose the aspects of the data that support it, while ignoring the rest”. I think that this is wrong.

What should you *ask* at an interview for a PhD position? (Part 1)

There are a lot of advice books out there which tell you what to answer to all those mean questions that you might be asked at a job interview.

They might be more or less useful. However, they never tell you what to *ask* in order to find out if a PhD-position is or isn’t a good one. (At least I haven’t found any book that tells that. And I read a lot of these books before applying to my PhD-position because relying on my social skills would have been foolish.)^[1]

But don’t worry, I’ll tell you ;).

Of course this list is subjective, but maybe it can give some ideas about what could possibly be relevant to someone who is looking for a PhD-position. Because at the time I didn’t ask any of these things. I was just glad they took me... :/

This first part is related to questions about the project you are applying for... further parts will be related to the supervisor/professor and the group you are (potentially) going to work with and the working environment.^[2]

Most often when you apply for a PhD-position in a specific project a short description of the project is given in the job-advertisement. But this description is generally very short and doesn't tell much at all what it's really about.

Therefore I think the supervisor or whoever does the interview should introduce you more to the topic, the background theories and how the specific question is planned to be answered.

With how I don't mean answers like "with fMRI" or "with TMS" (what most likely the job-description already tells), but a more specific explanation of the planned study design, so that you can (try to) judge if that is realistic to accomplish or not. For that purpose it would be useful to get a copy of the project-proposal. It would spare the supervisor time and you could assess by yourself if you think it's realistic. Yes, in theory any funded research project has already been through a selection process during which, among other things, the feasibility of the research idea should have been evaluated. However, that is the theory. It seems to me, that in practice no one in these selection committees does some basic reality checks. Therefore you’ve to do it by yourself. ^[3]However, a lot of researchers might not like to give their proposal out because they might fear that you steal their brilliant ideas. That is fine^[4], but in this case I still think they should give enough background information to make judgements possible.

Of course it is possible that the study design is not yet developed (though I think at least a rough study design should be part of the proposal). I think that that should be mentioned as it involves more uncertainty then.^[5]

So whether or not you get the research proposal, what are the critical points to check?

First, I think it is important information which population should be examined. Patients with rare disorders might be difficult or impossible to recruit for participation in a demanding experiment.^[6] That does not mean that such research should not be carried out. I doubt that difficulty in recruitment is related to the importance of the research, but it should be taken into account: You might end up with a very long recruitment phase and/or a very small sample size. So be prepared to conduct an underpowered study...

The other important factor is how many participants should take part in the study in total in comparison to the measurement time per participant and the available lab/MRI time. In general, the longer the experiment the lower the change to get enough time to do the measurements. Also take into account that one day has 24h available hours at maximum.

As next, I would consider number of methods to be learned. It takes time to learn to analyze an fMRI design, but there might be much more you have to learn. You might have to do additional MRI-recordings and analysis.^[7] Furthermore you may have to or want to run additional recordings in parallel to the MRI (like skin conductance, heart rate, breathing, or even EEG) or an additional part of the study might be done with EEG, TMS, eye-tracking, virtual reality or something like that. It might be exciting to learn a lot of methods; but it takes time. Therefore I think it is an important point to take into consideration. A finished PhD with just one or two methods is probably much more worth than an unfinished PhD with 10 half-learned methods. Of course a number of factors might influence how hard it is to learn various methods. I’ll speak about that in a later post.

It might also be important to ask if the project is already running and if so at which stage it is. I'm not saying it's good or bad if the project is already running, but it might be important information. You likely won't be involved in the experimental design then, but on the other hand data (to publish!) might be available faster (and: it's running!). The important question then is, though, to whom the data "belong", i.e. who is allowed to publish with them? This is a question that can lead to huge fights in some groups.

Speaking about already running experiments, I’d furthermore say that it is "dangerous" to start with any new method that has not been applied by anyone in the department/institute before. Even if it should work in theory (it surely should, otherwise no one would use it, right?) it might be that the hard- or software used at the institute is not compatible to what you (or the research proposal) plan. If you can check the compatibility, do so.

Also, whether you got the proposal to read or not, try to read background studies on the subject. If you got the proposal, you can find them by yourself, if you didn’t get it I think you should ask for them. The professor might tell you, that it is sufficient if you read them once you got the position (which may sound nice), but I think that is kind of too late.

Though it's still possible to terminate a PhD later (i.e. once you have enough insight into the project) it's harder because of the time you've invested.^[8]

So try to find out if the background studies are sound^[9], especially if it are background studies from the same department.^[10] And if you have to do a direct or indirect replication of work done previously by your supervisor, run away.^[11]

P.S.: Sorry for my english. I deleted about half of my text (bc too long), so I hope it makes still some sense...
And for the next post I'll complain about which handy feature the English language is missing.

P.S. 2: I really hope this is not misunderstood as advice on how to get an "easy" PhD. It's not meant to be! 

---

^[1] [Though, reading those books was already a good start into wasting a lot of time since I was not asked any of those weird questions during the job interviews (for what I should be glad). It seems social skills are not a selection criteria*. Obviously those questions might also be pretty unvalid and unreliable, especially if more people are like me and just learn the “correct” answer from a book, but considering all the unreliability of research I doubt that that is why they are not used to determine who’s a good candidate. *They might be success criteria though. More on that later.]

^[2] [This distinction is a bit artificial, but I didn't want to write an even longer post than I already wrote. Especially since not everything applies to everybody.]

^[3] [Although it would of course be better if a selection committee of experts did that and not a student who hasn’t even started a PhD-training yet.]

^[4] [I guess in some areas it's more realistic than in others.]

^[5] [If in need for a list what can go wrong, contact me ;) ]

^[6]  [For animal-studies it might be that some animals are difficult to get. For example mice with a specific gene modification. But I don’t work in animal-research, so I don’t know much about that.]

^[7] [Like, for example, morphometry, DTI (diffusion tensor imaging), functional connectivity (e.g. PPI), resting state analyses (e.g. ICA), MR spectroscopy, ASL (arterial spin labeling), or another fancy (new) method.]

^[8] [Time-investment sound way too nice... I don't want to scare anyone away from a PhD. Just choose the project carefully. Otherwise it might be more than just time.]

^[9] [or not less so than an “average” study]

^[10] [Which would not be surprising because it makes sense to build new work on previous work, but it can lead to a lot of trouble when they are... not good.]

^[11] [I think more replications should be done, because every study can have positive results just by chance. Together with a publication bias towards positive findings, this means that it is hard to know which findings to trust. However, I think that replications should be independent. Or preregistered with signed contracts that any result will get published from all parties involved.]

A little workshop on data-fabrication...?

I saw something in a paper which didn't make much sense to me... Of course it is also possible that I’m just stupid and overlooking explanations... so I’ll write here what I found and you can decide on your own.

This is a table I reproduced from that paper. The reason I don’t simply make a screenshot of the table in the paper is that I could be false. And in that case I don’t want to falsely accuse the paper (or its authors) of error. 

So I tried to provide some anonymity to the paper… (for now).

Pre are measurements taken before... something... and post afterwards (if you didn't guess it). 

The same people were tested in pre and post and the sample size was 21.

What surprised me were the tiny standard deviations for some of the Variable 1 and 2, especially in combination with the range given.

So I tried to find some numbers that would fulfill the criteria given (mean, standard deviation and range) for these 4 (or 8) variables. First my "results" for the first four variables pre:

As you can see I managed to find values (for each participant) that add up to the given mean for every variable. (I should note here that the third and the fourth variable are ratings on a numerical (not visual) rating scale, so values other than other than X.0 and X.5 might not be so likely.) 

I also tried to get the SD as much to the given value as possible. You see what I was able to get. 

(SD excel is the SD calculated by excel and SD me is the SD calculated by me with help of excel, i.e. I set up the formula myself cos I didn’t know what formula excel was using. I didn’t calculate it by hand though, because there would be no chance to get a correct result that way.)

The SDs (dark blue row) I got are much higher than the given SDs for the first two variables (variable 1 pre and 2 pre).

I don’t know, obviously there’s a chance that I’m just dumb and didn’t do it correct. 

In order to get a lower SDs than those I got, I then replaced these numbers (for 2 to 20) with the mean of the first and the last value. I can’t change the first and the last value, because they are given in the table. Obviously, when I do that the mean is not "correct" anymore, but I should get the lowest possible SD (min and max value fixed) with that, shouldn't I? 

Apparently, for the first two variables the SD is still a lot higher than the given SD. (For the third and fourth variable I was able to get the same (very close, but I could have tried longer) SD as given in the table anyways.)

In the next table you can see that this is similar for these variables in post:

Just for the variables 1 and 2 (pre and post) I couldn’t find any values that met the criteria. 

So I thought maybe the authors inadvertently reported the SEM instead of the SD. Maybe it is possible to call the SEM SD because after all the SEM is a measurement of deviation which is standardized too.

Therefore I tried to get the SEM to be the value reported for the SD:

That is possible for the first to variables (pre and post) but I didn’t manage to do that for the third and fourth variable.

 

Maybe they reported the SEM for Variable 1 and 2 and the SD for Variable 3 and 4. I don’t know. The other possibility is that I made a mistake, obviously.

Anyways… in the paper they did t-tests with these variables. So I tried to do that too… Obviously I don’t know in which order the values were… so if I just take my fabricated values I get a far off t-value.

Sorry for my SPSS speaking German, but it isn’t important anyways (as it is obvious that this gives the false t-values):

At least I saw some significances in that output window for once… :( 

And mean, SD and SEM are about the same as in my Excel calculation.

Fortunately the paper tells me which t-values I ought to get:

So I calculated what the Sum of the (distance to the mean)² of the differences between the two conditions (pre and post) needs to be in order to get 

* the "correct" mean difference between the samples 

* and therefore also the "correct" t-value.

Here’s a table for that:

Then I tweaked my… why not call them "data" ("data" not: data) to fit these criteria.

Although I was not able to reproduce exactly these numbers, it went quite well and I got pretty close. 

Obviously there are a lot of solutions, especially if I don't meet the SD-criteria for each Variable as specified in the table (I could have tried for the 3rd and 4th variable, but I didn't).


Here (one possible set of ) the individual "data"  I used to get this values:

Btw, I gained some data-fabrication knowledge doing this (which I had not previously had!! All my data are honestly p-hacked :( :( [1]). Want to hire me?

SPSS too thinks that the new t-values are much better:

I know this might seem like a really small point... since the t-values are all possible and hopefully correct. I don't know. It's relevant for effect-size estimation, for example. 

I can be false (just to say this again, because maybe I'm not seeing something really obvious).

Another point I noticed is that they report a mean of 0.36 (SD=0.29) in some questionaire data. I can't say anything about this, since I don't know their participants personally, but seriously... I doubt it.

I found this graph which represents the answer-distribution for a non-clinical population for either this scale or a quite similar one (it's not clear to me from the paper; there is a long and a short version):

The reported mean for that scale (out of that other study) is 9.4 with a SD of 7.83). My participants (healthy controls) have a mean value of 12.4 (with min=4, max=19, and SD=3.5). Of course the mean can be lower (then there) in different populations at a different time and if especially people with probably low values in that questionaire are recruited. But 0.36 seems really low to me.

I don't know if they did some transformations on that or whatever, but it is not reported.

Oh, and even though I obviously didn't report a lot of varibles/values from that study I didn't omit a control group... it's just pre and post. 

(Again, I don't exclude my stupidity at all. Sorry for the repetition.
I thought very long about whether or not to post this, but I think the SD and the mean of the questionaire are too low...)

---

[1] Obviously p-hacking is bad too! I’m well aware of that. I don’t even know what is worse…

edit/addition: 
1. I didn't want to imply that the data were fabricated. I don't think so!! The title refers to what I did in this post. I think it wouldn't make much sense if the data (of the SD-variables) were fabricated, because even then they should have an SD that is possible (if I'm right that the one they report is not possible with that mean and range).  (And the reported t-value is possible with much higher SDs.)
For the mean on the questionnaire-data I think it is most likely that it is either a mistake, the data were transformed (though that is not mentioned) or the participants were influenced.
And while I'm at it, I think there could be two other possible mistakes in that paper. One is relatively clear (because they report the study protocol in a slightly different order at different points in the paper) and the other it is not certain imo.

2. Christopher at the neurotroph-blog did some R-simulations on the Standard Deviations. You can read his blog-post here.