Opportunism in research

When I was in seventh grade I joint the youth-group of a political party. Approximately two years later we had the opportunity to meet with the leader of that party to “discuss”(1) the further directions said party could or should take.

The leader of that party had been of the view that the party should takes some positions of the green party, not of reasons related to the content of the positions, but because the green party had dropped these positions.

While I could understand that getting votes is important for a political party, I found that too opportunistic and left: I felt that you should argue for things bases on the arguments (and counterarguments) you have not based on popularity.

I changed a lot of my views, but I still believe that. However, maybe I understand better that it’s not always easy not to be opportunistic, because opportunisms seems to be what brings people forward. The questions is whether it brings you where you originally wanted to go.

I hope that research is primarily about learning new things about the world, but it’s also politics: Should you point your results just a little brighter than they really are? How much, if any, opposing literature should you cite? Change that effect size estimate a little to increase the chances of funding? Present only your positive sides in a CV?

Some people may say that none of these things are ok, ever, under any circumstances. I tend to agree, but then, how long do you search for counter evidence? For how long do you check if things aren’t worse than you think? Living, breathing, consuming is inherently opportunistic, isn’t it? Working is. If you can’t work in research, you’ve to work elsewhere, and I don’t know if selling burgers at McDonalds would be any less opportunistic?

So the question may be where to draw the line. That however is a difficult questions, especially when a lot of current problems in research may stem from people taking every chance (to get money, to publish, to stay in research) with no regard to the long term consequences… (2)

(1) Of course I know that such “discussions” are just for publicity/popularity as well.

(2) I don’t know if I could say what I want to say. Basically my question is: What is ok and what isn’t and how should we decide? It would be easy to say, all but perfect honesty is wrong, but a) perfect honesty is difficult to achieve, because you can never tell everything that *might possibly* be relevant and b) I think it would put most people out of research pretty quickly. Some people may say, the perfectionism shouldn’t be to goal, but then the question is where to draw the line? What is ok and what isn’t?

A (very) critical "review" of CBT: individual psychotherapy

Now, to say at least something nice about the CBT-program which I did: The psychotherapists were nice, kind and, I think, understanding. However, in the time I was there I had four different therapists and that’s not because I didn’t get along with one or the other, but because they have contracts which suck as well: Their contract ran out, so they switched to a different position. 

Therefore I obviously couldn’t really discuss anything with them in depth, due to the time constraints. However, that wasn’t too bad, because I wouldn’t have known what to discuss anyways. I told them about the struggle with my PhD, but I feel like twitter is the better place to discuss that, because the details are of course difficult to understand for people who don’t work in research.

So, it was nice that they were sympathetic and at least pretended to understand how I was feeling, but I’m not sure that did anything to help me with depression, though I do think it possibly made staying at the hospital (which is not the nicest place to be) a bit easier. But then, on the other hand, they try to figure out why you are depressed, i.e. which live events made you depressed. Maybe I’m just not good at arriving at hypotheses (regarding that) in which I feel some sense of certainty (though of course you fundamentally can’t know), but I don’t know why this happened to me. My working environment might be a candidate (for a cause) but I really don’t know, I liked my work and I liked what I was doing (mostly), I feel it would be unfair to blame that entirely. 

But then, psychologist come up with their own hypothesis, like maybe your parents avoided and ignored you as a child. Of course they state it as a hypothesis, not a fact. They barely know you, let only your parents, so how could they? But still, a hypothesis that is spoken out is something you think about, is it true, is it not true, how can I know whether it’s true or not, I only have my memories and not the ones of other children and no one did an experiment on my family (well, I hope ;)) to figure out if I was treated worse than other children. So how could I possibly know? And, I dunno, but I think this can be the side-effect of psychotherapy: That you search for a cause, because there has to be one (hasn’t there?) and start to reinterpret (normal or maybe not so normal) events in the light of that hypothesis. (Depression certainly can help with that…). And I just don’t know if much is gained with that; if I’d think of my parents as bad parents (I don’t!) how would that help me (other than avoiding the awkward moments when I’ve to explain that I don’t know how it happened)?

A (very) critical "review" of CBT: music therapy

Even though the psychotherapy program I did for depression was called "cognitive behavioral" it had some elements which I’d classify as psychodynamic, namely the "art-therapy" and the "music-therapy". Luckily I got rid of both of them pretty fast.

In the art therapy we had to do stuff like drawing our feelings. Surely you can make up a visual metaphor for everything, but to me it didn’t make a lot of sense, nor did I want to provoke the therapist with blank papers. The music-therapy however was a lot stranger than that, it didn’t have really anything to do with music, even though that’s what it was called. Well, to be fair, I could have played an instrument, the therapist had asked if I wanted, but I didn’t want. So she thought up her own therapy for me, which consisted of figuring out how many parts you (I) have and naming them. Since I told her, that I am, fortunately or not, only one person, who – as everyone else – of course has different personality traits or aspects but no different parts, she did the work for me and defined and named different parts. I don’t remember all of them, but one was the self and a supposed other one was a waiting person. She wrote these "parts" on individual pieces of paper and asked me to lay them down on the floor. So I did; I placed the stack of papers on the floor. Of course my stupid part didn’t get *how* she wanted the pieces to be laid down; I had to put them besides each other, kind of like a mind map. But even when I did that, it wasn’t right. The therapist didn’t feel that I’d placed them the right distances apart, so she asked me to step on one piece of paper while she stood on another and asked me if that was a comfortable distance. It was not, I don’t like standing face to face with another person, without any distance. But I’m sure this didn’t have to do anything with the pieces of paper on the floor. So I told her about my suspicions about cause and effect here. And that’s how I got rid of that therapy :).

To be honest it is quite mysterious to me why the health insurances even pays for a therapy like that. They don’t if you are not inpatient, but if you are in a psychiatry they pay for all kind of stuff. However, there were other patients (though not many!) who said that it helped them. But then, if it’s not evidence based (and it isn’t by the insurances own criteria; otherwise they had to pay it on an outpatient basis as well), you don’t know what else might have helped them….

A (very) critical "review" of CBT: mindfulness

In previous posts I told you that I did an inpatient CBT program for depression. I described the activity planning group here and the relaxation classes here. Today I’d like to share my experiences with the mindfulness group.

I doubt that these classes were according to any protocol, so I don’t want to extend my opinion on it to everything that’s called mindfulness. In the beginning of (almost) every class we were told the mindfulness or pleasure rules: Pleasure needs time, needs to be allowed, is individual… and the rest I don’t recall despite all the repetition. After that we had to focus on one sense, seeing, hearing, tasting, touching, and smelling. For example we were given an item in the hand with closed eyes and had to touch it to see how it feels, or we had to walk through the room and focus on the different colors we could see. I hope you get the point.

Now, I think it this *may* be *some* distraction, but it doesn’t really help. The mechanism by which it should help is not clear to me. When I’m feeling well, this would probably be an interesting task, but when I’m not it’s not. When I’m fine I like a lot of little things, an accidental splash of color on my water bottle, packaging foil (the sounds and looks of it), trees (they just look interesting and come in so many different forms), and lots, lots more. But part of depression is loss of pleasure and it doesn’t come back with just seeing things you might otherwise find beautiful or interesting. I can well describe the sensations (the roughness or softness of a material, the temperature of it, etc.) but it doesn’t mean anything. Chocolate tastes different than cardboard, but it is the same feeling. It’s just not nice. It’s effort. In some way I think the focus on what should be nice but isn’t is quite depressing in itself. It is like they try to tell you “but look, the world is nice, you’ve just to focus on that” but it doesn’t feel, look, take, sound nice.

Having a story to tell...

In a TV-documentary I once saw there was a prosecutor who basically said (smiling): “Yeah, I know that he fitted the criteria for insanity, everyone knew, it was so obvious. I didn’t think we’d succeed with our strategy but we had to build this *narrative* that he was sane.” And she seemed to be totally fine with it, she did her job and won, so that’s something to celebrate, isn’t it? Well, I don’t know. While she succeeded in her job, shouldn’t trials be about evidence and justice instead?

In science, I think this is even more complicated because there isn’t necessarily a representative for every side of an argument. While there of course is peer-review (which serves that purpose), peer-reviewers (most of the time) can’t know what is not reported, what is not part of the narrative that the authors tell. They don’t go and dig failed preliminary studies or replications out of the authors’ file-drawers; they don’t search for evidence against the authors claims.

Therefore I think that in research it is even more important that the full story, not just the prettiest element of it is told. However, that is not what we are taught. In 2015 I attended a Summer School for PhD-students. It was very interesting and I learned a lot, however we also were told that we have to tell a story, report the aspects that are most convincing and interesting to tell. While I understand and agree with some parts of that (data can be shown in confusing or less confusing ways) I think that this advice is mostly understood (and probably meant) as “out of the many things you could tell about your study/your data, you’ve to pick the most interesting part and choose the aspects of the data that support it, while ignoring the rest”. I think that this is wrong.

Who's responsible for flawed studies?

In his blog post about case of data-fabrication in which the (alleged) fabricator was not included as an author, Neuroskeptic notes that „Authorship means responsibility“. I agree!!

In this case, the grad student who collected and manipulated the data remained (so far) unnamed, because he is, rightly or not, not listed as an author. But while it should be clear that authors have to take responsibility for what is published under their name, I think that responsibility reaches further.

In any complex study naturally a lot of people are involved. Of course it’s not the fault of the person at Siemens who wrote the scanning sequence if we then go and use it incorrectly and – in my opinion – neither is it the fault of a supervisor if their PhD-student “cleverly” manipulates the data.(1) After all it’s not kindergarten; not everything can be checked.

But it should be (though I know that it not always is) by “surprise” if an aspect of a study turns out to be problematic after publication: You shouldn’t be able to know prior to publication that, say, the statistics are somewhat dubious; papers with fishy statistics shouldn’t be published. However, it might turn out later on that “consistent signals” may just be artifacts produced by the imaging techniques - and the fact that brains need a constant supply of blood. In such cases it would be useful to know which sequence was used and how – exactly - the resulting data were analyzed. To figure that out the people who chose the sequence or did the analyzes should be known – which in the current mess of unregistered studies is much more likely if they are (co-)authors. Of course they also deserve to be on the paper if they had (substantial) work with the study but for them to become important at some point it doesn’t really matter how “scientific” their work was. The purely technical aspects can be just as important: How long did it take before blood was frozen, how (exactly) where the patients immobilized in the scanner, which arm was the blood pressure apparatus on, did the optic cables reliable transfer the signal and how was that made sure, …? One might say, that all these kind of information should be part of the publication but often it isn’t.

Another point is which responsibilities people involved in research have beyond their specific duties. While I generally think that the responsibility doesn’t extent (much) above ones area of expertise, I do think that everyone does have the responsibility not to knowingly do or support unethical stuff – and any unjustified or flawed study is unethical in my opinion. In this sense funding agencies share a great deal of responsibility and, once known, the media are also responsible for reporting scientific misconduct. However, of course it can be difficult to judge which studies are unethical or flawed.


(1) If the data are manipulated “uncleverly” I do however think that the supervisor has some responsibility because he or she should be able to “see” that.

(Unfortunately English is still not my native language. I'm sorry about that!)

A (very) critical "review" of CBT: activity planning

I have been doing a cognitive behavioral therapy (CBT) program for depression. In yesterdays post I described the relaxation classes we had to do as part of the program. Today I’d like to talk about the “activity planning” group.

The group was based on the notion that actions, feelings and thoughts influence each other. Therefore – we were told – we just had to get doing (pleasurable!) stuff again in order to feel better and have more pleasant thoughts. We were handed a list of over 200 potentially pleasurable activities and had to do one such activity alone and one together with other people for each class. Each such activity furthermore had to be rated on a 1 to 10 rating scale in the dimensions “mood”, “drive” and “ease of mind” before, during and after the respective activity.

While I understand that it is important to get doing stuff – cos otherwise with absence of or low drive you’d just lie in bed and eventually die of thirst (or lack thereof) – I find that it is asked a bit much, that those activities should be perceived as pleasurable. For me, it just made me feel even more like a loser, because I couldn’t get any pleasure out of going for a walk, even though I tried. (“But it’s nice out here… I just have to tell myself… It’s nice… Sun and wind and…. But it sucks so much, why do I have to do this… I don’t have the energy, I’ll never arrive at the other end of the street…. And how do I know if it’s nice, the world is f*cked pretty much. It’s NOT nice. Climate change, and plastic in water, air pollution…. Why do I have to walk here, it’s so exhausting…. And the food we get here is wrapped all in plastic…. Which pollutes the seas….. But the river is NICE…. I’ve to tell myself…… But I don’t want to lie, not to others and not to me…… it’s not nice…. But maybe liars are the better people, at least they make others happy and aren’t such failures….. I hate this.” And so on….).

But what’s more important than that potentially nice activities didn’t seem to work for me is how to group was lead: It was instructed by whichever nurse had duty that day and they seemed to have very different opinions about what constitutes a pleasurable activity (independent of the list). While some accepted everything others had very specific ideas and would even get mad at you if you did an activity which they didn’t find pleasurable, for example tidying up or learning something (as an activity alone) or skyping and chatting or tweeting (as an activity with others). In their mind you had to get a massage or take a bath or color in something as a nice activity alone and an activity with others had to involve more than virtual contact (going to the cinema or a cafe would do). Now, I don’t find tidying up that pleasurable either, but at least something’s done afterwards, i.e. there’s less mess. If I’d get a massage I wouldn’t enjoy it and my place would be just as messy afterwards, so therefore nothing would be gained. Probably I’d feel guilty for spending the money on something I didn’t even enjoy, and like a failure for not enjoying something which should be enjoyed. Therefore I prefer tidying up – it has the better consequences. However, as I said, not everybody shares that opinion and while of course everybody is entitled to theirs I find it unhelpful decry the activities of people who tried their best to do (and enjoy...) them anyways. But of course not every nurse was like that, some didn’t really give a shit (so that the group ended within 5 minutes after everyone had recited their ratings) and others were trying to be helpful and encouraging (e.g. by suggesting in which other situations the said activity could be helpful or how the activity itself could be improved or what could be done instead).

Overall, is it helpful having to be doing a “pleasant” activity? Personally, I don’t think so, because it’s just not pleasurable and not feeling anything nice made me feel bad. *Just* doing some stuff, without any aspiration to finding it pleasurable *might* be helpful: Not eating or not drinking (water ;) ) doesn’t either, nor does not showering or not brushing teeth etc. But, in my opinion, it can be unhelpful to hope for any pleasure doing these things. Because if that’s the point and it’s too far away you might as well just not do it.

A (very) critical "review" of CBT: Progressive Muscle Relaxation

I did an inpatient CBT program for depression (as a patient) and I’d just like to express my opinion about it (because in a lot of ways it sucked … and apparently I’m not well enough to fill out their feedback form, so…). Of course it’s just that, just my personal opinion and just based on personal experiences at one time and at one place. Other people will have different experiences.

Whatever…

The program consisted of relaxation, activity planning, a mindfulness-group, music (and/or art) an “Info-Group” and of course the CBT itself (in a group and individually with a therapist).

In the relaxation group we did Progressive Muscle Relaxation according to Jacobsen (PMR). There you have to tense a specific muscle (like you hand, your upper or lower arm, your foot, etc.) for a few seconds and then relax it according to instructions. Then you should focus on the difference between the tension and the relaxation as well as the difference between (say) the left and the right hand. This relaxation technique is popular in Germany (I don’t know about elsewhere) however it does not work for me. (I tend to tension and relax my hands/fingers a lot anyways, especially when I'm not well, but I don't do it according to PMR guidelines; so therefore during the lessons I had to (try to) stop that.)  That’s not their fault, obviously, however I had to do the relaxation class during the entire time of my stay there, even though it had – if any – the opposing effect on me. Most of the time the instructor read the instructions from a piece of paper, but occasionally they just played a tape. Still, the fact that you had to go there may be helpful for people for whom the PMR works but who don’t have the motivation to do it on their own, when they are not forced to. (I find it hard to find any empirical evidence that it works for depression, though.) 

However, I think they vastly overestimate the amount of people from whom it works, because they did not accept any kind of negative feedback. At the end of each lesson there was a feedback round where you had to say how you are now and most people stated that they felt more relaxed. However, if you stated that you weren’t relaxed you still had to make up something which supposedly was better (which reminds me of research). My room-mate suggested I’d just say I’d me more relaxed to have my peace (which REALLY reminds me of research).

However, the one thing that was helpful about the PMR for me was having the appointment (where I had to go) itself; it structured the day a bit. But over all, I had preferred if I hadn’t had to go there.

I may or may not describe the other parts as well. 

Don’t know yet.

My dreams 2016

Hope you don't get nightmares...

1.

I have to do an MRI experiment, of course. For that purpose I have to poke my participants with needels, lots of them, cause it’s fun to investigate suffering. But – oh shit, I think - the needels are magnetic. They’ll slice my poor participant into thousand pieces while being scanned. I lean into the MRI-scanner to my screaming participant and try everything to grap as many needels as many needels as I can reach but this hurts my participant even more. 

Versions of this dream include me trying to convince my colleagues that they can’t put the participant in the scanner prior to him being sliced up or added fun with two participants simultaneously. 

2.

My supervisor wants me to build an inside fantasy world on top of the research building. With parks and waterfalls and stuff. And of course an airport, cos s/he is busy so how else should s/he get there. I keep thinking that that’s impossible, an airplane can‘t land inside the top floor of a building. Furthermore the groundplan of that planned top floor is much greater then that of the building itself. I mention that, but my supervisor just says that if I’m too dump for that I shall just ask the experts. Unfortunately I’m having a hard time finding experts on top story fantasy word building. However, to my surprise one day the the site is finished and it looks good. However, I know that it isn’t, I know that it’ll turn into a disaster and of course I’m right. The plane flights through a tunnel, and which is connected to that top floor to land on the airport and first it looks like it might just work out but then there’s a wind or something, the plane bounces from one site of the tunnel to the other and everything falls into pieces. 

Versions of that dream include a lot of stress because the human recouses depart has disabled my card to enter the building and especially the top floor or me going for a check in the fantasy dream world before my supervisor visits and discovering plies of dead (park-) workers in a sideway. 

3.

I try to get somewhere. Home, to work, where ever. But of course I don’t manage to get there. 

I may take a plane, which flies way too low between the buildings of a random city. The plan has a steering wheel and somehow, I’m the one who has to control it and try to avoid hitting any buildings. But it doesn’t react to what I do. Furthermore I have to run back and forth between the uncontrollable steering wheel and the back of the plane where people want something from me, but I don’t understand what, so they get angry and everyone screams at me. It’s windy, too. Sometimes strong winds hit the plane from street canyons, and the damn steering wheel doesn’t react. Eventually the plane crashes, with its nose into the asphalt of a street. People are not pleased with that.

Or, I may try to take a train (maybe a good idea after all these crashed planes, one would think). But of course, it doesn’t work out. Purchasing a ticket can be a complicated matter, especially in (my) dreams. The ticket machine may calculate a ridiculously long journey with several overnight stays at scary rail stations for an actually short distance. Or the ticket machine explodes when I try to purchase my ticket. Or, what could be a nice dream but isn’t, the ticket machine gives me back way too much money. I don’t know what to do with it i.e. how to handle the situation and get the money I falsely received back to its rightful owner(s). I can’t leave it there, because then obviously someone will take it. I can’t take it myself because I don’t have enough pockets and I’m clumsy and drop it everywhere. While I try to pick one coin up again I lose 10 others. People chase me for the money. I forget the rest in the ticket machine. My trains don’t arrive and the prediction on which track they’ll arrive changes constantly. After some chasing over some tracks I decide just to wait on the track I’m at till my train arrives there. (Good plan, no? :D). But police is after me for the money I just wanted to bring to the lost-properties office. (Another great plan, right? …. That reminds me of a real situation in Munich where I found a 50 Euro note in a train (S-Bahn) station on the moving stairs. I was so confused by that, that I turned to the person next to me and asked him if it was his…..). I get arrested, my attempts to explain myself fail.

Alternatively I have a ticket, but I need to bring my little sister with me and like always the announced tracks change constantly. We chase through station hall after station hall without getting anywhere. I realize we need to get underground but the underground part of the station is all flooded. It’s supposed to be like that, it’s just me and apparently my sister who can’t breathe under water. Not only that, but there’s also an underwater obstacle course which we have to solve in order to get somewhere. Unfortunately I fail and my little sister drowns.

I spare you the versions of that dream in which I accidentally kill my grandmother, my father or other random people.

Sleep well.

Bullshitting.

A few weeks ago Richard Morey wrote a blog post^[1] about how undergraduate students in psychology learn to bullshit (i.a.) by criticizing peer-reviewed papers (which are difficult to criticize because they shouldn’t contain any severe and easily detectable errors anymore). 

When I was studying we didn’t have to write any essays or to criticize any studies. So maybe I didn’t learn about the art of bullshitting well enough. However, I believe that the problem that bullshitting is rewarded is even greater than that: Guessing an answer in an exam and bullshitting a bit around is almost always better than stating that you don’t know. 

My first MSc-exam was in Clinical psychology. I think, I had learned really well^[2], but as this was an important – and also basically^[3] my first oral – exam, I was very nervous. The first question the professor asked me was: "Why is the prevalence of major depression increasing?" And I didn’t know! I didn’t even know it was increasing! But if s/he asked that, that must be, right? So I mentally went through all the text-books I’d worked through, searching for the part where they explained *why* more people today are depressed then in the past. I mean, it could be anything, or nothing. More divorces, different foods, more or different industries, different expectations or stressors, different lifestyles? Vaccinations? Exams?^[4] But my professor hadn’t asked for some hypotheses or my opinion but *why* that is. And I didn’t know! So I told him/her so, while I kept searching mentally in the text-books and all my notes. His/her next question was *if* I *knew* the theories for depression. As I was still desperately trying to come up with a better answer than "I don’t know" for his/her first question, I just said “yes”, because, well, I knew. At this point, I think s/he possibly felt pranked, because apparently s/he wanted me to name (or explain) those theories not be told that I knew them. Upon clarification I of course explained the theories to her/him, but I had real trouble understanding what s/he wanted from me throughout the whole exam.

It occurred later, much later, to me that s/he probably wanted to ask a "nice" first question whereby my task had been to bullshit something together, despite not knowing. Who knows *if* and *why* the prevalence truly is increasing and *if so* compared to when? 

I don’t blame that professor for my inability to understand my task correctly, so please don’t take me wrong. Of course I could have said: I don’t know… for this and that reason… one would need to know this and that to answer this question. Certainly that would have been much better, but I thought she wanted an answer to the question and not some justification why I don’t have it. 

A lot of people will say "Yeah, but you have to think beyond the facts and combine all the knowledge you have. Everybody can learn some facts so there’s no value in echoing some books without thinking about it by yourself!" And it's true, it's certainly an advantage to be able to think even during – or especially during – exams. But then at least frame the question like it. (E.g. "I'm interested in your opinion/thoughts. Let’s assume the prevalence of major depression was increasing. What could be possible factors that may – or may not – contribute to that? And how could one test if those factors actually do contribute to an increase in the prevalence? What would possible problems in testing those hypotheses be?") Otherwise it invites to bullshitting.

 

But then, on the other hand, bullshiting seems to be an essential skill in many aspects of life… maybe that is why they select by that….

[Sorry if that post is to bullshitty itself, obviously it’s just my opinion (and experience) and no knowledge… Also sorry for language-errors.]

---

^[1]  Link: http://bayesfactor.blogspot.de/2016/04/how-to-train-undergraduate.html 
^[2] Or at least A LOT! Probably much more than what was good for me… 
^[3] Apart from a tiny part of my final school exams. But that was just a tiny part and therefore not nearly as important.

^[4] Not really.

"I should sell tickets to everyone who wants to see my patient"...

A few weeks ago a colleague of mine, who is a psychotherapist, told my colleagues and me about a patient of his, because he needed some help with questionnaires and other things. 

He had to treat this patient because of our supervisors wish. However, he personally thought that his patient was totally crazy proceded to explain why. Due to his explanations some other people wanted to see the patient, and he joked he should sell tickets and get rich by showing the patient to others. When he left he mentioned that it’s just not possible to have empathy with someone like that, because he can’t put himself in the patient’s position. Everyone agreed.

While this may be the most insulting (that’s my opinion, obviously not the opinion of others) thing I’ve ever heard here about a patient, it happens quite regularly that people are making fun of or laughing about patients (not everyone! But some people.). Mostly (again, my opinion) for really minor things like talking too much or misunderstandings. I’ve always felt really uncomfortable with that.

However, what irritates me is that he mentioned that he just can’t have empathy with the patient because of the patient’s behavior. That may be. I can’t understand the patients behavior either, but my understanding was, that that is (in part) why it is a disorder and not just perfectly reasonable and understandable human behavior?

Sometimes I get the feeling that psychologist as well as doctors just "understand" situations (behaviors, thoughts, …) that seems to some degree reasonable to them, and, at best, perfectly fit the diagnostic criteria for a disorder.

I don’t understand any better. But I don’t understand the slander either.

What is your opinion on this?

What should you *ask* at an interview for a PhD position? (Part 1)

There are a lot of advice books out there which tell you what to answer to all those mean questions that you might be asked at a job interview.

They might be more or less useful. However, they never tell you what to *ask* in order to find out if a PhD-position is or isn’t a good one. (At least I haven’t found any book that tells that. And I read a lot of these books before applying to my PhD-position because relying on my social skills would have been foolish.)^[1]

But don’t worry, I’ll tell you ;).

Of course this list is subjective, but maybe it can give some ideas about what could possibly be relevant to someone who is looking for a PhD-position. Because at the time I didn’t ask any of these things. I was just glad they took me... :/

This first part is related to questions about the project you are applying for... further parts will be related to the supervisor/professor and the group you are (potentially) going to work with and the working environment.^[2]

Most often when you apply for a PhD-position in a specific project a short description of the project is given in the job-advertisement. But this description is generally very short and doesn't tell much at all what it's really about.

Therefore I think the supervisor or whoever does the interview should introduce you more to the topic, the background theories and how the specific question is planned to be answered.

With how I don't mean answers like "with fMRI" or "with TMS" (what most likely the job-description already tells), but a more specific explanation of the planned study design, so that you can (try to) judge if that is realistic to accomplish or not. For that purpose it would be useful to get a copy of the project-proposal. It would spare the supervisor time and you could assess by yourself if you think it's realistic. Yes, in theory any funded research project has already been through a selection process during which, among other things, the feasibility of the research idea should have been evaluated. However, that is the theory. It seems to me, that in practice no one in these selection committees does some basic reality checks. Therefore you’ve to do it by yourself. ^[3]However, a lot of researchers might not like to give their proposal out because they might fear that you steal their brilliant ideas. That is fine^[4], but in this case I still think they should give enough background information to make judgements possible.

Of course it is possible that the study design is not yet developed (though I think at least a rough study design should be part of the proposal). I think that that should be mentioned as it involves more uncertainty then.^[5]

So whether or not you get the research proposal, what are the critical points to check?

First, I think it is important information which population should be examined. Patients with rare disorders might be difficult or impossible to recruit for participation in a demanding experiment.^[6] That does not mean that such research should not be carried out. I doubt that difficulty in recruitment is related to the importance of the research, but it should be taken into account: You might end up with a very long recruitment phase and/or a very small sample size. So be prepared to conduct an underpowered study...

The other important factor is how many participants should take part in the study in total in comparison to the measurement time per participant and the available lab/MRI time. In general, the longer the experiment the lower the change to get enough time to do the measurements. Also take into account that one day has 24h available hours at maximum.

As next, I would consider number of methods to be learned. It takes time to learn to analyze an fMRI design, but there might be much more you have to learn. You might have to do additional MRI-recordings and analysis.^[7] Furthermore you may have to or want to run additional recordings in parallel to the MRI (like skin conductance, heart rate, breathing, or even EEG) or an additional part of the study might be done with EEG, TMS, eye-tracking, virtual reality or something like that. It might be exciting to learn a lot of methods; but it takes time. Therefore I think it is an important point to take into consideration. A finished PhD with just one or two methods is probably much more worth than an unfinished PhD with 10 half-learned methods. Of course a number of factors might influence how hard it is to learn various methods. I’ll speak about that in a later post.

It might also be important to ask if the project is already running and if so at which stage it is. I'm not saying it's good or bad if the project is already running, but it might be important information. You likely won't be involved in the experimental design then, but on the other hand data (to publish!) might be available faster (and: it's running!). The important question then is, though, to whom the data "belong", i.e. who is allowed to publish with them? This is a question that can lead to huge fights in some groups.

Speaking about already running experiments, I’d furthermore say that it is "dangerous" to start with any new method that has not been applied by anyone in the department/institute before. Even if it should work in theory (it surely should, otherwise no one would use it, right?) it might be that the hard- or software used at the institute is not compatible to what you (or the research proposal) plan. If you can check the compatibility, do so.

Also, whether you got the proposal to read or not, try to read background studies on the subject. If you got the proposal, you can find them by yourself, if you didn’t get it I think you should ask for them. The professor might tell you, that it is sufficient if you read them once you got the position (which may sound nice), but I think that is kind of too late.

Though it's still possible to terminate a PhD later (i.e. once you have enough insight into the project) it's harder because of the time you've invested.^[8]

So try to find out if the background studies are sound^[9], especially if it are background studies from the same department.^[10] And if you have to do a direct or indirect replication of work done previously by your supervisor, run away.^[11]

P.S.: Sorry for my english. I deleted about half of my text (bc too long), so I hope it makes still some sense...
And for the next post I'll complain about which handy feature the English language is missing.

P.S. 2: I really hope this is not misunderstood as advice on how to get an "easy" PhD. It's not meant to be! 

---

^[1] [Though, reading those books was already a good start into wasting a lot of time since I was not asked any of those weird questions during the job interviews (for what I should be glad). It seems social skills are not a selection criteria*. Obviously those questions might also be pretty unvalid and unreliable, especially if more people are like me and just learn the “correct” answer from a book, but considering all the unreliability of research I doubt that that is why they are not used to determine who’s a good candidate. *They might be success criteria though. More on that later.]

^[2] [This distinction is a bit artificial, but I didn't want to write an even longer post than I already wrote. Especially since not everything applies to everybody.]

^[3] [Although it would of course be better if a selection committee of experts did that and not a student who hasn’t even started a PhD-training yet.]

^[4] [I guess in some areas it's more realistic than in others.]

^[5] [If in need for a list what can go wrong, contact me ;) ]

^[6]  [For animal-studies it might be that some animals are difficult to get. For example mice with a specific gene modification. But I don’t work in animal-research, so I don’t know much about that.]

^[7] [Like, for example, morphometry, DTI (diffusion tensor imaging), functional connectivity (e.g. PPI), resting state analyses (e.g. ICA), MR spectroscopy, ASL (arterial spin labeling), or another fancy (new) method.]

^[8] [Time-investment sound way too nice... I don't want to scare anyone away from a PhD. Just choose the project carefully. Otherwise it might be more than just time.]

^[9] [or not less so than an “average” study]

^[10] [Which would not be surprising because it makes sense to build new work on previous work, but it can lead to a lot of trouble when they are... not good.]

^[11] [I think more replications should be done, because every study can have positive results just by chance. Together with a publication bias towards positive findings, this means that it is hard to know which findings to trust. However, I think that replications should be independent. Or preregistered with signed contracts that any result will get published from all parties involved.]

A little workshop on data-fabrication...?

I saw something in a paper which didn't make much sense to me... Of course it is also possible that I’m just stupid and overlooking explanations... so I’ll write here what I found and you can decide on your own.

This is a table I reproduced from that paper. The reason I don’t simply make a screenshot of the table in the paper is that I could be false. And in that case I don’t want to falsely accuse the paper (or its authors) of error. 

So I tried to provide some anonymity to the paper… (for now).

Pre are measurements taken before... something... and post afterwards (if you didn't guess it). 

The same people were tested in pre and post and the sample size was 21.

What surprised me were the tiny standard deviations for some of the Variable 1 and 2, especially in combination with the range given.

So I tried to find some numbers that would fulfill the criteria given (mean, standard deviation and range) for these 4 (or 8) variables. First my "results" for the first four variables pre:

As you can see I managed to find values (for each participant) that add up to the given mean for every variable. (I should note here that the third and the fourth variable are ratings on a numerical (not visual) rating scale, so values other than other than X.0 and X.5 might not be so likely.) 

I also tried to get the SD as much to the given value as possible. You see what I was able to get. 

(SD excel is the SD calculated by excel and SD me is the SD calculated by me with help of excel, i.e. I set up the formula myself cos I didn’t know what formula excel was using. I didn’t calculate it by hand though, because there would be no chance to get a correct result that way.)

The SDs (dark blue row) I got are much higher than the given SDs for the first two variables (variable 1 pre and 2 pre).

I don’t know, obviously there’s a chance that I’m just dumb and didn’t do it correct. 

In order to get a lower SDs than those I got, I then replaced these numbers (for 2 to 20) with the mean of the first and the last value. I can’t change the first and the last value, because they are given in the table. Obviously, when I do that the mean is not "correct" anymore, but I should get the lowest possible SD (min and max value fixed) with that, shouldn't I? 

Apparently, for the first two variables the SD is still a lot higher than the given SD. (For the third and fourth variable I was able to get the same (very close, but I could have tried longer) SD as given in the table anyways.)

In the next table you can see that this is similar for these variables in post:

Just for the variables 1 and 2 (pre and post) I couldn’t find any values that met the criteria. 

So I thought maybe the authors inadvertently reported the SEM instead of the SD. Maybe it is possible to call the SEM SD because after all the SEM is a measurement of deviation which is standardized too.

Therefore I tried to get the SEM to be the value reported for the SD:

That is possible for the first to variables (pre and post) but I didn’t manage to do that for the third and fourth variable.

 

Maybe they reported the SEM for Variable 1 and 2 and the SD for Variable 3 and 4. I don’t know. The other possibility is that I made a mistake, obviously.

Anyways… in the paper they did t-tests with these variables. So I tried to do that too… Obviously I don’t know in which order the values were… so if I just take my fabricated values I get a far off t-value.

Sorry for my SPSS speaking German, but it isn’t important anyways (as it is obvious that this gives the false t-values):

At least I saw some significances in that output window for once… :( 

And mean, SD and SEM are about the same as in my Excel calculation.

Fortunately the paper tells me which t-values I ought to get:

So I calculated what the Sum of the (distance to the mean)² of the differences between the two conditions (pre and post) needs to be in order to get 

* the "correct" mean difference between the samples 

* and therefore also the "correct" t-value.

Here’s a table for that:

Then I tweaked my… why not call them "data" ("data" not: data) to fit these criteria.

Although I was not able to reproduce exactly these numbers, it went quite well and I got pretty close. 

Obviously there are a lot of solutions, especially if I don't meet the SD-criteria for each Variable as specified in the table (I could have tried for the 3rd and 4th variable, but I didn't).


Here (one possible set of ) the individual "data"  I used to get this values:

Btw, I gained some data-fabrication knowledge doing this (which I had not previously had!! All my data are honestly p-hacked :( :( [1]). Want to hire me?

SPSS too thinks that the new t-values are much better:

I know this might seem like a really small point... since the t-values are all possible and hopefully correct. I don't know. It's relevant for effect-size estimation, for example. 

I can be false (just to say this again, because maybe I'm not seeing something really obvious).

Another point I noticed is that they report a mean of 0.36 (SD=0.29) in some questionaire data. I can't say anything about this, since I don't know their participants personally, but seriously... I doubt it.

I found this graph which represents the answer-distribution for a non-clinical population for either this scale or a quite similar one (it's not clear to me from the paper; there is a long and a short version):

The reported mean for that scale (out of that other study) is 9.4 with a SD of 7.83). My participants (healthy controls) have a mean value of 12.4 (with min=4, max=19, and SD=3.5). Of course the mean can be lower (then there) in different populations at a different time and if especially people with probably low values in that questionaire are recruited. But 0.36 seems really low to me.

I don't know if they did some transformations on that or whatever, but it is not reported.

Oh, and even though I obviously didn't report a lot of varibles/values from that study I didn't omit a control group... it's just pre and post. 

(Again, I don't exclude my stupidity at all. Sorry for the repetition.
I thought very long about whether or not to post this, but I think the SD and the mean of the questionaire are too low...)

---

[1] Obviously p-hacking is bad too! I’m well aware of that. I don’t even know what is worse…

edit/addition: 
1. I didn't want to imply that the data were fabricated. I don't think so!! The title refers to what I did in this post. I think it wouldn't make much sense if the data (of the SD-variables) were fabricated, because even then they should have an SD that is possible (if I'm right that the one they report is not possible with that mean and range).  (And the reported t-value is possible with much higher SDs.)
For the mean on the questionnaire-data I think it is most likely that it is either a mistake, the data were transformed (though that is not mentioned) or the participants were influenced.
And while I'm at it, I think there could be two other possible mistakes in that paper. One is relatively clear (because they report the study protocol in a slightly different order at different points in the paper) and the other it is not certain imo.

2. Christopher at the neurotroph-blog did some R-simulations on the Standard Deviations. You can read his blog-post here.