Data Collection - Interview

To do research about any aspect of the world, we need data about the world. There are many different ways to collect data for education research. One of them has to do with interviewing research participants, and this data collection method is the focus here. More specifically, I will look at some scientific weaknesses inherent to interviews, and how they can be minimized.

The weak link in the rigor of any scientific research is the human factor. An interview explicitly relies on that weak link - the interviewer. On the other hand, education research has to do with humans, so one cannot escape being heavily involved with that weak link. Even if we eliminate the weak-link effects of the researchers, we are left with the weak-link effects of the participants. For example, if we use a perfect questionnaire to collect data, the participants who fill out the questionnaire are still fallible humans.

It may be that an excellent interviewer can produce more accurate and deeper results than a self-report such as a questionnaire. But as in all professions, excellent practitioners are not common. And the slightly-less-than-excellent interviewer, even if he/she is pretty good, is likely to miss some cues from the participant, to behave in a way that affects the participant, to occasionally misinterpret what the participant says, and in general put themselves as part of the entity creating the information rather than remain an objective tool for collecting information. The theoretical excellent interviewer is transparent, while the real-life almost-excellent interviewer is opaque. The qualities of an excellent interviewer are discussed separately. Here I concentrate on what's inherent to the interview, and lies outside the interviewer's control.

The more qualitative the interview is - open questions, free-form follow up questions - the more the interviewer is active in the interaction, and the more the interviewer affects the results. However, if there is a need for open questions - for example when trying to check what impressions the participants have of a certain learning situation - a questionnaire may miss the point and there is no way to avoid using a human interviewer. Serious effort should be spent to design research in such a way that a minimal number of open questions. This way, the level of involvement of the interviewer can be minimized and the penalties in term of objectivity can be minimized. A second line of defense is having some open questions to be administered by an interviewer, but only in a preliminary part of the research, aimed at generating more accurate questions to be pursued in later parts of the research, using more objective methods. A third line of defense, when complicated open questions can’t be avoided, is to include - preferably close to the end of the interview - a few questions designed to check the level and direction of dependence created between the interviewer and participant. These extra questions should not be apparent to the participant. It would be great if they are also not apparent to the interviewer, but if the interviewer is an excellent one - they will know. Some examples of such questions are “Did you enjoy the interview?” or “Was the interview difficult?” or “Were there accurate enough options for the closed questions?” or “Do you feel the questions and answers in the interview capture what you wanted to say?”. If the average response for a certain interviewer differ from the average for the one excellent interviewer in the research - maybe there was too much of that interviewer in the interview.

A general tool that can help with pushing interviews towards the objective end of the scale is recording the interview and reviewing the recording after the interview. Since recordings can improve the results an interviewer gets from an interview, some detailed considerations about recordings are discussed in the context of the interviewer as a superhero.

Beyond the factors the excellent interviewer can control in an interview situation, the participant is also affected by the interviewer in ways that are outside the control of the interviewer. The interviewee may have judgments regarding how the interviewer looks, sounds, and smells - whether positive, negative or otherwise. The interviewer might remind the interviewee of someone or something that the interviewee liked or disliked: Maybe someone the interviewee is inclined to appease or to confront. A phone interview removes the interviewer a bit from the participant, therefore preventing some of the ways the interviewer may affect the participant: body-language, look, etc. It still leaves many ways the interviewer affects the participant: Voice inflections, pauses, choice of words in between questions (those words not prescribed by the interview protocol), etc. On the other hand, in a phone interview the interviewer misses the interviewee's body language information.

A hypothesis: The more open and in-depth interviews are required for a research, the more the main researcher tends to conduct the interviews personally, rather than using research assistants as interviewers. If this is true, it may mean the researchers themselves don’t believe in the ability of another interviewer to reach the same raw data as the researcher can. This would mean that the researchers themselves believe that the answers in the interview have a lot to do with the interviewer rather than relating purely to the participants. This in turn would indicate that such an interview’s meaning depends on the specific researcher conducting it. If you don’t happen to be that researcher, the interview is of very limited meaning for you.

In case this blog entry appears very negative: The fact that such weaknesses exist doesn't mean interviews shouldn't be used. It does mean they should be used with care, and whenever possible, safer methods should be used.