Choosing Your Research Methods

In practice six main methods tend to be used:

The following discussion looks at the main pros and cons of using these methods, some of the main design features of each and some issues relating to their administration.

Questionnaires

These are sets of questions and attitudinal statements administered to individuals for self-completion. They can be posted to respondents, sent by email or filled in by the researcher during face-to-face enquiries. They can be:

Potential advantages

  1. Questionnaires are a useful means of getting data from a relatively large number of people or from a representative sample of that population.
  2. Therefore they are very efficient in terms of the researcher's use of time and effort. That is, you can distribute hundreds of them to students, colleagues or clients and get them completed and returned in the time it takes to set up and do half a dozen interviews. Questionnaires are also, usually, quicker to code and analyse that semi-structured or unstructured interviews.
  3. Questionnaires are a means of standardising the data collection process, i.e. everyone who completes it is being asked exactly the same questions in the same sequence [but of course this does not mean that they will choose to answer every question].
  4. The self-completion questionnaire may provide the respondent with time to reflect on the questions before answering them [especially if they can complete them in their own time].
  5. Respondents may feel that they can say what they really think if the questionnaire can be completed in privacy and anonymously [especially if the researcher is known to them or might be thought to have a vested interest in their answers].

Potential disadvantages

  1. There are hidden costs in the use of questionnaires. It may be a quicker method of collecting data and the format may facilitate data analysed but the design of a good questionnaire with clear instructions and unambiguous questions can take a long time. Semi-structured and unstructured interview schedules can be developed more quickly because the researcher gets immediate feedback from the respondents if they do not understand the questions.
  2. You may not always know that your carefully constructed questionnaire is not asking the 'right' questions until you start analysing the data, i.e. when it is too late to do anything about it. If they have sufficient time researchers try to get round this problem by piloting the questionnaire with a small group drawn from the population they are interested in before they make the final adjustments to it. But this is time consuming and practitioners doing research on a part-time basis do not always feel that they can spare the time to do this. Pilot your questionnaire if you can. At the very least send it to some colleagues and your tutor for comment.
  3. Once you have sent out the questionnaire you have little control over the situation. Suppose only a few completed ones are returned, i.e. what researchers refer to as a low response rate. What can you do? You can try several things to increase the likelihood of a good response rate:
    1. Get a 'captive' population or sample to complete it, i.e. ask students to complete it during a seminar or clients to complete it in your presence;
    2. Explain at the beginning of the questionnaire (or in an accompanying letter if it is a postal questionnaire) why their responses are important and what use will be made of the data.
    3. Make the questionnaire as user friendly and attractive as possible by using different font sizes, colour printing, attractive layout and provide as many opportunities as possible to tick boxes or ring code numbers so that it can be filled in as quickly as possible.
    4. Distribute reminders perhaps with a second copy of the questionnaire.
  4. Avoid a lot of open-ended questions. As we have seen, one of the advantages of a self-completion questionnaire is that it pre-codes people's responses into a standardised set of answers to facilitate data analysis. Why lose that advantage? If you feel the need to have a lot of open-ended questions perhaps you have opted for the wrong method.

Design Issues

Please use this link to access a brief summary [PDF:44KB] of some of the key issues when designing a questionnaire.

There are several books that give useful advice on designing questionnaires. See, e.g.:

There are also some useful tips on questionnaire design in:

Interviews

A face-to-face conversation (or by telephone or video conference) which is structured around a set of questions, topics or themes. The degree to which it is structured can vary greatly. It can be highly structured (and then usually it looks like a questionnaire or checklist but administered by the researcher rather than self-completed). It can be semi-structured, mainly comprising a set of open-ended questions but often with follow-up probes and prompts; or it can be relatively unstructured - a list of themes or topics or headings which can be adjusted to each individual interviewee.

Potential advantages

  1. Provides an opportunity for the interviewee to give a more detailed response than in a questionnaire.
  2. The data will usually be richer and more full of contextual information than the data provided by a questionnaire.
  3. The data will be collected in a natural setting. For example, interviewing a lecturer immediately after a seminar will probably generate more data than a questionnaire which could be completed at any time. There is scope for the interviewer to cross-reference to the seminar, particularly if he or she also observed it [an example of triangulation at work].
  4. An interview is a particularly useful tool if you are trying to understand the experiences and actions of each individual respondent. The more unique each respondent's experiences are the more important it is to use a research technique which acknowledges and elicits that uniqueness.
  5. An interview is also a useful method if the respondents are not very literate or not used to expressing their views on paper.
  6. It provides an opportunity to probe respondents' views in ways that might be difficult to plan for in advance.
  7. It provides the researcher with an opportunity to also observe and record the non-verbal behaviour of the respondent as well.

Potential disadvantages

  1. The interview is not necessarily a cost effective method for a researcher who does not have any help in collecting data from a relatively large sample or population of respondents.
  2. The full transcription of interviews takes a lot of time. A one-hour tape recorded interview takes about 8-10 hours to transcribe.
  3. If you want to it is possible to quantify some of the answers (e.g. the number of respondents who answered this question positively or negatively) but other methods might be more appropriate for this purpose.
  4. Good interviewing requires expertise and experience. It is not an easy option for the inexperienced researcher. It requires, for example, good listening skills; body language that encourages the interviewee to relax and talk; a capacity to ask questions, perhaps take notes and yet maintain eye contact; an ability to prompt people who are not very responsive; knowing just how long to allow a silence to continue before intervening; an ability to probe sensitive areas and issues; being able to 'think on your feet' and be flexible in your questioning e.g. to recognise that someone has just answered 3 of your questions in one response and therefore being able to instantly adjust your schedule of questions accordingly to being able to decide on-the-spot whether an interviewee is leading you down a blind alley and needs to be brought back to the sequence of questions on your schedule or is raising interesting issues and should be encouraged to continue. In other words, interviewing is not a soft option. It requires planning (including planning for unexpected contingencies) and it requires skill.
  5. There is always a high risk with interviews that the researcher will unduly influence the responses of the interviewee: through the way the questions are asked, the researcher's body language and manner, the interviewer's position of authority in the interviewee's institution or vice versa (a potential problem in practitioner research).

Design Issues

Please use this link to access a brief summary of some of the key issues when designing an interview schedule.

For useful sources of advice on the technicalities of designing interview schedules see:

For a brief summary of the issues see:

Administrative issues

  1. Once you have explained what the interview is all about and why you are interviewing the respondent try one or two 'warm-up' questions to establish a good rapport.
  2. Listen, listen, listen! You can lose rapport if you ask someone a question which they have already answered when replying to an earlier question. Also you may want to cross-reference the answer to one question again the answers given to earlier questions, e.g. 'Earlier in the interview you said that...'
  3. Do not be afraid to leave out a question if you think it is not relevant to a particular interviewee.
  4. If your interviewee is talkative and giving you a lot of useful information do not hesitate to change the sequence of your questions if you think it would be appropriate (e.g. if the respondent has mentioned something that you were not originally intending to ask about until later in the interview) but remember to go back to the questions you have temporarily skipped when the time is right.
  5. Think about how you are going to end the interview. Don't let it simply 'trail off' because you have both run out of time or things to say.
  6. Even when you have switched off the tape recorder or put away your notebook you need to stay 'switched on'. A common phenomenon in interviewing is that people often give you very interesting and useful information once the formal interview is over. Write it down as soon as you can but don't foreclose the situation by either asking if you can switch on your recorder again or by saying 'I'm afraid that I'm in a bit of a hurry!'

Focused group discussions

These are widely used in market research, particularly for testing reactions to new products, and in politics for testing reactions to new policies. The method is also sometimes used in health promotion research and social work research. As an approach it tends to be used in combination with other research techniques.

Potential advantages

  1. A good medium for exploring a group of people's views without imposing your own agenda on them too strongly.
  2. Provides opportunities to explore the thinking behind the kinds of responses which might have been given to a questionnaire but without opting for the more expensive option of follow-up one-to-one interviews.
  3. In the early stages of a project it can be a useful means of identifying issues or areas of interest that could be followed-up using more quantitative research methods.
  4. A useful method of working with a group of people that operate as a group (a class of students, a department of colleagues, an interest or pressure group, etc)
  5. It provides the researcher with an opportunity to cross-check responses, e.g. 'Does everyone agree with that?'...Does that reflect your experience as well?...etc.

Potential disadvantages

  1. Difficult to follow-up the views of individuals during the group discussion, especially on topics which may be sensitive.
  2. Difficult to follow-up the views of individuals once the group discussion has been completed. If it was easy to do this why not opt for face-to-face interviews in the first place?
  3. This method can be heavily affected by the dynamics of the group, especially if some members of the group are senior to others. One or two people can easily dominate the proceedings if they have clear views and are articulate. It is possible as a facilitator to counter this tendency but it takes experience and self-confidence.

Design Issues

  1. Keep the size of the group fairly small (8-12).
  2. An hour is probably the minimum time to allow for the discussion to develop. It takes a while for the group to settle and for the right atmosphere to develop.
  3. Draw up a list of topics or themes to be covered. These should loosely guide the discussion. You want a balance between allowing the discussion to develop its own momentum and making sure your issues are addressed.
  4. Introduce yourself (if they do not already know you) and explain the purpose of the session, how long it will take and what you will do with the information.
  5. Get them to briefly introduce themselves (if they do not know each other).
  6. Decide how you will record the discussion. Will you tape it? Take notes? Get a colleague to take notes?
  7. Good chairing skills are essential (a) to ensure that the discussion is dominated by one or two and (b) to find out if there is a consensus within the group on a particular issue or theme and (c) to judge when to move on to the next theme or issue.

Observation

The systematic description of events and behaviours in their actual social setting. This tends to be classified along three dimensions. Firstly, whether the observer is a participant in what he or she is observing or is detached. Secondly, on whether or not those who are being observed are aware of this. Thirdly, the extent to which the observation is structured. The degree of structure can range from the observer taking notes or keeping a diary at one end of the spectrum to using coding schemes, checklists and category systems at the other.

Potential advantages

  1. The biggest advantage of collecting data through direct observation is that it is one of the most direct research techniques. You are not asking people what they would do or think you are watching what they do and listening to what they say.
  2. Used in combination with questionnaires or interviews, observation can therefore provide useful insights into the extent to which there is a correspondence or discrepancy between what people say and what they actually do. As such observation makes a useful contribution to the process of triangulating evidence.
  3. The researcher observing his or her own practice has an advantage because the situation to be observed is familiar (which in turn reduces preparation time).
  4. It allows the researcher to describe the full complexity of the situation, including the dynamics of the group being observed.

Potential disadvantages

  1. It is very time consuming. How many times do you need to observe a situation or a group before you can be sure that you are describing with confidence what is really happening.
  2. When the observer is also a practitioner observing his or her own practice then there is a potential problem of how to ensure that te observer can detach himself or herself sufficiently.
  3. It is difficult to estimate the effect of the observer on the situation being observed.
  4. The analysis of observational data is time consuming and often open to very different interpretations.

Design Issues

Here is a brief summary of some of the key issues when designing an observation schedule:

  1. The first key question is: 'Are you going to be a participant or detached observer? What are the advantages to either role in this particular setting?
  2. Linked to that question is: 'Are you going to observe your own practice or the practice of your colleagues and are you going to observe your students or a colleague's?'
  3. Are you going to tell the group that you will be observing them?
  4. Do you want a narrative description of what is happening or do you want to design a specific observation schedule. The former approach is often described as 'naturalistic' and used by social anthropologists and some sociologists. Some form of observation schedule is more often used by psychologists and educational researchers. Look at the different kinds of schedule in the recommended reading before making up your mind on this.
  5. What specifically do you want to observe:
    1. Do you need to describe the setting? [the layout of the room, where people are sitting, the equipment used, etc)
    2. Do you need to describe the people involved?
    3. Do you need to describe every event and activity over a specific time period or just a sample of events, or select the ones you are interested in?
    4. Are you just interested in the interactions between the tutor and the students or are you also interested in the interactions between the students?
    5. Will it be useful to time each event or activity? What will this tell you? How will you use this data?
  6. When will you record your observations: at the time or immediately afterwards or much later? [this can be a real problem when you are a participant observer].
  7. How will you record your observations? Pen and paper, video recording, audio recording?
  8. If you are using an observation schedule can you use a coding system to make the data collection process easier?
  9. Is observation the only source of data for your research or will it be complemented by interviews, questionnaires or other methods? If the latter how will you relate one kind of data to another?

References

There are at least three useful texts for classroom observation:

For more general guides to observation as a method of enquiry see:

Documentary Analysis

Practitioners can make use of a wide range of documents, including course and module outlines, course materials, staff development documents, inspection reports, examiners' reports, minutes of meetings, memoranda, letters, diaries, students' journals, administrative records, management information systems.

Potential advantages

  1. Unlike most research instruments and data sources, documents are not so prone to being influenced by the researcher.
  2. Documents enable the researcher to investigate the background and context of the situation and the specific problem they are interested in.
  3. A useful means of evaluating the extent to which the rhetoric (or the policy) is actually put into practice [but of course you also need other means and other sources for evaluating practice].
  4. Documentary analysis is a useful means of analysing the 'official' view and accessing the 'official' record of events, decisions and plans.
  5. Some documents [e.g. student assessments] can provide a measure of the impact of changes introduced during the Action Research process.

Potential disadvantages

  1. Research bias may be restricted in one sense because the researcher has not produced the documents [in the same way as he or she has devised the questionnaire or interview schedule] but bias can still be present in the selection of the documents.
  2. Documentary analysis - if it is to be systematic - is time consuming. Analysis can range from reading and commenting on it to undertaking a full-scale content analysis or from quoting a mission statement to undertaking a statistical analysis of student assessment results for the last five years.
  3. There is little guidance available from experienced researchers on how to analyse some kinds of documents [e.g. students' diaries or reflective journals].
  4. Documents require a critical reading similar to the skills employed by the historian when analysing primary sources. Each document was written for a particular purpose and within a specific context which may not necessarily be obvious from the document itself. Documents have ot be interpreted asd well as read and this calls for expertise and experience.

Design Issues

Historians, political scientists and economists may design their research around the analysis of documents, records and statistics but for most action researchers investigating their own and others' practice documentary analysis is mainly a supplementary method employed for collecting background information and for triangulating with the main methods used.

The classic text here, which may still be available in some libraries, is:

A useful guide to using some documentary sources is:

The best short summary of the issues and approaches is:

Experiments

The key feature of any experiment is that the researcher deliberately controls and manipulates the conditions under which the effects of a change or intervention can be measured. Cohen, Manion & Morrison (2000) provide a useful brief description of experimental research in the natural and physical sciences:

Imagine that we have been transported to a laboratory to investigate the properties of a new wonder fertiliser that farmers could use on their cereal crops, let us say wheat. The scientist would take the bag of wheat seed and randomly split it into two equal parts. One part would be the grain under normal existing conditions - controlled and measured amount s of soil, warmth, water and light and no other factors. This would be called the control group. The other part would be grown under the same conditions - the same controlled and measured amounts of soil, warmth and light as the control group but, additionally, the new wonder fertiliser. Then, four months later, the two groups are examined and their growth measured. The control group has grown half a metre and each ear of wheat is in place but the seeds are small. The experimental group, by contrast has grown half a metre as well but has significantly more seeds on each ear, the seeds are larger, fuller and more robust." (p.211)

The key features of the experiment are:

Will this classical experimental design still work when the subjects of the experiment are people rather than wheat seeds?

The experiment is still the norm in medical research and is widely used in all forms of psychology, research into health care and, to a lesser extent, research into social care. One particular form of the experimental design, the randomised controlled trial (through which, for instance, new drugs and forms of medical treatment are tested) is still generally regarded in those disciplines as the 'gold standard' of research. It is used to test new drugs and forms of medical treatment and to evaluate the effectiveness of new interventions in health and social care [Gomm & Davies (2000) p.xi].

In each case the experiment is designed in such a way that it reduces the likelihood that the prior knowledge of the subjects, the practitioners and the researchers taking part in the trial might unduly influence the results of the experiment.

However, in many social situations it is simply not practical (or sensible) to try to control all of the possible variables that might influence the outcomes of a specific change or intervention. Indeed, in some instances it would also be unethical to use a controlled experiment if, for instance, the subjects were not in a position to give their informed consent to participation in the experiment or if participation meant that they might suffer or be treated unfairly or if the experiment required them to do something illegal or immoral. Also, in the real world, it may not be possible to assign people randomly to either the experimental or the control group.

In such circumstances some researchers have introduced the idea of a quasi-experiment. Perhaps the most common kinds of quasi-experiment employed in social research are: Where the researchers collect data that enable them to compare the same subjects before-and-after an intervention or change has been introduced;

Where the researchers establish a pilot or study group (the subjects of their research) who experience the change or intervention and a reference or comparator group from whom data are collected using the same research instruments even though they are not experiencing the change or intervention.

Therefore the quasi-experiment retains the element of comparison which is so central to the experimental research design but subjects are seldom allocated to their groups and, if they are, this is rarely done at ransom. In practice, in most quasi-experiments the researcher does not have any control at all over the so-called 'control group' (or reference or comparator group).

Ultimately the central question for any researcher opting for a quasi-experimental design will be: 'Am I comparing like with like?' If the answer is: 'I believe so' then the follow-up question will inevitably be: 'How do I know?'.

There are a variety of different kinds of quasi-experimental research design and each has its own advantages and disadvantages. Rather than outline all of them here it is recommended that you read the relevant section in Colin Robson's book, Real World Research (1993 Oxford) Chapter Four: Experimental Design Outside the Laboratory, especially pp.96-108.

For those readers interested in the relative advantages and disadvantages of experimental and quasi-experimental designs for research in health or in social care see Chapter 3 in Gomm & Davies (2000).

References

Please use this link to return to the page containing the sitemap





Logo for UHI Logo for Perth College