Data-collection methods are too numerous to count.
Choosing the appropriate method is thus anything but easy, and in many cases requires some experience. Thus, it makes sense to talk first with experts or representatives from other similar projects. You need to consider which data you need and how relevant and meaningful these data are.
Chosing the right method
Methods differ greatly in terms of effort and expense; the amount of prior knowledge necessary; and the degree of detail, significance and reliability of findings.
If your budget is small, or you’re still at the very beginning of the project, it’s best to start with small, straightforward measures that you can expand little by little.
The decision for or against a particular method depends on:
- The questions you're trying to answer and
- The indicators you're using to determine whether a project objective has been reached.
The figure below gives an overview of analysis questions and should help you reducing complexity.
Testing your data collection method
Before you start to collect your data in full, we recommend that you test the chosen instruments – for example with the help of a questionnaire or interview guide. During this pre-testing phase, you should focus on:
- Are the questions clear and precise?
- Does the choice of words match the target group’s own communication style?
- Are the questions phrased positively? Are they formulated so as to avoid causing discomfort or defensiveness among respondents?
- Are the instructions for those carrying out the survey clear and unambiguous?
- Is the length and scope of the survey or questionnaire appropriate? Questionnaires and interviews that are too long can make respondents impatient and unfocused, which in turn has a negative effect on the answers’ utility
Less is more – at the beginning! Even for small organizations, there are methods of data collection that require little in the way of resources to use. Thus, don’t be intimidated; instead, start with small, clear measures that you can later expand. You can implement some of the data-collection methods yourself. The results may be less precise, but they will in many cases be sufficient to initiate processes of learning and improvement.
Overview on the most important data-collection methods
The following list contains all relevant (yet, certainly not all!) data collection methods you may use and lists the accodant advantages and disadvantages.
The most commonly used type of data collection, proven and effective. With questionnaires, you can find out how satisfied participants are with an offering, how much they’ve learned, and whether they’re actually acting on their new knowledge. Questionnaires with closed questions can be quickly evaluated in bulk; questionnaires with open questions generally provide useful additional information.
- Many surveys possible in a short time
- Data can be easily summarized
- Possible to ensure anonymity
- Preparation requires time and expertise
- Return rates may be low
- Answer options limited
- No opportunity for follow-up questions
YEA uses questionnaires once annually to determine whether participating youths are satisfied with the project. A second questionnaire assesses whether mentors are satisfied with the degree of support provided.
Individual interviews capture different perspectives on a particular subject. They are useful:
- For surveys during the needs-assessment process
- When a new project module is being developed
- To elicit individual feedback
- To identify areas that can be improved
The form of the interview depends on the interviewee, the questions being asked and the goal of the conversation. Semi-structured interviews – that is, those using both open and closed questions – are a good way to obtain deeper insight into a particular topic. Individual interviews are particular useful for the needs assessment or when a concept for a project component is being developed.
Choosing the right interviewer is also crucial. If project staffers lead the interview, there’s a danger that the interviewee will only give answers he or she thinks are desired. If someone external is doing the asking, respondents may wind up keeping their true opinions to themselves.
Be sure to survey a representative but not too homogeneous group. This is the only way to obtain statistically significant and reliable answers.
Interviews should always be designed around a few core questions. If the goal of the interview is unclear, the answers too are likely to be vague.
- Stakeholders are involved
- Provide relevant data
- Can reveal unexpected results, if any
- May elicit statements that wouldn’t be made in a group situation
- Enable follow-up questions to the interviewee
- Time intensive
- Interviewers must be trained
- Results may be difficult to evaluate and quantify
After the project’s end, YEA uses individual interviews to ask participating youths how they benefited from the project, and whether they have found a place in a vocational-training program.
This refers to interviews with experts, decision makers, and other people who can provide an assessment of the local situation and the target-group’s circumstances. In expert interviews, the priority isn’t the interviewees as individuals; rather, the focus is on their function as experts in a specific issue area, or as representatives of a group.
You can conduct one-on-one interviews with individual experts, or interview several at the same time, for example using a round-table format. This makes sense when planning a project, as well as at regular intervals while the project is running.
- Manageable in terms of effort and expense
- Provide a synthesis of opinions
- Involve decision makers
- Risk that the discussions remain too abstract
YEA uses expert interviews in various contexts. For example, during the needs-assessment process, the head of the youth-services agency, a staffer from the job center, a school principal and a businessperson are all asked for their opinions. In addition, interviews with class teachers are conducted regularly during the project.
A focus group is a moderated discussion between multiple participants who engage with each other around a narrowly defined question. Ideally, the participants inspire one another to provide deeper statements than would otherwise be the case.
Focus groups are particularly suitable when dealing with common experiences and problems, and when solutions must be developed.
The key factor is the choice of the panel: Can the participants speak openly with each other? Do group members represent different viewpoints? Can a fruitful discussion develop?
- Stakeholders are included
- Can also deliver unexpected results
- Added value provided by conversation between participants (different points of view)
- Follow-up questions possible
- Medium time expenditure
- Expertise necessary
- Results difficult to quantify, and potentially hard to evaluate
- Participants may not speak openly
- Individual opinions and experiences are pushed into background
Every six months, YEA invites the mentors to a group conversation in which experiences, problems, successes and the project’s logic model are discussed. The results of the focus group are used to improve the project on an ongoing basis.
Informal conversations / anecdotes
Informal conversations include all conversations with participants and stakeholders that extend beyond "Good morning" or "See you later". Any form of feedback, appraisal or evaluation can be helpful – regardless of whether it takes place in the lounge, in a tutorial session or on the way home. By their nature, informal conversations provide information particularly about unplanned results, both positive and negative.
It’s crucial that you collect such information from informal conversations regularly and systematically, throughout the entire course of the project.
Project diaries kept by all full-time and volunteer staffers have shown themselves to be extremely useful in this context. It is also helpful to encourage staffers to report on anecdotes in project meetings, and to make sure to record these. Then, rather than letting these records disappear into the bottom of a drawer somewhere, make sure they’re discussed at regular meetings between volunteer and full-time employees.
Informal conversations are particularly useful for eliciting opinions among members of the project’s indirect target group (e.g., parents).
- Direct contact with the target group
- Provide information about unplanned effects
- Provide content for project communications (storytelling)
- No previous experience is necessary, inexpensive to implement
- Results are highly subjective and
- Difficult to generalize
The way you ask a question directly affects the answer you receive. For a discussion of the advantages and disadvantages of open and closed questions, click here.
At YEA: Various mentors learn through incidental conversations with parents that their children have not only improved academically, but have also seen their aggression levels decline. Each mentor keeps a mentor diary to record both positive and negative events. Discussions on the contents take place at the mentor group’s regular meetings.
In some cases, instead of asking a barrage of questions, it can be more useful to make a careful observation.
In systematic observations, events, individuals, groups and social spaces are observed. Systematic observations are appropriate when you want to check answers from a survey, or hope to develop further, specific findings. In a participant observation, the observing person takes place more or less actively in the interactions taking place within the field of observation. In a non-participant observation, the observing person remains outside the field they’re observing. For more information on this topic, see wikipedia.
- Information obtained directly from the target group or social space
- Can provide communicative content (storytelling)
- Very time-consuming
- Observers must be trained
- Can be viewed as a violation of privacy
At YEA: The external evaluator audits the actual changes in the youths’ social behavior. To do so, she reviews the surveys completed by the youths and the teachers. In addition, she sits in on classes at the school in order to observe the youths’ social behavior in an academic context.
Tests and measurements
Tests and measurements provide important information at various points in the project cycle.
At the project’s beginning, for example, they offer information on the initial situation within the target group. Examples can range from math tests for students to the collection of health-status information for participants in a health program.
Tests often employ quantitative methods, but qualitative and mixed methods are also possible.
- Good at depicting changes over time
- Standardized tests provide high degree of comparability
- Medium time expenditure
- Require extensive expertise
- Standardized tests may not be applicable to the target group’s specific situation
Before the beginning and after the end of the job-application training, YEA tests participants to assess their level of knowledge.
In case studies, the focus is on individual participants or a very specific, narrowly defined group.
A variety of methods can be used for case studies, including semi-structured interviews, systematic observations, focus groups and so on.
Case studies are appropriate when the goal is to illustrate the results being achieved. By using figures and data combined with the stories of individual people, a very enlightening picture of a project’s social impact can be drawn.
- Information obtained directly from the target group
- Provide good content for project communications (storytelling)
- In some cases, it can be difficult to make generalizations
YEA tells the story of the formerly unruly Jacob, from his first conversation with the YEA mentor through the end of his trial period at a job. For the case study, the monitoring data is combined with qualitative statements from the mentor diaries.
Important information for the social-impact analysis can be found in many internal and external documents.
Internal documents can include project concepts, reports and meeting minutes; in these, you can find information on goals, results and deviations from the plan.
External documents can include studies, surveys and official statistical reports; these are interesting particularly for the needs-assessment and context-analysis process, and as comparative reference points.
- Usually close at hand
- Inexpensive to access, and can be quickly located
- Staff members are involved
- Inexpensive to access
- Methodologically most reliable
- For regular surveys, comparisons over time are possible
- May not be objective
- May lack information on cause-effect relationships
- Information may be out-of-date or incomplete
- Often not related to the target location, and highly aggregated
- Possibly out-of-date
YEA: For the external evaluation, the evaluator drew both on information from the project-design documents and monitoring data. At the beginning of the project, during the needs-assessment and context-analysis process, YEA used official statistics on youth unemployment in the region as well as documents with information on the regional vocational-training market.