An overview of data-collection methods

The most commonly used type of data collection, proven and effective. With questionnaires, you can find out how satisfied participants are with an offering, how much they’ve learned, and whether they’re actually acting on their new knowledge. Questionnaires with closed questions can be quickly evaluated in bulk; questionnaires with open questions generally provide useful additional information.

Advantages:

• Many surveys possible in a short time
• Data can be easily summarized
• Possible to ensure anonymity

Disadvantages:

• Preparation requires time and expertise
• Return rates may be low
• Answer options limited
• No opportunity for follow-up questions

YEA uses questionnaires once annually to determine whether participating youths are satisfied with the project. A second questionnaire assesses whether mentors are satisfied with the degree of support provided.

Individual interviews capture different perspectives on a particular subject. They are useful:

• For surveys during the needs-assessment process
• When a new project module is being developed
• To elicit individual feedback
• To identify areas that can be improved

The form of the interview depends on the interviewee, the questions being asked and the goal of the conversation. Semi-structured interviews – that is, those using both open and closed questions – are a good way to obtain deeper insight into a particular topic. Individual interviews are particular useful for the needs assessment or when a concept for a project component is being developed.

Choosing the right interviewer is also crucial. If project staffers lead the interview, there’s a danger that the interviewee will only give answers he or she thinks are desired. If someone external is doing the asking, respondents may wind up keeping their true opinions to themselves.

Be sure to survey a representative but not too homogeneous group. This is the only way to obtain statistically significant and reliable answers.

Interviews should always be designed around a few core questions. If the goal of the interview is unclear, the answers too are likely to be vague. 

Advantages:

• Stakeholders are involved
• Provide relevant data
• Can reveal unexpected results, if any
• May elicit statements that wouldn’t be made in a group situation
• Enable follow-up questions to the interviewee

Disadvantages:

• Time intensive
• Interviewers must be trained
• Results may be difficult to evaluate and quantify

After the project’s end, YEA uses individual interviews to ask participating youths how they benefited from the project, and whether they have found a place in a vocational-training program.

This refers to interviews with experts, decision makers, and other people who can provide an assessment of the local situation and the target-group’s circumstances. In expert interviews, the priority isn’t the interviewees as individuals; rather, the focus is on their function as experts in a specific issue area, or as representatives of a group.

You can conduct one-on-one interviews with individual experts, or interview several at the same time, for example using a round-table format. This makes sense when planning a project, as well as at regular intervals while the project is running.

Advantages:

• Manageable in terms of effort and expense
• Cost-efficient
• Provide a synthesis of opinions
• Involve decision makers

Disadvantages:

• Risk that the discussions remain too abstract

YEA uses expert interviews in various contexts. For example, during the needs-assessment process, the head of the youth-services agency, a staffer from the job center, a school principal and a businessperson are all asked for their opinions. In addition, interviews with class teachers are conducted regularly during the project.

A focus group is a moderated discussion between multiple participants who engage with each other around a narrowly defined question. Ideally, the participants inspire one another to provide deeper statements than would otherwise be the case.

Focus groups are particularly suitable when dealing with common experiences and problems, and when solutions must be developed.

The key factor is the choice of the panel: Can the participants speak openly with each other? Do group members represent different viewpoints? Can a fruitful discussion develop?

Advantages

• Stakeholders are included
• Can also deliver unexpected results
• Added value provided by conversation between participants (different points of view)
• Follow-up questions possible

Disadvantages

• Medium time expenditure
• Expertise necessary
• Results difficult to quantify, and potentially hard to evaluate
• Participants may not speak openly
• Individual opinions and experiences are pushed into background

Every six months, YEA invites the mentors to a group conversation in which experiences, problems, successes and the project’s logic model are discussed. The results of the focus group are used to improve the project on an ongoing basis.

Informal conversations include all conversations with participants and stakeholders that extend beyond "Good morning" or "See you later". Any form of feedback, appraisal or evaluation can be helpful – regardless of whether it takes place in the lounge, in a tutorial session or on the way home. By their nature, informal conversations provide information particularly about unplanned results, both positive and negative.

It’s crucial that you collect such information from informal conversations regularly and systematically, throughout the entire course of the project.

Project diaries kept by all full-time and volunteer staffers have shown themselves to be extremely useful in this context. It is also helpful to encourage staffers to report on anecdotes in project meetings, and to make sure to record these. Then, rather than letting these records disappear into the bottom of a drawer somewhere, make sure they’re discussed at regular meetings between volunteer and full-time employees.

Informal conversations are particularly useful for eliciting opinions among members of the project’s indirect target group (e.g., parents).

Advantages:   

• Direct contact with the target group
• Provide information about unplanned effects
• Provide content for project communications (storytelling)
• No previous experience is necessary, inexpensive to implement

Disadvantages:

• Results are highly subjective and
• Difficult to generalize

The way you ask a question directly affects the answer you receive. For a discussion of the advantages and disadvantages of open and closed questions, click here.

At YEA: Various mentors learn through incidental conversations with parents that their children have not only improved academically, but have also seen their aggression levels decline. Each mentor keeps a mentor diary to record both positive and negative events. Discussions on the contents take place at the mentor group’s regular meetings.

In some cases, instead of asking a barrage of questions, it can be more useful to make a careful observation.

In systematic observations, events, individuals, groups and social spaces are observed. Systematic observations are appropriate when you want to check answers from a survey, or hope to develop further, specific findings. In a participant observation, the observing person takes place more or less actively in the interactions taking place within the field of observation. In a non-participant observation, the observing person remains outside the field they’re observing. For more information on this topic, see wikipedia.

Advantages:

• Information obtained directly from the target group or social space
• Can provide communicative content (storytelling)

Disadvantages:

• Very time-consuming
• Observers must be trained
• Can be viewed as a violation of privacy

At YEA: The external evaluator audits the actual changes in the youths’ social behavior. To do so, she reviews the surveys completed by the youths and the teachers. In addition, she sits in on classes at the school in order to observe the youths’ social behavior in an academic context.

Tests and measurements provide important information at various points in the project cycle.

At the project’s beginning, for example, they offer information on the initial situation within the target group. Examples can range from math tests for students to the collection of health-status information for participants in a health program.

Tests often employ quantitative methods, but qualitative and mixed methods are also possible.

Advantages:

• Good at depicting changes over time
• Standardized tests provide high degree of comparability

Disadvantages:

• Medium time expenditure
• Require extensive expertise
• Standardized tests may not be applicable to the target group’s specific situation

Before the beginning and after the end of the job-application training, YEA tests participants to assess their level of knowledge.

In case studies, the focus is on individual participants or a very specific, narrowly defined group.

A variety of methods can be used for case studies, including semi-structured interviews, systematic observations, focus groups and so on.

Case studies are appropriate when the goal is to illustrate the results being achieved. By using figures and data combined with the stories of individual people, a very enlightening picture of a project’s social impact can be drawn.

Advantages:

• Information obtained directly from the target group
• Provide good content for project communications (storytelling)

Disadvantages:

• In some cases, it can be difficult to make generalizations

YEA tells the story of the formerly unruly Jacob, from his first conversation with the YEA mentor through the end of his trial period at a job. For the case study, the monitoring data is combined with qualitative statements from the mentor diaries.

Important information for the social-impact analysis can be found in many internal and external documents.

Internal documents can include project concepts, reports and meeting minutes; in these, you can find information on goals, results and deviations from the plan.

External documents can include studies, surveys and official statistical reports; these are interesting particularly for the needs-assessment and context-analysis process, and as comparative reference points.

Advantages:

Internal data:

• Usually close at hand
• Inexpensive to access, and can be quickly located
• Staff members are involved

External data:

• Inexpensive to access
• Methodologically most reliable
• For regular surveys, comparisons over time are possible

Disadvantages:

Internal data:

• May not be objective
• May lack information on cause-effect relationships
• Information may be out-of-date or incomplete

External data:

• Often not related to the target location, and highly aggregated
• Possibly out-of-date

YEA: For the external evaluation, the evaluator drew both on information from the project-design documents and monitoring data. At the beginning of the project, during the needs-assessment and context-analysis process, YEA used official statistics on youth unemployment in the region as well as documents with information on the regional vocational-training market.