Difficult-to-demonstrate results

Verifying results can be a major challenge for many different reasons. What to do in the case of results that can be identified only with significant effort and/or expense?

Verifying long-term results

Many social projects aim at results that occur only after considerable time has passed. Verifying such results is complex and difficult. The challenges begin with the collection of data, because it is often very difficult and costly to contact former participants after the passage of considerable time.

In addition, the degree to which the observed results are in fact attributable to the project can be open to question, since a variety of influences have shaped the (former) participants’ development in the time between the end of the project and the occurrence of the long-term result.

What you can do:

  • Keep an up-to-date contact database.
  • While the project is still ongoing, tell participants that there will be a follow-up survey in the future.
  • Ask survey respondents to assess how substantial the project’s influence on their current situation has been.
Illustration Rettungsring
  • Verification using the logic model

    If it can be determined that participants have shown results at outcome levels 4 (changes in knowledge) and 5 (changes in behavior and actions), it can be assumed with a certain probability that the project has also produced results at level 6 (change in living situations).

What if participants don’t want to be or cannot be surveyed

In some projects, it proves difficult to survey participants. Some have reservations with regard to participating in surveys, others fear repercussions (for instance, if they are at risk of violence, have themselves had problems with the law, etc.), while others are simply not in a position to provide information (small children, critically ill persons, etc.).

What you can do:

  • Assure and guarantee anonymity
  • Survey persons with a close relationship (e.g., parents or caregivers)

Campaigns & advocacy

The target of campaigns is to influence decision makers and public opinion, and initiate processes that express themselves in changed consciousness and behavior. Examples include campaigns against smoking, or in the areas of environmental protection, education policy or inclusion. Results in such areas are generally difficult to ascertain.

Determining whether a campaign has had the desired effect on the societal level is best accomplished using a large-scale study (e.g., on the health status of the population). However, this is unlikely to be feasible for an individual organization.

What you can do:

Using the various levels of the logic model, define partial objectives for which you can collect data. For example, influence on decision-makers or relevant multipliers can be verified on the basis of whether demands have been taken up or arguments adopted, or whether the media has begun reporting more intensively on the subject.

The target groups are organizations

In projects where the target groups are organizations, results take the form of the benefits these organizations derive from the project’s support.

An organization’s changes in knowledge and operational style represent medium-term outcomes.

A long-term result would be if the organization proves able to work more effectively. These results can be illustrated primarily by using qualitative data.

Suppose that a project supports organizations in obtaining and managing volunteer activists. Then:

  • A result at outcome level 4 would consist of the fact that the organization has learned how to acquire and support volunteers.
  • A result at outcome level 5 would be that the organization employs this knowledge on a daily basis.
  • Results at outcome level 6 could be identified if the organization implements projects more effectively due to its improved management of volunteers.

Results at the societal level

In many cases, it ranges from difficult to impossible to collect data on impact at the societal level. This is because changes at the societal level in most cases appear only after considerable time. In addition, addressing complex problems requires many impacts, which in turn are subject to many influences.

In this case, it is all but impossible to filter out the net contribution to change produced by an individual project. However, this doesn’t mean that you shouldn’t try!

What you can do:

  • Think about impact while still in the phase: Develop realistic indicators that you can use to determine whether your project has had an influence on overall societal structures.
  • Try to rein in funders’ expectations regarding impacts, in part by making reference to the complex interrelationships underlying the problem (problem tree). Sketch for them the part your project plays on the broader social stage, and what results are realistically possible.

 

Open offerings

Open offerings provided to children, youths or older people often have difficulties making reliable statements about their effectiveness, and therefore sometimes experience a certain pressure to justify their efforts.

Sometimes the group of participants is very heterogeneous, and sometimes participation is irregular, making it appear difficult to set project objectives and verify results.

What you can do:

  • Think creatively: Why are attendees using your offering?

A senior-citizen meeting group is not attended solely because the coffee is so good and the cake so inexpensive; rather, attendees want to make contact with other people and improve their quality of life. What would project objectives and indicators based on this insight look like?

Even in the case of events like afternoon playgroups or block parties, which are mainly about having fun, children and youths are at the same time building social, cultural and political skills, and learning to take responsibility for themselves. What might project objectives and indicators look like here?

  • Use qualitative surveys, case studies and anecdotes. Support the Error:there was a database conflict. Dear editor: are you sure you got the right keyword? output figures with statements from participants discussing how they felt about the offering.

 

Prevention projects

If youths don’t begin to smoke, or 14-year-old girls avoid becoming pregnant, prevention projects have achieved their goal. However, how can it be established that these results are attributable to a specific project?

What you can do:

  • Identify a comparison group that didn’t participate in the project. For example, this could be youths from another class in the same grade, or from another secondary school in the city district you’re targeting.
  • Let your logic model guide your thinking even for prevention projects (see “Verifying long-term results” above, on this page).
  • What are control groups?

    A control group does not take part in the project, thereby making a comparison with the participant group possible and allowing the derivation of statements on the project’s effectiveness. Carrying out an evaluation using control groups is quite complicated, but it delivers robust, resilient findings – otherwise, results attributed to the project could also be driven by other causes.