Goal-free evaluation

Goal-free evaluation (Often shortened to GFE) is any evaluation in which the evaluator conducts the evaluation without particular knowledge of or reference to stated or predetermined goals and objectives.^[1][2] This external evaluation model typically consists of an independent evaluator who is intentionally screened from the program's stated goals and objectives in hopes of reducing potential goal-related tunnel vision.^[3][4] According to Scriven,^[5] the logic behind avoiding stated goals and objectives has to do with "finding out what the program is actually doing without being cued as to what it is trying to do. If the program is achieving its stated goals and objectives, then these achievements should show up; if not, it is argued, they are irrelevant" . The goal-free evaluator attempts to observe and measure all actual outcomes, effects, or impacts, intended or unintended, all without being cued to the program's intentions. As Popham ^[6] analogizes, “As you can learn from any baseball pitcher who has set out in the first inning to pitch a shutout, the game’s final score is the thing that counts, not good intentions.”

Youker and Ingraham ^[7][8] summarize GFE's benefits which are based on:

• Controlling goal
• Uncovering side effects
• Avoiding the rhetoric of “true” goals
• Adapting to contextual/environmental changes
• Aligning goals with actual program activities and outcomes
• Supplementing goal-based evaluation (also known as objectives-based evaluation)

Development

Michael Scriven^[9] first introduced the concept of GFE in the 1970s. According to Youker, Ford, and Bayer, following GFE's introduction, there was moderate interest use ^{[10] [11] [12] [13] [14] [15] [16] [17]} .^[18] However, enthusiasm and interest waned, leaving few practitioners. Then in the 2000s, there was a small resurgence of interest in GFE. In a literature search of GFE from 1972 to 2012, Youker and Ingraham (2013) uncovered roughly a score of evaluators who purport to have conducted a GFE or others identified them as having conducted a GFE.

Description

Goal-free evaluation is not a comprehensive stand-alone evaluation model; rather it is considered either a perspective or position concerning an evaluator's goal orientation throughout an evaluation, or a technique or tool for evaluating without referencing goals. Scriven ^[19] claims GFE is methodologically neutral, which means that it can be used or adapted for use with several other evaluation approaches, models, and methods as long as the other approaches do not mandate goal orientation.

There are really only two methodological requirements of GFE. The first is that the goal-free evaluator be external from and independent of the program and its upstream stakeholders (program funders, designers, administrators, managers, staff, volunteers, vendors, etc.);^[20] the second is that someone be appointed a goal screener. A screener is an impartial party (i.e., someone who is not assigned to GFE design or data collection), such as an administrative assistant, a third party, or even the evaluation client.^[21] The screener intervenes between the evaluator and the program people to eliminate goal-oriented communications and documents before they reach the goal-free evaluator. The screener does not require extensive training; the screener should, however, have a basic understanding of GFE's purpose and methodology and be relatively familiar with the organization and its program to be able to identify program goals and objectives.

There are two types of GFE evaluations^[22] that are used. The first type being when an evaluator intentionally avoids the stated goal and objectives by taking special precautions. For example, the evaluation may take special precautions to hire goal-screeners who play the role as a liaison between the evaluator and the client. This helps with reducing bias because it allows the outcomes speak for themselves. The second type is goal-dismissive. A goal dismissive evaluator simply disregards the goals and objectives, instead of using special precautions to avoid it.

Full and partial GFE

GFE can be full and partial. During a full GFE, the evaluator is goal-free throughout the duration of the evaluation.^[23]

A partial GFE begins goal-free but at some point, during the evaluation, the evaluator learns the stated goals and objectives.

Principles

In a recent attempt to further articulate general principles for guiding the goal-free evaluator, Youker ^[24] proposed the following principles:

1. Identify relevant effects to examine without referencing goals and objectives.
2. Identify what occurred without the prompting of goals and objectives.
3. Determine if what occurred can logically be attributed to the program or intervention.
4. Determine the degree to which the effects are positive, negative, or neutral.

GFE in practice

Youker ^[25] introduced GFE dos and don'ts which are included below.

GFE dos

• Identify and use a screener (i.e., an intermediary who ensures that no goal- or objective-based information is communicated to the goal-free evaluator).
• Refer all communiqués to screener and involve the screener throughout the evaluation to protect from potential contamination.
• Have all written material screened for references to program goals or objectives prior to evaluator receipt.
• Advise all program people of goal-free nature and the parameters of goal-free evaluation. Ensure that they understand they are not to relay goal/objective-related information.
• Stop program staff if they begin talking about goal-oriented information.
• Identify potential areas in which to search for effects (in part through a needs assessment) and use these as the basis for criteria to be measured.
• Identify and select justifiable tools to measure performance and actual effects (i.e., tools that are reasonable with adequate grounds for use).
• Measure performance and actual effects/experience; observe the program as is.
• Compare factual information about the program effects/experiences with pre-identified needs to assess the program's impact on consumer needs.
• Offer a profile of the positive, negative, and neutral effects.

GFE don’ts

• Communicate with program staff regarding goals or objectives.
• Attempt to find stated goals and objectives.

Limitations

GFE has identified limitations.^[26] First, GFE is not appropriate in every circumstance.^[27] For example, when a client is solely interested in examining goal attainment. Second, it is advocated as a standalone evaluation approach.^[28] GFE is used as a supplement to goal-based evaluations strategies. Third, GFE may be efficient in assessing outcomes, as the evaluator casts a broad net in search for outcomes.^[29] Fourth GFE disregards the opinions and goals of the certain one stakeholder group,^[30] namely the administrators, designers, funders, etc.

References

1. Youker, Brandon. . Social Work Research. Archived from the original on 2014-08-13.
2. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
3. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
4. Youker, Brandon; Ingraham, Allyssa; Bayer, Nick (2014). "An assessment of goal-free evaluation: case studies of four goal-free evaluations". Evaluation and Program Planning. 46: 10–6. doi:10.1016/j.evalprogplan.2014.05.002. PMID 24863956.
5. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. p. 180. ISBN 9780803943643.
6. Popham, James (1974). Evaluation in Education. McCutchan Publishing Corporation. p. 58. ISBN 082111512X.
7. Youker, Brandon; Ingraham, Allyssa (2013). "Goal-free evaluation: An orientation for foundations' evaluations". The Foundation Review. 5 (4).
8. Youker, Brandon; Ingraham, Allyssa; Bayer, Nick (2014). "An assessment of goal-free evaluation: case studies of four goal-free evaluations". Evaluation and Program Planning. 46: 10–6. doi:10.1016/j.evalprogplan.2014.05.002. PMID 24863956.
9. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
10. Alkin, M.C. (1972). "Wider context goals and goal-based evaluators". Evaluation Comment: The Journal of Education Evaluation.
11. Evers, J.W. (1980). "A field of goal-based and goal-free evaluation techniques". Kalamazoo: Western Michigan University Unpublished Doctoral Dissertation.
12. House, E.R. (1980). Evaluating Validity. SAGE Publications, Inc.
13. House, E.R. (1991). "Confessions of a responsive, goal-free evaluation". Evaluation Practice. 12: 109–114. doi:10.1177/109821409101200118. S2CID 220434776.
14. Salasin, S. (1974). "Exploring goal-free evaluation: An interview with Michael Scriven". Evaluation. 2: 9–16.
15. Scriven, M. (1973). "Goal-free evaluation". School Evaluation: 319–328.
16. Scriven, M. (1974). "Pros and cons about goal-free evaluations". Evaluation in Education:Current Applications: 34–67.
17. Thiagarajan, S. (1975). "Goal-free evaluation of media". Educational Technology: 38–40.
18. Welch, W.W. "Goal-free evaluation report for St. Mary's Junior College (Unpublished report)". {{cite journal}}: Cite journal requires |journal= (help)
19. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
20. Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. SAGE Publications, Inc.
21. Youker, Brandon (2005). "Ethnography and evaluation: Their relationship and three anthropological models of evaluation". Journal of MultiDisciplinary Evaluation. 3: 113–132.
22. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
23. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
24. Youker, Brandon. . Social Work Research. p. 434. Archived from the original on 2014-08-13.
25. Youker, Brandon. . Social Work Research. Archived from the original on 2014-08-13.
26. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
27. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
28. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
29. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.
30. Scriven, Michael (27 August 1991). Evaluation Thesaurus (4th ed.). SAGE Publications, Inc. ISBN 9780803943643.