Course Glossary

Because the field of program evaluation has developed its own terminology over the decades, a glossary is included here for easy access as you read this Study Guide.

Applied Research. Research that is practical in its orientation and that has real-world application. Program evaluation is considered by many to be a specialized form of applied research. Others believe that the context-specific application of program evaluation results indicates that it should not be considered any form of research.

Assessment. While often used as a synonym for program evaluation, its meaning is usually limited to processes that are focused on quantitative and/or testing measures, and less on judgement.

Bias. There are two crucially distinct senses of this word. First of all, bias can mean prejudice. In this sense it is the opposite of fairness, impartiality, objectivity. But bias can also mean a preference or commitment. In this sense, its opposite is neutrality or disinterest.

Client. A person, group, or organization that commissions an evaluation, usually by developing and advertising a Request for Proposals (RFP) statement. This is the person, group, or organization to whom the evaluator has a legal contractual responsibility, and who receives the evaluation report(s).

Closed Response. Refers to questionnaire design - the type of questionnaire that permits respondents to answer only within a limited choice of pre-set answers. Learned select a response, rather than create one.

Control Group. Used in experimental research designs. The group that acts as a control in experiments does not receive the "treatment" that is received by the experimental group (in the case of educational experiments, the treatment is often the instruction). The function of the control group is to establish the extent to which the same effect or outcome occurs without the treatment. If the extent is the same, it would indicate that the treatment is not causing the changes in the experimental group, if all other variables are controlled. If the extent is lesser, it indicates that the treatment has been (probably) shown to have an effect. In evaluation there is rarely any use of control groups, because of practicality and ethics issues.

Criteria. Specific indicators of merit or worth. For each evaluation question or standard, a number of criteria might be applied to determine the program's success in relation to that question or standard.

Data. Data are nothing more than ordinary bits and pieces of information. They can be concrete and measurable (i.e. class attendance), or invisible and more difficult to measure (i.e. personal feelings). The interest and perspective of the evaluator is what turns these bits and pieces of information into evaluation data. Research data conveyed through words are referred to as qualitative data, and research data conveyed numerically are referred to as quantitative data.

Document Analysis. Documents refer to any written records of an entity - in the case of program evaluation, any paper or electronic sources of information on the program. Evaluators often complete document analysis as one form of data analysis.

Effectiveness. This refers to the achievement of outcomes that (a) may have been expected, or (b) may not have been anticipated. The judgement of success of a program is related frequently to evidence of effectiveness.

Efficiency. This goes beyond effectiveness measures by referring to the amount or quantity of resources involved in attaining the desired effect. It implies an absence of wastage of time, effort, and material resources in goal attainment.

Emergent Design. A characteristic of some models of evaluation; in particular participant-oriented evaluation. The design of a participant-oriented evaluation needs to remain sufficiently open and flexible (or emergent) to permit exploration of the program being evaluated as the data are being collected, so that the data themselves in part dictate what will be explored next.

Ethics. On a practical level, a set of rules governing behaviour, practice, and attitudes that are based on the primacy of the doctrine of equal rights and fair treatment.

Evaluand. A generic term used to refer to whatever is being evaluated; the program, project, policy, etc. being evaluated.

Evaluation Design. The process of stipulating the investigatory procedures to be followed in performing an evaluation - and also the product resulting from that process.

Evaluation Standards. There are two meanings to this term - external standards and internal standards. External standards are those developed by The Joint Committee on Standards for Educational Evaluation, and they are a set of principles and guidelines for the professional conduct of evaluators and clients. Internal standards are those developed within an evaluation, usually accompanied by measurable criteria, to act as measuring sticks against which to make judgments on the program.

Evaluator. A professional with responsibility for evaluating programs, who works internally within an organization or institution in an evaluation role, or externally on a contractual basis to complete evaluations.

External Evaluator. A contractual evaluator who is hired by the client for the purpose of evaluating a specific program, for a specific, short time period.

Formative Evaluation. It is conducted usually during the development or pilot-testing of a program for improvement purposes. It is usually conducted for in-house staff of the program only. However, formative evaluation can be done by an external evaluator once a program is in its first implementation, or even later. The distinguishing feature here is purpose - formative evaluation has the intent to improve.

Goals. General aims or desired outcomes of a program - more general than objectives.

Informed Consent. To meet ethical responsibilities, evaluators are bound to obtain consent from all participants before collecting data in an evaluation, and that consent should be informed. Informed consent implies that participants will be informed of why the data are being collected, how data will be used, what their contribution will involve, and how their anonymity will be protected.

Internal Evaluator. Program or project staff responsible for conducting an evaluation internally. They may be specialized evaluation staff, but they are internal to the organization.

Interviewing. A frequently used means of collecting data in performing evaluations. It can range from structured to unstructured. It can be conducted via telephone or face-to-face. The purpose is to discover what people know, think, and feel about a program or some element of it.

Likert Scale. Refers to evaluation or research instruments designed to measure attitudes. Likert scale items consist of a sentence, followed by a four or five point scale that commonly ranges from strongly agree to strongly disagree. Respondents select the scale item which best reflects their attitudes.

Measurement. The determining of the magnitude of a quantity - usually, although not necessarily, on a test or standardized instrument with a numerical scale.

Member Checks. This refers to checking back with participants who supplied data regarding the accuracy of the analysis of that data. Member checks are done to ensure that the evaluators' summary description and analysis are consistent with the participants' views.

Merit. The intrinsic value of a program (evaluand). Benefits that lie beyond application or use. For example, it might be argued that students should study or be exposed to classical music - not because they can apply it to their studies or work - but because it enriches their lives, stimulates their intellectual and emotional growth, and contributes to their overall knowledge.

Neutrality. Not supporting any view or opinion, or having no strong opinion about an entity or stance. It is often used synonymously with impartiality.

Objectivity. Implies unbiased and unprejudiced. It is commonly used also to mean uninvolved, on a personal basis. It is often mistakenly used as synonymous with neutrality, which in the strictest sense means having no strong opinion.

Observation. The process of direct sensory inspection, frequently involving trained observers. It is a valuable data collection method, but one that needs to be used carefully in terms of making interpretations from what is observed.

Open Response. Refers to questionnaire design - the type of questionnaire that permits respondents to answer in their own words. They create responses to any questions asked.

Outcomes. For the most part, post-program effects that can be measured, in some sense.

Pre-Post Design. An experimental design in which study participants are randomly assigned to levels of treatment (learning experience), and measures are collected before and after the treatment. Scores are then compared to determine if significant gains have occurred between pretests and posttests.

Qualitative Evaluation. Evaluation that is typically conducted in the program's setting, and that uses the evaluator as the main data collection instrument (i.e. s/he observes and asks questions over time). It employs an emergent design, multiple data gathering methods, and data are analysed using verbal descriptions, for the most part.

Quantitative Evaluation. Evaluation that employs specific tests and measures as data collection instruments, and that uses a structured (or a priori) evaluation design, and statistical analysis of numerical data.

Stakeholder(s). Any individual, or individuals, having a stake in, or being affected by the program being evaluated, or by the evaluation results.

Summative Evaluation. An evaluation conducted to provide decision-makers, stakeholders, and/or potential consumers with judgments about the program's merit and worth, in relation to the standards and criteria. The purpose is to determine the adoption, continuation or expansion, or termination of the program.

Triangulation. The practice of comparing results from data across different data sources or data collection methods to increase the credibility of the results. Data collected from instructors might be triangulated with data collected from administrators, or data from interviews might be triangulated with data from document analysis.

Worth. The value of an entity in terms of its utility, or extrinsic value. A program might have high worth value if graduates are well prepared for certain professional responsibilities, for example