Unit+1+Study+Questions

=Unit 1 - Readings=

In addition to reading this // Study Guide //, you should read the following textbook sections and articles from the // Readings // : 1. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). // Program evaluation: Alternative approaches and practical guidelines // (pp. 3-52). White Plains, NY: Longman. 2. Gredler, M. E. (1996). // Program evaluation // (pp. 3-19). Columbus, Ohio: Merrill. 3. Migotsky, C., et al. (1997). [|Probative, dialectic, and moral reasoning in program evaluation] .// Qualitative Inquiry, // 3(4), 453-467. 4. Levin-Rozalis, M. (2003). [|Evaluation and research: Differences and similarities.] // Canadian Journal of Program Evaluation //, 18(2), 1-31

Unit 1 - Issues to Think About
1. Think about our system of testing/evaluating students' work and assigning grades. This occurs from primary through post-graduate education. It is the focus of the educational measurement movement. How has this type of measurement/evaluation changed over the past century or more? What has it to do, if anything, with modern program evaluation? 2. If a program has a clearly delineated set of predetermined goals and objectives, and if the learners are identified and their prerequisite knowledge is determined, and if the program's content development is closely aligned with the goals and objectives, and furthermore, if the program is implemented as designed, isn't it apparent that the program must be a fine program - guaranteed success? What could possibly be a problem with such a program? 3. Think about the five theories that underlie evaluation as a field, as outlined in this unit's Commentary. Which of these theories do you identify with, as a fledgling student of program evaluation? Could you prioritize these theories, in order of impact? Should you?

** Fitzpatrick, Sanders & Worthen, 2004, p. 3-52. **
1. What are the 10 characters of a profession? 2. Is evaluation a profession, according to these 10 characteristics? (p.7) 3. List three examples of evaluation uses. 4. What is the basic difference between formative and summative evaluation? (p. 16/7) 5. What is the difference between program evaluation and evaluation research, according to Fitzpatrick, Sanders, and Worthen? (p.17) 6. Who are the audiences for formative evaluation? For summative evaluation? (p.18) 7. What is the difference between internal and external evaluation? (p.23) 8. Which era (in terms of decades) saw the most growth and development of program evaluation as a field of study and practice? (p.) 9. Could you make a time line of the key events in the history of program evaluation? What would be the highlights? (p.) 10. Is evaluation a profession or a discipline? Or both? (p.42 last paragraph)
 * formative: if the evaluation's primary purpose is to provide information for //program improvement//. Often, such evaluations provide information to judge the merit or worth of //a part// of a program.
 * summative: if the evaluation is concerned with providing information to serve decisions or assist in making judgments about a program's overall worth or merit in relation to important criteria. Scriven defines it as, " evaluation done for, or by, any observers or decision makers who need valuative conclusion for any other reasons beside development."
 * Robert Stake: "When the cook tastes the soup, that's formative evaluation; when the guest tastes it, that's summative evaluation."
 * formative: the audience is generally the people delivering the program or those close to it, (examples given: those responsible fore developing the new schedule, delivering the training program, or managing the mentoring program). Because formative evaluations are designed to improve programs, it is critical that the primary audience be people who are in a position to make changes in the program and its day to day operations.
 * summative: the audience include potential consumers (students, teachers, employees, managers, or health officials in agencies that could adopt the program), funding sources (taxpayers or a funding agency), and supervisors and other officials, as well as program personnel. The audiences for summative evaluations are often policymakers or administrators, but can, in fact, be any audience either the ability to make a "go or no go" decision.

** Gredler, 1996. **
1. What is the basic difference in assessment and evaluation, according to Gredler? (p.5 2nd last paragraph) 2. How does Gredler define evaluation - in one sentence? (p.) 3. What global event launched evaluation into national prominence in the USA? (p.) 4. Why is the // production factor of early evaluation studies a poor fit with social reality? // (p.9 end) 5. What are alternative assessments? (p.) 6. How does program evaluation relate to and differ from educational research, accountability, and accreditation movements? (p.)

** Migotsky, C., et al. (1997). **
1. What do these authors object to, in Scriven's position on program evaluation practice? 2. What do the authors mean by the term // probative? // 3. What do these authors mean by the term // dialectic? // 4. What are the four elements of Scriven's // evaluation logic? // -- Scriven talks about logical sequence of concepts that defines how people try to connect data to value judgments that the evaluand is good or bad, better or worse, -- passing or failing, or the like. [|Scriven outlined the four step]s: 5. Is Scriven promoting an absolute stance or a relative stance, in his determination of evaluation standards? Which stance are these authors promoting? 6. What argument does Schwandt make about moral reasoning and its dichotomy with Scriven's probative, value-controlled thinking? 7. Is judgment probative? Can it be? 8. Do these authors wish to expand the evaluator's ability and use of judgment, or to control it? 9. What is meant by // triangulation, and why is it valuable to program evaluators? // 10. Merit does not equal productivity, these authors claim. What does it equal?
 * They object to Scriven's persistent drive for objective, analytical tactics i the conduct of evaluations, in particular the final synthesis procedure. Scriven encourages the move away from a model of personal, intuitive judgment to one of analytic objective heuristics. (pp. 454/463)
 * probative: tending to prove or seek the truth, the objective, analytical, and rationale approach to program evaluation.
 * Scriven notes that probative inferences should be believed until discredited.(p.457)
 * Scriven argues that we need to (1) compile a comprehensive list of possible merits (using conceptual analysis and empirical needs assessment, (2) look at performance on each of the criteria, (3) in some cases a more complex inference is required - rework things, (4) ten criteria are weighted by relative importance.(p.457)
 * dialectic: A method of argument or exposition that systematically weighs contradictory facts or ideas with a view to the resolution of their real or apparent contradictions.
 * need to concentrate resources into providing better review mechanisms to improve the value resolution process: focus on opportunities and expectation for self-reflection, opportunities for professional critique (includes casual critiques and just talking it over). Sharpening judgmental powers of program evaluators and increase external critiquing of the judgments. (p. 460/1)
 * selecting criteria of merit, those things the evaluand must do to be judged good
 * setting standards of performance on those criteria, comparative or absolute levels that must be exceeded to warrant the appelation "good"
 * gathering data pertaining to the evaluand's performance on the criteria relative to the standards
 * integrating the results into a final value judgment.
 * Scriven claims we can rank simple cases and even reach some conclusions of "absolute" merit. The authors are promoting a more relative stance on merit.
 * Schwandt describes the process of interpreting ethical and moral dilemmas as technical problems as a denial of the "messiness" of social inquiry and thereby a limitation to the expertise of the professional evaluator and that such knowledge disengaged from the subjectivity of morality would, therefore, have little direct applicability in the competing values that make up human interaction and social life. (p. 458)
 * Judgment is an intrinsic process. Evaluation is a perception-judgment unity. There is no perception independent of conceptualization. There is no recognition without meaning; there is no meaning without valuing. Perception is the process through which sensations are interpreted, using knowledge and understandings of the world, so that they become meaningful experiences. Formal evaluation can emphasize either the suppressing of these personal judgments or the refining of them. Judging goodness requires cultivating perceptual awareness of those particularities. (p. 460)
 * Definitely expand the evaluator's ability and use of judgment.
 * experienced evaluators gains understanding by studying the evaluand from different point of view and from different frames of reference (sounds like social constructivism). In creating valid representation and value judgments, evaluators expose their interpretations to rival conceptual perspectives using different conceptual frames, modifying, and refining notions of merit. This triangulation process assures multiple viewpoints and provoking new interpretations. This sometimes can cause conflicting representations and create tension, but this can encourage deeper/fuller understanding of the evaluand. (p. 461)
 * a process for assuring multiple viewpoints rather than reducing to a single viewpoint. (p. 461)
 * see #8
 * (Reminds me of social negotiation in social constructivism.)
 * Scriven defines merit in terms of performance. (p.462)
 * Alternatively, the authors define merit not only in terms of outcomes, but summative evaluation also portrays how well the program was constituted and operated.
 * Programs that contribute to the improvement in social well-being are meritorious just by existing.
 * Following state-of-the-art practices to some extent is meritorious.
 * For different purposes/populations/contexts/times programs have different values, not as Scriven notes, in productivity alone. (p.462)

** Levin-Rozalis, M. (2003). **
1. Which domain signals the main difference between research and evaluation? 2. How does research look at knowledge? 3. How does evaluation look at knowledge? 4. What is theory-driven evaluation? 5. How is deductive logic in opposition to the logic used in evaluation? 6. Evaluation is not theory — dependent; it is field dependent. Do you agree with this statement made by the author? (yes) 7. What are the principles of abduction? 8. How do research methods and evaluation methods differ? 9. What is the function of internal and external validity in evaluation? > 1. the "cause" precedes the "effect" in time (temporal precedence), > 2. the "cause" and the "effect" are related (covariation), and > 3. there are no plausible alternative explanations for the observed covariation (nonspuriousness). > In scientific experimental settings, researchers often manipulate a variable (the independent variable) to see what effect it has on a second variable (the dependent variable). For example, a researcher might, for different experimental groups, manipulate the dosage of a particular drug between groups to see what effect it has on health. In this example, the researcher wants to make a causal inference, namely, that different doses of the drug may be held responsible for observed changes or differences. When the researcher may confidently attribute the observed changes or differences in the dependent variable to the independent variable, and when he can rule out other explanations (or rival hypotheses), then his causal inference is said to be internally valid. [|Wikipedia] 10. Is relevance an acceptable replacement for external validity, in terms of evaluation rigor? 11. Is causality an essential outcome of program evaluation? 12. The author sees research and evaluation as separate and distinct disciplines, despite their sharing concepts, data collection methods and instruments. Does her article provide solid argument for this stance? Has it convinced you of the separateness of these two disciplines?
 * In the domain of application, lies the main difference between research and evaluation . (p. 5)
 * research is meant to formulate general knowledge as general laws. The understanding of the knowledge generated is to create an abstract generalized law that can be applied to as many events as possible. (p. 5/6)
 * research is intended to enlarge a body of knowledge. (p. 6)
 * evaluation is intended to gather knowledge and understanding of a concrete activity/project and to give this understanding back to the project as feedback. (p. 6)
 * evaluation attempts to investigate the mutual influences between a maximum number of variables at a given time and place. (p. 6)
 * knowledge is only a means to an end and its value is first and foremost in the feedback for the evaluated project. (p. 6)
 * Some evaluators have found solutions for this problem by borrowing methods from other disciplines, others, by relating to theory. (p. 4)
 * Evaluators facilitate program stakeholders to clarify their program theory or model. (program theory: stakeholders’ implicit and explicit assumptions on what actions are required to solve the problem and how the problem will respond to the actions.) The program theory is then used as a framework to guide the design of evaluation design, the selection of research methods, and the collection of data. [|link to a PowerPoint] or (p. 9)
 * because the evaluation examines the object in order to reveal the variables and the elements that play a role and the connections between them. It does not use the object (evaluand) to validate variables and suppositions stemming from an existing theory. (p.9)
 * using deductive logic one would never know the real causes, because the theory leads to the mechanisms, contexts, and outcomes that are part of its frame of reference. (In reference to contraception use and counseling.) It is not important when examining a theory, but it is crucial when we want to know what it is that works in a project.
 * evaluators rely on theoretical knowledge, but do not attempt to validate theories.
 * attempting to draw hypotheses from some specific theoretical framework will limit the scope of the evaluation and prevent the evaluator from making hypotheses that do not arise from this theoretical framework.
 * theoretical frameworks dictate the concepts we use and their expected relationships with each other.
 * In an evaluation, a theory that is suitable to the project can be chosen at a later stage, when the evaluator is drawing conclusions and explaining the findings. (p. 11)
 * situated logic - look locally and then forward.
 * The principles of abduction are based on the notion that there are no a priori hypotheses, no presuppositions, and no advance theorizing. Each event is scrutinized and its importance examined. Hypotheses are then formed about the event: Is it connected to other events and, if so, how? Perhaps it is an isolated event and, if so, what is its meaning? The explanations we form for these new events are “hypotheses on probation,” and a cyclical process of checking and rechecking against our observations takes place, widening and modifying the explanation through this process. [|Link to another paper by Levin-Rozalis] or (p. 12)
 * The main aim of research is to increase our understanding and knowledge concerning humanity in general, and thus it must find variables that can be generalized and formulated into rules. In order to do this, the researcher must sieve all the variables connected to a specific event that are not subject to generalization or do not belong to the system of suggested explanations.
 * In contrast, the activity of evaluation tries to use all the phenomena that have been discovered in the examination in order to present a coherent system of explanation for the project being examined, thus diminishing the ability to generalize the findings. (p. 14)
 * Research stems from a theory and its hypotheses and empirical generalizations. It is the fruit of pedantic attention to planning a strategy for operationalizing the hypotheses - formulating the variables in observational terms. An observational term has to be anchored in reality; a suitable situation for observing must be found or artificially created before the appropriate scientific observations are carried out to test the hypothesis. To do this, the researcher must choose the variables that seem most suitable to the concepts of the hypothesis: the most suitable subjects, the most felicitous field and the most appropriate set-up.
 * Evaluators cannot choose the participants, set-ups, or variables of the project. The field is given, the participants are given, and, at least in part, the variables are not known in advance. There is a general definition of the evaluation questions, but they are not defined in terms of a hypothesis and the variables are not operational. The instruments of evaluation (interviews, discussions, observations, questionnaires, videos, analysis of protocols, dialogue analysis, or any other tool) are planned and chosen according to the population involved, the activities to be checked, the question to be evaluated, the time and money available, and the contract signed between the operators of the project or the initiators and the evaluation team.
 * Evaluators do not examine isolated variables; they examine events, which include most of the possible variables together with their interconnections and contexts, as well as factors that are not variables (the type of neighbourhood or the character of the school, for example). In their analysis of the events, evaluators define categories and variables, based on previous knowledge and according to either the aims of the project or the finding. For example, if a variable like opacity in definition of roles occurs repeatedly in different aspects and locales of the project, it then becomes possible to see if its occurrence and influence are identical in each case, what its origins are, and so on.
 * As in anthropological or qualitative research, the evaluator him/herself is an instrument. In addition to their academic and professional qualifications, it is important for evaluators to have the ability to make contact and communicate, for example, to win trust and to present their findings in a way that will make them not only understandable but also acceptable for application. Despite the importance of the evaluator’s personality in this process, evaluation must be systematic, structured, and professional, and not “art” or intuition-based. (p. 15/16)
 * A threat to validity would be any factor that influences the results of the evaluation.
 * Internal validity is the degree the treatment or intervention effects change in the dependent variable. The greater the ability a evaluator can attribute the effect to the cause, rather than to extraneous factors the higher the degree of confidence that the treatment or intervention caused the effect.
 * Inferences are said to possess internal validity if a causal relation between two variables is properly demonstrated. A causal inference may be based on a relation when three criteria are satisfied:
 * Inferences about cause-effect relationships based on a specific scientific study are said to possess external validity if they may be generalized from the unique and idiosyncratic settings, procedures and participants to other populations and conditions. Causal inferences said to possess high degrees of external validity can reasonably be expected to apply (a) to the target population of the study (i.e. from which the sample was drawn) (also referred to as population validity), and (b) to the universe of other populations (e.g. across time and space).
 * The most common loss of external validity comes from the fact that experiments using human participants often employ small samples obtained from a single geographic location or with idiosyncratic features (e.g. volunteers). Because of this, one can not be sure that the conclusions drawn about cause-effect-relationships do actually apply to people in other geographic locations or without these features. [|Wikipedia]
 * But evaluators only stress internal validity, as they interested in the here and now of a specific project and not the generalizability of the evaluation. (p.17)
 * However, evaluators are interested in if the effect of the project can be generalized or what the scope of the projects' application may be. (Generalize their conclusions, but not their findings.) (p. 18)
 * since most of the concepts of evaluation are indirect/abstract/theoretical constructs, one can never be completely confident that what they are measuring is indeed the thing that they want to measure. But this is okay because the operational definition and the question used in the evaluation stem from the field and not from a theory. (p. 17)
 * definitely, for the evaluation to be effective, the feedback and value embodied in it must be relevant. (p. 20)
 * A significant element in feedback from evaluation is causality. When we know the cause then we can try to control the effect. (p.23)
 * yes and yes.

The study questions for this course were written by: [|Mary Kennedy]