Unit+4+Study+Questions

=Unit 4 Issues to Think About=


 * 1) Your textbook authors (Fitzpatrick, Sanders, and Worthen) present separate chapters on quantitative data collection and qualitative data collection. Do you think later evaluation authors (i.e. participant-oriented evaluation authors and authors writing about evaluation in the 1980s and 1990s) would support this separation of methodologies? Explain.
 * 2) The application of experimental designs in program evaluation is problematic. Can you list some reasons why implementation of experimental designs are problematic?
 * 3) While experiments are quite useful for demonstrating cause and effect relationships, they suffer from some major disadvantages. First, good experiments are difficult to conduct. They require a lot of human energy and resources. Second, it takes a lot of ingenuity, cleverness, and experience to design experiments well. Third, experiments often take the behaviour we are interested in out of context. This sometimes produces considerable artificiality and some question how readily we can generalize any findings to other contexts. Finally, in some contexts, there are questions as to how ethical it is to manipulate people's exposure to the things being studied. For example, if we really believed that exposing children to violent TV does increase aggressiveness are we justified in deliberately exposing such children to the violence? ([|link])
 * 4) There are multiple methods of collecting data that can be used in implementing a program evaluation. Many evaluations make use of multiple methods, and one positive outcome of the multiple data collection approach is that data are triangulated. Can you think of one or two negative outcomes of using multiple data collection methods? In balance, which approach would you undertake?
 * “Subjectivity” of nature of data collection and analysis ([|link to ppt])
 * Observations may be selectively reported making it impossible to gauge the extent to which they are typical
 * Risk of collecting meaningless and useless information from participants.
 * Problems of objectivity vs detachment
 * Problems of ethics: Entering the personal world of the participant
 * Very time consuming

Unit 4 Readings
In addition to reading this Study Guide, you should read the following textbook sections and articles from the Readings:
 * 1) Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). //Program evaluation: Alternative approaches and practical guidelines// (pp. 260-300). White Plains, NY: Longman.
 * 2) Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). //Program evaluation: Alternative approaches and practical guidelines//(pp. 301-374). White Plains, NY: Longman.
 * 3) Janesick, V. J. (1998). //Stretching exercises for qualitative researchers// (pp. 13-43). Thousand Oaks, CA: Sage.
 * 4) Poulin, M. E., Harris, P. W., & Jones, P. R. (2000). [|The significance of definitions of success in program evaluation]. //Evaluation Review, 24//(5), 516-536.

Unit 4 Study Questions

 * Fitzpatrick, Sanders & Worthen, 2004, p. 260-300**

Stufflebeam's (1973a) resultant structure for developing evaluation designs includes these six activities/functions: (p. 260)
 * 1) //Focusing// the evaluation
 * 2) //Collecting// information
 * 3) //Organizing// information
 * 4) //Analyzing// information
 * 5) //Reporting// information
 * 6) //Administering// the evaluation

(Questions 1 to 4 are not from the assigned readings, not sure about question 5.) 1. The authors delineate criteria and standards hierarchically, with criteria being more general than standards. Is this representative of the way most evaluators use these terms? 2. What is the stakeholders' function in delineating evaluation questions? 3. Is the process of determining standards and criteria dependent on type of evaluation model selected? 4. In evaluation planning, is the client involved? 5. Are research designs useful in evaluation contexts?

6. What do the authors suggest as a classification scheme for data collection methods? (p. 268) 7. What are the elements of an evaluation management plan, as described by the authors? (p. 277) - A management plan is needed to structure and control resources, including time, money, and people. - A good management plan must specify for each evaluation question the following:
 * I Data collected directly from individuals identified as sources of information
 * **-** A. Self-reports (1) Paper-and-pencil methods (e.g., structured questionnaires, unstructured surveys, checklists, inventories, rating scales) (2) Interviews (structured or unstructured, personal or telephone) (3) Focus groups (4) Personal records kept at evaluator's request (e.g., diaries, logs)
 *  B. Personal products (1) Tests: a. Supplied answer (essay, completion, short response, problem solving) b. Selected answer (multiple-choice, true-false, matching, ranking) (2) Performances (simulations, role-playing, debates, pilot competency testing) (3) Samples of work (portfolios, work products of employees)
 * II. Data collected by an independent observer
 * **-** A. Narrative accounts
 * - B. Observation forms (observation schedules, rating scales, checklists)
 * III. Data collected by a technological device
 * **---**-- A. Audiotape
 * - B. Videotape
 * - C. Time-lapse photographs
 * - D. Other devices (1) Physical devices (blood pressure, air quality, blood-alcohol content, traffic frequency or speed) (2) Graphic recordings of performance skills (3) Computer collation of participant responses
 * IV. Data collected with unobtrusive measures
 * V. Data collected from existing information resources or repositories
 * **--**--- A. Review of public documents (federal, state, or local department reports, databases, or publications)
 * - B. Review of organizational documents or files (files of client records, notes or products of employees or program deliverers, manuals, reports, audits, publications, minutes of meetings)
 * - C. Review of personal files (correspondence or E-mail
 * 1) the tasks to be performed and the time lines for each task,
 * 2) the personnel and other resources required to complete the task, and
 * 3) the cost.

Beginning & Ending Dates ||  Personnel Involved & Estimated Costs ||  Other Resources Needed & Costs ||  Total Task Cost ||
 * Sample headings for a management plan worksheet,**
 * <span style="background-color: rgb(255, 255, 255);"> Evaluation Question || <span style="background-color: rgb(255, 255, 255);"> Tasks || <span style="background-color: rgb(255, 255, 255);"> Estimated Task

Fitzpatrick, Sanders & Worthen, 2004, p. 301-374 <span class="__mozilla-findbar-search" style="padding: 0pt; background-color: yellow; color: black; display: inline; font-size: inherit;"> Data Collection Designs
 * **Case studies** may be used to accumulate evidence for //causal// relationships and are invaluable for exploring issues in depth, providing "//thick descriptions//" of programs in implementation, different outcomes, contextual issues, and needs and perspectives of various stakeholders.

>> groups as a result of dropouts. Many use pretests as benchmarks to report the change that has occurred in those participating in the program from before the program to its conclusion. These reports are often appealing to stakeholders; however, pre-post comparisons can be misleading because the change from pre to post can he due to the program and/or other factors in the participants' lives (e.g., natural changes that occur with the passage of time, other learning, and intervening events). Instead, the post measure of the comparison group is generally the more appropriate comparison because it better represents what the treatment group would have been like-at that point in time-if they had not received the new curriculum or program. In other words, the comparison group experienced the same other factors and passage of time that contributed to change from the pre to the post measure in the group receiving the new program; they simply did not experience the program. So, the difference between the comparison group and the treatment group more >> clearly reflects the effect of the program than does a comparison of the change >> from the pre to post measures.
 * **Experimental designs**, may be used when the stakes are high, a research design to investigate cause and effect relationships between interventions and outcomes. Experimental designs are case controlled studies which use **random sampling practices** to place subjects in **control groups** and **experimental groups** and then compare the differences in outcomes. Experimental designs, if feasible, are preferable to quasi-experimental designs in that they can counter more threats to the internal validity of the study. (p. 311)
 * //Posttest-only designs//: (1) decide what comparison are desired and meaningful, (2) assure that the 'students' in the two (or more) comparison groups are similar. (3) collect information at the end after the program ends (posttest) to determine whether **difference occurred**. The name of the design, posttest-only, does not dictate the measure to be used. Post-treatment measures can be surveys, interviews, observations, tests, or any other measure or measures deemed appropriate for getting a full picture of the outcomes. (p 311/312)
 * //Pre-post designs//: when a pretreatment measure can supply useful information. For example, it the groups are small, there may be concern about their equivalence. A pretest can help confirm their equivalence, though only on the measures collected. If there is **concern that many participants may drop out** of the program, and, thus, scores on the posttest may not represent equivalent groups, pretest scores can be used to examine differences in the two


 * **Quasi-experimental designs** can be useful when **random assignment is difficult or inappropriate**. A research design that does not involve random allocation of subjects between control and experimental groups, but approximates that design in other respects, e.g., using a comparison group that is matched on all apparently relevant characteristics. The term, due to Campbell and Cook, is misleading because it suggests that these designs are less adequate for establishing causal connections, when in fact they are often just as adequate, and often have other huge advantages (e.g., reduced cost, shorter timelines, less ethical problems). The key error here is to think that 'adequacy' involves some standard that goes beyond the scientific and legal requirement of 'beyond reasonable doubt' which can readily be obtained by other designs. ([|link])
 * //Interrupted time-series designs// (p. 316) to **establish a trend for change/cause**
 * Random assignment is inappropriate or impractical (e.g., program is a policy or law that applies to everyone)
 * **Existing data** are available that have consistently measured the construct of interest
 * Quite a few data collection points exist to establish a trend prior to the new program or policy
 * Few, it any, other factors are anticipated to occur concurrently that could also change the construct
 * The program or policy should have a relatively **quick impact**
 * //Nonequivalent comparison group design// is **similar to pre-post** design, but there is no random assignment to groups. Try to find an **existing group** very similar to the one that will receive the new program. The pretest is a more important component of this design than it is in the experimental designs because it helps us examine **similarities between the groups.** Of course, the goal is to establish the equivalence of the groups, if only on the pre-measure. (p 316)
 * //Regression-discontinuity design// is used when eligibility for the program to be studied is determined by a person's "scoring" above or below a certain point on the eligibility criterion (e.g., high blood pressure or cholesterol levels). The design then compares outcomes for people in the program with outcomes for people who were not eligible for the program, using regression methods. A 'discontinuity" in the line, or a difference in the regression line, for the two groups suggests a program effect. This design can be useful when programs are limited to those most in need or most qualified, such as a program for highly gifted students, and eligibility is determined by a clearly defined cut point.(p. 316/7)

>> to examine the impact of an intervention, but simply to explore and describe changes in the construct of interest. The results of a time-series design can be very useful at the beginning stage of a case study if the evaluator explores with stakeholders their interpretations of the ups and downs exhibited in the results. Their perspectives may point the way to the next steps in data collection. (p. 318)
 * **Descriptive Designs**//:// Case studies are often used for descriptive purposes when the desire is to examine an issue from many different perspectives. Unlike the qualitative case study design, these designs do not provide in-depth descriptions. They are fairly simple designs but are used frequently to answer rather straightforward questions.
 * //Cross-sectional designs// provide useful **quantitative** information on large numbers of individuals and groups to show a **'snapshot in time."** Cross-sectional studies (also known as Cross-sectional analysis) form a class of research methods that involve observation of some subset of a population of items all at the same time, in which, groups can be compared at different ages with respect of independent variables, such as IQ and memory. The fundamental difference between cross-sectional and longitudinal studies is that cross-sectional studies take place at a **single point in time** and that a longitudinal study involves a series of measurements taken over a period of time. Cross-sectional research takes a 'slice' of its target group and bases its overall finding on the views or behaviours of those targeted, assuming them to be typical of the whole group. ([|link])
 * //Time-series design// is intended to demonstrate **trends or changes over time**. Unlike the interrupted time-series design, the purpose of the design is not

>>>> These results might then be used to conduct focus groups or more intensive interviews with subsets of the population tested or surveyed. This stage of data collection can probe for greater understanding of the survey results. (p. 320)
 * **Mixed Methods Designs:** There is no one design that is best for all settings. A good design is one that matches the evaluation questions developed during the planning phase, the context of the evaluation, and the information needs and values of stakeholders. When using mixed methods, the evaluator should consider her purpose or purposes in using those mixed methods and select the design or approach most appropriate for achieving that purpose. (p. 319)
 * Component Design: These designs are common, and do expand our knowledge of the program, but are not necessarily the best use of mixed methods designs because the contrasting methods focus on such different issues. (p. 320) [I am not sure if they include triangulation in this, as it measures the //same thing// using tools that have different biases.]
 * **Triangulation** refers to the convergence or corroboration of data gathering and interpretation about the same phenomenon. The exact approach or form of data gathering and/or interpretation can vary. For example, researchers sometimes state they are using data triangulation, investigator triangulation, theoretical triangulation, or methodological triangulation. Data triangulation refers to the convergence or corroboration of data about the same phenomenon. Investigator triangulation refers to the collaboration of two or more investigators to gather and interpret the data. Theoretical triangulation refers to the use of more than one theoretical framework to guide the conceptualization of the study and the interpretation of the data. And, methodological triangulation refers to the use of more than one method to gather the data. The terms methodological triangulation and triangulation are often used by different researchers as being synonymous with the broader designation of mixed or multiple methods. The use of these terms can be confusing. ([|link])
 * **Complementarity** reaches beyond triangulation by focusing not only on overlapping or converging data, but also on the different facets of phenomenon, providing a greater range of insights and perspectives.([|link]) It is designed to gain a fuller understanding or picture of the construct of interest. Methods with different biases may still be selected, but not with the hope of results converging and increasing validity. Instead, the hope is for somewhat different results that, when combined across methods, will provide a fuller picture of the abstract constructs we tend to examine in evaluation. (p. 306)
 * **Expansion** is the overall widening of the scope, breadth, or range of a study. ([|link]) and provides a fuller picture of the program, but not of any individual construct. (p. 306)
 * Integrated Designs mix methods and paradigms at many different stages. Caracelli and Greene see these designs as more desirable, writing that they "have the potential to produce significantly more insightful, even dialectically transformed, understandings of the phenomenon under investigation." (p. 320)
 * [[image:Iterative_Design_Process.jpg caption="Iterative Design"]]
 * **Iterative/spiral design** ([|link])
 * The evaluator uses different methodologies, from different paradigms, in sequence with the results of each informing the next stage of data collection and interpretation. Thus, interviews might be conducted to begin tapping the construct and to provide information for constructing a survey or some other paper-and-pencil measure such as a test. This measure could be given to a broader sample of people to further explore perspectives gained from the initial interviews and study the degree to which such views are held across many different people and subgroups.
 * [From gaming] Iterative design, graphic above, is a design methodology based on a cyclic process of designing, testing, analyzing, and refining a work in progress. In iterative design, interaction with the designed system is used as a form of research for informing and evolving a project, as successive versions, or //iterations// of a design are implemented. Test; analyze; refine. And repeat. Because the experience of a viewer/user/player/etc cannot ever be completely predicted, in an Iterative process design decisions are based on the experience of the prototype in progress. The prototype is tested, revisions are made, and the project is tested once more. In this way, the project develops through an ongoing dialogue between the designers, the design, and the testing audience. ([|link])
 * Embedded - Different methods are embedded within each other (p. 320)
 * Holistic - Use program theory or concept mapping as structure for integrating mixed methods throughout (p. 320)
 * Transformative - Mix methods, values, stakeholders; use participatory, empowerment, action-oriented (p. 320)

The qualitative researcher concentrates on the instance, trying to pull it apart and put it back together again more meaningfully--analysis and synthesis in direct interpretation. The quantitative researcher seeks a collection of instances, expecting that, from the aggregate, issue--relevant meanings will emerge. (p. 360)

1. Define quantitative methods.
 * Quantitative research uses methods adopted from the physical sciences that are designed to ensure objectivity, generalizability and reliability. These techniques cover the ways research participants are selected randomly from the study population in an unbiased manner, the standardized questionnaire or intervention they receive and the statistical methods used to test predetermined hypotheses regarding the relationships between specific variables. The researcher is considered external to the actual research, and results are expected to be replicable no matter who conducts the research.
 * The strengths of the quantitative paradigm are that its methods produce quantifiable, reliable data that are usually generalizable to some larger population. Quantitative measures are often most appropriate for conducting needs assessments or for evaluations comparing outcomes with baseline data. This paradigm breaks down when the phenomenon under study is difficult to measure or quantify. The greatest weakness of the quantitative approach is that it decontextualizes human behavior in a way that removes the event from its real world setting and ignores the effects of variables that have not been included in the model. ([|link])

2. Define qualitative methods.
 * Qualitative research methodologies are designed to provide the researcher with the perspective of target audience members through immersion in a culture or situation and direct interaction with the people under study. Qualitative methods used in social marketing include observations, in-depth interviews and focus groups. These methods are designed to help researchers understand the meanings people assign to social phenomena and to elucidate the mental processes underlying behaviors. Hypotheses are generated during data collection and analysis, and measurement tends to be subjective. In the qualitative paradigm, the researcher becomes the instrument of data collection, and results may vary greatly depending upon who conducts the research.
 * The advantage of using qualitative methods is that they generate rich, detailed data that leave the participants' perspectives intact and provide a context for health behavior. The focus upon processes and "reasons why" differs from that of quantitative research, which addresses correlations between variables. A disadvantage is that data collection and analysis may be labor intensive and time-consuming. In addition, these methods are not yet totally accepted by the mainstream public health community and qualitative researchers may find their results challenged as invalid by those outside the field of social marketing. ([|link])

3. What is critical multiplism?
 * Critical implies that, as in positivism, the need for rigor, precision, logical reasoning and attention to evidence is required, but unlike positivism, this is not confined to what can be physically observed. Multiplism refers to the fact that research can generally be approached from several perspectives. Multiple perspectives can be used to define research goals, to choose research questions, methods, and analyzes, and to interpret results (Cook 1985). ([|link])
 * The method of critical multiplism involves testing competing models in order to determine the most inclusive and parsimonious [frugal or economical] explanation.
 * Critical Multiplism is a framework that constitutes the combination of two prescription for research planning: critical thinking and multiple representation. The critical part refers to the fact that in planning a research effort, decisions about how to carry out the research should be made critically; the opposite would presumably be mindlessly. For instance, one must decide what the outcome measure will be for an intervention study. One might simply elect to use whatever outcome measure is common in the field and let is go at that. That could be fairly mindless decision making if one knew that the common outcome measure had deficiencies or if it were unlikely to capture exactly the outcomes intended from the intervention. If one were interested in the effect of treatment of seizure disorder on reading ability of children, one would use as research subjects of children of one gender or the other, or else children of both genders. Critical thinking would lead to an active, rather than a passive, decision about what to do, along with the articulated reasons for doing is that way. Less critical approaches would be on the "convenient," or 'seemed like a good idea at the time' variety.
 * The multiplism component has to do with the fact that only if facets of research are allowed to vary or are planned to vary will we have good grounds for generalizing findings. Multiplism is precisely the thinking behind recent federal legislation that mandates that research supported by National Institute of Health (NIH) will be expected, unless otherwise justified, to include both male and female subjects from minority groups as well as from Whites. Too much important biomedical research was being done on White males and was regarded as dubiously applicable to females and minorities. That thinking, multiplism, is also behind frequent complaints that so much of psychological research is done on college sophomores.([|link]: search for Critical Multiplism)

4. How is critical multiplism related to triangulation? (p. 224 of [|Philosophical and theoretical perspectives for advanced nursing practice])
 * Critical multiplism has been called a method of 'elaborated triangulation' and, as such, is purported to add nothing new to contemporary research methodology. Post positivists counter that triangulation is part of, but not equal to, the critical multiplist approach. Triangulation has been defined as the combination of two or more theories, data sources, methods or investigators in the study of a single phenomenon. The goal of triangulation is to circumvent the personal biases of investigators and overcome the deficiencies intrinsic to a single-investigator, single-theory or single-method study to promote greater confidence in the observed findings.
 * Critical multiplism is also concerned with reducing bias, in the recognition that no one approach or measure is perfect or without bias. As a result, both triangulation and critical multiplism seek to eliminate inherent bias in the research methods chosen. However, critical multiplism goes further in that it encourages the exhaustive study of phenomena from as many different perspectives as possible, given that the recognition that theory is a huge fishnet of complex, mutually interacting relationships among constructs or variables. Further, triangulation frequently is presented as being conducted by a lone researcher of a group of researchers working in tandem to study a phenomenon. In contrast, critical mutiplism does not require that researchers work in tandem, rather they may be working in vastly different regions of the globe, working in isolation, yet studying similar phenomena in different ways.
 * Is is the essential critique and scrutiny of the multiple ways of studying the phenomena that distinguishes critical multiplism from triangulation. Critical multiplism is an encouragement to all researchers to be open to all of the possible ways of examining phenomena in an effort to arrive at warranted knowledge claims. This openness is part of the reason for critical multiplists' desire to involve multiple stakeholders, beyond mere investigators, in the process of research.

5. List a few major technical problems with data collection in evaluation. (p. 358)
 * Unclear directions lead to inappropriate responses, or the instrument is insensitive or off-target. (Always pilot-test your methods.)
 * Inexperienced data collectors reduce the quality of the information being collected. (Always include extensive training and trial runs. Eliminate potential problem staff before they hit the field. Monitor and document data collection procedures.)
 * Partial or complete loss of information occurs. (Duplicate files and records; keep records and raw data under lock and key at all times.)
 * Information is recorded incorrectly. (Always check data collection in progress. Cross-checks of recorded information are frequently necessary.)
 * Outright fraud occurs. (Always have more than one person supplying data. Compare information, looking for the "hard to believe."
 * Procedures break down. (Keep logistics simple. Supervise while minimizing control for responsible evaluation staff. Keep copies of irreplaceable instruments, raw data, records, and the like.)
 * One of the newer issues in experimental designs concerns the failure to adequately consider statistical power in planning designs. As a result, Type II errors, or failure to find significant differences between groups when such differences really exist, occur far more frequently than we are aware. Such errors can cause us to reject beneficial programs because we believe they make no difference when, in fact, small sample sizes and/or large group variability may have limited our ability to detect differences. Lipsey (1990) discusses methods for planning designs to avoid such problems. (p. 317)

6. How are experimental designs related to data collection methods employed? > I think this question should be, "How are experimental designs, related to data collection methods, employed?" (insert commas) > Experimental designs are case controlled studies which use **random sampling practices** to place subjects in control groups and experimental groups and then compare the differences in outcomes. (p. 311) 7. Is sampling an issue in performing program evaluations, normally? When is it not an issue? > To answer most evaluation questions, data will be gathered from the entire population because the population of interest is relatively small and external validity, or generalizability, beyond the group of interest is not a priority. Methods of random sampling can be used when the group is large and generalizability is important. Purposive sampling is useful when information is being collected from small numbers of individuals or units. In these cases, the purpose is not generalizability, but description and generation of new ideas. (not an issue: small units, versus it is an issue:larger units) (p. 331/2) > Must choose the appropriate sample size to have confidence in validity and credibility for the general audience for the study. (p. 322) > When groups are small it is best to collect information from all of them. However, when data collection becomes costly, as with interviews or observations, sampling is necessary. Some might think that random sampling should always be used. However, random sampling for interviews and observations could result in data that are representative but not very useful for the intended purpose. (p. 330) 8. Define cost-benefit analysis, cost-effective analysis, cost-utility analysis, and cost -feasibility analysis. >> or stakeholder is to choose from among several different ways to achieve the same goal, this method would be the correct choice. Here we simply provide a list of all costs and all benefits, when it is not possible to reduce them all to money terms. This greatly improves the quality of choices made by those who see the analysis since it usually uncovers many hidden costs and hidden benefits.([|link] and page 326)
 * cost analysis- The process of determining all the significant costs of something. This is a highly skilled process and for a long time in the history of evaluation was ignored or treated as a minor consideration, probably because in the early days, the evaluands were all programs already paid for by the state. Various species of cost analysis are useful for different purposes ([|link]):
 * **cost-benefit analysis** - This approach requires, and so will only work when it is possible to reduce all costs and benefits of an option to monetary terms (this can be difficult). It has the advantage of giving a simple answer. Each alternative is examined to see whether benefits exceed costs, and the ratios of the alternatives are compared. The alternative with the highest benefit-to-cost ratio is then selected. When that is impossible, use cost-effectiveness analysis. Often, simple cost analyzes will suffice to satisfy the client. Given their costs, cost-benefit studies are only cost-effective when stakeholders are trying to make summative decisions about programs with quite different outcomes. Should we rebuild the playground or purchase new hooks? When a choice is to be made among programs with like outcomes, other types of cost studies that do not require monetizing benefits can be more appropriate. ([|link] and p. 325)
 * **cost-effective analysis** involves comparing the costs of programs designed to achieve the same or similar outcomes. When the task for the administrator
 * **cost-utility analysis** is used to analyze alternatives by comparing their costs and their utility as perceived by users. Utility can be measured by assessing users' preference for or satisfaction with each option. The results are ratios quite similar to cost-effectiveness ratios except the ratio reflects cost for satisfaction, not effect. (p. 327)
 * **cost-feasibility analysis** - The most modest use of cost analysis is to determine whether in fact one can afford a particular option at all, when all the costs-monetary and other-are correctly computed.([|link])

9. What is the focus of a case study? (p. 307/8) > The focus of a case study is on the case itself. Such an approach may be particularly appropriate in evaluation when there is a need to provide in-depth information about the unit, or case, at hand, and not so much to generalize to a larger population. Because evaluation is typically intended to he situation specific, a case study design provides the opportunity to discover the unique attributes of an individual case.
 * **description** - describe something in depth
 * **explanation** - give the reader a real understanding of the program and the many different ways it might me viewed. The voices and perspectives of many different stakeholders involved with the program are heard.
 * **exploration** - exploring the "hows" and "whys" of a evaluand, and encourage deeper exploration of the issues, recognizing that there are many different perspectives.

10. Define focus groups. > This term refers to an important part of what is conventionally classified as qualitative methodology and to the small groups involved. Considerable expertise and a number of highly evolved techniques are involved in focus group methodology, which offers many advantages over one-on-one interviewing, such as lower cost (per head), resilience to cancellations, and the chance to draw out comments that would not surface in the personal interview, to set off against the weaknesses of reduced privacy, complex scheduling, and the need for a different kind of specialized skill. ([|link])
 * Focus groups are particularly useful in needs assessments and monitoring studies and for formative evaluations. (p. 351)
 * Focus groups can help confirm or disconfirm program theories during the planning stages of programs.
 * They can raise novel ideas based on participants' own experiences.
 * Focus groups can also be useful in discovering more about program outcomes, such as how participants have used what they gained, what barriers they faced, or what changes they would make in the program.

11. Is focus group interviewing the same as interviewing any small group of program participants together? > Focus groups are like an interview in that they involve face-to-face interaction, but they build on the group process. A skilled focus group facilitator will make use of ideas or issues raised by participants in the focus group to obtain reactions from others in the group. Discussion in focus groups is not always interviewer to interviewee, but often dialogue continues among focus group participants themselves. Thus, the interview is very much of a group process. (p. 351) > The role of the leader is to facilitate discussion by posing initial and periodic questions, moderating the responses of more vocal members, and encouraging responses of quieter members. The leader may also ask questions to clarify ambiguities or get reactions from other group members. Participating in groups can be intimidating for some. Sensitive topics can be difficult and there can be a tendency for individuals to acquiesce by agreeing with the majority of the group, i.e., group think. Fontana and Frey (2000) note that the skills required for leading a focus group are similar to those required for a good interviewer, but the leader must also be knowledgeable about methods for managing group dynamics. (p. 351) > Focus on higher order questioning or the the focus group then really becomes a structured group interview, not a locus group, because it has lost the key focus group characteristics of member interaction, openness, and exploration. (p. 351)

12. What is content analysis, and how is it applied in program evaluation? > > Content analysis is a special type of analysis of qualitative information collected in textual form (e.g., field notes, narrative interviews, newspaper articles, minutes of meetings). These procedures may be used to describe, analyze, and summarize the trends observed in these documents. Coding categories may focus on either the actual content of the document ("what is said") or underlying motives, emotions, or points of view ("how it is said"). (Remember the set of content analysis categories for newspaper articles on sex education.) (p. 362) > > Advantages of Content Analysis ([|link to ppt])
 * Transparent, replicable method
 * Counting involves minimal interpretation
 * Allows for longitudinal analysis
 * Relatively unobtrusive - no reactive effects
 * Flexible - can be applied to various texts
 * Provides information about populations that are difficult to access directly

> Disadvantages of Content Analysis([|link to same ppt])
 * Only as good as the quality of the documents
 * Coding manuals have to be interpreted
 * Variant interpretations of latent content
 * Descriptive rather than explanatory--no answers to ‘why’ questions
 * Atheoretical? [Unrelated to or lacking a theoretical basis]

13. What are mixed-method evaluation designs? > <span class="wiki_link_new">See above.

1. How does the author define empirical? > "The meaning of the term empirical at this point means, "relying on direct experience and observation" and is the cornerstone of qualitative work." (p. 30) > I am not sure what she means by //at this point//; however, I have just started reading the article.
 * Janesick, 1998, p. 13-58**

2. What are the six types of interview questions described by the author? (p. 46)
 * 1) Basic Descriptive Questions: Can you talk to me about your car accident? Tell me what happened on that evening. Describe how you felt that evening.
 * 2) Follow-up Questions:You mentioned that "planning time" is important to you. Can you tell me how you use planning time?
 * 3) Experience/Example Questions:You mentioned that you loved going to Paris. Can you give me an example or two of what made you love Paris? Talk about your impressions of Paris.
 * 4) Simple Clarification Questions: You have used the term constructivist teacher today. Can you clarify that for me? What exactly can you tell me about your constructivist teaching?
 * 5) Structural/Paradigmatic Questions: You stated that this class was a problematic one. What would you describe as the cause of these problems?Of all the things you have told me about being a critical care nurse, what is the underlying premise of your workday? In other words, what keeps you going every day?
 * 6) Comparison/Contrast Questions: You said there was a big difference between a great principal and an ordinary principal. What are some of these differences? Can you describe a few for me?

3. Does the author's interview question types mesh with Patton's four categories, presented in the Commentary of this unit? > Patton's four categories seem to divided into the cognitive/affective domains and are very general. Whereas Janesick's interview questions all deal with the cognitive domain (describe, plan, experience,..) and are more specific, thus can guide more easily novice program evaluators. [Probably more to it, but can't think of anything else right now.]

4. What are the four advantages and four disadvantages of focus groups, as listed by the author? (p. 36) > Strengths > Trade-offs/Disadvantages
 * 1) The major strength of focus groups is the use of the group interaction to produce data that would not be as easily accessible without the group interaction.
 * 2) Focus groups combine elements of both individual interviews and participant observation, the two principal data collection techniques of qualitative researchers.
 * 3) One can observe a great deal of interaction in a given limited time period on a particular topic.
 * 4) Participants' interaction among themselves replaces the interaction with the interviewer, leading to a greater understanding of participants' points of view.
 * 1) Focus groups are fundamentally unnatural social settings, when compared to participant observation.
 * 2) Focus groups are often limited to verbal behavior.
 * 3) Focus groups depend on a skilled moderator, not always available when needed.
 * 4) Do not use focus groups if the intent is something other than research; for example, conflict resolution, consensus building, staff retreats, and work to change attitudes.

5. How does the author describe analysis of interview data?

1. Why does the author emphasize the need for evaluators to know the organization's goals - both formal stated goals and informal goals? > Knowledge of goals gives insight into why an organization functions a particular way. Understanding the goals may also allow the evaluator to understand why actual organizational conduct differs from official policy. Having this knowledge allows the evaluator to recommend appropriate implementation strategies for change. Furthermore, awareness of organizational goals suggests to the organization (those responsible for implementing change) that the evaluators are committed to using the goals specifically identified by the organization when suggesting appropriate indicators of success. (p. 516)
 * Poulin, Harris & Jones, 2000, p. 516-535**

> In addition, the formal goals of an organization may not be representative of the actual behavior of the organization. Everyday decision making and behavior of members of the organization may conflict with the formal goals. (p. 517)

> Without measuring the stated goals of the organizations within the system, we risk two things: (a) not learning key pieces of information about the desired outcomes of organizations and (b) producing an evaluation that will be ignored by the organizations we seek to aid in development. (p. 517)

2. Does the approach outlined in this article apply to internal evaluation, external evaluation, or both?

3. How does the author define evaluability assessment? > An evaluability assessment is a program analysis tool that helps the evaluator learn about the program in practice in addition to the formal, theoretical program or the program "on paper." This tool is used as a means to identify the stated goals of a program. (p 519)

> Other definitions are,
 * a study to determine whether or not a program or project can be evaluated
 * a systematic process used to determine the feasibility of a program evaluation. It also helps determine whether conducting a program evaluation will provide useful information that will help improve the management of a program and its overall performance. ([|link])

4. Who made decisions on definitions of success in the case cited? (p. 519/520) The process was > policy making in the larger system of prevention services in Philadelphia.
 * 1) implement an evaluability assessment to help identify the state goals of a program (admin/staff were interviewed and asked to specify the most important goals of their program and what factors they would use to define a successful youth at their program)
 * 2) Review of the goals revealed an overlap existed in definitions of success across agencies. The selection of outcome measures were based on the most frequently cited definitions of success.
 * 3) A pilot test of potential outcome measures as well as a survey of program staff was conducted to determine which measures worked best to collect the information desired and to gather program staff opinions regarding the measures. Measures included in the pilot test were those that measured the concepts desired with sufficient validity and reliability as tested by other studies.
 * 4) Final selection of measures was based on program staff members' sense of their face validity, how they seemed to work with youths, and their ability to gather the information desired in a concise manner. The measures finally selected were those that would fit with how programs felt they should be evaluated. That is. outcome measures were //not imposed// on programs without their input but, rather, were selected based on what the programs said that they actually tried to achieve. In this way, program evaluations could be conducted that would report information that the programs would find useful for program development and

5. How are definitions of success used by evaluators? (p. 517) > Examining programs' definitions of success, over time, facilitates program development and policy making.

> Organizations are dynamic. Continual review of the goals of an organization is necessary to remain "in the know" and to respond to changes in the environment that affect the organization. Adaptation of the evaluation to the changing needs and context of the organization will optimize the proposed reforms and produce information for the organization to make informed choices about change.

> The evaluator should try to be aware of what each organization states it is actually trying to do and why it is trying to achieve these goals. Without this knowledge it is likely that the suggested reforms would not be viewed as viable or appropriate by the organizations involved, and it is unlikely that change will occur. Each organization acts as a key stakeholder in the system. These stakeholders are responsible for the implementation of changes within the system and within their organization. If the evaluator ignores the stated goals of each organization within the system, it wiIl likely produce an evaluation of little value to these organizations, an evaluation that will be ignored.

6. How are definitions of success used to assist the organization and its programs? > The inclusion of program goals with each program outcomes report informs the reader of the report with what the program is trying to do, how well they are doing it, and delineates the programs top goals (which may not be apparent). (p. 528)

> Also, periodic review of providers' definitions of success renews confidence that the outcome reports produced contain the information about which programs are most concerned. (p. 529)

7. Is the author's view of program development aligned with Owen's view of program development?
 * I see a connection between Owen's Monitoring Form - provides insight into how programs can be fine-tuned and justify their existence.
 * Participant-orientated evaluations - working cooperatively with stakeholders to gain evidence and empirically based knowledge to enhance decision-making and the effectiveness of organizations.
 * Anyone else have other ideas??? The words "program development" throws me off as Owen's book is focused on program evaluation not development.

The study questions for this course were written by: [|Mary Kennedy]