MDDE617+Unit+5+Study+Questions

=Unit 5 - Readings =

 In addition to reading this Study Guide, you should read the following textbook sections and articles from the Readings: 
 * 1) Achtemeier, S. D., Morris, L. V., & Finnegan, C. L. (2003). Considerations for developing evaluations of online courses. I(1). Available at []
 * 2) Cyrs, T. E. (2001). //Evaluating distance learning programs and courses//. Available at: [] (June 2003).
 * 3) Joint Committee on Standards for Educational Evaluation. (1994). //The program evaluation standards: How to assess evaluations of educational programs// (pp. 23-24, 63, 81-82, 125-126). Thousand Oaks, CA: Sage.
 * 4) Lockee, B., Moore, M., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. //EDUCAUSE Quarterly, 25//(1). Available at: []
 * 5) Moore, M. G. (1999). Editorial: Monitoring and evaluation. //American Journal of Distance Education, 13//(2) p. 1-5.
 * 6) Whalen, T., & Wright, D. (1999). Methodology for cost-benefit analysis of web-based tele-learning: Case study of the Bell Online Institute. //American Journal of Distance Education, 13//(1), p. 22-44.

Unit 5 - Issues to Think About

 * As the integration of online and web-based activities and resources into conventional learning is becoming prevalent at all levels of education, the distinction between distance teaching/learning and conventional teaching/learning is blurring. Does that have implications for program evaluation? If so, what are these implications?
 * Cyrs states, "Evaluation should always be based on the objectives of a program or course. Without the specification of objectives, there are no criteria on which to base an evaluation." Which evaluation model author would agree wholeheartedly with Cyrs? Which evaluation model authors would disagree with Cyrs? Do you agree, or disagree with Cyrs' statement?
 * Some readings support a comparative approach to evaluation in distance education - that is, comparing outcomes of distance education programs with outcomes of conventional education programs. What do you see as problems with this comparative stance? Would it be a disservice to distance education? Or is it just not feasible?

Unit 5 - Study Questions
<span style="background-color: rgb(255, 255, 255);">**Achtemeier, Morris & Finnegan, 2003, online** 1. What do these authors believe is the focus of evaluation plans? > <span style="background-color: rgb(255, 255, 255);">The focus of the evaluation plans should be the //Seven Principles for Good Practice in Undergraduate Education//, as they reflect good practice in teaching and learning. <span style="background-color: rgb(255, 255, 255);"> 2. The authors concentrate their efforts on the development of a course evaluation instrument. What do they base their instrument development on? > <span style="background-color: rgb(255, 255, 255);">They based their instrument development on principles of effective teaching identified in the literature, <span style="background-color: rgb(255, 255, 255);"> 3. What do student evaluations of courses usually measure? <span style="background-color: rgb(255, 255, 255);"> 4. The authors use what process in trying to define quality distance education? <span style="background-color: rgb(255, 255, 255);"> 5. Is the authors' process, as described, a sound process? > <span style="background-color: rgb(255, 255, 255);">It seems to be a sound process, as they attempt to create 'benchmarks' for the design of evaluation instruments that can be used to create valuable feedback loops for course and teaching improvement. <span style="background-color: rgb(255, 255, 255);"> 6. List the seven principles for good practice in undergraduate distance education. (p. 3) <span style="background-color: rgb(255, 255, 255);"> 7.Is student evaluation of courses or programs considered to be program evaluation? (p. 6) > <span style="background-color: rgb(255, 255, 255);">It is only meant to be a part of a multiple-methods assessment and evaluation process. <span style="background-color: rgb(255, 255, 255);"> 8. The researchers found a "great disjuncture." What was the great disjuncture between? (p. 11) > <span style="background-color: rgb(255, 255, 255);">This research found a great disjuncture between the guidelines suggested for effective teaching and learning and the principles that were evaluated by the end-of-course evaluation instruments. The absence of questions dealing specifically with the online environment suggests that many instruments used in the evaluation of online instruction were likely taken from traditional course settings and applied directly to evaluate computer-mediated instruction. <span style="background-color: rgb(255, 255, 255);">
 * <span style="background-color: rgb(255, 255, 255);">Seven Principles for Good Practice in Undergraduate Education
 * <span style="background-color: rgb(255, 255, 255);">Principles of Effective Teaching in the Online Classroom
 * <span style="background-color: rgb(255, 255, 255);">Tom C. Reeves' fourteen pedagogical dimensions (p. 6)
 * <span style="background-color: rgb(255, 255, 255);">Alexander Astin’s input-environment-outputs (frequently designated as I-E-O) model (p. 6)
 * <span style="background-color: rgb(255, 255, 255);">...
 * <span style="background-color: rgb(255, 255, 255);">amount of student-faculty contact that took place during the course (p. 8)
 * <span style="background-color: rgb(255, 255, 255);">student’s time on task
 * <span style="background-color: rgb(255, 255, 255);">questions pertaining to effective teaching in the online environment
 * <span style="background-color: rgb(255, 255, 255);">course goals were clearly articulated
 * <span style="background-color: rgb(255, 255, 255);">what degree the student was satisfied with the learning activities in the course
 * <span style="background-color: rgb(255, 255, 255);">questions about having necessary skills and equipment (few)
 * <span style="background-color: rgb(255, 255, 255);">format easy to use (few)
 * <span style="background-color: rgb(255, 255, 255);">having sufficient instructions (very few)
 * <span style="background-color: rgb(255, 255, 255);">questions pertaining to concerned with students’ perceptions of the appropriateness, reasonableness, and fairness of the course (p. 9)
 * 1) <span style="background-color: rgb(255, 255, 255);">to investigate the definitions and principles of effective teaching and learning in undergraduate education, generally, and distance education, specifically;
 * 2) <span style="background-color: rgb(255, 255, 255);">to perform a **content analysis** of instruments currently in use in the online environment using as a frame of reference the concepts and principles drawn from the literature; and
 * 3) <span style="background-color: rgb(255, 255, 255);">to develop considerations for the design of evaluation instruments in the online environment.
 * 1) <span style="background-color: rgb(255, 255, 255);">Encourage student-faculty contact,
 * 2) <span style="background-color: rgb(255, 255, 255);">Encourage cooperation among students,
 * 3) <span style="background-color: rgb(255, 255, 255);">Encourage active learning,
 * 4) <span style="background-color: rgb(255, 255, 255);">Give prompt feedback,
 * 5) <span style="background-color: rgb(255, 255, 255);">Emphasize time on task,
 * 6) <span style="background-color: rgb(255, 255, 255);">Communicate high expectations, and
 * 7) <span style="background-color: rgb(255, 255, 255);">Respect diverse talents and ways of learning.


 * Cyrs, 2001, online**

<span style="background-color: rgb(255, 255, 255);"> 1. According to Cyrs, what is the purpose of evaluating distance education programs and courses? <span style="background-color: rgb(255, 255, 255);"> 2. What is Cyrs stance on the formative/summative debate in evaluation? > <span style="background-color: rgb(255, 255, 255);">Formative evaluation is conducted formally and informally throughout a course/program to provide corrective feedback to the stakeholders that need the data. This can be accomplished through scored tests and quizzes, self-tests that are not scored, and one-minute evaluations given at the end of a class. The latter asks one pertinent question such as “ What was the most important thing that you learned in class today?” <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">Summative evaluation takes place at the end of a course or program. These data are used to re-design a course or program. This type of evaluation includes attitudes towards the course/program as well as learning outcomes. In addition, summative evaluation would also include administration of the program/course. Sample summative evaluation questions could include open ended constructed response questions such as: <span style="background-color: rgb(255, 255, 255);"> 3. Would you label Cyrs' stance on evaluation to be traditionalist or modern? Why? > <span style="background-color: rgb(255, 255, 255);">On the continuum of traditionalist and modern stance of evaluation, I would place Cyrs more towards the modern stance. Cyrs delineates his reasons for instructors choose to present a course on the WWW. I have made bold his reasons that demonstrate recent shifts in pedagogy/andragogy. <span style="background-color: rgb(255, 255, 255);"> 4. What are the www criteria suggested by Cyrs? <span style="background-color: rgb(255, 255, 255);">
 * <span style="background-color: rgb(255, 255, 255);">Who are the stakeholders that need to know the outcomes of a program or course?
 * <span style="background-color: rgb(255, 255, 255);">What needs to be known? What is the purpose 2. of the evaluative data?
 * <span style="background-color: rgb(255, 255, 255);">Why do these stakeholder need to know?
 * <span style="background-color: rgb(255, 255, 255);">When do the stakeholders need to know-during and/or after the completion of a distance learning program or course?
 * <span style="background-color: rgb(255, 255, 255);">How should the data be presented?
 * <span style="background-color: rgb(255, 255, 255);">Is the evaluation design empirical or anecdotal?
 * <span style="background-color: rgb(255, 255, 255);">How often do the stakeholders need the data?
 * <span style="background-color: rgb(255, 255, 255);">How will the data be used?
 * <span style="background-color: rgb(255, 255, 255);">Identify the strengths and weaknesses of the course/program.
 * <span style="background-color: rgb(255, 255, 255);">Would you recommend this course/program to your colleagues or other students?
 * <span style="background-color: rgb(255, 255, 255);">What would you do differently?
 * <span style="background-color: rgb(255, 255, 255);">What would you add or eliminate?
 * <span style="background-color: rgb(255, 255, 255);">How relevant and useful was the content?
 * <span style="background-color: rgb(255, 255, 255);">What are some of the benefits that you gained during the course/program?
 * 1) <span style="background-color: rgb(255, 255, 255);">The student has access to the WWW on demand. The student will explore web resources independently or as guided learning with specific **guidance from the instructor.**
 * 2) <span style="background-color: rgb(255, 255, 255);">Web-based instruction functions best in a constructionist environment.
 * 3) <span style="background-color: rgb(255, 255, 255);">**The instructor is not the sole source of information and therefore becomes a guide on the side-a facilitator rather than disseminator of information.**
 * 4) <span style="background-color: rgb(255, 255, 255);">The amount of time that a student has to learn something is variable unless for some reason it is specified.
 * 5) <span style="background-color: rgb(255, 255, 255);">Students must accept responsibility for their own learning.
 * 6) <span style="background-color: rgb(255, 255, 255);">**Learning does not take place in a fixed location. It takes place at home, at work, in a library, as well as in a classroom.**
 * 7) <span style="background-color: rgb(255, 255, 255);">The primary content resource shifts from a single text and teacher to a variety of information resources, multimedia as well as print, available worldwide.
 * 8) <span style="background-color: rgb(255, 255, 255);">Content is presented using a hypertext format with links to further levels of detail and elaboration that is under the control of the student.
 * 9) <span style="background-color: rgb(255, 255, 255);">Student identified resources (URLs) must be evaluated as to validity and reliability.
 * 1) <span style="background-color: rgb(255, 255, 255);">The class size was appropriate.
 * 2) <span style="background-color: rgb(255, 255, 255);">There was a reasonable balance between real and delayed time classes.
 * 3) <span style="background-color: rgb(255, 255, 255);">The course was totally asynchronous.
 * 4) <span style="background-color: rgb(255, 255, 255);">The instructor used:
 * 5) <span style="background-color: rgb(255, 255, 255);">listservs
 * 6) <span style="background-color: rgb(255, 255, 255);">bulletin boards
 * 7) <span style="background-color: rgb(255, 255, 255);">chat rooms
 * 8) <span style="background-color: rgb(255, 255, 255);">audio conferencing
 * 9) <span style="background-color: rgb(255, 255, 255);">e-mail
 * 10) <span style="background-color: rgb(255, 255, 255);">voice-mail
 * 11) <span style="background-color: rgb(255, 255, 255);">Student e-mail messages were answered within 24 hours.
 * 12) <span style="background-color: rgb(255, 255, 255);">Navigation through the course was easy.
 * 13) <span style="background-color: rgb(255, 255, 255);">Students were able to communicate with each other.
 * 14) <span style="background-color: rgb(255, 255, 255);">Students received adequate feedback on assignments and projects.
 * 15) <span style="background-color: rgb(255, 255, 255);">Students worked primarily in teams.
 * 16) <span style="background-color: rgb(255, 255, 255);">The navigation icons were consistent through all of the web pages.
 * 17) <span style="background-color: rgb(255, 255, 255);">Graphics were effective.
 * 18) <span style="background-color: rgb(255, 255, 255);">Graphics and pictures were easily downloaded.
 * 19) <span style="background-color: rgb(255, 255, 255);">All computer conferencing dialogue was available to all students at any time.
 * 20) <span style="background-color: rgb(255, 255, 255);">The instructor provided some useful URLs.
 * 21) <span style="background-color: rgb(255, 255, 255);">There was good discussion among teams during project work.
 * 22) <span style="background-color: rgb(255, 255, 255);">The course syllabus was clear and directive.
 * 23) <span style="background-color: rgb(255, 255, 255);">There was adequate real-time interaction with the instructor.
 * 24) <span style="background-color: rgb(255, 255, 255);">Students were taught how to identify, access, and evaluate URLs.
 * 25) <span style="background-color: rgb(255, 255, 255);">WWW contextual assumptions
 * Joint Committee on Standards, 1994, p. 22-23, 63, 81-82, 125-126**

<span style="background-color: rgb(255, 255, 255);"> 1. The Joint Committee has developed professional evaluation standards for what purpose? > <span style="background-color: rgb(255, 255, 255);">The goal of The Program Evaluation Standards is the development of evaluation standards to help ensure useful, feasible, ethical, and sound evaluation of educational programs, projects and materials. Taken as a set, thirty Standards provide a working philosophy for evaluation. They define the Joint Committees’ conception of the principles that can guide and govern program evaluation efforts. They are intended for both users of evaluation and for evaluators. ([|link to doc]) <span style="background-color: rgb(255, 255, 255);"> 2. What are the four categories of Joint Committee Standards? > <span style="background-color: rgb(255, 255, 255);">The four attributes of sound program evaluation are: <span style="background-color: rgb(255, 255, 255);"> 3. How might an evaluator make use of these standards in his/her professional practice? ([|link to same doc above]) > <span style="background-color: rgb(255, 255, 255);">These standards guide the decisions, employment, and assessment of evaluations. > <span style="background-color: rgb(255, 255, 255);">Stufflebeam (1992) describes sets of standards as noteworthy because they provide: > Fournier (1994) recommends that “The Program Evaluation STANDARDS is a “must have” book for anyone responsible for reviewing evaluation proposals, planning and conducting evaluations, managing evaluation projects, or judging the merit and worth of evaluations once completed. For experienced practitioners, it provides a set of values and principles by which to guide successful practice… For newcomers and the less experienced who may be responsible for commissioning and using evaluations, the STANDARDS supply a useful framework for generating a list of questions to raise about any evaluation plan… an invaluable “how to” resource for graduate students venturing out into the field, and it instills a sense of what it means to be a responsible evaluator… > > Patton (1994), states that, “Certainly no contemporary student of evaluation should come through a training program without studying the STANDARDS.” (p. 195) “There can be no question that these are the evaluation profession’s definitive statement of STANDARDS. I use knowledge of the STANDARDS as an indicator for knowledge of evaluation… the STANDARDS represent much more than a set of professional guidelines; they constitute a philosophy of evaluation that emphasizes and values utility, feasibility, propriety, and accuracy.” (p. 198) <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">...Read the document for more information. ([|link to same doc above]) <span style="background-color: rgb(255, 255, 255);"> 4. What does the Joint Committee mean by meta-evaluation? <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">For more info on meta-evaluation read Stufflebeam's [|META-EVALUATION] paper or this [|online resource]. <span style="background-color: rgb(255, 255, 255);">
 * 1) <span style="background-color: rgb(255, 255, 255);">The **utility** standards are intended to ensure that an evaluation will serve the information needs of intended users.
 * 2) <span style="background-color: rgb(255, 255, 255);">The **feasibility** standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.
 * 3) <span style="background-color: rgb(255, 255, 255);">The **propriety** standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.
 * 4) <span style="background-color: rgb(255, 255, 255);">The **accuracy** standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.
 * <span style="background-color: rgb(255, 255, 255);">an operational definition of … program evaluation;
 * <span style="background-color: rgb(255, 255, 255);">evidence about the extent of agreement concerning the meaning and appropriate methods of educational evaluation;
 * <span style="background-color: rgb(255, 255, 255);">general principles for dealing with a variety of evaluation problems;
 * <span style="background-color: rgb(255, 255, 255);">practical guidelines for planning evaluation;
 * <span style="background-color: rgb(255, 255, 255);">widely accepted criteria for judging evaluation plans and reports;
 * <span style="background-color: rgb(255, 255, 255);">conceptual frameworks by which to study evaluation;
 * <span style="background-color: rgb(255, 255, 255);">evidence of progress ... to professionalize evaluation;
 * <span style="background-color: rgb(255, 255, 255);">content for evaluation training;
 * <span style="background-color: rgb(255, 255, 255);">descriptions of “best evaluation practices”
 * <span style="background-color: rgb(255, 255, 255);">Meta-evaluation is the process of delineating, obtaining, and applying descriptive information and judgmental information about an evaluation’s utility, feasibility, propriety, and accuracy and its systematic nature, competence, integrity/honesty, respectfulness, and social responsibility to guide the evaluation and publicly report its strengths and weaknesses.
 * <span style="background-color: rgb(255, 255, 255);">Formative meta-evaluations—employed in undertaking and conducting evaluations—assist evaluators to plan, conduct, improve, interpret, and report their evaluation studies.
 * <span style="background-color: rgb(255, 255, 255);">Summative meta-evaluations—conducted following an evaluation—help audiences see an evaluation’s strengths and weaknesses, and judge its merit and worth.
 * <span style="background-color: rgb(255, 255, 255);">Meta-evaluations are in public, professional, and institutional interests to assure that evaluations provide sound findings and conclusions; that evaluation practices continue to improve; and that institutions administer efficient, effective evaluation systems. (l[|ink to an article abstract])


 * Lockee, Moore & Burton, 2002, online**

<span style="background-color: rgb(255, 255, 255);"> 1. Which four groups of issues do the authors include in evaluating distance education? > <span style="background-color: rgb(255, 255, 255);">A few of the factors to consider are instructional, technological, implementation, and organizational issues. <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">Formative evaluation issues: largely fall into the two primary categories (p. 22) <span style="background-color: rgb(255, 255, 255);"> Summative evaluation issues. (But remember if a program is on-going, summative evaluation cam be used as formative evaluation for course/program improvement. Also, the dichotomy of evaluation into the polarized view formative and summative is not a true reflection of reality.) >> the student support services. <span style="background-color: rgb(255, 255, 255);"> 2. How do the authors view these four sets of issues? (p. 21) > <span style="background-color: rgb(255, 255, 255);">While these factors can be isolated and itemized, by no means are they independent of each other. As in any system, the separate components must work together effectively so that the whole DE system can operate holistically. When DE delivery technologies break down, distance learners cannot engage in the planned instructional event. Without institutional policies that provide for online support services, distance learners can find it difficult or impossible to get assistance with matters necessary for their basic participation in a higher education program. Thus, a comprehensive review of DE efforts must not only scrutinize the individual system components, but also attempt to get a clear picture of how the parts work together as a whole to create positive outcomes (learning, satisfaction, matriculation, and so on). <span style="background-color: rgb(255, 255, 255);"> 3. What do the authors mean by incremental analysis?(p. 21)
 * <span style="background-color: rgb(255, 255, 255);">**Instructional design issues** (such as teaching strategy choices and assessment methods). Evaluators seek answers to the primary question of learning effectiveness. If these questions can be addressed within the formative evaluation stage, then corrective measures can produce more effective learning experiences for distance students
 * <span style="background-color: rgb(255, 255, 255);">Did students learn what the goals and objectives intended? If not, why?
 * <span style="background-color: rgb(255, 255, 255);">Was the instruction well written?
 * <span style="background-color: rgb(255, 255, 255);">Were the objectives clearly stated and measurable?
 * <span style="background-color: rgb(255, 255, 255);">Were appropriate instructional strategies chosen?
 * <span style="background-color: rgb(255, 255, 255);">Was there enough practice and feedback? Were examples provided?
 * <span style="background-color: rgb(255, 255, 255);">Did assessment methods correlate with instructional content and approaches?
 * <span style="background-color: rgb(255, 255, 255);">**Interface design issues** (Web site navigation, aesthetics, and so forth). In evaluating the interface design of a Web-based course, a few simple questions can provide insight into the strengths and weaknesses of a site’s look and feel.
 * <span style="background-color: rgb(255, 255, 255);">Was the Web site easy to navigate?
 * <span style="background-color: rgb(255, 255, 255);">Was it aesthetically pleasing, as well as legible?
 * <span style="background-color: rgb(255, 255, 255);">Did each page in the site download easily?
 * <span style="background-color: rgb(255, 255, 255);">If special plug-ins were needed, were links provided to acquire them?
 * <span style="background-color: rgb(255, 255, 255);">Also, consideration of learners with special needs should be addressed at this stage.
 * <span style="background-color: rgb(255, 255, 255);">If graphics or images were used, were alternative ways provided for sight-impaired learners to get the intended information?
 * <span style="background-color: rgb(255, 255, 255);">If course information was presented using audio, could hearing-impaired learners access transcriptions?
 * <span style="background-color: rgb(255, 255, 255);">Was information clearly available to learners with disabilities on where to get assistance if needed?
 * <span style="background-color: rgb(255, 255, 255);">**Organizational issues**.
 * <span style="background-color: rgb(255, 255, 255);">(One example: use of faculty time.) Many critics have raised the issue of the time it takes to deliver a course online, given the increased amount of direct communication with students, plus the frequent increase in student numbers. Good questions to ask relate to the balance of workload and efficient use of time by faculty who teach at a distance. (p. 24)
 * <span style="background-color: rgb(255, 255, 255);">Is time spent on DE courses significantly detracting from research and scholarship?
 * <span style="background-color: rgb(255, 255, 255);">Are DE faculty designing efficient strategies to implement their courses?
 * <span style="background-color: rgb(255, 255, 255);">An example of an organizational issue from the article. It became clear to us upon review of initial assignments in the first online course that no mechanism existed to receive and manage student work. the university established a filebox server system for all students that we leveraged for our distance learners. dealing with this issue during the formative stage of evaluation led to a workable solution before it became a crisis. (p. 25)
 * <span style="background-color: rgb(255, 255, 255);">**Implementation issues**: the process of DE has a variety of stakeholders, from students to faculty to support personnel to the host institution itself. (p. 25)
 * <span style="background-color: rgb(255, 255, 255);">Some implementation concerns are shared by **all stakeholders**, such as the reliability of the delivery technology and the accessibility and effectiveness of
 * <span style="background-color: rgb(255, 255, 255);">Other concerns are **specific to individual stakeholders**.
 * <span style="background-color: rgb(255, 255, 255);">For example, distance learners must understand the distance environment and be prepared to engage in self-directed learning.
 * <span style="background-color: rgb(255, 255, 255);">Also, distance learners should clearly understand faculty expectations and know who to contact for technological and instructional needs.
 * <span style="background-color: rgb(255, 255, 255);">Regarding faculty concerns, their preparedness to teach in distance settings is important, while accessibility to appropriate professional development activities is essential. These factors, as well as incentives and rewards for teaching at a distance, are very real issues that faculty face, hence worthy of evaluation.
 * <span style="background-color: rgb(255, 255, 255);">Finally, educational providers, such as institutions of higher education, are concerned with quality assurance. Are our distance courses and programs of strong quality and rigor? Do they meet our professional accreditation criteria? These questions can also be answered within a summative evaluation effort.

4. How do these authors view formative and summative evaluation? > <span style="background-color: rgb(255, 255, 255);">**Formative** evaluation serves to improve products, programs, and learning activities by providing information during planning and development. Data collected during the design and development process provides information to the designers and developers about what works and what doesn’t, early enough to improve the system while it remains malleable. <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">**Summative** evaluation determines if the products, programs, and learning activities, usually in the aggregate, worked in terms of the need addressed or system goal. imply, formative and summative evaluations differ in terms of the audience for the information collected, the time in the development cycle when the information is collected, and the intention behind the data collection. Summative evaluation is information provided to audiences external to the design and development team (such as funding agencies, clients, or accreditation agencies) about how the entire package works in a real setting. Although this information might be used to suggest changes, additions, segmentations, and such, it’s more likely that the information will be used to make fiscal and policy decisions to use, or continue funding, a learning system. <span style="background-color: rgb(255, 255, 255);"> 5. Do the authors have anything in common with Owen, in terms of the program aspect of evaluation? If so, what?


 * Moore, 1999, p. 1-5**

<span style="background-color: rgb(255, 255, 255);"> 1. Why does Moore believe that monitoring is required in distance education? (p. 1) > <span style="background-color: rgb(255, 255, 255);">A good monitoring system tells administrators what problems instructors and students are experiencing and indicates if delays or breakdowns occur in the communication systems-while there is still enough time to take remedial action. <span style="background-color: rgb(255, 255, 255);"> 2. Is Moore's idea of monitoring the same as Owen's idea of monitoring evaluation?

3. What is the one generalization that Moore suggests holds true for any distance education program? > <span style="background-color: rgb(255, 255, 255);">One of the few generalizations that can be made about any distance education program-whatever the communications media used and the content level-is that a good monitoring and evaluation system is likely to lead to a successful program, and a poor system is almost certain to lead to failure. <span style="background-color: rgb(255, 255, 255);"> 4. List the three elements of a program that are essential for effective monitoring, according to Moore. <span style="background-color: rgb(255, 255, 255);">
 * <span style="background-color: rgb(255, 255, 255);">first is the preliminary specification of good learning objectives (p. 1)
 * <span style="background-color: rgb(255, 255, 255);">second key to successful monitoring and evaluation is the construction and, later, the handling of the products submitted by students or trainees as (evidence of learning--commonly known as assignments (p. 2)
 * <span style="background-color: rgb(255, 255, 255);">third key to good monitoring and evaluation is a good data gathering and reporting system (p. 3)


 * Whalen & Wright, 1999, p 22-44**

<span style="background-color: rgb(255, 255, 255);"> 1. What are the two common measures incorporated in cost-benefit analysis? > <span style="background-color: rgb(255, 255, 255);">Two common measures are the <span style="background-color: rgb(255, 255, 255);"> 2. Define fixed costs and variable costs. ([|link to a glossary]) <span style="background-color: rgb(255, 255, 255);"> > <span style="background-color: rgb(255, 255, 255);">Thus, variable costs increase with the number of students, while fixed costs are incurred before a course is even offered. <span style="background-color: rgb(255, 255, 255);"> <span style="background-color: rgb(255, 255, 255);"> <span style="background-color: rgb(255, 255, 255);"> 3. In web-based education and training, what does the term capital costs refer to? (P. 28/9) <span style="background-color: rgb(255, 255, 255);"> 4. In comparing web-based education/training and conventional education, which has the higher fixed cost? (p. 42) > <span style="background-color: rgb(255, 255, 255);">Web-based training has higher fixed costs than classroom based training; however, these higher course development costs are offset by lower variable costs in course delivery. This is primarily due to the reduction in course delivery time (course compression) and the potential to deliver courses to a larger number of students than is possible in a traditional classroom without incurring significant incremental costs. Realizing savings for Web-based courses requires a sufficient number of students in order to recover course development costs. Since employees must be paid for time they spend taking a course, student salaries are an important consideration in this costing study. <span style="background-color: rgb(255, 255, 255);"> 5. In comparing web-based education/training and conventional education, which has the higher variable cost? > <span style="background-color: rgb(255, 255, 255);">Conventional education has higher variable costs, see above. <span style="background-color: rgb(255, 255, 255);"> 6. Is web-based training cost-effective, according to the authors? <span style="background-color: rgb(255, 255, 255);">
 * <span style="background-color: rgb(255, 255, 255);">break even point, the point at which costs are recovered, and
 * <span style="background-color: rgb(255, 255, 255);">return on investment, which illustrates the economic gain or loss from having undertaken a project.
 * <span style="background-color: rgb(255, 255, 255);">fixed costs: costs, which are incurred by a business whether it is operating to generate income or not and which do not necessarily increase or decrease as a total volume of production, increases or decreases. Rent, for example, must be paid whether or not any business is accomplished.
 * <span style="background-color: rgb(255, 255, 255);">example, fixed costs for videoconferencing as: videoconferencing equipment, technicians' salaries for running the equipment, installation costs, and fees for basic telephone lines.
 * <span style="background-color: rgb(255, 255, 255);">variable costs: the costs additional to fixed costs of running a business, that can vary depending on the level of demand and activity.
 * <span style="background-color: rgb(255, 255, 255);">Variable costs for videoconferencing delivery of distance courses included fees for long distance network usage, shipping charges for supplementary print materials, honoraria for professors, and salaries paid for the preparation of course materials.
 * <span style="background-color: rgb(255, 255, 255);">Examples of costs for traditional classroom delivery include
 * <span style="background-color: rgb(255, 255, 255);">instructor's salary and benefits; number of courses taught by the instructor; costs of course development, course materials, administrative support, classroom overhead, and any additional time the instructor spent on the course for activities such as grading and meeting with students
 * <span style="background-color: rgb(255, 255, 255);">development time, classroom overhead costs, instructors costs, and travel costs for the participants
 * <span style="background-color: rgb(255, 255, 255);">high travel costs for students and length of time spent away from the job
 * <span style="background-color: rgb(255, 255, 255);">Examples of costs for web-based courses include
 * <span style="background-color: rgb(255, 255, 255);">development time was high a CD-ROM course, totaling $1,205,394 over three years.
 * <span style="background-color: rgb(255, 255, 255);">equipment costs and course development costs
 * <span style="background-color: rgb(255, 255, 255);">License Fees for Learning Platform Software
 * <span style="background-color: rgb(255, 255, 255);">Hardware
 * <span style="background-color: rgb(255, 255, 255);">Content Development. Content development variables include: instructional design; multimedia design; the production of text, audio, motion video, graphics, and photos in machine-readable format; course authoring; software development; integration of content and testing; modification/adjustment; training; course testing; and motion video elements.
 * <span style="background-color: rgb(255, 255, 255);">Developer Salaries.
 * <span style="background-color: rgb(255, 255, 255);">note an example: synchronous course required far less development time primarily due to less use of multimedia
 * <span style="background-color: rgb(255, 255, 255);">employees must pay employees for the time they spend in training, student salary costs are a significant factor in costing analysis. More time spent in course delivery translates into higher student salary costs and less cost savings.
 * 1) <span style="background-color: rgb(255, 255, 255);">Capital costs include the server platform shared by all courses mounted on that server as well as the cost of the content development shared by all students taking that course. Operating costs represent the costs for the time students and instructors spend using the courses.
 * 2) <span style="background-color: rgb(255, 255, 255);">Content development includes six items:
 * 3) <span style="background-color: rgb(255, 255, 255);">instructional and multimedia design;
 * 4) <span style="background-color: rgb(255, 255, 255);">the production of text, audio, video, graphics, and photographs;
 * 5) <span style="background-color: rgb(255, 255, 255);">the development of authoring and delivery software, or the cost of licensing commercial software;
 * 6) <span style="background-color: rgb(255, 255, 255);">the integration, modification, and testing of course content;
 * 7) <span style="background-color: rgb(255, 255, 255);">student and instructor training; and
 * 8) <span style="background-color: rgb(255, 255, 255);">course testing.
 * <span style="background-color: rgb(255, 255, 255);">Web-based training can be cost-effective is the return of investment (ROI) is greater than 1 (100%).
 * <span style="background-color: rgb(255, 255, 255);">The return on investment (ROI) is the percentage that represents the net gain or loss of using Web-based training instead of classroom delivery.

The study questions for this course were written by: [|Mary Kennedy]