The course‎ > ‎

Week 7: Evaluate

21 - 27 February 2013
Twitter hashtag: #oldsmooc_w7

Bart van de Biezen
Welcome to Week 7 led by Professor Thomas Reeves from the University of Georgia, US and Dr. Yishay Mor from The Open University, UK. 

Evaluation of learning designs is critically important. Around the world, each month sees the introduction of numerous commercially produced or locally developed learning designs, OERs, and other learning events promoted as engaging, effective, and efficient. Yet systematic evaluation of these is often lacking and evidence for their efficacy is weak. To conduct a useful evaluation of learning designs requires a "triangulation" approach whereby multiple strategies and tools are applied. This week is specifically designed to establish evaluation as a key strategy throughout the learning design process.  Our specific focus will be on "formative evaluation" for the purpose of improving a learning design as it is being developed.

Learning outcomes
  • Recognize formative evaluation as one of several evaluation functions application during the learning design process, including:
  1. review;
  2. needs assessment;
  3. formative evaluation;
  4. effectiveness evaluation;
  5. impact evaluation; and
  6. maintenance evaluation.
  • Distinguish between   
research and evaluation
assessment and evaluation
formative and summative evaluation
  • Prepare a formative evaluation plan for a prototype learning design, involving a heuristic evaluation and at least one other strategy
  • Implement formative evaluation plan
  • Report and act upon formative evaluation plan
Expected time required to engage minimally with all activities is 10+ hours.
  • Short route 3+ hours (Activities 1, 2, 4, 5 & 6).  This involves thinking, reading and discussing about the role of evaluation in learning design.
  • Alternatively you can (re-)design, schedule, enact and share your own routes through these learning opportunities to learn about evaluation, just as many of you have during the first six weeks of the OLDS MOOC.
Note that if you do not have a project to evaluate, you can either evaluate any previous learning design activity you have been involved in, or you can evaluate this MOOC.


1. Begin by viewing the video introduction to the week’s activities (5 mins)
(download the transcript of the introduction video)

2. Watch the Terminology Review Slidecast 
(10 mins) that will help you make some important distinctions in a few concepts related to evaluation (direct link: of the slidecast)
  • research and evaluation
  • assessment and evaluation
  • formative and summative evaluation
3. Read the formative evaluation paper (45 mins) (download the paper here)

Post a cloud to the "Evaluation experiences" cloudscape, sharing your experience of evaluating a learning design / educational project.

Evaluation Targets
5. Identify a list of evaluation decisions that you anticipate having to make concerning your prototype learning design (30 mins)

Please note that anticipating decisions is not an easy task, but if you can do so in advance, your evaluation activities will be much more focused. Examples of evaluation decisions that might be anticipated:
  • Should the content identified for this learning design be expanded, reduced, or remain the same?
  • Should any media incorporated into our learning design be expanded, reduced, or remain the same?
  • Once the learning design is completed, how can we best share it with others?
  • What strategies can we identify to maintain our learning design in the future?
6. Prepare a list of evaluation questions that must be answered to make the decisions identified in Task 5 above (30 mins)

Please note that each decision identified in Task 5 above may have more than one question associated with it. For example, if the anticipated decision is: “Should the content identified for this learning design be expanded, reduced, or remain the same?,” then the following questions might be addressed by your formative evaluation:
  • To what extent, if any, do content experts recommend expanding or reducing the content encompassed in our prototype learning design?
  • To what extent, if any, do instructors recommend expanding or reducing the content encompassed in our prototype learning design?
  • To what extent, if any, do learners recommend expanding or reducing the content encompassed in our prototype learning design?
7. Post a cloud to the "Evaluation targets" cloudscape and to your project portfolio, describing the evaluation decisions and evaluation questions you have listed, and any reflections you have on the process.  Comment on one or two other clouds.

Evaluation Toolbox

8. We have started a toolbox of evaluation instruments in Cloudworks. There are further tools described at
Review and comment on these tools, and add any other tools that you are aware of to the toolbox. You can create a cloudscape for your own evaluation toolkit, and add the clouds of the tools relevant to your project to that cloudscape.

9. Prepare a draft formative evaluation plan for your learning design (90 mins)

You can find a template for an evaluation plan here: 

Please note that this plan template is probably more extensive than you’ll have time to write, but it can at least suggest some organization for your plan. Please post your plan in the "Evaluation plan" Cloudscape.

10. Comment on one or two other evaluation plan clouds (90 mins)

Monday and Tuesday   

11. Review the heuristic evaluation protocol at: (30 mins)

Please note that the heuristics presented in this protocol should be modified and/or reduced to best fit the nature of your learning design. For example, if your learning design is focused on the development of an Open Educational Resource (OER) that does not include any learning assessment functionality, then the assessment heuristic (14. Assessment: The e-learning program provides assessment opportunities that are aligned with the program objectives and content.) can be deleted from the protocol.

12a. Implement a heuristic evaluation of your learning design (120 mins)

Please note that this can be done online as well as in person.

- OR -

12b. Implement any other evaluation strategy to enhance your learning design prototype (120 mins)

Please note that here are many options here, but two that may be especially appropriate would be Formative Review done by someone representative of your target learners or an Expert Review by another learning designer who is not a member of your design team per se.

You can download a Formative Review Log here:
You can download an Expert Review Checklist here:

Please post your findings to the "Evaluation Results" cloudscape.

13. CONVERGE: Tuesday 26th February 12 NOON GMT Google hangout discussion. This will be a synchronous (recorded) session, which will provide an opportunity to ask questions of this week's team and reflect with others on the activities.

Reflect on your perspective regarding the merits and challenges involved in formative evaluation, by writing a short reflective blog post on the experience or adding your reflective account of week 7 to the Week 7 Cloudscape.

Please note that formative evaluation does not have to end here just because Week 7 is over. You are encouraged to continue to conduct formative evaluation activities so that your Learning Design can be improved as much as it is feasible. Formative evaluation is like exercise….you can always do more and get fitter.

Yishay Mor,
Feb 17, 2013, 6:12 AM