Tag: assessment

Microsoft Forms Updates – July 18, 2022

Microsoft has announced some great new features for Microsoft Forms. Some of these have been implemented and we are still waiting on some.

Some notable additions are that the new form and new quiz buttons have been separated into two distinct buttons and forms will now accept up to 5 million responses. One addition that is very exciting is that it is now possible to import a form or quiz from a Word or PDF document! This should make drafting forms and quizzes much easier. Check out the video below for more details.

Table of contents
0:00 Introduction
0:09 Forms home page improvements
1:20 Forms templates
2:34 Forms Quiz – Smart Convert from Word or PDF document
4:43 “Collect Responses” rename
5:46 5M max response
6:13 Polls app rename in Teams meeting
7:07 Word Cloud in Teams meetings

[UPDATE May 25, 2022] Canvas: New Quizzes

A couple months ago I shared a note about New Quizzes – a new quiz engine being developed in Canvas. In that post I share a couple known issues. One of those was that manually graded questions were not appearing on the Dashboard in the To Do list for instructors.

I am pleased to share that the Instructure development team has implemented this capability for New Quizzes. It is now possible to receive a to-do notification for manually graded items within quizzes. You can see an example notification below.

A notification link to grade a New Quizzes quiz question.
New Quizzes To Do List item

Another update Instructure shared relates to the Fill In the Blank question type. It is now possible for instructors to use the rich content editor as they create fill-in-the-blank questions. The provides more formatting options and the ability to integrate formulas into the questions. This video (2:22) demonstrates the expanded capability.

Canvas: Group Quiz, Multiple Attempts with Penalties

Team-based learning is a well established active learning pedagogical strategy. I am aware of one instructor who is using team based learning at Lawrence. In his course he has students take a quiz individually, then the same quiz as a team. The second team based quiz allows multiple attempts but penalizes the students for each subsequent attempt. The purpose of this is to encourage students to discuss what the correct answer should be before submitting. He invited me and my supervisor to sit in on a class. It was amazing to see how much discussion happened in the breakout rooms. This approach works great in Zoom, and can be equally effective in face-to-face environments as well.

Canvas is currently developing a new quiz engine, New Quizzes. It is currently available in our instance of Canvas, but not the default quiz creation tool. It looks like New Quizzes will replace Classic Quiz in December 2022.

Canvas doesn’t currently support this quizzing behavior. One main mechanism to communicate with Instructure and effect change is the Community Forums. If you find this capability compelling and useful for you or other instructors, please add your voice to this community post.

Moodle: Demystifying the Gradebook Session Summary

The Moodle gradebook can be a confusing, headache-inducing, tangle. But it doesn’t have to be. The strategies shared in this session and post can help make the Moodle gradebook more clear to instructors and students about how grades are calculated. Bringing clarity increases transparency making the gradebook more usable and inclusive.

There are a number of different methods to arrive at a grade, called aggregation methods. There are three which were the focus during the session: weighted mean of grades (WM), simple weighted mean of grades (SWM), and Natural aggregation. These three methods should be able to address the majority of desired grading scenarios.

Each of these may be used to arrive at the same grade. But they each calculate grades in a slightly different way. One big difference between WM, SWM, and Natural is that the first two normalize the grades to be out of 100 (points or percent). This can potentially create lack of clarity about how a grade is calculated. Natural used to be called “Sum of grades” and at default functions in the same way. For Natural then, the category and course totals are a sum of all the grades contained within. You can see a comparison of the three methods in the session video beginning at 0:07:50.

Strategies

Whether an instructor chooses WM-SWM or Natural grading there are some strategies that can be utilized to make the Moodle gradebook more usable.

Start with your syllabus – What is meant by this is that an instructor should review how their grading strategy is laid out in their syllabus and replicate this in the Moodle gradebook. Many instructors use categories of grades i.e. Assignments, Quizzes, Exams, Final Exam, or Participation. Create these categories in the gradebook first, before adding activities or grade items. In addition to creating an organizing structure this approach allow an instructor to add activities directly to categories when they are created making less work down the road.

Use numbers – The Moodle gradebook calculates most accurately with numbers. It can use and display letter grades, but doing so can introduce some variability. For instance, does an “A” mean 100, or 94, or… ? Scales (check, check minus, or satisfactory/unsatisfactory) are difficult or impossible in some cases for Moodle to use in calculation. If an instructor desires the grades be displayed in certain way there are options for controlling this.

Keep it simple – While it is possible to mix and match aggregation methods, to use extra credit, drop the lowest ‘x’ grades, and nest categories, doing so can make it less clear to students how a grade is calculated. Anything an instructor can do to make this is as clear as possible limits challenges to grades, and through understanding created through transparency, allows the student to engage more fully.

If you have questions about setting up your gradebook, or about grading in Moodle please contact Jedidiah Rex.

Session Video

The recording of the session (1:18:12) is posted below. It is only available to Lawrence University faculty and staff.

Resources

https://docs.moodle.org/38/en/Grade_aggregation
https://docs.moodle.org/38/en/Grades_FAQ
https://docs.moodle.org/38/en/Grader_report

Moodle: H5P update and Grading

I recently discovered that there was an update for the H5P interactive video content type. After installing the update there was a change to how H5P passed grades to the Moodle gradebook. Based on my investigation I have been able to get the interactive video to pass grades successfully to the Moodle gradebook. The update requires that students click a submit button at some point in the video (figure 1). Instructors must configure when this button appears. My notes on how this may be done follow.

H5P interactive video submit answers screen containing a green submit button.
Figure 1: H5P Interactive video submit answers button.

What the instructor needs to do

You will need to create a point in the video for the students to click a submit button. (instructions below)

What students need to do

Students will need to click a submit button (figure 1) in order for their grades to be passed back to the gradebook.

To add a Submit button

  1. Turn editing on
  2. Open the activity and then click Edit > Edit Settings
  3. Click on the “Interactions” section within the activity editor.
  4. Scrub to the point in the video you wish to place the submit button. If you want to encourage the students to watch the entire video I recommend placing this at the end.
  5. Click the ‘star” icon, then click the “+” icon to add a submit button (Figure 2)
    Scroll to the bottom of the page and click the button to “Save changes and return to course.”
Green star and plus icons highlighted inside a red rectangle.
Figure2: Star icon to place Submit button.

Options

The instructor may place the submit screen/button at any point in the video timeline. There are a couple places and reasons for placing the option.

  • Right after the last question – Doing this ensures that the answers are recorded in the gradebook.
  • At the end of the video – doing it this way may encourage students to view the entire video (is this is the instructors goal.

I don’t have a strong recommendation for either option. Choose the option that best aligns with your desired end, learning objection or outcome.

IDC04 – Project-based Assessment: Interview with Dr. Kristin Shingler

Listen to ID Corner:
Apple Podcasts | Google Play Podcasts | Spotify | RSS
All Episodes|Music provided by Kimiko Ishizaka

Transcript

In this episode I interview Dr. Kristin Shingler, Teaching Specialist in the University of Minnesota School of Dentistry about the project-based assessment she uses in her classes.

Resources

Microbial Murders Crime Scene Investigation: An Active Team-Based Learning Project that Enhances Student Enthusiasm and Comprehension of Clinical Microbial Pathogens

Kristin’s Documents

IDC01-Peer Assessment

Listen to ID Corner:
Apple Podcasts | Google Play Podcasts | Spotify | RSS
All Episodes|Music provided by Kimiko Ishizaka

IDC01 Transcript (PDF)

Welcome to the first episode of the ID Corner podcast.
The ID Corner podcast is a place where I will share and invite you to think about anything and everything related to teaching and learning. I plan to release three episodes a term. My hope is that this podcast will be a resource to teachers and a source of discussion. I want to provide space for thinking and talking about teaching. I invite you to comment below.

Peer Assessment

In his article published in Theory and Practice, Keith Topping defines peer assessment (PA) as, “… an arrangement for learners to consider and specify the level, value, or quality of a product or performance of other equal-status learners.” (Topping, 20) Why would an instructor want to use PA? What are the benefits and challenges? If I decide to use PA in my course, how should I go about doing so?

PA can be used as a means of engaging students in the learning process (Allberg and Lorås, 6) especially when the instructor deliberately communicates to students about the reasons for and benefits from PA. (Topping, 25) It has been shown that there can be gains for both assessors and assesees due to PA. (Topping, 22)

Benefits

What are the benefits of peer assessment? In the following section, we will look at benefits in three areas: academic benefits and social skill benefits for students, and benefits of PA for teachers.

Academic

Topping lists a number of benefits that students may enjoy because of participating in a PA exercise. Some of those listed are: time on task, increased sense of accountability, increased assessment of understanding, earlier error identification and analysis, and increased metacognition. (Topping, 23) These are all areas that a teacher should want to see gains in their students. There is some correlation between a high PA and high performance on exams. (Allberg and Lorås, 10) This means that those students who receive a high grade in the PA perform well on the exams. Another study by van Zundert et al. reports that students work revisions improved based on feedback from PA. (273)

Social

Students may also realize gains in social skills. One such gain is the ability to judge good work. (van Zundert et al., 270) Related to criticism Topping reports that, “Learning how to give and accept criticism, justify one’s own position, and reject suggestions are all useful, transferable social skills.”(24) PA helps the student practice these skills. Lastly, Aalberg, and Lorås share that, “Research also shows that peer assessments, if applied successfully, can have positive effects on self-confidence, transferable skills and social skills.”(9)

Teacher

Benefits to students should weigh heavy in favor of use, but there are specific benefits that a teacher may realize with PA. One benefit of using PA for the teacher is an economy of scale. Using the students as assessors creates that many more assessors and shares the responsibility of assessment. (Topping, 22) In addition, when taken as a whole, the sum of all the assessments may provide a deeper level of detail than with a single assessor. Aalberg and Lorås report that teachers found that the comments provided by the peer assessors seemed to, “have a higher level of detail than what the teacher would have been able to identify, and if when all comments are put together they gave a good overview.”(9) Related to PA of group work, the work by van Zundert et al. quotes a study mentioning that PA provides teachers, “a more accurate picture of individual performance in group work (Cheng & Warren, 2000).”(270) Another final benefit that teachers may realize is that the process of preparing for and administering PA may cause them to take a focused look at the course objectives and assessment, bringing clarity. As Topping mentions, “”Peer assessment can lead teachers to scrutinize and clarify assessment objectives and purposes, criteria, and grading scales.” (24)

Challenges & Questions

PA is not without its challenges. Students may feel anxious about assessing and being assessed by peers. (Topping, 24) Some students may feel that the assessment is not fair or that the assessment is not valid because their peer is not an expert in the area being assessed. (Allberg and Lorås, 8) They may also feel that the time and effort to complete a PA exercise is not worth what they perceive they may get out of the experience. (Allberg and Lorås, 8) One question related to PA is, “Are the grades valid and reliable?” It has been shown that PA grades are generally higher than a teacher may give. (Allberg and Lorås, 8) Can a teacher trust the PA enough to include the grades in the overall course grade? Aalberg, and Lorås share that PA may be used in grading, a finding that was confirmed by Topping. (9)  There are conflicting findings about whether and to what degree social processes may affect PA. Topping reports that social processes can, “influence and contaminate the reliability and validity of peer assessments.” (24)  He elaborated that, “Peer assessments can be partly determined by friendship bonds, enmity, or other power processes, the popularity of individuals, perception of criticism as socially uncomfortable, or even collusion to submit average scores, leading to lack of differentiation.” (24) A study referenced by van Zundert et al. conversely shares that the fairness of PA affected by personal relationships is negligible. (van Zundert et al., 272) There may be strategies that an instructor may use to minimize the possibility of these issues becoming a factor in the PA exercise.

Strategies

How do we maximize benefits and minimize challenges associated with PA? As we will see in the following section, there are strategies that an instructor may employ to make the best use of PA in the learning experience. With forethought and preparation, PA can be an effective pedagogical tool.

Beginning

The success of the PA activity is due largely to how much time is spent in preparation. As with any other learning activity, a teacher should define the objectives, what she or he wants students to get out of the experience, and why it is important for the students to participate. (Topping, 25) Helping students understand the ‘why’ of the activity will go a long way in alleviating some of the negative feelings students may have about the PA exercise.  Communicating the importance of the place the students are taking within the PA exercise can help them to be more engaged. As Topping states, “…students should be told that peer assessment involves students directly in learning, and should promote a sense of ownership, personal responsibility, and motivation.”(24) Providing support for the students in a PA experience should begin with defining how the students will be assessed and creating a rubric for doing so. The creation of a rubric can help bring focus on those most important take-aways, and provide language that the students may use in their assessment of peers. (Allberg and Lorås, 2) Providing guidance and opportunities to practice PA can have a great positive impact on both students’ perception of PA and ability to carry out a PA. (van Zundert et al., 270, 274-5)  Another strategy that can help students as they begin practicing a PA exercise is to have them give positive feedback initially. This can help to reduce anxiety they may have about assessing their peers. (Topping, 24]

During

A teacher must monitor and coach the PA activity in order for it to be successful. (Allberg and Lorås, 9) Providing guidelines and rubrics are one step in the preparation, but to be successful the instructor needs to continue to support the students through the experience. Students need direction from the teacher and opportunities to practice writing appropriate and helpful assessment comments. (Allberg and Lorås, 11) One strategy for reducing assesse anxiety in the initial PA experience is to give positive feedback first. (Topping, 24) In addition, providing multiple opportunities to practice PA will help students build skills for accurately assessing the work of their peers. Evaluating the quality of the PA is another way that teachers may support students through the experience and help them build good assessment skills. (Allberg and Lorås, 11)

In his article “Peer Assessment” Keith Topping provides eleven steps to executing a successful PA activity. I will not reproduce the entire list here, but there are a few items from the list that are important to recount here. The first recommendation is to find a partner. Do not implement PA on your own. I think this is important to share the intellectual load of talking through and determining objectives, rubrics, guidelines, and structure. This is work and many hands are helpful. The second recommendation mentioned is, “Clarify purpose, rationale, expectations, and acceptability with all stakeholders.” Determining the reason for and what you want students to get out of the PA experience is crucial for making the experience a success. Communicating these with students early and throughout the experience can increase their engagement, accountability and sense of ownership. Similar to this point is to include the students in identifying the assessment criteria. This is especially the case for PA of team or group work. Next is practice, practice, practice. Training students on how you want them to do PA is the best way to ensure that they do it the way that you want and to have a successful experience. To that end, providing exemplars, guidelines for and rubrics for the assessment you are asking them to do provides them a solid grounding in what criteria you want them to assess. This also provides a foundation for the sort of language that is appropriate and desired for them to use in the PA. Lastly, instructor should make a point to evaluate the assessments. This can be a graded exercise or not. Doing so provides a good means of determining if the students understand the objectives of the PA activity. This can also serve as a means of gathering exemplars for future iterations of the activity. (Topping, 25)

Peer assessment can be a great tool for formative assessment as well as benefitting students in a number of other ways. Preparing adequately and following the strategies listed her can help your PA activity be a success.

References

Aalberg, Trond, and Madeleine Lorås. 2018. Active Learning and Student Peer Assessment in a Web Development Course ; Active Learning and Student Peer Assessment in a Web Development Course. http://hdl.handle.net/11250/2582857.

Topping, Keith J. 2009 “Peer Assessment.” Theory Into Practice, vol. 48, no. 1, Winter , pp. 20–27. doi:10.1080/00405840802577569.

van Zundert, Marjo, et al. 2010. “Effective Peer Assessment Processes: Research Findings and Future Directions.” Learning and Instruction, vol. 20, no. 4, pp. 270–79. doi:10.1016/j.learninstruc.2009.08.004.

Resources

Discussion

UDL Workshop Summary

What is UDL?

Image of the UDL Guidelines. Link to web version.
UDL Guidelines. Click the image to view an interactive version.

“Universal Design for Learning (UDL) is a set of principles for curriculum development that give all individuals equal opportunities to learn. ”1 

UDL has its beginning in the universal design architectural movement which came out of “barrier free” movement. The term universal design was coined by Ronald Mace (North Carolina State University) in the 1960’s. Some examples of universal design are:

  • Smooth, ground level entrances without stairs
  • Lever handles for opening doors rather than twisting knobs

Universal Design for Learning was created by David H. Rose, Ed.D. of the Harvard Graduate School of Education and the Center for Applied Special Technology (CAST) in the 1990’s. UDL, based on research in neuroscience,  uses three principles. These principles are to provide: 2

  1. Multiple means of representation
  2. Multiple means of action and expression
  3. Multiple means of engagement

The three UDL principles are based on three learning networks. The three networks are recognition, strategic, and affective.

  1. The recognition network deals with, “how we gather facts and categorize what we see, hear, and read.”
  2. The strategic network deals with how we plan and perform tasks,” and “how we organize and express our ideas.”
  3. The affective network deals with how, “learners get engaged and stay motivated.”

There are other uses of universal design in teaching and learning:

  • UD for Instruction (UDforI)
  • UD of Instruction (UDI)

While similar, they are not the same as universal design for learning. UDL is distinctly focused on the student experience and attending to  varied ways students may approach a learning experience.

Two common ideas and terms withing the UDL framework are the ‘myth of the average‘ or ‘learner variability.’ These terms point to the idea that students vary (in strengths, background, experience, perspective, and many more)  from context to context, and from day to day.

Some examples of variability are:

  • background knowledge
  • gender
  • socio-economic status
  • culture
  • non-native language learners
  • learning exceptions: physical disability, mental illness, anxiety

UDL is meant to address this variability by providing as many on-ramps to the course material as possible, and as many means for the students to demonstrate what they have learned. You may be saying, “Well isn’t this just differentiated instruction?” One analogy in, Design & Deliver: Planning and Teaching Using UDL3, by Loui Lord Nelson, compares differentiated instruction to creating single meals for each student while UDL is a buffet. The buffet allows each person to choose those things that the like and that work best for them.

Here is a link to the guidelines. The way that this chart is organized is by principle. Under each principle are three guidelines. Each guideline then has a few supporting checkpoints. Each of the checkpoints in this chart are linked to further explanation. The three levels on the left-hand side, access, build, and internalize, speak to increasing levels of engagement of students with the learning materials. The checkpoints are what we  use to identify the specific strategies we will employ as we design a specific learning experience. I have attached PDF copies in the “resources” section of the guidelines and the graphic organizer to this post.

Let’s dig a little deeper into the checkpoints to see how we might apply them. Some examples are below.

  • 1 – Provide options for perception.
    • The mighty PDF. Quite ubiquitous in the higher-ed environment and at the core and inaccessible file. The PDF file format is technically an image. Even though we can see text, there isn’t any underlying text that may be manipulated. Optical character recognition, or OCR, is the process of identifying or extrapolating the text within an image.  Adobe Acrobat Pro is one example of software that can do this for PDF documents. Once this process has been completed other options for interacting with the document become available. Such affordances are:
      • having the document read aloud – providing another mode of interaction. Toward this end there are programs, web tools, and mobile apps that can help facilitate this.
      • searching the text – making the document a study/research tool.
      • Redeeming time – the ability for a student to listen to the document on a mobile device allows the student to listen when they have time e.g. student athlete traveling to/from and event.
  • 3.3 – Guide information processing and visualization.
    • Mind mapping or concept mapping provides a way for the student to organize information and make connections between the ideas.
  • 5.1 – Use multiple media for communication.
    • Providing the student audio or video feedback allows tone and non-verbal cues to be communicated. Making this an option for students can provide assistance to those students who are better explaining ideas verbally than in writing.
  • 5.2 – Use multiple tools for construction and composition.
    • Allowing students to create a podcast as an option to writing a research paper gives the students a creative alternative.
  • 6.1 – Guide appropriate goal setting.
    • Providing time estimates for readings/materials supports the student in managing their own work, planning and prioritizing for when works best for them.
  • 6.3 – Facilitate managing information and resources.
    • Chunking, or breaking a larger task or piece of information supports the student in managing the task and/or understanding the larger idea.
  • 6.4 – Enhance capacity for monitoring progress.
    • Helping students to see their progress in a course can allow them to better assess their work and the learning they are doing. Course completion tracking in Moodle is a tool that could be leveraged to support this.
  • 7.1 – Optimize individual choice and autonomy.
    • Providing alternative formats of course materials and providing options for assignments and activities support this checkpoint. Other strategies that may do so also are creation of class norms, collaborative assessment/rubric design in which the instructor and class work together to create the assessment or rubric.
  • 7.2 – Optimize relevance, value, and authenticity.
    • Whose voices are represented? Are the materials relevant to all the learners? How can they made to be relevant?
  • 8.1 – Heighten salience of goals and objectives.
    • The more an instructor can deliberately share of the “why” of the task/assignment/activity/process, the better the students will understand the importance of the said task/assignment/activity/process, and most importantly be more engaged and motivated in the learning process.
  • 8.3 – Foster collaboration and community.
    • Shared class notes are a way to foster collaboration and to support students who may not be as strong in this area.
  • 8.4 – Increase mastery-oriented feedback.
    • Providing learners feedback that a particular response is correct or incorrect really only provides limited information. Creating feedback that guides a student toward discovering the correct answer supports this checkpoint. Utilizing Moodle quiz feedback and the  question bank are examples of a tool that an instructor might use.
  • 9.3 – Develop self-assessment and reflection.
    • Including metacognitive prompts – “What worked?,” “What didn’t?,” “Why?,” “What would help you learn?” – Can support the student in taking control of their learning, knowing themselves better and provide them the means to identify and search after those things that help them the best/most.

Some things to keep in mind or questions to ask as you consider adapting a learning experience within the UDL framework:

  • What does this design change require? Technology? Time? Funds?
  • What are challenges against this design change?
  • How easily could this change be implemented in a course?
  • How might you assess the impact of the change?

Take-Away’s

The hour for the workshop went by very quickly. Everyone was able to get something out of the time together. One common thread was appreciation for the analogy of UDL being a “buffet” – providing options for the students to choose what they like and works best for them, versus differentiated instruction where you are making separate meals to fit each student. Others from the workshop voiced appreciation for the time and space to talk with peers about what colleagues are doing and realizing that they are dealing with a lot of the same issues.

This workshop was a great experience for me as a facilitator and I look forward to discussing these ideas further and working with instructors to implement these and other strategies into the teaching and learning experience.

Resources

Web

Books

References

  1. udloncampus.cast.org
  2. Burgstahler, Sheryl, Universal Design in Higher Education: from Principles to Practice, 2015.
  3. Lord Nelson, Loui, Design & Deliver: Planning and Teaching Using UDL,” by Loui Lord Nelson