Imagine that you are assessing a learner’s ability to provide bag-mask ventilation. Your assessment tool includes a simple item: “Selects the proper mask size.”
The learner approaches the patient, picks up the mask, and begins ventilating. The mask fits appropriately. You check “yes” and move on.
But there was only one mask in the room.
Did the learner demonstrate the ability to select the correct mask? Or did the simulation make the selection for them?
I would argue that we have observed the learner using an appropriate mask, but we have not created a meaningful opportunity to assess selection. That small distinction illustrates a much larger issue in simulation-based assessment: the way we design the simulation determines what assessment data we can collect.
The Simulation as an Assessment Data Generator
When designing a simulation, we spend considerable time thinking about the clinical story, equipment, patient findings, and progression of events. For assessment purposes, each of those decisions has another consequence. It influences which aspects of the learner’s performance become observable or measurable.
I find it useful to think of the simulation as an assessment data generator. We deliberately create conditions in which learners can demonstrate the behaviors, decisions, interpretations, and skills that we intend to assess.
In a previous post, I introduced the RST approach, which considers three places where we can intervene to improve simulation-based assessment: the Rater, the Simulation, and the Tool. These components interact. A well-written tool and a well-trained rater still need a simulation that makes the relevant performance accessible.
Here, I want to focus on the relationship between the T and the S. The question is straightforward:
What does the assessment tool require the simulation to do?
Of course, our assessment goals should guide the development of both the tool and the simulation. But whether we are creating a new assessment or reviewing an existing one, the items in the tool give us a practical way to examine whether the simulation can produce the evidence we need. Think of the assessment tool as a requirements document for the design of your simulation scenario.
Let’s consider four examples.
If You Want to Assess Selection, Provide a Choice
Returning to our mask example, the word “selects” creates a design requirement. The learner needs to discriminate among plausible options.
Providing several mask sizes creates that opportunity. Now the rater can observe whether the learner chooses an appropriate size for the patient.
This does not require an elaborate redesign. It requires us to recognize that using an item supplied by the simulation and selecting that item are different performances.
The same question applies whenever our tool asks whether a learner chooses a medication, a piece of equipment, or a course of action. Have we provided a meaningful choice, or have we arranged the environment so that the answer is effectively supplied?
If You Want to Assess Communication, Create the Interaction
Consider an item that asks whether the learner communicates clinical information in an organized manner.
The learner might manage the patient very effectively. But if the scenario never creates a reason to present the patient to another clinician, we may reach the end with very little evidence for that item.
Here, the intervention involves the flow of the scenario. We can create a clinically appropriate reason to contact a consultant. An embedded participant on the telephone can receive the presentation, allowing the learner to organize the clinical information, explain the concern, and communicate the request.
The interaction also needs deliberate design. What does the consultant ask? How much information do they volunteer? Do they allow the learner to complete the presentation, or repeatedly interrupt and redirect it?
These details influence the performance we observe. For assessment, we need a sufficiently consistent opportunity across learners. One consultant should not organize the presentation for the learner while another waits silently for a complete report.
If You Want to Assess Interpretation, Make It Accessible
Now consider an item that says, “Correctly interprets wheezing.”
The learner places a stethoscope on the patient’s chest. We can observe auscultation. But what did the learner hear, and how did they interpret it?
Watching the action does not answer that question.
One approach is to ask learners to think aloud throughout the scenario. In my experience, this can be useful with some learners, but it can also feel artificial and add to the demands of the encounter. More experienced clinicians may find it particularly disruptive to narrate every thought while managing a patient.
Another option is to create a natural clinical exchange. Immediately after auscultation, an embedded nurse might ask, “What are you hearing, doctor?”
The learner now has an opportunity to articulate the finding. The rater has evidence that corresponds more directly to the assessment item.
Notice that the nurse asks for the learner’s interpretation without supplying it. “Are you hearing wheezing?” would create a different assessment opportunity because the question itself suggests the answer.
The goal is to make the learner’s interpretation accessible while preserving the performance we intend to assess.
If You Want Precise Measurement, Provide the Capability
Suppose the assessment tool requires us to determine whether chest compressions meet specified rate and depth criteria.
We can watch a learner perform compressions. However, visual observation alone may not provide the precision required to score those criteria confidently, particularly compression depth.
In this situation, the assessment requirement creates a technical requirement. We need a simulator or measurement system capable of recording the relevant performance with sufficient accuracy.
This is a useful way to approach technology decisions. The question becomes, “What measurement capability does this assessment require?” That gives us a concrete reason to select a particular simulator or additional measurement device.
Sometimes the assessment requires very little technology. Sometimes a specific measurement capability is essential. The intended use of the data should guide that decision.
What If We Cannot Observe It During the Scenario?
Some aspects of reasoning are difficult to make accessible during an active scenario without disrupting the encounter or influencing subsequent decisions.
For example, several diagnoses might lead a learner to take similar initial actions. Those actions alone may not tell us which diagnosis the learner considered most likely.
We can design a structured question immediately after the scenario, before teaching or feedback begins: “What diagnosis did you believe was most likely, and what findings supported it?”
If that response contributes to the score, the question belongs in the assessment design. We should plan the wording, timing, permitted follow-up questions, and scoring criteria in advance.
A planned question can also be incorporated into the debriefing, but we need to be clear about when we are collecting assessment evidence and when we are helping the learner develop their understanding. A response offered after coaching tells us something different from a response given independently.
Similarly, a retrospective explanation provides evidence about what the learner can articulate afterward. It does not give us direct access to every thought they had during the scenario. Our interpretation of that evidence needs to respect that limitation.
The active scenario can end before assessment data collection ends. However, we should decide how to collect that additional data before the assessment begins.
Put Your Assessment Tool and Simulation Side by Side
A practical next step is to review your assessment tool one item at a time. For each item, ask:
- What performance or response do we need in order to score this item?
- Where does the learner have an opportunity to demonstrate it?
- What must the simulation provide to create that opportunity?
- Can the rater actually observe it, or can our equipment measure it?
- Will different learners receive sufficiently equivalent opportunities?
Then walk through the simulation with those questions in mind. Look at the equipment, the patient findings, the embedded participant scripts, the event sequence, and any questions asked afterward.
If you cannot identify how an item will generate evidence, revisit the simulation, the item, or both. An absent opportunity is a design problem; it should not automatically become a judgment that the learner lacks the ability.
The mask, consultant, wheezing, and compression examples require different interventions. Yet each begins with the same examination of what the tool asks us to assess and what the simulation must provide.
When reviewing your next assessment, put the tool beside the scenario and ask: Have we designed an opportunity to generate the evidence needed for every item we plan to score?
Until next time, Happy Simulating!