Category Archives: scenario design

Designing for Learner Success: 5 Ways of Ensuring Simulations are Effective Student-Centered Learning Environments

Soooooo, first off. Happy 2026 to you! I trust it is going well so far. I have been doing some studying about some of the learning theories and educational principles associated with simulation and higher education recently. As such, I have been making many new connections in my mind about the ways that healthcare simulation deeply connects to evidence and theory regarding higher education, and surprisingly to the K-12 education world. So this post is a bit more theoretical than usual, but it is representative of some of the course work that I am currently enjoying! I hope you share the same enthusiasm.

If you have worked in healthcare simulation for any length of time, you may have heard the phrase “guide on the side, not sage on the stage” which comes from Alison Kings 1993 publication (1) that suggested college level teaching move toward a constructivist theory design. Some argue she set the stage for the flipped classroom, where lecture content is moved outside the classroom to make room for active, guided learning during class time. It is often repeated, but do we truly practice it? In the rush to employ high-technology manikins or arrange the perfect clinical fidelity, it is easy to focus on the teaching rather than the learning.

To truly maximize the potential of our simulation center education programs, we need to shift our perspective toward Student-Centered Learning Environments (SCLEs) as described by Jonassen & Land (2). According to the learning sciences, SCLEs are not just about letting learners “figure it out” on their own; they are grounded designs where learners negotiate meaning, engage in authentic problems, and utilize scaffolding to bridge the gap between novice and expert where we serve as facilitators.

So then, how do we ensure our healthcare simulations function as true student-centered environments? Here are a few approaches that blend educational theory with practical simulation design.

1. Respect the “Learner’s Scenario”

In a previous post, I discussed how the word “scenario” means different things to different people. To the educator, it is a blueprint; to the operations specialist, it is a technical playbook. But to the learner, the scenario is the experience.

A core assumption of SCLEs is the “centrality of the learner”. While we may set external goals, the learner ultimately determines how to proceed based on their individual needs and the questions they generate. To support this, we must design scenarios that allow for agency. We must move away from linear, step-by-step exercises, which behaviorist theories might favor, and toward open-ended inquiry where learners identify gaps in their own knowledge and seek evidence to resolve them. After all, such design mimics the real-world practice of medicine and diagnostic processes in all of healthcare.

2. Recognize Scenarios as “Practice Fields” (But Mind the Cognitive Load!)

We often strive for realism, but we must be careful. Student-centered learning is rooted in “situated cognition,” meaning knowledge is inextricably tied to the context in which it is used. We want our simulations to act as “practice fields”, or environments where learners can engage in the authentic work of professionals. Think diagnosing a condition, managing a code, displaying empathy, or any other aspect of healthcare that we wish our learners to show us their abilities.

However, authenticity does not mean clutter. As I have written before, cognitive load is a currency that must be spent wisely. If we overload a scenario with irrelevant noise or “too much stuff” in the name of realism, we risk overwhelming the learner. A true student-centered design simplifies the authentic practice just enough to make it accessible, reducing the complexity without removing the core challenge. This allows the learner to focus on the learning objectives rather than processing extraneous details.

3. Anchor Learning in Prior Experience

You cannot center a curriculum on a student if you do not know where they are starting. Learners come to us with tacit, often naive beliefs rooted in their everyday experiences. Effective SCLEs use “anchored instruction” to connect new concepts to these familiar contexts.

In the absence of harboring true expertise in the needs of your intended learners, a robust needs assessment is the cornerstone of simulation design. As I have discussed in previous posts there are many pathways by which this can be accomplished. By gathering data through surveys or interviews, we identify the specific gaps between current outcomes and desired goals. This allows us to tailor the simulation to the learner’s “Zone of Proximal Development” ensuring the challenge is neither too boring nor too overwhelming. When we validate a learner’s prior experience, we empower them to take ownership of the inquiry.

4. Scaffolding: The Art of Optimal Guidance

There is a misconception that student-centered learning means “minimal guidance.” In reality, it requires optimal guidance. In the simulation world, we often provide this through scaffolding, or constructing the learner journey in a building block adventure that ultimately help learners manage the complexity of the task.

Scaffolding in simulation takes many forms:

  • Pre-Simulation Learning Assignments: Helps learners activate prior knowledge, identify gaps, and form initial mental models before entering the simulation. By establishing a shared baseline of concepts, terminology, and expectations, these assignments help level the playing field among participants, allowing the simulation itself to focus on higher-order reasoning,  and sensemaking rather than uneven content familiarity.
  • Conceptual Guidance: Helping learners organize their thoughts, perhaps through “argument structuring tools” or specific prompts that help them distinguish between conflicting ideas.
  • Debriefing: This is perhaps our most powerful scaffold. Using the HUMBLE approach (Humility, Understanding, Mindfulness, Balance, Learning, Engagement), we can guide learners to reflect on their performance. Reflection allows students to compare their internal ideas with the evidence generated during the simulation, leading to a more coherent understanding.

5. Leverage Multiple Perspectives

Deep understanding rarely emerges from a single point of view. It develops when learners are exposed to, and must reconcile, multiple perspectives. Well-designed simulation environments are uniquely positioned to support this kind of learning.

In healthcare simulation, perspective-taking happens at several levels. Within a single discipline, learners are often exposed to differing clinical interpretations, prioritization strategies, or communication styles. Two clinicians may look at the same evolving scenario and arrive at different conclusions about what matters most in that moment. Simulation creates a safe space for these differences to surface, be examined, and be discussed, without the risk of patient harm and through the operative lens of a safe learning environment. This kind of cognitive diversity encourages learners to move beyond “the right answer” and toward deeper clinical reasoning and judgment.

Interprofessional simulation amplifies this effect even further. When nurses, physicians, pharmacists, respiratory therapists, and other professionals train together, learners gain direct insight into how roles, responsibilities, and mental models differ across the care team. What one profession sees as a priority may not align with another’s perspective, and simulation makes those differences visible. Rather than flattening these viewpoints, effective SCLEs use them as learning assets.

From a student-centered perspective, the power of simulation lies in shifting knowledge construction from the individual to the group. Learners are not simply absorbing expert explanations; they are actively negotiating meaning with peers who bring different training backgrounds, experiences, and assumptions to the scenario. Over time, this shared sensemaking helps build a true learning community, one in which understanding is co-constructed and collective competence exceeds what any single learner could achieve alone.

Importantly, structured debriefing plays a critical role in solidifying this learning. When facilitators intentionally invite multiple voices into the conversation, asking “What were you seeing?” or “How did your role shape that decision?”, learners begin to appreciate not only what decisions were made, but why they differed. This reflective dialogue reinforces perspective-taking as a core professional skill, not an optional add-on.

In this way, simulation-based SCLEs mirror the realities of clinical practice itself: complex, collaborative, and shaped by multiple viewpoints. By embracing, not minimizing, these differences, simulation helps learners develop the adaptability, empathy, and team-based reasoning required for real-world patient care.

Conclusion

Transforming a simulation program into a student-centered learning environment (SCLE) requires more than just high-tech equipment. It requires a shift in mindset from the outset of the design. We must view the simulation ecosystem through the lens of the learner. By balancing authentic practice with cognitive load management, respecting prior experience, and providing robust scaffolding, we empower our learners to become autonomous, lifelong problem solvers.

Until next time, Happy Simulating!

  • (1) King, A. (1993). From sage on the stage to guide on the side. College Teaching, 41(1), 30–35.
  • (2) Jonassen, D. H., & Land, S. M. (2012). Student-centered learning environments (pp. 3–25). In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (2nd ed.). Routledge.

Leave a comment

Filed under Curriculum, design, scenario design, simulation

What’s a Scenario? The Word That Means Many Things, To Many People in Healthcare Simulation

Definition of 'scenario' showing its etymology and meaning related to scripts and instructions.

If you’ve worked in healthcare simulation for any length of time, you’ve probably used the word “scenario” countless times. “Let’s build a new scenario.” “We’re running the sepsis scenario this afternoon.” “That scenario went great!”

But have you ever stopped to think about how differently that same word means to each person involved? The word “scenario” is a perfect example of how language in simulation can unite us, or possibly confuse us,  depending on our perspective. Additionally those creating said “scenarios” need to be keenly aware of these implications.

In truth, “scenario” represents something unique to different members of the simulation ecosystem: learners, educators, technicians, and administrators. Understanding these different lenses can help strengthen teamwork, communication, and the overall impact of our simulation programs.

The Learner’s Scenario: The Clinical Experience

For learners, the scenario is the experience itself. It’s the unfolding clinical-like moment that challenges their knowledge, judgment, and communication skills in an effort to improve.

In the learner’s mind, the scenario “is” the simulation. It’s what they see, hear, and feel—the patient’s distress, the team dynamics, the need to make decisions under pressure. The learner rarely thinks about the planning that went into it; they simply step into a space that hopefully they were well oriented, feels real enough and is relevant to their goals.


For them, the scenario represents an opportunity: a chance to act, reflect, and learn in a safe environment. When done well, it becomes a memorable and emotionally resonant learning event that bridges the gap between classroom knowledge and clinical performance along with providing a stimulus for self-improvement.

The Educator’s Scenario: The Blueprint for Learning

For the educator or faculty member, the scenario is not just an experience—it’s a design.

To the educator, the scenario is the blueprint for what the learner will encounter. It contains the story arc, learning objectives, key events, and expected actions. It guides how pre-learning will be incorporated or reinforced to prepare the learner, how the simulation unfolds, and how the debriefing reinforces the lessons afterward as well as how assessment strategies and tools are incorporated into the learning encounter.

A well-constructed scenario is both an art and a science. It is an instrument that balances operations with realism and  educational intent. It requires alignment between objectives, assessment, and debriefing. The educator’s scenario document might include everything from patient history and vital sign trends to faculty prompts, checklists, and suggested debriefing strategies and topics.

In this view, the scenario becomes a curricular instrument, a tool that translates educational goals into lived experience.

The Simulation Operations Team’s Scenario: The Technical Playbook

For the simulation operations specialist or technician, the scenario is a technical plan, a script for how to bring the educator’s vision to life.

This version of the scenario includes the logistics that make the experience possible, for example:
– Scheduling and room reservations
– Equipment and supply lists
– Simulator programming and physiological responses
– Audio-visual configurations
– Staffing assignments and role descriptions

For the operations team, precision is everything. A single oversight—an unplugged cable, a missing monitor, or a mistimed vital sign change, can derail the encounter and disrupt the learning flow along with the concentration of the learners and faculty alike.

Their scenario isn’t about learning objectives; it’s about execution. It ensures that the right tools, people, environments, and technology align perfectly at the right moment to make the educational magic happen. In many ways, their scenario is the stage directions that make the play run seamlessly. Or to borrow a piece from a previous blog post of mine, it is the music that plays to allow the learners to dance and be evaluated.

The Administrator’s Scenario: The Unit of Measurement

To program administrators and simulation center leaders, the word “scenario” carries yet another meaning.

From this vantage point, the scenario represents a unit of activity. Think of it as a quantifiable event tied to scheduling, staffing, and financial data. It’s a building block for understanding center utilization, cost recovery, and return on investment.

An administrator may see a scenario not only as an educational event but also as a data record in a management system: duration, participants, faculty hours, resource use, and consumables. From these data points come critical insights such as how much it costs to deliver a course, how often equipment is used, and where efficiencies or resource gaps exist.

This administrative view ensures that simulation programs remain sustainable, scalable, and aligned with institutional goals.

One Word, Many Worlds

The fascinating thing about the word “scenario” is that all these definitions are correct, utilized every day in the simulation world and essential. Each reflects a different dimension of the same phenomenon.

For the learner, it’s an experience.
For the educator, it’s a design.
For the technician, it’s an operation.
For the administrator, it’s a metric.

Together, these perspectives form the ecosystem that allows simulation to thrive. The most successful programs are those where these views overlap and inform one another—where educators appreciate the operational complexity, technicians understand the learning goals, and administrators recognize the educational and patient-safety impact that justify the resources.

When those perspectives align, the word “scenario” transforms from a simple script or event into a powerful tool for advancing healthcare education and safety.

Director’s Reflection

In my years of working with simulation programs around the world, I’ve learned that the strength of a simulation scenario isn’t found in just the documents or the technology’s, but it also in the shared understanding among the people who create, deliver, and learn from it.

A scenario is a bridge connecting intent to experience, vision to execution, and learning to improvement. Whether you’re writing one, running one, or analyzing its data, remember that every scenario represents a small but meaningful step toward better healthcare.

Until Next Time,

Happy Simulating!

Leave a comment

Filed under Curriculum, debriefing, design, operations, scenario design, simulation

Debugging Simulation: How Alpha and Beta Testing Strengthen Scenario Success

In the world of healthcare simulation, our goal is to create meaningful learning experiences that improve the safety and quality of patient care. Achieving that goal requires careful planning, thoughtful design, and rigorous evaluation of our simulation scenarios. One concept borrowed from the world of software and technology development—but often overlooked in healthcare education—is the process of alpha and beta testing.

By understanding and applying these concepts to simulation scenario design, educators can significantly enhance the efficiency and effectiveness, and overall impact of their programs. Let’s take a closer look at what alpha and beta testing mean, why they matter in healthcare simulation, and how they can help elevate both the learner as well as the facilitators experience.


What Do We Mean by Alpha and Beta Testing?

The terms alpha testing and beta testing originate from the software development industry. Before an application is released to users, developers put it through multiple rounds of trials to identify problems, fine-tune functionality, and ensure that it behaves as intended. Healthcare simulation, while a very different domain, benefits from the same structured approach.

  • Alpha testing is the internal trial run. In the simulation context, this means running a new scenario with the development team or a small group of faculty before exposing it to actual learners. The purpose is to check for errors, gaps, or inconsistencies in the scenario design. Are the case details clear? Do the vital signs respond correctly to learner interventions? Does the simulator technology function as intended?
  • Beta testing is the external pilot run. This step introduces the scenario to a limited group of learners—often peers, or learners similar to those whom the scenario is intended. The purpose is to observe how real participants interact with the scenario. Do they engage in the way you intended? Do the prompts drive the critical thinking skills you were hoping to elicit? Are they interpreting the simulated aspects of the scenario in the manner which they are intended? Are the debriefing points aligning with your learning objectives?

When done well, these stages help identify potential pitfalls, correct technical issues, and refine educational flow before the simulation reaches a larger audience.


Why Alpha Testing Matters

Alpha testing is your chance to work out the “kinks” of a simulation in a controlled environment. Think of it as a rehearsal where mistakes are not only acceptable but expected.

Consider a scenario where learners are expected to diagnose sepsis in an unstable patient. During alpha testing, your faculty team might discover that the simulator’s vital signs do not update quickly enough when fluid resuscitation is administered. Or perhaps the timing of lab results makes it impossible for learners to reach the intended diagnosis within the allotted session. Identifying these issues before learners arrive saves both time and frustration. However, always remember that those who participated in the design often have developed a shared mental model and may miss the fact that some things are misinterpreted by actual intended learners.

Some examples of key questions to ask during alpha testing include:

  • Do the scenario instructions match the programmed mannequin responses?
  • Are embedded participants (e.g., a nurse or family member role) clear on their scripts?
  • Does the timing of critical events support the learning objectives?
  • Are there any “gotchas” that could derail learner engagement?
  • Did the pre-briefing take longer than expected?

By the end of alpha testing, the simulation team should have a scenario that is technically functional, logically sound, and aligned with its stated goals that runs in the approximate amount of time that it was designed.


Why Beta Testing is Crucial

Once the internal checks are complete, it is time to see how the scenario performs in the real world. Beta testing is the first opportunity to expose the simulation to actual learners, albeit on a smaller and more controlled scale.

Imagine your team has developed a scenario for emergency airway management. The alpha test confirmed that the mannequin responds appropriately to intubation attempts and that medications are available in the correct doses. During beta testing with a group of residents, however, you observe that they consistently miss an early cue about airway edema. This could mean your prompts are too subtle—or that your learners need more scaffolding. Either way, the feedback allows you to adjust before rolling it out widely.

Beta testing provides answers to questions such as:

  • Are learners engaging with the scenario in the way we anticipated?
  • Do the actions of participants align with the intended outcomes? competencies?
  • Does the scenario create opportunities for meaningful debriefing?
  • What unexpected challenges or learner behaviors emerge?

In essence, beta testing allows the scenario to “fail safely” in front of a pilot group so that the eventual cohort benefits from a polished and purposeful experience.


Lessons from Software Development

In software engineering, skipping alpha and beta testing is a recipe for disaster—think buggy apps, frustrated users, and poor reviews. The same risks apply to simulation. Without proper testing, scenarios can fall flat, confuse learners, or even undermine the credibility of your program.

Borrowing these terms reminds us that scenario design is not a one-and-done activity. It is an iterative process where feedback loops play a central role in quality improvement. Just as developers patch software bugs, simulation educators refine scenario elements until they function smoothly.


Practical Tips for Implementing Alpha and Beta Testing

  1. Schedule testing time. Don’t assume you can “test on the fly” before learners walk in. Build alpha and beta testing into your development timeline.
  2. Use checklists. Structured tools can help your team evaluate everything from simulator programming to alignment with learning objectives.
  3. Capture feedback systematically. During beta testing, request that observers take notes on learner behaviors, timing, and unintended outcomes. Post-scenario surveys can also capture learner perceptions.
  4. Iterate, don’t improvise. Resist the urge to “fix” problems on the fly during a live teaching session. Incorporate changes based on alpha/beta feedback before the full rollout.

How This Benefits Learners

Ultimately, alpha and beta testing serve a dual role about making faculty feel more comfortable as well as enhancing the learner experience. A well-tested scenario ensures that:

  • Learners are immersed in a coherent case that is relevant to their learning needs.
  • Technical glitches do not distract from critical thinking.
  • Debriefing discussions flow naturally from the scenario, rather than being forced or disconnected.

In other words, when educators invest in testing, learners reap the rewards through higher-quality education and, by extension, safer patient care.


Conclusion: Test Early, Test Often

Healthcare simulation has matured into a vital component of modern education. But as with any educational tool, its effectiveness depends on the rigor of its design. By embracing alpha and beta testing, simulation teams can identify weaknesses, refine strengths, and deliver scenarios that consistently meet their objectives.

The lesson from software holds true: the more you test before release, the fewer problems you encounter afterward. In healthcare simulation, that means fewer distractions, more meaningful learning, and ultimately better outcomes for patients.

So the next time you’re preparing to debut a new scenario, pause and ask: Have we really tested this? If the answer is no, it may be worth an extra round of alpha or beta testing. Your learners, as well as your participating faculty, and technical staff will thank you.

Leave a comment

Filed under Curriculum, design, scenario design, simulation, Uncategorized

Improving Interrater Reliability in Healthcare Simulation-Based Assessments: The RST Approach

Achieving high interrater reliability (IRR) is a cornerstone of any effective medium or high stakes assessment in healthcare simulation. Without consistent and dependable scoring across multiple raters, the validity of an assessment can be called into question. Interrater reliability ensures that evaluations are fair, objective, and truly reflective of the participant’s performance rather than the subjective biases or variability among raters.

For simulation-based assessments, however, maintaining IRR can be particularly challenging due to the complex, dynamic, and multifaceted nature of healthcare scenarios. This is where the RST approach—focusing on changes to the Rater, the Simulation, and the Tool—can offer a systematic and impactful framework for improvement. In this post I’ll walk you through this approach, providing insights and practical strategies for applying RST to your simulation programs.


The R in RST: Changing the Rater

One of the most straightforward avenues to improve IRR is addressing variability related to the rater. This is critical because raters bring their own perspectives, experiences, and biases to the evaluation process, all of which can affect their scoring.

Strategies for Enhancing the Rater’s Consistency:

  1. Rater Calibration Sessions
    Conducting rater calibration sessions is one of the most effective ways to ensure raters have a shared understanding of the evaluation criteria. These sessions involve reviewing sample performances as a group and discussing scoring rationales to align perceptions. This shared experience helps raters interpret assessment tools in the same way, leading to more consistent scoring.
  2. Rater Selection and Expertise
    Consider who is performing the assessment. Are they subject matter experts? Are they trained educators? Selecting raters with relevant expertise and familiarity with the assessment content can reduce variability. Alternatively, inexperienced or overly diverse rater pools may introduce inconsistencies.
  3. Addressing Rater Bias
    Even with calibration, unconscious biases can creep into assessments. Training raters to recognize and mitigate biases—such as favoring individuals who perform similarly to the rater’s own practice style—can improve consistency.
  4. Changing Raters
    If specific raters consistently show discrepancies in their scoring compared to others, it may be necessary to replace them or limit their participation in high-stakes assessments. Using multiple raters per simulation and averaging scores can also dilute individual biases.

The S in RST: Changing the Simulation

The second dimension of the RST approach involves modifying the simulation itself to make it more assessable. By carefully designing simulations to make critical behaviors, thought processes, and decisions more observable, you enhance the ability of raters to evaluate participants consistently.

Strategies for Simulation Adjustments:

  1. Prompting Observable Actions
    Simulations can be structured to encourage participants to verbalize their thought processes or articulate their decisions. For instance, during a scenario involving a critical diagnosis, asking participants to “think aloud” as they interpret clinical findings can provide raters with clear evidence of decision-making skills, making scoring more straightforward.
  2. Embedding Structured Checkpoints
    Building structured checkpoints into the simulation—such as specific moments when participants are asked to summarize their findings or outline their next steps—creates clear opportunities for assessment. This reduces ambiguity for raters.
  3. Standardizing Simulation Flow
    Variability in how simulations unfold can lead to scoring challenges. Using standardized patient scripts, consistent cues, and fixed timing for critical events ensures that all participants encounter the same conditions, making assessments more comparable. If high technology simulators are being used for the simulation, consider the use of preprogram scenario to ensure the physiology changes are consistent across all episodes of the same scenario.
  4. Revisiting Scenario Complexity
    While realism is a hallmark of effective simulation, excessive complexity can overwhelm raters and obscure key performance indicators. Simplifying scenarios to focus on specific competencies can improve the clarity and reliability of evaluations.

The T in RST: Changing the Tool

The assessment tool is often an overlooked factor in achieving IRR, yet it plays a pivotal role in how raters interpret and apply scoring criteria. A well-designed tool minimizes ambiguity and makes scoring intuitive, even for less experienced raters.

Strategies for Tool Optimization:

  1. Behavioral Anchors for Rating Scales
    Adding specific behavioral examples or descriptors to rating scale items helps raters apply the scales consistently. For instance, instead of a vague “Good” rating, an anchored descriptor like “Effectively communicates diagnosis and treatment plan to patient” provides clarity.
  2. Item Grouping and Ordering
    Organizing items logically—for example, grouping communication skills, clinical decision-making, and procedural skills separately—makes it easier for raters to focus on one domain at a time. A cluttered or disorganized tool can lead to confusion and inconsistent scoring.
  3. Simplifying Language
    Ensure that the language in the tool is straightforward and free of jargon. If raters struggle to interpret an item, their scoring may vary widely.
  4. Usability Enhancements
    Small changes, like improving the font size, using bullet points, or incorporating intuitive layouts, can significantly reduce rater fatigue and errors during scoring. A user-friendly tool ensures raters stay focused on the participant’s performance rather than grappling with the mechanics of the tool.
  5. Pretesting the Tool
    Conduct pilot assessments using the tool to identify problematic items or inconsistencies. This feedback loop allows you to refine the tool before deploying it in high-stakes simulations.

Putting It All Together: The RST Approach in Action

To illustrate how the RST approach works holistically, imagine a healthcare simulation designed to assess a participant’s ability to manage a cardiac arrest scenario:

  • Rater: You organize a calibration session where all raters review a sample video of a cardiac arrest scenario and agree on scoring criteria. You also ensure raters have experience in emergency medicine and provide bias-awareness training.
  • Simulation: The scenario is adjusted to include a structured moment where the participant is required to verbalize their reasoning for choosing a particular medication. Additionally, standardized cues are used to ensure all participants face identical conditions.
  • Tool: The assessment tool is revised to include behavioral anchors, such as “Identifies and administers epinephrine within 3 minutes” for procedural accuracy. The tool’s layout is simplified, grouping items under headings like “Clinical Judgment” and “Communication.”

With these changes, the IRR for this simulation-based assessment improves, as raters now have a shared understanding, participants’ actions are more easily observable, and the tool provides clearer guidance.


Conclusion: Adopting the RST Approach for Better Assessments

While I will agree, improving interrater reliability in healthcare simulation assessments is no small task, but the RST approach offers a structured framework to tackle the challenge. By focusing on the Rater, the Simulation, and the Tool, you can systematically address the factors that contribute to variability and ensure more consistent, fair, and accurate evaluations. For more on this see my previous blog post on interrater reliability.

Whether you are designing a new assessment or refining an existing one, considering how changes in these three areas might influence IRR is a worthwhile investment. With reliable assessments, we not only enhance the quality of simulation-based education but also uphold the integrity of our evaluations—ultimately contributing to better-prepared healthcare professionals.

Are you ready to elevate your simulation assessments? The RST approach is here to guide your journey.

Please like and comment if you would like to see more topics like this in my blog!

Until next time, Happy Simulating!

Leave a comment

Filed under assessment, design, scenario design, simulation

Cognitive Load as a Currency: Spend it WISELY in Simulation Scenario Design

In the world of healthcare education, we know that simulation-based training is a powerful tool, allowing students to experience real-life scenarios in a controlled environment. Simulation not only bridges the gap between theory and practice but also builds confidence and competence in a safe space. However, as with all educational tools, there’s a delicate balance to maintain regarding design decisions, particularly when it comes to the concept of cognitive load.

Cognitive Load: A Precious Resource

Cognitive load refers to the amount of mental effort being used in the working memory. It is, in essence, the currency of the mind—a finite resource that, when spent wisely, can lead to effective learning and retention. But, just like any currency, it can be squandered if not managed properly.

In our healthcare simulations, participants are asked to perform tasks that mimic real-life situations. They must think critically, make decisions quickly, and often work under pressure—all while processing the simulated environment around them. Every element in a simulation scenario demands a portion of the participant’s cognitive load. When this load becomes too heavy, it can overwhelm the learner, leading to confusion, errors, and, ultimately, a less effective educational experience.

The Hidden Costs of Over-Designing Simulations

In an effort to make simulations as realistic as possible, educators sometimes introduce elements that, while seemingly beneficial, can actually detract from the learning experience. These can include irrelevant information, extraneous equipment, or overly complex scenarios that do not directly contribute to the learning objectives. While the intention is often to enhance the realism of the scenario, the reality is that these additional elements force participants to expend cognitive energy on processing what is simulated and why it is being simulated.

For example, consider a scenario designed to teach students how to manage a patient in cardiac arrest. The core learning objectives might include recognizing signs of cardiac distress, performing CPR, and administering appropriate medications. However, the students might find themselves distracted if the scenario also includes irrelevant background noise, additional non-essential equipment, or extraneous patient history that doesn’t contribute to the learning objectives. They may spend valuable cognitive resources trying to process this irrelevant information rather than focusing on the critical tasks at hand.

The Art of Simplification: Less is More

To maximize the effectiveness of simulation, it’s essential to streamline scenarios, focusing on the elements that directly support the learning objectives. This doesn’t mean stripping away all realism, but rather, carefully curating the scenario to include only those aspects that enhance understanding and practice of the targeted skills. The goal is not to make it real but to make it real enough. Our goal is not to recreate reality but to provide an environmental milieu that supports the tasks at hand and allows the scenario to achieve intended objectives.

When designing a simulation, ask yourself:

– What are the primary learning objectives?

– What elements of the scenario directly support these objectives?

– Are there any elements that, while realistic, do not contribute to the learning goals and could potentially distract or overwhelm the students?

By answering these questions, you can begin to design scenarios that are both effective and efficient, ensuring that students’ cognitive resources are spent on mastering the intended skills rather than getting bogged down by unnecessary details.

A Practical Approach to Cognitive Load Management

1. Clear Objectives: Begin with a clear understanding of what you want your students to learn. Every element of the simulation should tie back to these objectives.

2. Essential Information Only: Include only the information and equipment necessary to achieve the learning goals. Avoid adding extras that don’t directly contribute to the scenario’s success.

3. Sequential Learning: If multiple skills need to be practiced, consider breaking them down into separate scenarios. This allows students to focus on one set of objectives at a time, reducing cognitive overload.

4. Debrief Thoughtfully: Use the debriefing session to reinforce learning objectives and clarify any confusion. This helps students consolidate what they’ve learned and understand the relevance of each element in the simulation.

5. Feedback and Iteration: Regularly gather feedback from participants and use it to refine your scenarios. What seems beneficial in theory might not always work in practice, and being open to adjustments is key to effective simulation design. Further, I fstudents stumble in the same point in the scenario, look for potential design flaws or elements that might be adding confusion.

Conclusion: Design the Scenarios to allow the participant to Spend Wisely

Cognitive load is a valuable resource that must be managed carefully in healthcare simulation design. By focusing on what is essential and stripping away the non-essential, educators can create scenarios that are not only realistic but also aligned with the primary learning objectives. This approach ensures that students can devote their cognitive energy to mastering the skills that matter most, leading to more effective learning and better outcomes in real-life situations.

In the end, the key to successful simulation design is not in how much you can add, but in how much you can refine and simplify. By spending cognitive load wisely, you enable your students to thrive in a simulated environment, fully prepared to face the challenges of the real world.

Until Next Time, Happy Simulating!

Leave a comment

Filed under Curriculum, scenario design

Not Every Simulation Scenario Needs to Have a Diagnostic Mystery!

It is quite common to mistakenly believe that there needs to be a diagnostic mystery associated with a simulation scenario. This could not be further from the truth.

Sometimes it arises from our clinical hat being confused with our educator hat (meaning we let our view of the actual clinical environment become the driving factor in the design of the scenario.) We must carefully consider the learning objectives and what we want to accomplish. One of the powerful things about simulation is that we get to pick where we start and where we stop, as well as the information given or withheld during the scenario.

Let us take an example of an Inferior Wall Myocardial Infarction (IWMI). Let us imagine that we desire to assess a resident physician’s ability to manage the case. Notice I said to manage the case, not diagnose, then manage the case. This has important distinctions on how we would choose to begin the scenario. If the objectives were to diagnose and manage, we might start the case with a person complaining of undifferentiated chest pain and have the participant work towards the diagnosis and then demonstrate the treatment. Elsewise, if we were looking to have them only demonstrate proficiency in the management of the case, we may hand them an EKG showing an IMWI (or maybe not even hand them the EKG) and start the case by saying, “your patient is having an IWMI” and direct them to start the care.  

What is the difference? Does it matter?

In the former example of starting the case, the participant has to work through the diagnostic conundrum of undifferentiated chest pain to come up with the diagnosis of IWMI. Further, it is possible that the participant does not arrive at the proper diagnosis, in which case you would not be able to observe and assess them in the management of the case. Thus, your learning objectives have become dependent on one another. By the way, there’s nothing wrong with this as long as it is intended. We tend to set up cases like this because that is the way that the sequencing would happen in the actual clinical environment (our clinical hat interfering). However, this takes up valuable minutes of simulation, which are expensive and should be planned judiciously. So, my underlying point is if you deliberately are creating the scenario to see the diagnostic reasoning and treatment, then the former approach would be appropriate.

The latter approach, however, should be able to accomplish the learning objective associated with demonstrating the management of the patient. Thus, if that is truly the intended learning objective, the case should be fast-forwarded to eliminate the diagnostic reasoning portion of the scenario. Not only will this save valuable simulation time it will also conceivably lead to more time to carefully evaluate the treatment steps associated with managing the patient. Additionally, it will eliminate the potential of prolonged simulation periods that do not contribute to accomplishing the learning objectives and/or get stuck because of a failure to achieve the initial objective (in this case, for example, the diagnosis.)

So, the next time you make decisions in the scenario’s design, take a breath and ask yourself, “Am I designing it this way because this is the way we always do it? Am I designing it this way because this is the way it appears in the real clinical environment?”

The important point is that one is asking themselves, “How can I stratify my design decisions so that the scenario is best crafted to accomplish the intended learning objectives?” If you do, you will be on the road to designing scenarios that are efficient and effective!

Leave a comment

Filed under scenario design, simulation

Sherlock Holmes and the Students of Simulation

I want to make a comparison between Sherlock Holmes and the students of our simulations! It has important implications for our scenario design process. When you think about it, there’s hypervigilance amongst our students, looking for clues during the simulation. They are doing so to figure out what we want them to do. Analyzing such clues is like the venerable detective Sherlock Holmes’s processes when investigating a crime.

Video version of this post

This has important implications for our scenario design work because many times, we get confused with the idea that our job is to create reality when in fact, it is not that at all our job. As simulation experts, our jobs are to create an environment with the reality that is sufficient to allow a student to progress through various aspects of the provision of health care. We need to be able to make a judgment and say, “hey, they need some work in this area,” and “hey, they’re doing good in this area.”

To accomplish this, we create facsimiles of what they will experience in the actual clinical environment transported into the simulated environment to help them adjust their mindset so they can progress down the pathway of taking care of those (simulated) patient encounters.

We must be mindful that during the simulated environment, people engage their best Sherlock Holmes, and as the famous song goes, [they are] “looking for clues at the scene of the crime.”
Let’s explore this more practically.

Suppose I am working in the emergency department, and I walk into the room and see a knife sitting on the tray table next to a patient. In that case, I immediately think, “wow, somebody didn’t clean this room up after the last patient, and there’s a knife on the tray. I would probably apologize about it to the patient and their family.”

Fast forward…..

Put me into a simulation as a participant, and I walk into the room. I see the knife on the tray next to the patient’s bed, and I immediately think, “Ah, I’m probably going to do a crich or some invasive procedure on this patient.”

How does that translate to our scenario design work? We must be mindful that the students of our simulations are always hypervigilant and always looking for these clues. Sometimes when we have things included in the simulation, we might just have there as window dressing or to try to (re)create some reality. However, stop to think they can be misinterpreted as necessary to be incorporated into the simulation by the student for success in their analysis.

Suddenly, the student sees this thing sitting on the table, so they think it is essential for them to use it in the simulation, and now they are using it, and the simulation is going off the tracks! As the instructor, you’re saying that what happened is not what was supposed to happen!

At times we must be able to objectively go back and look at the scenario design process and recognize maybe just maybe something we did in the design of the scenario, which includes the setup of the environment, that misled the participant(s). If we see multiple students making the same mistakes, we must go back and analyze our scenario design. I like to call it noise when we put extra things into the simulation scenario design. It’s noise, and the potential for that noise to blow up and drive the simulation off the tracks goes up exponentially with every component we include in the space. Be mindful of this and be aware of the hypervigilance associated with students undergoing simulation.

We can negate some of these things by a good orientation, by incorporating the good practice into our simulation scenario design so that we’re only including items in the room that are germane to accomplishing the learning objectives.

Tip: If you see the same mistakes happening again and again, please introspect, go back, look at the design of your simulation scenario, and recognize there could be a flaw! Who finds such flaws in the story?  Sherlock Holmes, that’s who!

1 Comment

Filed under Curriculum, design, scenario design, simulation

5 Tips to Improve Interrater Reliability During Healthcare Simulation Assessments

One of the most important concepts in simulation-based assessment is achieving reliability, and specifically interrater reliability. While I have discussed previously in this blog every simulation is assessment, in this article I am speaking of the type of simulation assessment that requires one or more raters to record data associated with the performance or more specifically an assessment tool.

Interpreter reliability simply put is that if we have multiple raters watching a simulation and using a scoring rubric or tool, that they will produce similar scores. Achieving intermittent reliability is important for several reasons including that we are usually using more than one rater to evaluate simulations over time. Other times we are engaged in research and other high stakes reasons to complete assessment tools and want to be certain that we are reaching correct conclusions.

Improving assessment capabilities for stimulation requires a significant amount of effort. The amount of time and effort that can go into the assessment process should be directly proportional to the stakes of the assessment.

In this article I offer five tips to consider for improving into rate of reliability when conducting simulation-based assessment

1 – Train Your Raters

The most basic and overlooked aspect of achieving into rate and reliability comes from training of the raters. The raters need to be trained to the process, the assessment tools, and each item of the assessment that they are rendering an opinion on. It is tempting to think of subject matter experts as knowledgeable enough to fill out simple assessments however you will find out with detailed testing that often the scoring of the item is truly in the eye of the beholder. Simple items like “asked medical history” may be difficult to achieve reliability if not defined prior to the assessment activity. Other things may affect the assessment that require rater calibration/training such as limitations of the simulation, and how something is being simulated and/or overall familiarity with the technology that may be used to collect the data.

2 – Modify Your Assessment Tool

Modifications to the assessment tool can enhance interrelated reliability. Sometimes it can be extreme as having to remove an assessment item because you figure out that you are unable to achieve reliability despite iterative attempts at improvement. Other less drastic changes can come in the form of clarifying the text directives that are associated with the item. Sometimes removing qualitative wording such as “appropriately” or “correctly” can help to improve reliability. Adding descriptors of expected behavior or behaviorally anchored statements to items can help to improve reliability. However, these modifications and qualifying statements should also be addressed in the training of the raters as described above.

3 – Make Things Assessable (Scenario Design)

An often-overlooked factor that can help to improve indurated reliability is make modifications to the simulation scenario to allow things to be more “assessable”. We make a sizable number of decisions when creating simulation-based scenarios for education purposes. There are other decisions and functions that can be designed into the scenario to allow assessments to be more accurate and reliable. For example, if we want to know if someone correctly interpreted wheezing in the lung sounds of the simulator, we introduced design elements in the scenario that could help us to gather this information accurately and thus increase into rater reliability. For example, we could embed a person in the scenario to play the role of another healthcare provider that simply asks the participant what they heard. Alternatively, we could have the participant fill out a questionnaire at the end of the scenario, or even complete an assessment form regarding the simulation encounter. Lastly, we could embed the assessment tool into the debriefing process and simply ask the participant during the debriefing what they heard when I auscultated the lungs. There is no correct way to do this, I am trying to articulate different solutions to the same problem that could represent solutions based on the context of your scenario design.

4 – Assessment Tool Technology

Gathering assessment data electronically can help significantly. When compared to a paper and pencil collection scheme technology enhanced or “smart” scoring systems can assist. For example, if there are many items on a paper scoring tool the page can sometimes become unwieldy to monitor. Electronic systems can continuously update and filter out data that does not need to be displayed at a given point in time during the unfolding of the simulation assessment. Simply having previously evaluated items disappear off the screen can reduce the clutter associated with scoring tools.

5 – Consider Video Scoring

For high stakes assessment and research purposes it is often wise to consider video scoring. High stakes meaning pass/fail criteria associated with advancement in a program, heavy weighting of a grade, licensure, or practice decisions. The ability to add multiple camera angles as well as the functionality to rewind and play back things that occurred during the simulation are valuable in improving the scoring accuracy of the collected data which will subsequently improve the interrater reliability. Video scoring associated with assessments requires considerable time and effort and thus reserved for the times when it is necessary.

I hope that you found these tips useful. Assessment during simulations can be an important part of improving the quality and safety of patient care!

If you found this post useful please consider subscribing to this blog!

Thanks and until next time! Happy Simulating.

Leave a comment

Filed under assessment, Curriculum, design, scenario design

Cognitive Load Control and Scenario Design in Healthcare Simulation

As the design architects of simulation scenarios, we must remain cognizant of our ability to have influence over the cognitive load of those experiencing our simulations in the role of learners.

When caring for patients in real life, we expend cognitive energy in doing so to ensure we make the right decisions to provide the absolute best care for every patient. We engage in critical thought processes, that guide our interpretation of the enormous number of facts surrounding each patient so we can make further decisions to provide various therapies, or advice to the patient.

Headache brain in a clamp isolated grey background

When we design simulations for our learners, we are creating similar environments noted above that demand a significant amount of cognitive workload to be endured for the participant to successfully navigate the case and care the [simulated] patient. In addition, I argue that we are adding additional cognitive workload by subjecting someone to the simulated environment insofar as they are engaged in a conscious or perhaps subconscious pursuit of deciding what is simulated and what is not. I have previously written about this and dubbed it the cognitive third space of simulation.

Nonetheless, there is mental energy spent in the care of the patient as well as the interpretation of the simulation. We also must realize that our design choices inside of the scenario contribute to the adjustment of the cognitive load endured by the learner(s) associated with our simulations. It is important that we be deliberate in our design to ensure that we are allowing all involved to achieve the desired learning outcomes.

Some specific examples of this cognitive load influence may help to bring forth an understanding. Take a test result for example. If one looks in the electronic health record and sees the values reported for a simple test, like a basic metabolic profile (which consists of a sodium, chloride, potassium, CO2, BUN, creatinine and glucose) there is a certain amount of mental energy goes into the interpretation of the numeric data presented for each of the seven items of the basic metabolic profile. Some electronic health records may color-code the results to assist in the processing of normal versus normal, and some may not.

Such a decision involved in the human factors design of electronic health record actually influences the amount of cognitive spend on the interpretation of the given value. Further, as experienced clinicians are keenly aware, we must interpret the lab value in the context of the patient for whom the test has been ordered. What is normal for one patient, may not be normal for another. Thus, even in the interpretation of a simple test, there is a significant amount of cognitive process (critical thought) that should be applied.

How does this relate to simulation scenario design? We have the ability to engineer the scenario design to help the participants channel cognitive energy into those things that are important and away from those those things that are not. If we continue to run with the example of the basic metabolic profile as an example, we have choices on how said values are reported to the participants of our simulation.

We could have the participants look it up in the simulated electronic health record which takes time and cognitive processing as described above. We could give them a piece of paper or display the results on a screen demonstrating the seven values. This still takes significant cognitive processing to interpret the data. We could simply indicate that the basic metabolic profile result was “normal”.  This method significantly decreases the cognitive processing associated with the seven values of the basic metabolic profile and how it is to be interpreted into the context of the scenario. Also, one could make the argument that we are offering subtle, or perhaps not-so-subtle clues to the case that the basic metabolic profile is not a major part of what needs to be processed in the care of this particular patient.  

It is important to realize that all the examples above are viable options and there is not one that is superior to another. It is important that the decision is made during the design of the case that allows the participant(s) of the scenario to focus the appropriate cognitive spend on that which the designers of the scenario feel are most important. In other words, if it is part of the learning objectives that the participant should evaluate the actual values of the basic metabolic profile, then of course it would be appropriate to provide the requisite information at that level of detail. If, however, the results of the same test are perfunctory to the bigger picture of the case then one should consider a different mechanism of resulting values to the simulation participant.

A common misperception in the design of healthcare simulation scenarios is to try to re-create the realistic environment of the clinical atmosphere. While this is always a tempting choice, it is not without consequences. It comes from the mistaken belief that the goal of simulation scenarios is to re-create reality. Modern, successful simulationists need to recognize this outmoded, immature thought process.

In the context of a case where the basic metabolic profile is not significantly important that we should not design the “dance” (scenario) to include the steps of looking in the electronic health record and making determinations of the values associated with the test. It is a waste of time, and more importantly a waste of cognitive processing which is already artificially increased by the participant being involved in the simulation in the first place. It is in my opinion a violation of the learner contract between faculty and students.

While I am focusing on a simple example of a single test, I hope that you can imagine that this concept extrapolates to many, many decisions that are made in the scenario design process. For example, think about a chest x-ray. Do you result a chest x-ray as “normal”, “abnormal” or otherwise during the run time of the scenario? Or do you show an image of a chest x-ray and have your participants interpret the image? One answer is not superior to the other. It is just critically important that you evaluate what is best for the cognitive load of the learners involved in your scenario and how the decision relates to the details of the learning objectives you wish to achieve during the course of the simulation activity.

In moderate to complex cases associated with healthcare simulation the designer of the simulation, or architect, has a responsibility to craft the scenario to accomplish the learning objectives that are intended. In many scenarios, hundreds of decisions are made in terms of how participants extract data from the experience to incorporate into their performance of the simulation. It is critically important that as the designers of such learning events that we remain cognizant of the cognitive load placed upon our learner(s) that is associated with the normal care of patients, as well as the extra that is imposed upon them from participating in a simulation-based case.

Many of the decisions that we incorporate into the design of our scenarios have significant influence over this cognitive load, and the mental energy participants will spend to engage in the participation. We need to understand the impact of our choices and be deliberate with our design decisions to enhance the overall simulation-based learning process efficiency and effectiveness.

Leave a comment

Filed under Curriculum, design, scenario design, simulation, Uncategorized

Exploring the Elements of Orientation and (Pre)Briefing in Simulation Based Learning Design

AdobeStock_119412077

I want to explore a little bit about orientation and (pre)briefing(s) associated with simulation based education design concepts. The words are often tossed about somewhat indiscriminately. However it is important to realize they are both important elements of successful healthcare simulation and serve distinct purposes.

When we look in the Healthcare Simulation Dictionary, we find that the definition of Orientation is aligned with an overview preparation process including “… intent of preparing the participants.” Examples include center rules, timing and the simulation modalities.

On the other hand, according to the same dictionary the definition of the word Briefing includes “An activity immediately preceding the start of a simulation activity where participants receive essential information about the simulation scenario….”

I look at orientation as the rules of engagement. I like to think of orientation linked to the overall educational activity in total. Some essential components include orientation to the simulation center, the equipment, the rules, and the overall schedule for the learning activity.

At a somewhat deeper level of thought I think the orientation is linked to the learning contract. What do I mean by that?

I think it is essential that we as the faculty are establishing a relationship with our learners and begin to establish trust and mutual respect. To that end, we can use orientation to minimize surprises. Adult learners do not like surprises!

We need to have the adult learner understand what they can expect. I always orient the learners as to what will feel real, and I am similarly honest with them about what will not feel real. If they will be interacting with a computerized simulator for example, I orient them to the simulator before the start of the program.

In the simulation world we throw around words like debriefing, scenario and task training. To clinical learners these terms may be unfamiliar, or have different contexts associated with them. This for example, can cause anxiety and during the orientation we need to walk them through the experience they are about to embark upon.

Some factors can influence the amount and depth of the orientation. Variables such as the familiarity your participants have with simulation, your simulation center, and your simulation-based encounters. For example, learners who come to your center on the monthly basis probably need less total orientation than those who are reporting for the first time. Learners familiar with the fact that debriefings occur after every simulation may already be acclimated to that concept, but people coming to the sim center for the first time may not be aware of that at all.

Participants just meeting you for the first time they might need a little bit more warming up and that an come in the form of orientation. Overall though it is not just about telling them what’s going on, as it is using the opportunity toward earning their trust and confidence in the simulated learning encounter(s) and the value associated to them as a professional.

BriefingGraphic3Switching the focus to the brief, briefing or (pre)briefing. The briefing is more linked to the scenario as compared to the orientation. The briefing should focus on the details of the case at hand introducing components of information that allow one to acclimate to what they going to need to accomplish during the simulation. What is their role and goals in this scenario they are about to embark upon? If you are going to ask people to play different roles then they are in real life, it is very important that this fact is crystal clear in the briefing.

I think that the briefing should also bring the context to the healthcare experience. It is important to orient the learner for the impending encounter what they are to perceive and think of as real as they are experiencing what is in the simulation. You as a simulation faculty may think that it is obvious that a room in your simulation center is an ICU bed. The participant may not and deserves clarity prior to the start of the simulation so they do not feel like they are being tricked or duped. During the briefing the statement “You are about to see a patient in the ICU…..” can remove such ambiguity.

Another critical briefing point is to clarify the faculty-student engagement rules that should be expected during the scenario runtime if it was not covered in the orientation. There are many correct ways to conduct simulation scenarios. There are varying levels of interaction between faculty members running the simulation and the learners that are participating. This should be clarified before the scenario starts.

For example, are you going to let the learners ask questions of the of the faculty member during the simulation? Or not? This should be upfront and covered in the briefing, and perhaps even aspects of that in the orientation.

While not a requirement I think that parameters associated with time expectations are always good to give in a briefing. For example stating “You are going to have 10 minutes in the scenario to accomplish X,Y and Z, and then we will have a ten minute debriefing before the next scenario.”

Remember our adult learners don’t like surprises! I always use the briefing before a scenario to remind the participant(s) that afterward we are going to have a debriefing. I remind them of that so that they know that they should collect her thoughts and ideas and be ready to have this discussion. Secondly, I am saying in any unspoken way, that if they are uncomfortable about something, or have questions, that there will be an opportunity for discussion during the debriefing. (In other words, your sort of giving some control back to the learner…. Helping to build the trusting relationship.)

Some of the variations of the briefing are similar to that of the orientation mentioned above. People who are more familiar to simulation, your particular programs, your style, may require slightly less of a briefing than others. Additionally, if you are running multiple scenarios as part of a simulation-based course, after the first couple of scenarios you will find that the briefing can be shortened as compared to the beginning of the day.

So, in summary, orientation and briefings are different elements of simulation-based learning that are useful for different things that will contribute to the success of your simulations.

Think of orientation linked to the bigger picture and the learner contract that contributes to making the relationship comfortable between the participants and the faculty. The orientation is the rules of engagement and orientation to the technology and being explicit as to what is to be expected of the participant. Think of the briefing as linked more to the scenario roles, goals, and introduction to patient and environment information to help the participant mentally acclimate to what they are about to dive into.

Leave a comment

Filed under Curriculum, scenario design, simulation, Uncategorized