Tag Archives: scenario design

Embedding Forcing Functions into Scenario Design to Enhance Assessment Capabilities

shutterstock_316201547Many people design scoring instruments for simulation encounters as part of an assessment plan. They are used for various reason ranging from a tool to help provide feedback, research purposes, to high stakes pass/fail criteria. Enhancing the ability for assessment tools to function as intended may often be closely linked to scenario design.

Often times checklists are employed. When designing checklists is critical that you are asking the question “Can I accurately measure this?”. It is easy to design checklists that seem intuitively simple and filled with common sense (from a clinical perspective) but are not actually able accurately measure what you think you are evaluating.

It is quite common to see checklists that have items such as “Observes Chest Rise”; “Identified Wheezing”; “Observed Heart Rate”. During faculty training sessions focusing on assessment tool development we routinely run scenarios that contain deliberate errors of omission. These items, some are routinely scored, or “checked” as completed. Why is this? Part of the answer is we are interjecting our own clinical bias into what we think the simulation participant is doing or thinking. This raises the possibility that we are not measuring what we are intending to measure, or assess.

Consider two checklist items for an asthma scenario, one is “Auscultates Lung Sounds”; another item is “Correctly Interprets Wheezing”. The former we can reasonably infer from watching the scenario and see the participant listen to lung fields on the simulator. The latter however is more complicated. We don’t know if the participant recognized wheezing or not by watching them listen to the lungs. Many people would check yes for “Correctly Interpreted Wheezing” if the next thing the participant did was order a bronchodilator. This would be an incorrect assumption, but could be rationalized in the mind of the evaluator because of a normal clinical sequence and context.

However, it may be completely wrong and the participant never interpreted the sounds as wheezing, but ordered a treatment because of a history of asthma. Or what would happen if the bronchodilator was ordered before auscultation of the lungs? What you have by itself, is an item on your checklist that seems simple enough, but is practically unmeasurable through simple observation.

This is where linking scenario design and assessment tools can come in handy. If the item you are trying to assess is a critical element of the learning and assessment plan perhaps something in the simulation, transition to, or during the debriefing can cause the information to be made available to more correctly or accurately, assess the item.

A solution to a real-time assessment during the flow of the scenario is possible within the design of the scenario. Perhaps inserting a confederate as a nurse caring for the patient that is scripted to ask “What did you hear?” after the participant auscultates the lungs fields. This will force the data to become available during the scenario for the assessor to act upon. Hence the term, forcing function.

Another possibility would be to have the participant complete a patient note on the encounter and evaluate their recording of the lung sounds. Another possibility would just be to have the participant write down what their interpretation of the lung sounds were. Or perhaps embed the question into the context of the debriefing. Any of these methods would provide a more accurate evaluation of the assessment item ““Correctly Interpreted Wheezing”.

While not trying to create a list of exhaustive methods I am trying to provide two things in this post. One is to have you critically evaluate your ability to accurately assess something that occurs within a scenario with higher validity. Secondly, recognize that creation of successful, reliable and valid assessment instruments are linked directly to scenario design. This can occur during the creation of the scenario, or can be as a modification to an existing scenario to enhance assessment capabilities.

This auscultation item just serves as a simple example. The realization of the challenges of accurate assessment of a participants performance is important to recognize to allow for the development of robust, valid and reliable tools.  The next time you see or design a checklist or scoring tool think in your own mind….. Can I really, truly evaluate that item accurately? If not, can I modify the scenario or debriefing to force the information to be made available?


Leave a comment

Filed under scenario design

The Contract Essential to the Parties of Simulation

If you think about it an agreement needs to exist between those whom facilitate simulation and those who participate. Facilitate the purposes of this discussion is referring to those who create and execute simulation based learning encounters. Sometimes the agreement is more formal other times more implied. This phenomenon has been described in many ways over the yearsshutterstock_226296865 having been branded by such descriptors as fiction contract, psychological contract, or learning contract.

Why does this need to be the case? A contract or agreement is generally called for when two or more parties are engaging in some sort of collaborative relationship to accomplish something. Often times these type of contracts spell out the responsibilities of the parties involved. If you think about simulation at a high level the facilitator side is agreeing to provide learning activities using simulation to help the participant(s) become better healthcare providers. The participants are engaged at the highest level because they want to become better healthcare providers. While not trying to hold a comprehensive discussion, let’s explore this concept and the responsibilities of each party a bit further.

Facilitators are designing simulation activities with a variety of tools and techniques that are not perfect imitators of actual healthcare. They are crafting events for which the participant to a greater or lesser extent immerse themselves in, or at a minimum simply participate. Some of these activities are designed to contain diagnostic mystery, some demand specific knowledge, skills and attitudes be known or developed to successfully complete the program. Facilitators are also putting participants in situations that the must perform in front of others and that can create feelings of vulnerability. So all toll, the role of the facilitator comes with enormous responsibility.

Facilitators are also asking the participants to imagine part of what they are engaging in is a reasonable facsimile of what one may encounter when providing actual healthcare. Therefore another tenet of the agreement is that the facilitator will provide an adequate orientation to the simulation environment pointing out what is more and less real including the role that the participant may be playing and how their role interacts with the environment outside of the simulation, if at all. (I.e. define any communications that may occur during the simulation between the participants and the facilitator.

Facilitators trained in simulation know that mistakes occur sometimes due to a lack of knowledge, incorrect judgement or unrelated issues such as a poorly designed simulation. Facilitators thereby commit to not judge the participant in anything other than their performance during the simulation. While diagnostic conundrums are inevitable in many types of simulations the facilitator should not try to unnecessarily trick or mislead the participant in any way that is not directly contributing to helping the participant(s) improve. The facilitator must attempt to use the time of the participants wisely and responsibly.

The role of the participant shares responsibilities as a part of the agreement as well. Participants agree to a commitment to become better healthcare providers through continuous learning and improvement. This is inherent in a professional, but there are some likely good reasonsshutterstock_147464348 to remind participants of this important premise.

Participants must agree to the use of their time to participate in the simulation. The participants are also agreeing to an understanding that they know the environment of the simulation is not real, and that there will be varying levels of realism employed to help them perform in the simulation. But to be clear they agree to this tenet predicated on the trust that that facilitators are having the participant experience simulations that are relevant to what they do, with an underlying commitment to help them get better. In simulations involving multiple participants, they must also agree to similarly not judge others on what occurs in the simulation, as well as keeping the personal details of what they experience in the simulation confidential.

So in closing, successful simulation or other immersive learning environments require an agreement of sorts between those who create and execute the simulation based learning environments as well as those who participate in them. Each party brings a set of responsibilities to the table to help to ensure a rich learning environment with appropriate professional decorum and commitment to improvement. The agreements range from implicit to explicit, but when they exist and are adhered to will continue to allow the recognition of value that can arise from simulation to help improve the care ultimately delivered to our patients. After all, isn’t that our highest goal?

1 Comment

Filed under Uncategorized

Extra Cognitive Processing Associated With Simulation – The Cognitive 3rd Space

This is a concept I’ve observed over the years. In the design and conducting of simulation we as facilitators and faculty members develop a shared mental model in what we see in a simulated environment, how we act in a simulated environment, aCognitiveGuynd how we anticipate that our trainees will act in the simulated environment.

Embedded in the latter assumption is what I call the cognitive 3rd space of simulation. Conceptually this refers to the fact that participants of simulation in healthcare have a background thought process that is continuously assessing what they are seeing in the simulated environment and trying to decide what the facilitators are trying to indicate with the presence, and sometimes absence, of the various pieces of equipment, clinical finding replicas, and other accoutrements of the environmental stimulus associated with simulation. In other words there is a continuous background thought process trying to figure out is this that I am seeing supposed to be simulated or not.

In the real clinical environment where healthcare providers are gathering data from interviews, observations, physical examinations, test results etc. that feed into the eventual analysis which leads to a decision-making plan. This cognitive 3rd space associated with simulation is the fact that this continuous reconciliation of “what are they trying to simulate for me?” question that is continuously active in the mind of the trainees during simulation encounter in addition to the traditional process of data gathering analysis and treatment planning associated with the provision of real healthcare.

The degree of which a participant manages this third space is multifactorial and relates to many things including experience in the simulated environment, the orientation, the environment itself, their own confidence as well as the degree of buy-in that they have for the overall experience.

Reconciling this requires us to make a conscious understanding that when we provide stimulus in the simulated environment it may or may not be interpreted by the participant of simulation in the same way that it was intended. Helping to control the potential variation and confusion that can result from this is embedded into the design of our simulations, briefings and orientation, equipment selection and the interactions that go on between participants and facilitators of simulation events.

A variant to this also relates to the environmental set up of the simulation space. Participants are often focusing on “clues” in the surroundings of the simulated clinical environment. For example, if they notice an intubation set up on the bedside tray table they may think “this scenario requires an intubation.” While in the simulation they may or may not perform an intubation in the patient as a result of the observation, however this thought process or separate thread of thought is extramural to the normal cognitive processing that might go on in caring for a real patient.

As designers of simulations we must work to ensure that try to keep this interpretive grey zone minimized. This often runs afoul of the desire for many who try to recreate reality and go onto to develop the theatrics of simulations with clever remedies that may actually introduce further confusion into the mindset of the participant. The result may be an impediment in the ability to evaluate the performance in terms of the ability of the decisions to translate to the real care environment.

Leave a comment

Filed under Uncategorized

Ebola and Fidelity

hazmat_shutterstock_135522821_aThose of you who are used to my normal musings and rants against perfecting the “fidelity” and realism used in simulations might be surprised to hear me speak of examples of simulations where perfect/near perfect fidelity does matter.

Various association social forums are abuzz with people talking about simulations involving personal protective equipment in the light of the current unfolding of the Ebola crisis. It is important to differentiate this type of simulation and recognize the importance of re-creating the aspects of the care environment that is the subject of the education in the most highly realistic way available. In this case we are probably talking about using the actual Personal Protective Equipment (PPE) equipment that will be used in the care of the patient suspected of Ebola at any given facility.

This is a high-stakes simulation where the interaction with the actual equipment that one will be using in the care environment is germane to a successful outcome of such interaction. In this case the successful outcome is keeping the healthcare worker safe when caring for a patient with a communicable disease.  More broadly this falls under the umbrella of simulation for human factors.

Human factors in this context being defined as “In industry, human factors (also known as ergonomics) is the study of how humans behave physically and psychologically in relation to particular environments, products, or services.” (source: searchsoa.techtarget.com/definition/human-factors)

Other examples of when human factors types of simulation are employed are in areas such as product testing, equipment familiarization objectives, environmental design testing. So for instance if we are evaluating the number of errors that occurs in the programming of a specific IV pump in stressful situations, it would be important to have the actual IV pump or a highly realistic operational replica of the same. This is in contrast to having the actual IV pump used in a hospital for scenario focused on an acute resuscitation of the sepsis patient, but not specifically around the programming of the IV pump. The latter example represents more of when the IV pump is included more as a prop in the scenario versus that of the subject of the learning objectives and inquiry on the safety surrounding its programming.

So yes world, even I fully believe that there are some examples of simulations where a re-creation of highly realistic items or elements is part and parcel to successful simulations. The important thing is that we continuously match the learning objectives and educational outcomes to those elements included are simulations so that we continue to be most efficient and efficacious in our designs of simulation-based education encounters. What I continue to discourage is a simple habit of spending intense time and money in highly realistic re-creations of the care environment when they are not germane to the learning objectives and educational outcomes.

1 Comment

Filed under Uncategorized

Beware of Some of the Costs of Fidelity and Simulation

Fidelity is a concept that is often not thought about in detail and can contribute to being a barrier in the adoption simulation. It is also one of the most misused words in simulation, and can unnecessarily contribute to raising the costs and complexities of simulations. It can results in a significant desire to make every attempt to re-create the reality of the patient care environment without consideration of the necessity related to accomplishing the learning objectives. This has tremendous implications. Trying to overdo fidelity can cause unnecessary set up time, cleanup time, and therefore add significant costs and inefficiencies to accomplishing a simulation-based encounter. This is not to suggest tat we shouldn’t consider planning for elements that help create a more realistic presentation f what we are trying to simulate, it is just suggesting caution that each element be carefully considered for necessity and overall impact to the entire scenario including costs.BloodyMess

This set of costs is very concerning because they are in a category of what I consider add to the hidden barriers of simulation. This arises from imbedding elements of simulations in to the design that contribute to the complexity, overall costs and don’t add to the educational outcomes.

Furthermore, attempts at trying to re-create fidelity can often add confusion for the simulation participants in trying to interpret what it is that is being simulated. As I have discussed previously, there is no such thing as suspension of disbelief. Participants of simulations are constantly having to interpret their environment. Since they are engaged in a simulation they are always hyperaware of the fact that some things are simulated and some things are attempts at creating normalcy. The take-home message is that the learning objective should be carefully considered to determine what aspects of a corresponding actual situation needs to be simulated in the laboratory environment.

A careful orientation of the learner along with describing the capabilities and limitations of the simulation will engage the participant in a way that you could accomplish the learning objectives with a minimal amount of work placed into recreating the fidelity of the situation. This is particularly true of trying to re-create the environment, physical exam findings or situation that a patient is likely to be found in when compared to an actual clinical encounter.

Leave a comment

Filed under Uncategorized

Feedback – More Than Just Debriefing


Too often in the designs of scenarios for simulation there is a lack of attention to the fact that there are many forms of feedback that occurs during a simulation than the debriefing. Debriefing is certainly an important part of any learning encounter, but in reality represents only one type of feedback.

As you think about sources of feedback I ask you to be both creative and attentive. I like to think of feedback in two broad categories, intrinsic and extrinsic. The latter being the more commonly thought of mechanisms such as debriefings, video reviews, and simulator log file reviews etc. with the former being the topic of this post.

What I find to be of significant interest as it related to the design of scenarios is the feedback that occurs intrinsically. That is clues, or changes that occur during the scenarios that are available to the participant to incorporate in their understanding of how their decisions, treatments, or lack thereof, are affecting the statues of the patient.

Many of you might be saying, what are you talking about???? Ha!!! Now on to my favorite part which is providing concrete examples to help explain myself further.  Let’s say you are simulating a pelvic fracture case with hypotension and shock. The vitals’ of the high technology simulator that you may be using for the case would likely show tachycardia and hypotension etc. Now lets say the participant(s) place a pelvic binding device and give a unit of blood. You may include changes in the vitals appearing on the monitor that indicated that there was mild to moderate improvement of the patient. Perhaps the tachycardia would decrease and the blood pressure may improve over a set period of time.

During the design process of the scenario many people may create the changes in the vitals thinking they are mimicking reality of what may occur.  More importantly I think those involved in the design of the scenario should realize that the changes in the vitals referred to above are a source of important intrinsic feedback. The participants should be able to make the observations and decide they are helping the patient to improve.

This can be powerful feedback that links together successful performance with particular behaviors or decisions that were made. It is self-discovery, it will help to guide further care and decisions if the scenario continues. If the designer of the scenarios recognizes this intrinsic feedback in the design phases, additional creative solutions can be implemented to reinforce the learning.

While my pelvic fracture example shows a positive change tin the patient based on correct actions, the converse example could be true if incorrect care is being rendered. Consider that if you have a heart attack case with hypotension and the patient is administered aspirin and nitroglycerin. You would likely worsen the shock from a physiological perspective. Seeing this change will provide intrinsic feedback to the learners(s).

This is not to say that it all has to do with fancy feedback from high technology simulators. The same could be accomplished with showing a worsening EKG on paper if treatment is incorrect or delayed, or conversely showing and improving EKG for the right treatment given within the appropriate time.

Approaching feedback from a deliberate perspective can be helpful in reinforcing learning. It should be recognized as a design tool and carefully integrated into the core deign of any scenario. Debriefing can be linked to these feedback areas in the scenario. This can provide valuable links or areas ripe for discussion to assist in accomplishing the learning objectives.

1 Comment

Filed under Uncategorized

Simulation: It’s Not All About Teamwork


Competent Professionals. That’s the thing that ALL health care teams are made of!

Those of us involved in simulation know that it is a powerfully, wonderful tool for educating and assessing people with regard to teamwork skills, responsibilities and communications. However that does not mean that every implementation of simulation needs to involve teamwork and communication skills.  Since the famous Institute of Medicine report came out in 1999 citing critical deficiencies in teamwork in healthcare environment and how they can be attributed to medical errors and less than quality patient care, as well as a recommendation for simulation to fill some of these gaps, many people have taken it as a mandate that simulation MUST include team training and communications. I say, Not the case.

In fact, I would argue it’s detrimental to think that way. Teams are built upon the pillars and shoulders of competent healthcare professionals. There are still gaps in educational needs in the ability to more efficiently educate individual healthcare providers, as well as provide assessment into their overall competence based on where they are in the spectrum of entry-level student to practicing clinician. While I would certainly advocate for teamwork and communication skills to be incorporated into any healthcare curriculum from beginning to end, I think there are plenty of appropriate times to ensure there is a mastery of individual competence in those goals and objectives that are surrounding knowledge, skills critical thinking and attitudes that we expect a competent professional to bring to a team.

The design and execution of the innovative education programs involving simulation has many different variables when we consider the needs of that of ensuring individual or personal confidence, versus that of a team. In fact the learning objectives would look quite different if we compare evaluating that of the responsibility of an individual person providing care, to that of the skill set that person should develop to be an integral part of the healthcare team, as further compared to the skill set that we would expect a team of professionals to demonstrate when involved in the care of the patient(s).

The detrimental part that I speak of I see far too commonly in my travels and observations of simulation efforts. There are many times when simulations are conducted that are largely designed to educate, and/or assess aspects those objectives that are largely individual in nature, such as decision-making, analysis, or perhaps psychomotor motor skills. Then at the last minute there is a design decision to include more students and literally “throw in” some elements of teamwork and her communications into the design of the scenario that ends up not being well thought out.

The end result is not hard to predict. Such simulations usually produce less than desired outcomes with regard to ensuring individual competence improvement, and are often confusing and/or inefficient with regard to being able to impart the skills associated with teamwork and communications.

So I do think that teamwork and communications, as well as team leadership skills are an important set of competencies that we need to continuously teach and evaluate across the spectrum of healthcare education and believe that simulation is one of many excellent tools to accomplish this.

However, I advocate that you proceed carefully with a well vetted list of objectives that lead to adequate scenario and curriculum design. I think you should carefully determine whether your overall educational effort is best suited for the competency of an individual, or is it largely directed towards focusing on teamwork and interpersonal communication efficiencies and effectiveness.

I do believe we can design encounters that effectively involve elements of both, but not without careful planning and forethought. In such attempts there is always a natural tension between those elements of design detail and assessment that can be applied to individual competence and those that can be applied to the team components.

Attention to these details will assist you in creating more robust educational encounters for your learners. It will also afford you better outcomes for your overall investment of time and money into your education program.

Leave a comment

Filed under Uncategorized