4-6 Design-Based Research for Health Professions Education
Arianna Mazzeo and Teresa Chan
Overview
Design fiction can be harnessed as a methodology to help elicit needs and engage in futurism. Aligned with applied research techniques such as design-based research, practitioners aim to exploring and criticizing possible futures by creating speculative scenarios narrated through designed artifacts. Design-based methodology allows researchers to go beyond simply studying a phenomenon or an outcome – instead, it encourages researchers to think and act like designers by engage knowledge end-users via meaningful interactions (discussion, observation, etc..) and using insights from these interactions to create better outputs that can be used to assist with knowledge translation.
Key Points of the Chapter
By the end of this chapter the learner should be able to:
- Describe design-based research methodologies and how they might be used in their health professions education research.
- Use design principles and tools to discover new research opportunities and questions.
- Articulate how these approaches can help with knowledge translation and/or co-creation phases.
Vignette
Andy, a lecturer in the Faculty of Health Sciences, was attending a technology conference. While wandering the exhibit hall between sessions, he found himself drawn to the vendor booth of an AR/VR technologies firm that he’d been following since has a trainee. His old friend, and former fellow postdoctoral fellow Devin was stationed at the booth and waved at Andy.
“Hi there Andy! It’s been a while… I hope you’re doing well?”
“I’ve been well… Survived the pandemic and got myself a job as a lecturer at our old alma mater. To be honest, I’ve been watching your career for a while and been quite proud of your success. It’s been very cool to see your start-up take off! Virtual reality and augmented reality are now so trendy. But I didn’t know you were trying to make a play into health professions education… Honestly, do you see a lot of applications in health professions education?”
“We are gaining marketshare in customizing content for immersive environments,” replied Devin. “That said, we don’t have much a presence in health professions education right now. Would you be interested in partnering with us to engage in research around the use of some of our new technologies?”
Andy scratched his head. He wasn’t really sure how augmented reality/virtual reality (or as the students often called it, “AR/VR”) could really be used yet, but perhaps it would be clearer if he asked a few more questions. He asked: “I don’t have a lot of budget to build AR/VR type immersive environments yet. I mean, we could certainly partner to see if we could get some grants to develop these, but that might take a while. Are there a low stakes way to engage in developing possible research applications without having to know how to program? I’ve been quite intrigued by this whole space, but I’m not 100% sure how we would use it in my area yet. Maybe we could use fictional personas and some design-based research techniques to develop these ideas out? What do you think?”
Now it was Devin turn to be confused. “What is a fictional persona? An actor?”
“A persona is a fictional character which you create based upon your research in order to represent the different user types that might use your service, product, site, or brand in a similar way. Creating personas helps the designer, in particular, of your VR/AR technology to understand users’ needs, experiences, behaviours and goals.”
“Ok! Yes… That is fascinating. Definitely could partner with you to employ personas and fictional characters to flesh this out. Let’s explore this more!”
Deeper Dive into this Concept
Design-Based Research (DBR) is a methodological framework used by researchers in the learning sciences, as applied methodology to teach and learn in education (1-3). The basic process of DBR involves developing solutions, called design interventions, for complex and systemic problems. Then, the design interventions are put to use to test how well they work and further refined. Through iterations, these interventions can be adapted and re-tested to gather more data. The purpose of this approach is to generate new theories and frameworks for conceptualizing learning, instruction, design processes, and educational learning innovation for complex societal problems.
The merits of the DBR methodology to research interventional designs to change learning environments have contributed to significant understanding and development of Healthcare educational practice. As a change methodology, DBR is well-suited for inquiry beyond the evaluation of teaching and learning environments.
The rationale for using DBR-type methodologies would be to better integrate discovery scholarship with application scholarship. First, the future is uncertain and unpredictable, and yet, we can be fairly certain that for HPE the future will continue to unfold and be unknown. The forces shaping the future will emerge unevenly across geographies, industries and populations, and DBR allows HPE researchers to tap into the creative parts of their process to engage in foresight and change that is ahead of the curve. The decisions we make today will play out in the future – and not always as intended – in a world that is complex, diverse and dynamic. DBR is a useful tool for both researchers and innovators alike, among other strategies for engaging in strategic foresight and other futurist strategies.
Similarities to Other Approaches
For those familiar with quality improvement and program evaluation, this iterative approach may feel similar to concepts such as the Plan-Do-Study-Act (PDSA) approach (4) or a Continuous Quality Improvement (CQI) cycle for an educational program (5). It is also notable that the act of engaging stakeholders in co-design via cyclical iterations of consultation will have shades of overlap with participatory action research and other forms of stakeholder consultation processes that are increasingly employed in clinical research. There are extensive overlaps between DBR and design thinking (which is an increasingly popular technique for determining educational needs (6)). In fact, the overlap is so extensive that in many circles DBR and design thinking are thought of interchangeably.
Personas & Scenarios
Personas are fictional cases that act as exemplars and archetypes for the design challenge. These personas are often developed with some basic demographic details – enough so that the participants in the design challenge can engage with the persona, but not so detailed that they are daunted by the intricacies of the case.
Classically, the personas are then placed in various situations to develop scenarios. Scenarios are stories about how the future might unfold for an organization, a community, a nation or even the world. Scenarios are anchored by an important exploratory or strategic question (e.g. What is the future of artificial intelligence in health professions education? Should we build an online medicine program in Tunisia?). Scenarios are created and used in sets of three or four to express a range of future possibilities, undergirded by major uncertainties that are critical to the focal question. Good scenarios for health education are both provocative and plausible. They are hypotheses, not predictions. Scenarios usually tell a story looking back from some point in the future – most often 10 years out. Scenarios can be descriptive or character-driven. As a set, the scenarios describe how and why particular futures have unfolded in ways that are meaningfully different from one another. This technique can help to generate foresight into possible futures, and help you to anticipate pain points or problems before they happen. An example of this can be found in a recent commentary paper within Academic Medicine where authors anticipated multiple possible futures for the use (or misuse) of learning analytics in the near future (7).
Examples of application of Design Based Research in Health Professions Education
Dr. Brent Thoma and his team have used Design-Based Research (DBR) in the creation of several dashboard visualization tools to assist educators and trainees in the age of Competency-Based Medical Education (7). From this teams’ work, we can better understand how Competence Committees need to visualize learner assessment data effectively to support their decision making. Dashboards play an integral role in decision support systems in other fields. DBR allows the simultaneous development and study of educational environments.
Thoma’s team used a DBR process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements (8). Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings (8). Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard (8). This team has also used DBR to elucidate the needs of trainees (9) and also those of faculty developers (10) who might be able to use this data in new ways.
In addition, there are several ways that DBR can be used in HPE research and scholarship. See Table 4.6.1 for cases examples of how DBR can be used to enhance the ways in which scientists can engage with end-users to close the knowledge translation gap, but also engage in methods to change their research course with stakeholder consultation.
| Phase of Research | Use Case | Example of an HPER project |
| 1.Before you start designing a project or program | a) Identifying end-user pain points | Chan TM, Mercuri M, Turcotte M, Gardiner E, Sherbino J, de Wit K. Making decisions in the era of the clinical decision rule: how emergency physicians use clinical decision rules. Academic Medicine. 2020 Aug 1;95(8):1230-7. https://doi.org/10.1097/ ACM.0000000000003098 (11) |
| b) Determine new research directions such as new clinical implementation strategies | Kamhawy R, Chan TM, Mondoux S. Enabling positive practice improvement through data‐driven feedback: A model for understanding how data and self‐perception lead to practice change. Journal of Evaluation in Clinical Practice. 2021 Aug;27(4):917-25. https://onlinelibrary.wiley.com/ doi/abs/10.1111/jep.13504 (12) |
|
| 2. Needs Assessment for Designing a new Product or Service | Identifying needs of your target audience or end-users | Munford V. Designing a centralized faculty performance dashboard: Optimizing feedback and scholarly data reporting. Thesis. eHealth Program; McMaster University. 2021. https://macsphere.mcmaster.ca/ bitstream/11375/27045/2/ Munford_Vanessa_2021Sept_eHealth.pdf (13) |
| Chorley A, Azzam K, Chan TM. Redesigning continuing professional development: Harnessing design thinking to go from needs assessment to mandate. Perspectives on Medical Education. 2020 Aug 12:1-6. https://link.springer.com/article/ 10.1007/s40037-020-00604-1 (14) |
||
| 3. For stakeholder consultation to ensure research is relevant | Knowledge user engagement strategy for facilitating improved end-of-project knowledge uptake | Zarabi S, Chan TM, Mercuri M, Kearon C, Turcotte M, Grusko E, Barbic D, Varner C, Bridges E, Houston R, Eagles D. Physician choices in pulmonary embolism testing. CMAJ. 2021 Jan 11;193(2):E38-46. https://www.cmaj.ca/ content/193/2/E38.abstract (15) |
| 4. To engage in evaluation of a project/program/service | Auditing existing resources within a program | The McMaster Program for Faculty Development Faculty Journey Vignette Project (https://journey.macpfd.ca ) |
| 5. Iterative, Co-Design of an Education Product | Improving Knowledge User & Stakeholder Engagement via early involvement and active co-design. | Thoma B, Bandi V, Carey R, Mondal D, Woods R, Martin L, Chan T. Developing a dashboard to meet Competence Committee needs: a design-based research project. Canadian medical education journal. 2020 Mar;11(1):e16. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7749685/ (8) |
| Carey R, Wilson G, Bandi V, Mondal D, Martin LJ, Woods R, Chan T, Thoma B. Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project. Canadian Medical Education Journal. 2020 Dec;11(6):e31. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc7082472/ (9) | ||
| Yilmaz Y, Carey R, Chan TM, Bandi V, Wang S, Woods RA, Mondal D, Thoma B. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Canadian Medical Education Journal. 2021;12(4):48-64. https://europepmc.org/article/med/34567305 (10) |
Key Takeaways
- Be concise. Brevity is the soul of wit and describing the scenario or using visuals to help and share your idea.
- Be culturally and socially appropriate and sensitive. Make sure that your characters/situations avoid cultural stereotypes. Vignette stories and language should be checked for bias, encourage participants to lay aside their innate judgement. Ensure that you avoid racist or discriminatory lexicon at all times.
- Have fun and enjoy the journey.
References
- Anderson T, Shattuck J. Design-based research: A decade of progress in education research?. Educational researcher. 2012 Jan;41(1):16-25. http://edr.sagepub.com/content/41/1/16
- Dai D. Design Research and Twice Exceptional Children: Toward an Integration of Motivation, Emotion, and Cognition Factors for a Technology-based Intervention: Brenda Bannan. In: Design Research on Learning and Thinking in Educational Settings 2012 Apr 23 (pp. 111-138). Routledge.
- Barab S, Squire K. Design-based research: Putting a stake in the ground. The journal of the learning sciences. 2004 Jan 1;13(1):1-4. https://www.tandfonline.com/doi/pdf/10.1207/s15327809jls1301_1
- Leis JA, Shojania KG. A primer on PDSA: executing plan–do–study–act cycles in practice, not just in name. BMJ quality & safety. 2017 Jul 1;26(7):572-7. http://dx.doi.org/10.1136/bmjqs-2016-006245
- Stratton TD. Legitimizing continuous quality improvement (CQI): Navigating rationality in undergraduate medical education. Journal of general internal medicine. 2019 May;34(5):758-61. https://doi.org/10.1007/s11606-019-04875-1
- Gottlieb M, Wagner E, Wagner A, Chan T. Applying design thinking principles to curricular development in medical education. AEM education and training. 2017 Jan;1(1):21-6. https://onlinelibrary.wiley.com/doi/abs/10.1002/aet2.10003
- Thoma B, Ellaway RH, Chan TM. From Utopia through Dystopia: charting a course for learning analytics in competency-based medical education. Academic Medicine. 2021 Jul 1;96(7S):S89-95. https://journals.lww.com/academicmedicine/Fulltext/2021/07001/From_Utopia_Through_Dystopia__Charting_a_Course.16.aspx
- Thoma B, Bandi V, Carey R, Mondal D, Woods R, Martin L, Chan T. Developing a dashboard to meet Competence Committee needs: a design-based research project. Canadian medical education journal. 2020 Mar;11(1):e16. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7749685/
- Carey R, Wilson G, Bandi V, Mondal D, Martin LJ, Woods R, Chan T, Thoma B. Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project. Canadian Medical Education Journal. 2020 Dec;11(6):e31. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc7082472/
- Yilmaz Y, Carey R, Chan TM, Bandi V, Wang S, Woods RA, Mondal D, Thoma B. Developing a dashboard for faculty development in competency-based training programs: a design-based research project. Canadian Medical Education Journal/Revue canadienne de l’éducation médicale. 2021;12(4):48-64. https://europepmc.org/article/med/34567305
- Chan TM, Mercuri M, Turcotte M, Gardiner E, Sherbino J, de Wit K. Making decisions in the era of the clinical decision rule: how emergency physicians use clinical decision rules. Academic Medicine. 2020 Aug 1;95(8):1230-7. https://doi.org/10.1097/ACM.0000000000003098
- Kamhawy R, Chan TM, Mondoux S. Enabling positive practice improvement through data‐driven feedback: A model for understanding how data and self‐perception lead to practice change. Journal of Evaluation in Clinical Practice. 2021 Aug;27(4):917-25. https://onlinelibrary.wiley.com/doi/abs/10.1111/jep.13504
- Munford V. Designing a centralized faculty performance dashboard: Optimizing feedback and scholarly data reporting. Thesis. eHealth Program; McMaster University. 2021. https://macsphere.mcmaster.ca/bitstream/11375/27045/2/Munford_Vanessa_2021Sept_eHealth.pdf
- Chorley A, Azzam K, Chan TM. Redesigning continuing professional development: Harnessing design thinking to go from needs assessment to mandate. Perspectives on Medical Education. 2020 Aug 12:1-6. https://link.springer.com/article/10.1007/s40037-020-00604-1
- Zarabi S, Chan TM, Mercuri M, Kearon C, Turcotte M, Grusko E, Barbic D, Varner C, Bridges E, Houston R, Eagles D. Physician choices in pulmonary embolism testing. CMAJ. 2021 Jan 11;193(2):E38-46. https://www.cmaj.ca/content/193/2/E38.abstract
About the Authors
name: Arianna Mazzeo
institution: McMaster University
Postdoctoral FellowArianna Mazzeo (she/her), is a postdoctoral fellow, Office of Continuing Professional Development, Faculty of Health Sciences, McMaster University. Prior to this position, she taught the course Methodologies and Practice in Design Engineering Research at Harvard University’s Master in Design Engineering Program, where she was also an advisor for Independent Design Engineering Projects (IDEP). Previously, she was an Associate Senior Research Fellow at RISD and at NYU TISCH Performance Studies, where she works on introducing innovative design research and experimental pedagogies of embodiment for social change in online education.
Grounded in critical and “artistic intelligence” as well as open learning design intervention in the public domain, her research focuses on design fiction scenario and video-based learning as tools to deliver interactively in the online environment enabling virtual collaboration. Her practice-based design research is aiming at the creation of a new public and lexicon for transformation of collaborative pedagogies for all where creative arts and science education meet.
name: Teresa Chan
institution: McMaster University
Teresa Chan (she/her) is an Associate Professor in the Department of Medicine within the Division of Emergency Medicine and cross-appointed to the Division of Education & Innovation. She is an associate member of the Department of Health research methods, Evidence, and Impact. She is also the Associate Dean, Continuing Professional Development Office (CPD) and a Clinician Scientist, McMaster Education Research, Innovation & Theory (MERIT) Program, Faculty of Health Sciences, McMaster University.
