Interactive Individualized Instruction: A Guide to Best Practices from the Council of Emergency Medicine Residency Directors

Molly Estes, MD; Puja Gopal, MD; Jeffrey N. Siegelman, MD; John Bailitz, MD; Michael Gottlieb, MD 


Over the last several years, there has been increasing interest in transitioning a portion of residency education from traditional, lecture-based format to more learner-centered asynchronous opportunities. These asynchronous learning activities were renamed in 2012 by the Accreditation Council of Graduate Medical Education (ACGME) as Individualized Interactive Instruction (III). The effectiveness and applicability of III in residency education has been proven by multiple studies, and its routine use has been made officially acceptable as per the ACGME. This article provides a review of the current literature on the implementation and utilization of III in emergency medicine residency education. It provides examples of currently implemented and studied III curricula, identifies those III learning modalities which can be considered best practice, and provides suggestions for program directors to consider when choosing how to incorporate III into their residency teaching. 


One of the most recent trends in medical education is the transition from traditional didactics (ie, lecture-based classroom teaching) to online learning modules, collectively referred to as asynchronous learning. Over the last several years, asynchronous learning has been shown to be a successful learning style for many learners. For example, Liu and colleagues performed a meta-analysis of what the authors termed “blended learning” (ie, the combination of traditional teaching methods with asynchronous learning) throughout all health professional learners. Their review found that blended learning consistently performed better than no intervention and that it did not perform inferiorly to traditional “non-blended” learning.1 A host of additional data exists, demonstrating that learners prefer smaller learning environments2 and that these methods can address the challenge of teaching physician self-assessment and fostering the practice of lifelong learning.3 

It can be seen that there is a broad range of the use of asynchronous learning across the field of medicine. Looking specifically at resident training, a survey of internal medicine program directors revealed that out of the 214 responding programs, 71.5% used III sometimes, somewhat often, or very often.4 Examples of asynchronous learning curricula can be found in nearly every medical area and specialty, from a pediatrics gastroenterology subspecialty rotation,5 microsurgery competencies in plastic surgery,6 radiology residents receiving more real-time feedback on radiographic reads,7 to journal club for general surgery.8 There are examples for the training of fellows9 and faculty.10 There are even examples of all learners, laypeople and medical professionals, participating in a basic life support class11 and for interprofessional learners from all levels of training and fields participating in teamwork training.12 

The early 2000s to 2010s saw a unique challenge to residency programs specifically as Free Open Access Medical Education (FOAM) resources saw an exponential rise.13 Anecdotal evidence at that time suggested that residents were utilizing these resources for their own asynchronous education, with or without residency program oversight. Programs were faced with the decision to either begin vetting and incorporating these resources into their curricula or to maintain a more traditional didactic approach. Questions were raised whether time spent in asynchronous learning could even be counted as part of ACGME required didactic time. 

In 2008, The Council of Emergency Medicine Residency Directors (CORD) in conjunction with a task force from the Residency Review Committee for Emergency Medicine (RRC-EM) set out to critically evaluate the Accreditation Council of Graduate Medical Education (ACGME) Emergency Medicine (EM) Program Requirements specifically pertaining to educational conferences.  One of the suggestions to come from that task force was for residency programs to actively consider incorporating asynchronous learning as an educational tool.3 Not long after the task force’s recommendations, the RRC-EM published criteria allowing up to twenty percent of conference didactic time to be spent in asynchronous learning, which was renamed as Individualized Interactive Instruction (III).14 A subsequent publication from the same group further defined specific requirements of a valid III program (see section on “Cautions of Implementation”).15 

Since then, there has been increasing research into how and which aspects of EM residency teaching can be transitioned to III.16 Some programs have applauded it as the way of the future,17-18 while others have advised caution in implementation.19-20 Multiple ideas have been published on how to incorporate III such as flipped classroom,21 journal article discussion boards,22 or a series of varied online learning tasks.23 Comprehensive databases have emerged offering vetted sources, centralized information, and access to experts.24 

Surveys have shown extensive utilization of III among residents,25 as well as extensive incorporation into EM training programs.26-27 A survey by Waxman and colleagues in 2014 showed that 63% of programs were incorporating III into residency training; however, they noted there were significant variations in the structure of the curricula. Of the 37% who were not using III, 71% had concerns related to the understanding and implementation of III within the ACGME/RRC-EM criteria.26 The purpose of this article is to provide a review of the current literature on III and best practices recommendations for programs to consider as they refine their already existing III curricula or implement a curriculum for the first time. 


This article is the second in a series of best practice reviews from the CORD Best Practices Subcommittee. The first three authors performed a search of PubMed for articles published from inception to March 31, 2018 using the same keywords “asynchronous learning” and “interactive individualized instruction.” Bibliographies of all relevant articles were reviewed for additional studies. Articles were screened by the search authors to evaluate for any articles addressing the specific topics of implementation and utilization of III curricula within the field of emergency medicine. 

The search yielded a total of 664 articles, of which 19 articles were deemed to be directly relevant to the field of emergency medicine and for inclusion in this review. When supporting data was not available, recommendations were made based upon the authors’ combined experience and consensus opinion.  Prior to submission, the manuscript was reviewed by the entire CORD Best Practices Subcommittee. 


In 2015, the CORD III Task Force performed an updated survey of program directors of ACGME-accredited emergency medicine residency programs on their current use of III (unpublished data).  Of the 77 unique programs which responded (approximately 46% response rate), 74% reported incorporating III into their programs.  More four-year format programs utilized III than 3-year programs (91% compared to 67%).  Programs implementing III were divided among those who offered either four or five hours of synchronized didactics weekly, or some variation thereof.  Of those who reported not using III, the most cited rationale was an unclear definition of what constituted III. Other programs were concerned about compliance or the resources required for implementation.  Offerings for III credit were quite diverse.  Many programs offered online learning modules, FOAM resources, and board review sessions for III credit.  Some utilized simulation, journal club, and attendance at national or regional meetings.  This survey shows that although there is a high rate of utilization of III amongst programs, there still remains a wide variation in qualifying activities.

While there is a significant amount of literature on the importance and acceptance of III as a learning tool, there is no standard or consensus method of implementation in emergency medicine. In addition, there is a dearth of information (only the single survey as described above) in the published literature as to how individual EM residency programs specifically implement III and significant variation amongst programs based on qualitative preliminary surveys. Some research even suggests that III may not be an adequate replacement for all of the didactics in a traditional curriculum, specifically for novice learners, concerns namely being their ability to identify specific knowledge gaps and their need to have adequate expert oversight to ensure true knowledge acquisition and retention.20 Several publications in recent years highlight examples of how EM residency programs nationwide have and are utilizing III; some selected examples are discussed below.

Wray and colleagues implemented an III curriculum in 2013 and measured the effect on in-training exam (ITE) scores. There were four modules per month created by faculty and chief residents, each designed to be completed in less than 1 hour. Educational content included journal articles, audio and video lectures, podcasts, links to FOAMed resources, and modules linked to quizzes. Residents were required to complete these modules and their progress was monitored in addition to ITE exam scores. The group found that despite the decrease in traditional conference hours, time now allotted to III, there was no negative impact on resident ITE scores.28   

Pensa and colleagues created a digital course for residents in 2014 and surveyed residents to assess satisfaction. The program educational material was curated by faculty from various FOAMed/digital resources and participation was optional. The modules included an assignment page with the content; a discussion page, which was a mandatory component of the module, and allowed for learners to post queries and for faculty members to answer questions; and a multiple choice quiz page for assessment. Thirty three of 48 residents participated in the survey in the first year and appeared to overall find the course useful, although there were significant variations in time spent participating in the course both among residents as well as faculty. The biggest barrier to participation residents identified was lack of time.29    

Kornegay and colleagues developed an III curriculum implemented during the 2011-2012 academic year. Faculty members identified gaps in the pre-existing synchronous curriculum and topics better suited for independent learning then developed a web-based platform consisting of curated content and an evaluation component, namely a reflective writing assignment or quiz. Of responding residents, about eighty percent were satisfied, very satisfied or extremely satisfied with the new modality. The group also analyzed conference attendance and ITE scores and found that PGY-1 resident attendance rate significantly improved from the prior year (85% vs 62% mean) though other curricular changes in the program (e.g. small group-based learning, interactive case-based conferences and changes in off-service rotations) may have also enhanced participation. There was no statistically significant difference in mean ITE scores pre- and post-intervention. Faculty reported a time commitment of about 4-8 hours per month, which was comparable to the time spent to prepare 1 hour of instruction for weekly conference pre-intervention.30 

Kothari and colleagues designed an III curriculum based on the Academic in Life Emergency Medicine (ALiEM) popular Approved Instructional Resources (AIR) series. The AIR series curates FOAM content from the top 50 open-access EM and critical care blog and podcast sites, provides associated core teaching points and multiple-choice questions for residents, and tracks resident participation to provide residency program directors with resident progress.24 Kothari and colleagues then implemented a second component to their III curriculum, which consisted of two high-impact journal articles selected by faculty on a monthly basis. The group found that introduction of the III did not negatively affect residency educational conference; attendance across all PGY levels was comparable to the year before.31 

Other innovative strategies and formats to implement III in EM have been centered upon discrete, focused topic areas within the larger EM curriculum, such as pediatrics,32-33 palliative and end-of-life care,34 and disaster medicine35 to mention a few examples. Commonalities exist amongst these examples, namely facilitators’ deliberate choosing of either a specific asynchronous learning program or a specific topic to be taught using asynchronous learning depending on their program’s needs. 


  1. III should be used cautiously with the novice learner.
  2. When deciding to develop or implement an III curriculum, first identify gaps in the current curriculum or those topics which may be best transitioned to an III format.  This is likely to vary between programs.
  3. A combination of available III (e.g. online blogs + podcasts + journal articles) seems to attract a greater number of residents to participate, likely as this variety addresses a broader span of individual learning preferences.
  4. Transition to III does not seem to negatively affect resident ITE scores or weekly conference attendance rates. 


The ACGME policy statement on the use of III within emergency medicine residency education is very strict as to the criteria that must be met in order for an activity to be considered III. Given that up to 20% (one out of every five hours) of previously considered core curriculum time can now be spent as III, there may be a natural inclination amongst programs to begin to cut back on planned traditional educational activities. This is a fallacy, and there exist several ways that implementation of what might be considered III can go wrong (see Table 1).15 Below are listed some common pitfalls encountered when implementing III. 

Table 1: ACGME Criteria for III15

1. The Program Director must monitor resident participation.
 2. There must be an evaluation component.
 3. There must be faculty oversight.
 4. The activity must be monitored for effectiveness.

 Independent reading and use of question banks: The ACGME places particular emphasis on any potential III being a planned activity that is tailored for the individual’s level of learning.  Resident-directed reading is not considered to be a planned activity. Additionally, independent use of a question bank is not directed to the individual’s particular needs, even if the astute resident is choosing specific topics to review.  Faculty may perhaps choose a specific reading or set of questions to include as a part of III, but these by themselves do not qualify.

Resident attestations of completion: An attestation of completion of an III activity is not considered to be adequate enough to prove resident participation. There must be a separate, tangible source of evaluation. Tracking quiz completion/participation after an online module or required reading would provide ample proof of activity completion just as a sign-in sheet before a simulation does the same.

Audio, video, or podcasts: These learning methods are considered to be passive learning, and use of them alone does not qualify as III. However, they can be combined with other learning modalities, such as a particular question set from an online question bank, to include an active component.

Monitoring for effectiveness: At the time of implementation of the chosen curricula, program directors must have a plan for how they will go about tracking the effectiveness of the III program. This can take many different forms: use of periodic review quizzes, objective clinical performance, test scores on the in-training exam, etc. However, this type of evaluation must be planned over several generations of residents to account for individual class variation and ensure the III program itself is not causing knowledge gaps. Regular check-ins with residents to additionally ensure their continued perspective of the curricula as beneficial are also recommended.


  1. Before designing or implementing an III curriculum, carefully review the ACGME criteria to ensure compliance.
  2. Resident driven use of question banks not not meet III criteria.
  3. An attestation of completion does not meet III criteria for participation.
  4. Use of passive learning methods alone (e.g. podcasts) does not meet III criteria.
  5. Regular curriculum assessment is essential to ensure adequate instructional merit and continued benefit to resident learning.


Several best practices have emerged from surveying EM program directors implementing III, both with respect to high quality, effective educational programming as well as compliance with RRC regulations.27

Simulation: Simulation activities easily satisfy the requirements of III. They can provide an individual resident the opportunity for self-directed work on a particular area of improvement with direct faculty supervision and immediate feedback. These work best when a resident identifies a particular case, topic, or procedure on which he or she would like to focus. 

Online Resources: A wealth of freely accessible material is available for III learning via podcasts, blogs, and online modules.  Program directors need to creatively consider how they will allow for the use of such material for III while maintain compliance with RRC regulations. Additionally, care must be taken to appropriately vet all resources to ensure credibility and academic rigor.36-37 Perhaps the most widely adopted single resource is the ALIEM-AIR Series,24 which as of that group’s publication in 2016 had been implemented in 65 programs. This group rigorously selects the highest quality online resources, as judged by EM faculty, provides a quiz for an evaluative component, and allows for online discussion. Individual program directors are able to monitor both the modules as well as their residents’ participation. Other best practices include discussion sessions with a faculty lead about a particular podcast or blog post. 

National/Regional Conferences: Attendance at specialty society meetings offers many learning opportunities.  To rise to the level of III, and to meet the criteria set forth by the RRC, programs have instituted a number of policies for such activities. Monitoring participation and faculty oversight are key areas of concern, and can be addressed by checking in with faculty who are also attending or presenting at a particular session. Some programs require discussion or written assignments following the session or conference.

Question Banks: Multiple question banks are available online and in print for residents’ use in preparing for standardized tests. While answering questions alone does not meet criteria for III (see “Cautions of Implementation” above), reviewing specific questions missed or themes with a faculty member certainly would be acceptable. 

Other Opportunities: Multiple other activities are in use in emergency medicine programs for III including journal clubs, research and teaching activities, oral boards practice, and many others.


  1. When designing an III curriculum, many options for learning activities are available to be included: simulation, online resources, national/regional conferences, question banks with faculty oversight, etc.
  2. When choosing online sources, care must be taken to ensure credibility and academic rigor; scoring methods exist and can be used to assess this.


While all attempts have been made to create an inclusive review of the current use of III in emergency medicine residency education, limitations must be acknowledged. In the identification of pertinent articles for inclusion, although multiple search terms were used and bibliographies cross-referenced, it is possible that some articles may not have been identified by the current review. Articles were chosen based on their primary relevance to the field of emergency medicine, not as an expansive review of the history of the use of III or its current use in other medical fields or specialties. The primary limitation to this data analysis is the relative paucity of data available on the direct implementation or utilization of full III curricula within emergency medicine residency programs. Multiple sources have supplied information pertaining to the use of specific topic-based curricula, but few show analysis of a more extensive use of III as might pertain to what can be considered a core curriculum. 


This article provides a review of the literature currently available on the implementation and utilization of III in emergency medicine residency education. It can be said conclusively that III has been proven to be an accepted part of modern residency education. Preliminary data suggests III may very well augment resident learning without negatively affecting standardized testing scores or resident participation in other traditional didactics. Care must be given to choose the appropriate learning level of the resident and ensure ACGME compliance with curricular activities. However, despite multiple sources of curricula options, there remains a paucity of information as to the effectiveness of specific III as pertaining to resident knowledge acquisition and retention.  More research is needed into this area to further refine what we determine to be gold standard III modalities.  Until then, it is the authors’ intention that readers will be more aware of the ACGME guidelines and the III options that exist in order to avoid the potential pitfalls of implementation at their home institutions.


  1. Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. The Effectiveness of Blended Learning in Health Professions: Systematic Review and Meta-Analysis. J Med Internet Res. 2016 Jan 4;18(1):e2.
  2. Graffam B. Active learning in medical education: strategies for beginning implementation. Med Teach. 2007;29:38-42.
  3. Sadosty, A. T., Goyal, D. G., Gene Hern, H., Kilian, B. J. and Beeson, M. S. (2009), Alternatives to the Conference Status Quo: Summary Recommendations from the 2008 CORD Academic Assembly Conference Alternatives Workgroup. Academic Emergency Medicine, 16: S25–S31.
  4. Wittich CM, Agrawal A, Cook DA, Halvorsen AJ, Mandrekar JN, Chaudhry S, Dupras DM, Oxentenko AS, Beckman TJ. E-learning in graduate medical education: survey of residency program directors. BMC Med Educ. 2017 Jul 11;17(1):114.
  5. Feist M, Ciccarelli M, McFerron BA, Molleston JP. Methods and effects of a case-based pediatric gastroenterology online curriculum. J Pediatr Gastroenterol Nutr. 2013 Feb;56(2):161-5.
  6. Satterwhite T, Son J, Carey J, Zeidler K, Bari S, Gurtner G, Chang J, Lee GK. Microsurgery education in residency training: validating an online curriculum. Ann Plast Surg. 2012 Apr;68(4):410-4.
  7. Choi HH, Clark J, Jay AK, Filice RW. Minimizing Barriers in Learning for On-Call Radiology Residents-End-to-End Web-Based Resident Feedback System. J Digit Imaging. 2017 Aug 24.
  8. Hammond J, Whalen T. The electronic journal club: an asynchronous problem-based learning technique within work-hour constraints. Curr Surg. 2006 Nov-Dec;63(6):441-3.
  9. Inra JA, Pelletier S, Kumar NL, Barnes EL, Shields HM. An active learning curriculum improves fellows' knowledge and faculty teaching skills. Adv Med Educ Pract. 2017 May 26;8:359-364.
  10. Bucklen KA, Carlson DW, Shah N, Pruitt C. Development of a pediatric hospitalist curriculum to promote faculty development, teaching excellence, and evidence-based care. Hosp Pediatr. 2014 Nov;4(6):387-92.
  11. Hsieh MJ, Bhanji F, Chiang WC, Yang CW, Chien KL, Ma MH. Comparing the effect of self-instruction with that of traditional instruction in basic life support courses-A systematic review. Resuscitation. 2016 Nov;108:8-19.
  12. Umoren RA, Poore JA, Sweigart L, Rybas N, Gossett E, Johnson M, Allen M, Scott PJ, Truman B, Das R. TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners. Creat Nurs. 2017 Aug 1;23(3):184-191.
  13. Cadogan M, Thoma B, Chan TM, Lin M. Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013). Emerg Med J. 2014;31:76–77.
  14. Accreditation Council for Graduate Medicine Education. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. Sept 30, 2012. Updated July 1, 2017. Accessed Mar 1, 2018.
  15. Review Committee for Emergency Medicine. Frequently Asked Questions: Emergency Medicine. Updated Oct 2017. Accessed Mar 1, 2018.
  16. Toohey SL, Wray A, Wiechmann W, Lin M, Boysen-Osborn M. Ten tips for engaging the millennial learner and moving an emergency medicine residency curriculum into the 21st century. West J Emerg Med. 2016;17:337–343.
  17. Reiter DA, Lakoff DJ, Trueger NS, Shah KH. Individualized interactive instruction: an innovative enhancement to resident education. Ann Emerg Med. 2013 Jan;61(1):110-3.
  18. Pourmand A, Lucas R, Nouraie M. Asynchronous web-based learning, a practical method to enhance teaching in emergency medicine. Telemed J E Health. 2013 Mar;19(3):169-72.
  19. Brydges R, Dubrowski A, Regehr G. A new concept of unsupervised learning: directed self-guided learning in the health professions. Acad Med. 2010;85(10 Suppl):S49–S55.
  20. Jordan J, Jalali A, Clarke S, Dyne P, Spector T, Coates W. Asynchronous vs didactic education: it’s too early to throw in the towel on tradition. BMC Medical Education. 2013;13:105.
  21. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014 Nov;15(7):938-44.
  22. Stoneking LR, Grall KH, Min AA, Panchal AR. Online research article discussion board to increase knowledge translation during emergency medicine residency. Adv Med Educ Pract. 2013 Jan 29;4:17-21.
  23. Ashton A, Bhati R. The use of an asynchronous learning network for senior house officers in emergency medicine. Emerg Med J. 2007 Jun;24(6):427-8.
  24. Lin M, Joshi N, Grock A, et al. Approved instructional resources series: A national initiative to identify quality emergency medicine blog and podcast content for resident education. J Grad Med Educ. 2016;8(2):219-225.
  25. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M, Fix M. A survey of the current utilization of asynchronous education among emergency medicine residents in the United States. Acad Med. 2014 Apr;89(4):598-601.
  26. Waxman M, Jiang J, Sawtelle S. Utilization of Independent Individualized Instruction (III) in United States Emergency Medicine Residency Programs: Results of the Council of Emergency Medicine Residency Directors (CORD-EM) III Task Force Survey. West J Emerg Med. 2014;15(5.1).
  27. Council of Residency Directors Individualized Interactive Instruction Task Force. Best Practices in Individualized Interactive Instruction. 2015.
  28. Wray A, Bennett K, Boysen-Osborn M. Efficacy of an asynchronous electronic curriculum in emergency medicine education in the United States. J Educ Eval Health Prof. 2017 Dec;14:29.
  29. Pensa G, Smith J, McAteer K. Calling all curators: a novel approach to individualized interactive instruction. West J Emerg Med. 2018 Jan;19(1):169-171.
  30. Kornegay J, Leone K, Wallner C, et al. Development and implementation of an asynchronous emergency medicine residency curriculum using a web-based platform. Intern Emerg Med. 2016;11:1115-1120.
  31. Kothari A, Breaud A, Manasco A, et al. Resident attendance at weekly conferences after implementation of an optional asynchronous learning curriculum. Intern J of Med Edu. 2017;8:163-164.
  32. Burnette K, Ramundo M, Stevenson M, et al. Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students. Acad Emerg Med. 2009;16.
  33. Chang TP, Pham PK, Sobolewski B, et al. Pediatric emergency medicine asynchronous e-learning: a multicenter randomized controlled Solomon four-group study. Acad Emerg Med. 2014;21(8):912-9.
  34. Gisondi MA, Lu DW, Yen M, et al. Adaptation of EPEC-EM Curriculum in a Residency with Asynchronous Learning. West J Emerg Med. 2010;11(5):491-9.
  35. Lund A, Lam K, Parks P. Disaster Medicine Online: evaluation of an online, modular, interactive, asynchronous curriculum. CJEM. 2002;4(6):408-13.
  36. Thoma B, Sanders JL, Lin M, Paterson QS, Steeg J, Chan TM. The social media index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16:242–9.
  37. Grock A, Paolo W. Free open access medical education: a critical appraisal of techniques for quality assessment and content discovery. Clin Exp Emerg Med. 2016 Sep; 3(3): 183–185.
Comment Form is loading comments...