Naturalistic Decision Making

Date: 5/22/2020 Show Description: “Jan Maarten is Principal Scientist at TNO and Professor of Applied Cognitive Psychology at the University of Twente, The Netherlands. His research interests include resilience engineering, team communication processes, and human-machine teaming. He is the lead editor on two influential volumes: Cognitive Task Analysis (2000) and Naturalistic Decision Making and Macrocognition(2008). He is co-editor of the recently released Oxford Handbook of Expertise (2020). He is editor in chief of the Journal of Cognitive Engineering and Decision Making. Dr. Schraagen holds a PhD in Cognitive Psychology from the University of Amsterdam, The Netherlands.” Where to find Jan Maarten: University of Twente TNO LinkedIn JCEDM The Oxford Handbook of Expertise Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: What is the first paper you ever published? [2:00] Experience conducting research with children [4:11] Nature of Jan Maarten's 1993 research study on expertise [6:15] Working with Herb Simon at Carnegie Mellon [10:57] What led Jan Maarten to attend the 1994 NDM conference in Dayton [13:00] Discussing the birth and significance of the book "Cognitive Task Analysis" [17:25] Experience and advice for navigating difficult research interviews [20:35] Interviewing technique advice for students [23:52] The most exciting project Jan Maarten is working on right now [27:47] Goals and directions for future work [36:50] What is a "cyber-physical system"? [40:20] Industries that could benefit from an NDM perspective [42:30] Which project has been the most rewarding for you? [47:50] Evaluating the success of Dutch navy training simulations [54:50] One questions that can reveal if someone is an NDM practitioner [57:06] Two truths and a lie [58:30]

Date: 5/15/2020 Show Description: “Colonel Shattuck is Senior Lecturer and Director of the Human Systems Integration Program at the Naval Postgraduate School in Monterey, California. Dr. Shattuck holds a faculty appointment in the Operations Research Department where he teaches human systems integration and human factors engineering. Colonel Shattuck graduated from the United States Military Academy in 1976, and following distinguished service as a signal officer and receiving his PhD from the Ohio State University in Cognitive Engineering, he returned to his alma mater to teach Engineering Psychology until his retirement from the Army in 2005. He has been at NPS since, he co-directs the Human Systems Integration Program. Larry has been an active researcher in the domain of military command and control for nearly two decades. He is unique among his peers in the NDM community for his ability to draw on deep operational experience when studying processes such as how commanders communicate their intent, the manner in which decision makers fuse data in tactical environments, and the ways in which data and information flow through technological system elements to the human agents in that system.” Where to find Larry: Naval Postgraduate School HSI DL Certificate Program HSI DL Master’s Degree Program Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: How Larry's military operational experience has informed his research findings [2:20] Making the shift to engineering psychology [7:47] Takeaways from the Army research laboratory's advanced decision architecture program [10:12] Examples of the program's contributions to the Army [13:58] Making progress with NDM solutions within a government context [19:10] Explaining the relationship between NDM and Human Systems Integration [29:07] Making the NDM research process more efficient [34:40] How NDM perspectives have influenced Larry's approach to teaching [38:26] The aspect of Larry's work that has been most rewarding [40:10] Lessons learned from students [43:45] Larry's top three mentors [45:46] Influences outside of NDM [49:25] One question that can determine if someone is an NDM practitioner [51:07] Two truths and a lie [52:08]

Date: 5/15/2020 Show Description: “Today we welcome Julie Gore. Julie Gore is a Reader in Organizational Psychology, at the School of Management, University of Bath in the UK. She is a Chartered Psychologist and Fellow of the British Psychological Society. Her research focus is on the psychology of expertise and Naturalistic Decision Making (NDM) across a range of professions working under uncertainty. She is Associate Editor for Journal of Occupational and Organizational Psychology and serves on the boards of the British Journal of Management and Frontiers in Organizational Psychology. Julie is also an academic advisor for Nasdaq’s Behavioural Science Lab. Dr Gore received one of the world’s first NDM PhD degrees in Applied Cognitive Psychology from Oxford Brookes University, UK.” Where to find Julie’s work: The Oxford Handbook of Expertise  The University of Bath  Naturalistic Decision Making and Uncertainty Naturalistic decision making: navigating uncertainty in complex sociotechnical work SPRITE+  Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Experience as one of the first NDM PhD program participants [1:46] How Julie initially heard about NDM [3:03] Julie's PhD research question [3:55] Discoveries made during PhD research [5:18] Advice for young NDM PhD students [6:15] Common barriers students need help managing [8:02] Surprises and concerns about today's generation of NDM researchers [10:10] Examples of PhD students whose perspectives have been transformed by NDM models [12:05] Making space for NDM research papers in journals and publications [14:52] Nature of Julie's work with NASDAQ [16:32] Exciting research on decision-making ethics and accountability in the digital space [19:23] Organizations' and outsiders’ attitudes toward NDM [23:17] Showcasing the utility of NDM methods [25:18] A particularly rewarding project [27:10] An explanation of "adaptive expertise" [29:12] Opportunities to advance NDM in the context of today's novel circumstances [32:37] Three people who influenced Julie's career [33:47] Influences from outside NDM [34:43] Changes in the NDM community [36:35] One question that can determine if someone is an NDM researcher [37:30] Directions for future research [39:37] Two truths and a lie [41:02]

Robert Hutton, co-founder of Trimetis and a cognitive systems engineering expert, dives into the challenges of establishing his consultancy in England, often working with the Ministry of Defence. He shares fascinating stories from intense field research in places like Iraq and Lebanon, highlighting interactions with local security forces. Emphasizing the importance of bridging theory with practice, Hutton also discusses innovative teaching methods that connect psychology to real-world applications while tackling the complexities of cognitive task analysis.

Date: 5/1/2020 Show Description: “Today we welcome our pal, Jennifer Phillips. Jenni is the CEO of the Cognitive Performance Group, which she cofounded in 2006. Her work focuses expert decision-making, primarily in the military training community. She and her colleagues at CPG have pioneered research into the development of skilled performance, and designed innovation in the areas of decision-centered training and assessment, including the development and application of mastery models. Among her many publications is a chapter in the Oxford Handbook of Expertise exploring mastery models and their value in supporting the acquisition and assessment of expertise.” Learn More About Jenni: Cognitive Performance Group Developing Mastery Models to Support the Acquisition and Assessment of Expertise Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Jenny describes her focus of expertise, including tactical thinking in the marine corps and mastery models [1:30] Jenni breaks down the term “mastery model” and its stages [3:45] Jenni gives a mastery model example from her work with marine corps instructors [6:05] Work on assessment of decision skills, performance rubrics, and the process of adaptability [9:50] Major influences in Jenni’s career in NDM [14:05] Other people and organizations outside the NDM community who have impacted Jenni’s work [16:30] What makes Jenni’s approach and expertise unique in her field [19:15] Challenges and nuances of working in a military environment [21:30] Difficulty getting buy-in to conduct research in law enforcement [23:45] A favorite anecdote from research in IED defeat work [24:45] Some distinct and difficult nuances of IED defeat work in Afghanistan and Iraq  [29:45] Meaning and application of “cognitive fidelity” [32:50] A surprising characteristic of young researchers [35:20] One single question that can determine if someone is an NDM practitioner [39:05] Two truths and a lie [39:45] Excitement about research on making mastery models work at a large scale [42:10]

Date: 4/24/2020 Show Description: “Today we are talking with our friend and colleague, Emilie Roth. I first met Emilie at the second Naturalistic Decision Making meeting in Dayton, OH in 1994, but she had been studying how people manage complexity long before that meeting. In fact, I remember at that meeting thinking: ‘I want to be like her.’ Emilie is one of the people that inspired me to think I really could be a scientist. And I know Emilie has inspired many others over the course of her career. She is one of the architects of the cognitive systems engineering movement. She had an important role in redesigning nuclear power plant control rooms in the wake of Three Mile Island. And that was just the beginning; her contributions have been felt far and wide. She now lives in Palo Alto, CA where she runs Roth Cognitive Engineering.” Where to find Emily: Roth Cognitive Engineering Resources: Oxford Handbook of Expertise Designing collaborative planning systems: Putting Joint Cognitive Systems Principles to Practice Cognitive work analysis Facets of complexity in situated work Learn more about NDM: NaturalisticDecisionMaking.org Journal of Cognitive Engineering and Decision Making Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Timestamps: Emily's first published paper [1:30] Events that shifted Emilie’s focus from experimental psychology to NDM [7:30] Developing the observation tools and methods for developing systems of expertise [12:25] Challenges in creating expert decision-making models for nuclear operations at Westinghouse [17:15] Experience navigating a world of engineers as a female psychologist [20:00] The benefits of work culture at Westinghouse [21:45] Observations of railroad industry culture, practices, and informal procedures [23:00] Optimistic outlook on specific research and applications of NDM [28:45] Two things about Emilie’s personal life that others might not know [34:20] Three books that influenced Emilie’s work [38:00] Associative thinking exercise [44:15] Elaborating on automation pros and cons [45:25]

Robert Hoffman, a luminary in the NDM community, talks about his early career, naturalistic decision making, troubling experiences in research, the importance of education in psychology and computer science, love for professional football, and applying cognitive task analysis in sports and policing.

Show Description: “Today we are talking with my friend and business partner, Gary Klein. Gary is one of the pioneers of the Naturalistic Decision Making movement. He is known for moving the study of decision making out of the laboratory and into the wild. In 1989, he invited a small group of international researchers to meet in Dayton, OH to talk about decision making in natural environments and how to study this phenomenon. It was that group of 32 researchers who came up with the term Naturalistic Decision Making. Gary's many contributions include the recognition-primed decision model that has greatly influenced how we all think about decision making and expertise. He and his colleagues developed early cognitive task analysis methods that have been a core set of tools for studying decision making in real world environments. He has mentored countless researchers, including both of us. Gary now lives in Boston, MA. He runs a company called ShadowBox LLC, and is also part-owner and advisor to Applied Decision Science, LLC.” Where to find Gary ShadowBox LLC Website LinkedIn Psychology Today Column Learn more about NDM: Website: https://naturalisticdecisionmaking.org/ Where to find hosts Brian Moon and Laura Militello: Brian’s website Brian’s LinkedIn Brian’s Twitter Laura’s website Laura’s LinkedIn Laura’s Twitter Resources mentioned during the show: Sources of Power by Gary Klein Seeing What Others Don't: The Remarkable Ways We Gain Insights by Gary Klein Timestamps: Gary's first published paper [1:36] How Gary got started in Naturalistic Decision Making [4:48] Gary's research on Israeli aircraft maintainers' coping methods during Yom Kippur War [7:55] Current NDM-related research that Gary is most excited about [12:10] Two of the biggest mistakes of Gary’s career and how he recovered from them [14:53] Gary shares his favorite research stores: -A firefighter whose quick thinking saved several of his colleague’s lives [22:53] -A British naval officer whose quick, life-saving decision was at first mistaken for ESP [28:00] -A neonatal nurse's expert instincts helped save a baby's life [33:33] Suggestions for how the NDM community can help during today's coronavirus crisis 37:15] Where Gary would prioritize funding and research investment right now [44:05] Associative thinking exercise [46:18] How the scientific method is “often a trap” that can limit researchers' learning opportunities [48:55]