AI in Education Podcast

Dan Ingvarson, a prominent figure in AI education, delves into how generative AI can revolutionize assessment practices, offering personalized evaluations for students. The discussion covers challenges teachers face, the potential of AI in creating effective rubrics, and the importance of evolving curricula to incorporate advanced AI methodologies for enhanced student learning and teacher workload.

Content warning! This episode talks about an academic research paper titled "ChatGPT is bulls**t", and we've not edited the word out - in fact, we've gone to town with it, talking about the different types of it (in the strictest academic sense). So you may not want to play this in the car on your school run! The news item discussed is: Student crafts elaborate AI scheme to pass university exam, gets arrested https://cybernews.com/news/turkish-student-found-using-ai-arrested/ This week's papers discussed are: Developing evaluative judgement for a time of generative artificial intelligence https://www.tandfonline.com/doi/full/10.1080/02602938.2024.2335321 Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization https://arxiv.org/abs/2404.04941 Working Alongside, Not Against, AI Writing Tools in the Composition Classroom: a Dialectical Retrospective https://uen.pressbooks.pub/teachingandgenerativeai/chapter/working-alongside-not-against-ai-writing-tools-in-the-composition-classroom-a-dialectical-retrospective/ GPT versus Resident Physicians — A Benchmark Based on Official Board Scores https://ai.nejm.org/doi/pdf/10.1056/AIdbp2300192 Evaluating General Vision-Language Models for Clinical Medicine https://www.medrxiv.org/content/10.1101/2024.04.12.24305744v1 Re-evaluating GPT-4's bar exam performance https://link.springer.com/article/10.1007/s10506-024-09396-9 Automated Social Science: Language Models as Scientist and Subjects https://arxiv.org/abs/2404.11794 Large language models cannot replace human participants because they cannot portray identity groups https://arxiv.org/abs/2402.01908 I also mentioned the article about the the Infinite Focus Group, which is here: https://www.ai-mindset.ai/ai-mindset-newsletter/the-infinite-focus-group The impact of large language models on university students' literacy development https://www.tandfonline.com/doi/epdf/10.1080/07294360.2024.2332259?needAccess=true Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays https://www.sciencedirect.com/science/article/pii/S2666920X24000109 Feedback sources in essay writing: peer-generated or AI-generated feedback? https://link.springer.com/article/10.1186/s41239-024-00455-4 ChatGPT is bullshit https://link.springer.com/epdf/10.1007/s10676-024-09775-5?sharing_token=0CIhP_zo5-plierRq8kkDPe4RwlQNchNByi7wbcMAY77xTOWyddkW01qGFs1m5zuuoZGBctVlsJF8SbYqcxWi-XzgEYEPiw7xwWi4bMYXJ_1JARDrER9JGdWZOW-UGSkrk_tXPjPh-XWvFNoiFzNlnDUUUEBAztiX9PtP2p6jfI%3D

Wow, this week we have a bumper episode with more resources than a GPT factory! Any time we get a guest making their second appearance, and therefore enter our Hall of Fame, then we officially dub them "Friend of the Show". And so this week, we've got Friend of the Show Leon Furze sharing his experiences and expertise. Here are the links and posts related to our conversation with Leon around assessment. Leon's blog - for all the updates and posts that he is working on https://leonfurze.com/blog Leons's free e-Book on assessment can be found here: https://mailchi.mp/leonfurze/assessment (free ebook on assessment) Leon Furze Linkedin profile is here if you want to follow his stream of thoughts, and to connect with him: Leon Furze - Furze Smith Consulting | LinkedIn The Artificial Intelligence Assessment Scale (AIAS) paper we discussed can be found here: https://open-publishing.org/journals/index.php/jutlp/article/view/810 The online course around practical AI strategies: https://practicalaistrategies.com/p/practical-ai-strategies And, even better, he's given listeners a discount code that will save you 25% of the cost. Just use the magic word 'AIPODCAST' The blog post we mentioned a couple of times during the episode: https://leonfurze.com/2024/05/27/dont-use-genai-to-grade-student-work/ Leons new book and course can be also found on his main site here: https://practicalaistrategies.com/

This week we set the episode timer for 15 minutes, and managed to get through just five papers before the buzzer went off! So we have plenty more papers to discuss in future episodes... ENHANCING K-12 STUDENTS' PERFORMANCE IN CHEMISTRY THROUGH CHATGPT-POWERED BLENDED LEARNING IN THE EDUCATION 4.0 ERA https://library.iated.org/view/ORTIZDEZARATE2024ENH Empowering student self-regulated learning and science education through ChatGPT: A pioneering pilot study https://bera-journals.onlinelibrary.wiley.com/doi/pdfdirect/10.1111/bjet.13454 ChatGPT "contamination": estimating the prevalence of LLMs in the scholarly literature https://arxiv.org/abs/2403.16887 Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews https://arxiv.org/abs/2403.07183 Large language models are able to downplay their cognitive abilities to fit the persona they simulate https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0298522

Assessment - Chris Goodall In this episode of the AI Education Podcast, host Dan converses with Chris Goodall, the head of digital education at the Bourne Education Trust in England. They discuss the integration of AI into education, how it can be used to enhance teaching and learning processes, and the impact of personalized AI tools on students and educators. The conversation covers practical applications of AI, the ongoing need for teacher and student adaptation to new technologies, as well as ethical considerations and future possibilities for AI in education. Chris Goodalls Linkedin profile here: Chris Goodall | LinkedIn Practical Advice for embedding IT in school: Embedding AI use in school

This podcast discusses the latest research in AI education, including topics like remote proctoring debates, chatbot support for international students, using chat GPT in veterinary medicine, and exploring the impact of generative AI on education. It also covers computing students' attitudes towards large language models, ethical considerations of AI in education, and the benefits of AI in enhancing creative learning in children.

This week we continue our series on Assessment and AI. Ray talks with Jason Lodge from The University of Queensland, and who must have the longest business card in Australia, as he's Associate Professor of Educational Psychology in School of Education and Deputy Associate Dean in the Faculty of Humanities, Arts and Social Sciences! The conversation talks about the challenges of assessment, and the options for rethinking assessment - and then we go deeper into Jason's views on the future of learning and assessment. Jason's a great guest to share his experiences, as during 2023 he was on the TEQSA group of experts that came together to produce a report on assessment for Australian universities, Assessment reform for the age of artificial intelligence https://www.teqsa.gov.au/guides-resources/resources/corporate-publications/assessment-reform-age-artificial-intelligence Working on policy and guidance in an area where technology is developing so rapidly - and students are racing ahead of institutions, was interesting and Jason talks about the group dynamic. One of the interesting notes he talks about is the mindset: "The mantra we kept returning to is that we weren't trying to develop a map, but a compass. This is the direction we think we might need to head here."

AI and the Future of Assessment: Transforming Educational Practices Episode Overview: In this episode of the AI Education Podcast, hosts Dan and Ray, alongside guests Adam Bridgman and Danny Liu, dive into the evolving landscape of academic assessment in the age of artificial intelligence. Recorded in the University of Sydney's own studios, this discussion explores the significant shifts in assessment strategies and the integration of AI in educational settings. Guest Introductions: Professor Adam Bridgeman: Pro Vice Chancellor Educational Innovation at the University of Sydney - focused on enhancing teaching quality across the university. [University bio] Professor Danny Liu: Professor of Educational Technologies - dedicated to empowering educators to improve their teaching methods through innovative technologies. [University page - LinkedIn page] Key Topics Discussed: The Persistence of Traditional Assessment Models: Despite the push to digital platforms during the COVID-19 pandemic, traditional assessment methods have largely remained unchanged, continuing the practice of replicating physical exam environments online. AI's Role in Rethinking Assessment: The guests discuss how AI challenges the conventional reasons for assessments, advocating for a paradigm shift towards assessments that truly measure student understanding and application of knowledge. Two-Lane Assessment Approach: Adam introduces a dual-lane strategy for assessment: Lane One: Ensures the rigorous verification of student competencies necessary in professional fields. Lane Two: Uses AI to foster skill development in using technology effectively, moving beyond traditional assessment forms to embrace innovative educational practices. Implementation Challenges and Solutions: The transition to new assessment models is recognised as a gradual process, needing careful planning and support for educators in rethinking their assessment strategies. Inclusivity and Access to Technology: Ensuring equitable access to AI tools for all students is highlighted as a critical aspect of the evolving educational landscape, emphasizing the need to support diverse student backgrounds and technological proficiencies. Future Outlook: The discussion concludes with reflections on the potential long-term impacts of AI on educational practices, the necessity of ongoing adaptation by educational institutions, and the importance of preparing students for a future where AI is seamlessly integrated into professional and everyday contexts. Further Reading: We recommend these three articles from the team, that give more detail on the topics discussed Where are we with generative AI as semester 1 starts? What to do about assessments if we can't out-design or out-run AI? Embracing the future of assessment at the University of Sydney

It's time to start a new series, so welcome to Series 8! This episode is the warm up into the series that's going to be focused on Assessment. We'll interview some fascinating people about what's happening in school and university assessment, how we might think differently about assessing students, and what you can be thinking about if you're a teacher. There's no shownotes, links or anything else for your homework for this episode - just listen and enjoy! Dan and Ray

The season-ending episode for Series 7, this is the fifteenth in the series that started on 1st November last year with the "Regeneration: Human Centred Educational AI" episode. And it's an unbelievable 87th episode for the podcast (which started in September 2019). When we come back with Series 8 after a short break for Easter, we're going to take a deeper dive into two specific use cases for AI in Education. The first we'll discuss is Assessment, where there's both a threat and opportunity created by AI. And the second topic is AI Tutors, where there's more of a focus on how we can take advantage of the technology to help improve support for learning for students. This episode looks at one key news announcement - the EU AI Act - and a dozen new research papers on AI in education. News EU AI Act https://www.europarl.europa.eu/news/en/press-room/20240308IPR19015/artificial-intelligence-act-meps-adopt-landmark-law The European Parliament approved the AI Act on 13 March and there's some stuff in here that would make good practice guidance. And if you're developing AI solutions for education, and there's a chance that one of your customers or users might be in the EU, then you're going to need to follow these laws (just like GDPR is an EU law, but effectively applies globally if you're actively offering a service to EU residents). The Act bans some uses of AI that threaten citizen's rights - such as social scoring and biometric identification at mass level (things like untargeted facial scanning of CCTV or internet content, emotion recognition in the workplace or schools, and AI built to manipulate human behaviour) - and for the rest it relies on regulation according to categories. High Risk AI systems have to be assessed before being deployed and throughout their lifecycle. In the High Risk AI category it includes critical infrastructure (like transport and energy), product safety, law enforcement, justice and democratic processes, employment decision making - and Education. So decision making using AI in education needs to do full risk assessments, maintain usage logs, be transparent and accurate - and ensure human oversight. Examples of decision making that would be covered would be things like exam scoring, student recruitment screening, or behaviour management. General generative AI - like chatgpt or co-pilots - will not be classified as high risk, but they'll still have obligations under the Act to do things like clear labelling for AI generated image, audio and video content ; make sure there's it can't generate illegal content, and also disclose what copyright data was used for training. But, although general AI may not be classified as high risk, if you then use that to build a high risk system - like an automated exam marker for end-of-school exams, then this will be covered under the high risk category. All of this is likely to become law by the middle of the year, and by the end of 2024 prohibited AI systems will be banned - and by mid-2025 the rules will start applying for other AI systems. ResearchAnother huge month. I spent the weekend reviewing a list of 350 new papers published in the first two weeks of March, on Large Language Models, ChatGPT etc, to find the ones that are really interesting for the podcast Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges arXiv:2401.08664 A Study on Large Language Models' Limitations in Multiple-Choice Question Answering arXiv:2401.07955 Dissecting Bias of ChatGPT in College Major Recommendations arXiv:2401.11699 Evaluating Large Language Models in Analysing Classroom Dialogue arXiv:2402.02380 The Future of AI in Education: 13 Things We Can Do to Minimize the Damage https://osf.io/preprints/edarxiv/372vr Scaling the Authoring of AutoTutors with Large Language Models https://arxiv.org/abs/2402.09216 Role-Playing Simulation Games using ChatGPT https://arxiv.org/abs/2402.09161 Economic and Financial Learning with Artificial Intelligence: A Mixed-Methods Study on ChatGPT https://arxiv.org/abs/2402.15278 A Study on the Vulnerability of Test Questions against ChatGPT-based Cheating https://arxiv.org/abs/2402.14881 Incorporating Artificial Intelligence Into Athletic Training Education: Developing Case-Based Scenarios Using ChatGPT https://meridian.allenpress.com/atej/article/19/1/42/498456 Incorporating Artificial Intelligence Into Athletic Training Education: Developing Case-Based Scenarios Using ChatGPT https://meridian.allenpress.com/atej/article/19/1/42/498456 RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education https://arxiv.org/abs/2403.08272 Comparison of the problem-solving performance of ChatGPT-3.5, ChatGPT-4, Bing Chat, and Bard for the Korean emergency medicine board examination question bank https://journals.lww.com/md-journal/fulltext/2024/03010/comparison_of_the_problem_solving_performance_of.48.aspx?context=latestarticles Comparing the quality of human and ChatGPT feedback of students' writing https://www.sciencedirect.com/science/article/pii/S0959475224000215