Instructional Impact Studies

Ever since ChatGPT arrived, there has been a wave of excitement, skepticism, and curiosity about whether - and how - it actually helps students. Now, a systematic review and meta-analysis by Deng et al. in Computers & Education has pulled together findings from 69 experimental studies, shedding new light on what ChatGPT means for teaching and learning. What Did the Research Reveal? 1️⃣ Stronger Academic Performance Studies show that ChatGPT-assisted interventions often lead to higher grades and better written work - especially in language-rich subjects. One caveat? Many experiments did not make it clear whether students were allowed to use ChatGPT during exams, raising questions about genuine mastery versus AI-assisted output. 2️⃣ Positive Motivation - But Mostly for College Learners University students typically felt more engaged and motivated. In K-12 settings, however, the motivational boost was not as pronounced - suggesting a need for age-appropriate strategies and scaffolds. 3️⃣ Perceived Gains in Higher-Order Thinking Learners reported enhanced creativity and critical thinking. The big “but”: most studies relied on self-reports, so future work needs objective assessments (e.g., problem-solving tasks or performance-based measures) to confirm actual skill growth. 4️⃣ Reduced Mental Effort, Uncertain Self-Efficacy ChatGPT may lighten cognitive load - learners felt tasks were less “taxing.” At the same time, studies showed a mixed or non-significant effect on self-efficacy, implying we need a deeper look at whether students gain real confidence or just convenience. What This Means for Educators & Academics? 1️⃣ Design Rich Assessments: To spot genuine skill gains, use project-based tasks that demand application and originality. 2️⃣ Spell Out Tech Policies: Clearly specify whether and how learners can use ChatGPT - especially for graded work. 3️⃣ Look for Long-Haul Impact: Do not just check excitement levels right after introducing ChatGPT; measure whether those positive vibes (and scores) persist weeks or months down the road. 4️⃣ Mind the Methods: If you are studying ChatGPT’s educational impact, conduct power analyses (to ensure you have enough participants) and randomize group assignments to get the most reliable data. This meta-analysis provides early - but promising - evidence that ChatGPT can enrich students’ learning experiences. The next step? Refining the methods, tracking long-term outcomes, and ensuring actual learning gains are assessed - not just AI’s ability to produce polished outputs. Reference: Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2024). Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers & Education, 105224. https://lnkd.in/eXe8agAT

Unpacking the impact of digital technologies in Education This report presents a literature review that analyses the impact of digital technologies in compulsory education. While EU policy recognizes the importance of digital technologies in enabling quality and inclusive education, robust evidence on the impact of these technologies is limited especially due to its dependency from the context of use. To address this challenge, the literature review presented here, analyses the focus, methodologies, and results of 92 papers. The report concludes by proposing an assessment framework that emphasizes self-reflection tools, as they are essential for promoting the digital transformation of schools. The literature review on the impact of digital technologies in education revealed several key findings: - Digital technologies influence various aspects of education, including teaching, learning, school operations, and communication. - Factors like digital competencies, teacher characteristics, infrastructure, and socioeconomic background influence the effectiveness of digital technologies. - The impact of digital tools on learning outcomes is context-dependent and influenced by multiple factors. - Existing evidence on the impact of digital tools in education lacks robustness and consistency. The assessment framework proposed in the report offers a structured approach to evaluating the effectiveness of digital technologies in education: 1. Identify contextual factors influencing technology impact. 2. Map stakeholders and their characteristics. 3. Assess integration into learning processes and practices. 4. Utilize self-reflection tools like the Theory of Change. 5. Provide evaluation criteria aligned with the framework. 6. Adapt existing tools for technology assessment. 7. Consider digital competence frameworks for organizational maturity. Implications and recommendations for policymakers and educators based on the report findings include: - Recognizing the contextual nature of technology use. - Focusing on creating rich learning environments. - Adopting a systems approach to studying technology impact. - Ensuring quality implementation and professional development. - Developing policies for monitoring and evaluation. - Encouraging further research on technology impact. By following these recommendations, stakeholders can leverage digital technologies effectively to improve teaching and learning outcomes in educational settings. https://lnkd.in/eBEN5XQg

How do you implement virtual reality effectively in an actual classroom? Many asked, we know the answer! 👇 In our latest article, published in Journal of Computer Assisted Learning we investigated this in a quasi-experimental design. 211 middle school students were distributed over three groups: 1) teacher-led instruction and discussion + VR, 2) VR + teacher-led instruction and discussion, 3) VR + VR (no teacher-led instruction and discussion, serving as a control group). We tested what the effect of these sequences were on students' performance, self-efficacy, and intrinsic motivation in a pre-post test design. Our findings indicate that there was a significant effect of adding generative learning strategies (in this case teacher-led instruction and discussion) over the VR-only group. Second, we noted a significant effect of pre-training on students' performance and intrinsic motivation (class-first group), compared to the group of students who received instruction and discussion after the VR-experience (VR-first group). No significant effect could be retrieved for self-efficacy. What does this mean? From a theoretical point of view, this reinstates previous studies which distinguish between media and method: it is NOT the tool (here VR) itself, but how it is implemented that matters; adding generative learning strategies thus matter; our findings corroborate the importance of pre-training referring to the concept of cognitive load. From a practical perspective, it highlights the importance of teachers thinking wisely about their instructional design when adopting immersive technologies: pre-training about the concepts and the main ideas is essential; a 'simple' teacher-led instruction and classroom discussion on the topic, engaging students in generative learning is sufficient: there is no need for video creation, gamified simulations or others. In short, the teacher thus matters, and virtual reality should not replace them. We are very excited and honored to have our study published in the renowned Journal of Computer Assisted Learning by Wiley and would like to explicitly thank editor Paul A. Kirschner for his efforts during the review process. A big shout out too to my co-autors Tijs Rotsaert Martin Valcke and Tammy Schellens (and Yves Rosseel for his guidance during the design and the analysis of the data). I would also like to thank all students, parents, teachers, principals, and supporting institutions that contributed to the experiment of this study. Finally, a warm thank you to my former colleagues from Thomas More Research (Dieter Struyf, Marijke Lemal, Alexander Vanhulsel, Sarah Talboom) who provided me with the opportunity to carry out this study. We hope that our study is valuable to various stakeholders, including teacher, instructional designers, innovation managers, trainers, technology providers, and VR developers. #VR The full paper is available from now via: https://lnkd.in/eu_7j5eP

📊 Learning Gains in Islamabad — What the Data is Telling Us We recently conducted an impact evaluation of our program deployment across schools in Islamabad and the findings were telling. ✅ The biggest learning gains were seen in Grades 4 and 5, especially in Urdu and Math. Grade 3 also saw a sharp rise in Urdu performance (+28 percentage points). These gains point to the strength of our upper primary interventions. 🚩 But there’s a red flag we can’t ignore: Grade 1 is struggling. English, Urdu, and Math all show learning losses compared to baseline. Why? Here’s what the data and classroom realities suggest: 🔹 Curriculum shock — The jump from Early Years to Grade 1 is steep. Students go from play-based learning to reading long paragraphs, multiple subjects, and formal assessments, often without adequate scaffolding. 🔹 Early years bypassed — Many parents enroll children directly into Grade 1, skipping the foundational early years altogether. 🔹 Language & migration — A large number of students are from refugee or multilingual backgrounds, making comprehension even harder at entry points like Grade 1. This tells us something critical: Grade 1 is a gateway year. If we don’t get this transition right, by smoothing the curriculum, strengthening early years, and supporting multilingual learners, we risk setting children up for long-term learning gaps. #EducationPolicy #FoundationalLearning #ImpactEvaluation #CurriculumReform #Taleemabad #EarlyYears #EdTech #PakistanEducation National Institute of Excellence in Teacher Education - NIETE Taleemabad

I’m pleased to share my first Impact Evaluation study publication that I coauthored with Noviandri Nurlaili Khairina Noah Yarrow and Jacobus Cilliers on one of Kementerian Pendidikan dan Kebudayaan (Ministry of Education and Culture Republic of Indonesia) ‘s top priority programs: Program Guru Penggerak. 𝐌𝐞𝐭𝐡𝐨𝐝𝐨𝐥𝐨𝐠𝐲: Quasi-experimental using RDD (Regression Discontinuity Design) approach, examining the cut-off score to select the control and treatment groups. 𝐒𝐚𝐦𝐩𝐥𝐞 𝐬𝐢𝐳𝐞 & 𝐚𝐜𝐭𝐢𝐯𝐢𝐭𝐢𝐞𝐬: 3 rounds of phone surveys (400 teachers and 10 pengajar praktik/mentors), school visits (350 schools: students, peer teachers, school principals, PGP teachers surveys), and in-person classroom observations. Data collection activities took place from May 2022 - April 2023. 𝐋𝐨𝐜𝐚𝐭𝐢𝐨𝐧: 5 cities/regencies for impact evaluation, and 13 cities/regencies across Indonesia for implementation survey. 𝐑𝐞𝐬𝐮𝐥𝐭𝐬: - Overall, we found evidence of positive impacts of the program in improving teaching practices and leadership skills, with some areas for potential improvements. - The program was well-implemented and well-received by teachers, though additional support for pengajar praktik is still needed. - Teaching practices improved substantially, especially in classroom culture and positive discipline. However, there is no evidence of improvements in differentiated instruction or in teachers’ socio-emotional skills in the classroom. - There is evidence of a spillover effect: PGP teachers provided instructional support to their peers at school. - The program improved teachers’ enjoyment of teaching, aspirations to become school principals, and self-confidence as leaders. Read further here: https://lnkd.in/gbkDN44T Summary also available in Bahasa Indonesia: https://lnkd.in/gqAtBQJq #teachertraining #teachingpractice #MerdekaBelajar #educationreform #programgurupenggerak

5,800 course completions in 30 days 🥳 Amazing! But... What does that even mean? Did anyone actually learn anything? As an instructional designer, part of your role SHOULD be measuring impact. Did the learning solution you built matter? Did it help someone do their job better, quicker, with more efficiency, empathy, and enthusiasm? In this L&D world, there's endless talk about measuring success. Some say it's impossible... It's not. Enter the Impact Quadrant. With measureable data + time, you CAN track the success of your initiatives. But you've got to have a process in place to do it. Here are some ideas: 1. Quick Wins (Short-Term + Quantitative) → “Immediate Data Wins” How to track: ➡️ Course completion rates ➡️ Pre/post-test scores ➡️ Training attendance records ➡️ Immediate survey ratings (e.g., “Was this training helpful?”) 📣 Why it matters: Provides fast, measurable proof that the initiative is working. 2. Big Wins (Long-Term + Quantitative) → “Sustained Success” How to track: ➡️ Retention rates of trained employees via follow-up knowledge checks ➡️ Compliance scores over time ➡️ Reduction in errors/incidents ➡️ Job performance metrics (e.g., productivity increase, customer satisfaction) 📣 Why it matters: Demonstrates lasting impact with hard data. 3. Early Signals (Short-Term + Qualitative) → “Small Signs of Change” How to track: ➡️ Learner feedback (open-ended survey responses) ➡️ Documented manager observations ➡️ Engagement levels in discussions or forums ➡️ Behavioral changes noticed soon after training 📣 Why it matters: Captures immediate, anecdotal evidence of success. 4. Cultural Shift (Long-Term + Qualitative) → “Lasting Change” Tracking Methods: ➡️ Long-term learner sentiment surveys ➡️ Leadership feedback on workplace culture shifts ➡️ Self-reported confidence and behavior changes ➡️ Adoption of continuous learning mindset (e.g., employees seeking more training) 📣 Why it matters: Proves deep, lasting change that numbers alone can’t capture. If you’re only tracking one type of impact, you’re leaving insights—and results—on the table. The best instructional design hits all four quadrants: quick wins, sustained success, early signals, and lasting change. Which ones are you measuring? #PerformanceImprovement #InstructionalDesign #Data #Science #DataScience #LearningandDevelopment

This study represents a rare attempt to understand the role instructors and teaching play in determining student academic success, analyzing data for 22,827 students in 704 gateway math courses at four California Community Colleges. Key Take-aways: 1. Math faculty are the most important factor in determining a student’s successful completion of gateway math. 2. Specific instructional practices emphasizing transparency, support, empathy, and human connection can reduce racial disparities in gateway math course outcomes. #TeachingMatters #HumanizeOL

📚 Early-literacy interventions have the potential to prevent reading difficulties, but their large-scale effects often remain unclear. A recent study on the Dutch early-literacy intervention Build! offers valuable insights into the importance of long-term evaluations. The study compared schools implementing Build! with those that did not, using data from 6 years and applying difference-in-difference models to track literacy outcomes. Here’s what they found: ✅ No immediate impact was observed upon introducing the intervention. ✅ However, after 2 years of continuous implementation, notable improvements emerged: 📉 Decrease in students with reading, spelling, and comprehension difficulties. 📈 Increase in average reading and spelling abilities. 💡 Key Takeaway: Large-scale interventions often show their true impact over time. Evaluating outcomes beyond the initial years is crucial for understanding long-term effectiveness. This research highlights the importance of persistence and patience in educational reforms—real change takes time! ⏳ In #Latam we have identified key characteristics of educational reforms and time is one of them. Read full blog here: 👉 https://lnkd.in/gWtF8ZSi And read Build! paper here 👉 Implementation Takes Time: Reduction of Literacy Problems in Schools Implementing an Early-Literacy Intervention https://lnkd.in/e4kHxP3V #Education #EarlyLiteracy #Intervention #Research #EdTech #EducationalImpact #EDUKnowledge

As part of Rise Together's investment in edtech company BookNook, we pledged to support a gold-standard randomized control trial in partnership with Teacher's College, Columbia University, to collect data on the efficacy of high-impact tutoring on student literacy achievement. We are excited to finally share the results of the study - read my blog post below with links to the full study and press release. Two key take-aways from the study: 1) Statistical Significance in Reading Gains: The study revealed that students who engaged with BookNook for the recommended 20 or more sessions experienced statistically significant reading gains. This finding not only highlights the importance of consistent and frequent use of the platform but also demonstrates BookNook's capability to foster developmental advancements in literacy. 2) Wide Applicability Across Demographics: A crucial aspect of the study's findings is the platform's adaptability and effectiveness across diverse demographic groups. Regardless of varying backgrounds and learning needs, students across the board showed marked improvements, highlighting the platform's adaptability and versatility in addressing literacy gaps in different contexts. While the study was confined to Rocketship schools, its implications are far-reaching. It demonstrates the potential for online platforms to support reading growth across diverse educational settings. It also emphasizes the necessity of structured tutoring approaches to realize significant learning advancements. https://lnkd.in/gVxHdjih