Does generative-ai feedback in formative assessment improve higher-order thinking: motivational pathways in vocational safety education

  • Waskito Waskito 
    Universitas Negeri Padang
    Indonesia
  • Tri Juliantika
    Universitas Negeri Padang
    Indonesia
  • Refdinal Refdinal
    Universitas Negeri Padang
    Indonesia
  • Yeka Hendriyani
    Universitas Negeri Padang
    Indonesia

Abstract

Occupational health and safety (OHS) education in vocational high schools requires higher-order thinking skills (HOTS), yet formative assessment is often limited by delayed teacher feedback. This study examined whether generative AI-supported formative assessment using ChatGPT and Gemini improved HOTS and learning motivation compared with conventional teacher-led assessment, and explored the relationship between perceptions of AI-assisted assessment, motivation, and HOTS. A posttest-only nonequivalent control group quasi-experimental design involved 60 Grade X Machining Engineering students at SMK Negeri 1 Padang, divided into two intact classes of 30 students each. HOTS was measured using a 20-item reasoned multiple-choice test covering Bloom’s cognitive levels C4–C6, while motivation and perceptions of AI-assisted assessment were measured using validated questionnaires. Independent-samples t-tests were used for the primary analysis, followed by exploratory partial least squares structural equation modelling (PLS-SEM) in the experimental group. No significant differences were found between the AI-assisted and conventional assessment groups in HOTS scores (81.73 vs. 80.87, p = .417, d = 0.21) or motivation (54.03 vs. 54.83, p = .497, d = −0.18). Within the experimental group, perceptions of AI-assisted assessment were positively associated with motivation (β = 0.743, p < .001), and motivation was positively associated with HOTS (β = 0.476, p = .023). Although the indirect effect was statistically significant (β = 0.354, p = .042), its bias-corrected confidence interval included zero, making the mediation evidence inconclusive. These findings do not demonstrate that AI-assisted formative assessment improves HOTS or motivation beyond conventional teacher feedback. However, the observed motivational relationships warrant further investigation using larger samples, baseline measurements, and longer interventions.

Keywords
  • Formative assessment
  • Artificial intelligence
  • Learning motivation
  • Higher-order thinking skills
  • Occupational health and safety
  • PLS-SEM
Lisensing Information
Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

References
  1. Adaby, M. A., Sugiarto, T., Arif, A., & Muslim. (2026). Analisis tingkat pengetahuan pada soal UAS produktif otomotif semester genap 2024/2025 kelas XII TKR SMK Negeri Teknologi di Kota Padang [Analysis of knowledge levels in productive automotive final-semester examination items for Grade XII light-vehicle engineering students]. Ensiklopedia of Journal, 8(2), 98–105. https://doi.org/10.33559/eoj.v8i2.3710
  2. Aldahlaki, Z. H. A. (2026). The role of artificial intelligence in enhancing learning motivation through self and immediate feedback. Journal of Technology and System Information, 3(1), 1–7. https://doi.org/10.47134/jtsi.v3i1.5570
  3. An, P. M. N., Anh, T. N. P., & Dinh, C.-T. (2025). Investigating the mediating effects of motivation on academic performance in a synchronous EFL online learning. Journal of e-Learning and Knowledge Society, 21(1), 224–235. https://doi.org/10.20368/1971-8829/1135976
  4. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
  5. Cao, C., Taddese, E. T., Gebresilase, B. M., Shorouk, A., & Berhanu, K. Z. (2025). The influence of learning motivation on academic performance in Chinese vocational college students: A self-determination theory perspective. International Journal of Innovative Research and Scientific Studies, 8(6), 3297–3309. https://doi.org/10.53894/ijirss.v8i6.10339
  6. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
  7. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153
  8. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
  9. Fidan, M., & Gencel, N. (2022). Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: The effects on learning performance and intrinsic motivation. Journal of Educational Computing Research, 60(7), 1716–1741. https://doi.org/10.1177/07356331221077901
  10. Garzón, J., Patiño, E., & Marulanda, C. (2025). Systematic review of artificial intelligence in education: Trends, benefits, and challenges. Multimodal Technologies and Interaction, 9(8), 84. https://doi.org/10.3390/mti9080084
  11. Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2017). A primer on partial least squares structural equation modeling (PLS-SEM) (2nd ed.). SAGE Publications.
  12. Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203
  13. Hanazahra, A., Efriyanti, L., Satria, E., & Indra, Y. (2025). Evaluasi implementasi pembelajaran adaptif berbasis artificial intelligence (AI) di SMKN 1 Bukittinggi [Evaluation of the implementation of artificial intelligence-based adaptive learning at SMKN 1 Bukittinggi]. Jurnal Pendidikan Teknologi Informasi dan Vokasional, 7(2), 1–13. https://doi.org/http://dx.doi.org/10.23960/jpvti
  14. Hanis, M., & Wahyudin, D. (2024). Pemanfaatan artificial intelligence (AI) dalam penyusunan asesmen pembelajaran bagi guru sekolah dasar [Utilizing artificial intelligence (AI) in developing learning assessments for elementary school teachers]. Jurnal Ilmiah Profesi Pendidikan, 9(2). https://doi.org/10.29303/jipp.v9i2.2252
  15. Hapsari, D. D., Ramadhani, G. Y., & Ikramullah, N. I. (2024). Literature review: Pengaruh artificial intelligence (AI) terhadap motivasi belajar peserta didik [Literature review: The effect of artificial intelligence (AI) on students’ learning motivation]. Jurnal Empati, 13(4), 313–324. https://doi.org/10.14710/empati.2024.46697
  16. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
  17. Hidayati, H. N. (2025). Pemanfaatan artificial intelligence pada penilaian pembelajaran mendalam [Utilizing artificial intelligence in deep learning assessment]. Afeksi: Jurnal Penelitian dan Evaluasi Pendidikan, 6(5), 900–908. https://doi.org/10.59698/afeksi.v6i5.531
  18. Hukom, J. (2025). Model integrasi feedback digital berbasis AI untuk pembelajaran berorientasi kompetensi [An AI-based digital feedback integration model for competency-oriented learning]. IDENTIK: Jurnal Ilmu Ekonomi, Pendidikan dan Teknik, 2(4), 169–174.
  19. Ji, H., Suo, L., & Chen, H. (2024). AI performance assessment in blended learning: Mechanisms and effects on students’ continuous learning motivation. Frontiers in Psychology, 15, 1447680. https://doi.org/10.3389/fpsyg.2024.1447680
  20. Jian, Z. (2022). Sustainable engagement and academic achievement under impact of academic self-efficacy through mediation of learning agility: Evidence from music education students. Frontiers in Psychology, 13, 899706. https://doi.org/10.3389/fpsyg.2022.899706
  21. Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2–10. https://doi.org/10.1007/BF02905780
  22. Keong, L. M. (2025). AI-personalised learning in higher education: A study on learning outcomes and motivation among university students. International Journal of Research and Innovation in Social Science, 9(5), 6075–6085. https://doi.org/10.47772/IJRISS.2025.905000473
  23. Kock, N., & Hadaya, P. (2018). Minimum sample size estimation in PLS-SEM: The inverse square root and gamma-exponential methods. Information Systems Journal, 28(1), 227–261. https://doi.org/10.1111/isj.12131
  24. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2
  25. Larasati, S. P., Sukma, D., Karneli, Y., Fikri, M., Amat, M. A. C., & Rismi, R. (2026). AI-driven feedback and its impact on student motivation in online learning. G-COUNS: Jurnal Bimbingan dan Konseling, 10(2), 1071–1088. https://doi.org/10.31316/g-couns.v10i02.8584
  26. Nafiati, D. A. (2021). Revisi taksonomi Bloom: Kognitif, afektif, dan psikomotorik [Revision of Bloom’s taxonomy: Cognitive, affective, and psychomotor]. Humanika: Kajian Ilmiah Mata Kuliah Umum, 21(2), 151–172. https://doi.org/10.21831/hum.v21i2.29252
  27. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
  28. Pratama, D. H., & Sugiarto, T. (2025). Literasi keselamatan dan kesehatan kerja sebagai prediktor kedisiplinan praktikum siswa SMK program Teknik Kendaraan Ringan [Occupational health and safety literacy as a predictor of practical-work discipline among light-vehicle engineering students]. JTPVI: Jurnal Teknologi dan Pendidikan Vokasi Indonesia, 3(4), 891–902. https://doi.org/10.24036/jtpvi.v3i4.222
  29. Rahmawati, Y. (2025). Kecerdasan buatan di ruang kelas: Analisis penurunan motivasi belajar dan kemampuan berpikir kritis siswa SMP Negeri 1 Ganding Sumenep [Artificial intelligence in the classroom: An analysis of declining learning motivation and critical-thinking ability among students of SMP Negeri 1 Ganding Sumenep]. Jurnal Teknologi Pendidikan, 2(4), 1–8. https://doi.org/10.47134/jtp.v2i4.1640
  30. Roediger, H. L., III, & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
  31. Sari, I. P., Hatta, T. A., Manurung, A. A., & Dachi, S. W. (2024). The influence of a high-order thinking (HOTS)-based learning model on students’ midterm question analysis. Aksaqila International Humanities and Social Sciences Journal, 3(2), 58–66. https://doi.org/10.30596/aihss.v3i2.505
  32. Simões, S., Oliveira, T., & Nunes, C. (2022). Influence of computers in students’ academic achievement. Heliyon, 8, e09004. https://doi.org/10.1016/j.heliyon.2022.e09004
  33. Wahyu, R., Pertiwi, L., Kulsum, L. U., & Hanifah, I. A. (2024). Evaluating the impact of artificial intelligence-based learning methods on students’ motivation and academic achievement. International Journal of Post Axial: Futuristic Teaching and Learning, 2(1), 49–58. https://doi.org/10.59944/postaxial.v2i1.279
  34. Waskito, Irzal, Wulansari, R. E., & Ya, K. Z. (2022). The adventure of formative assessment with active feedback in the vocational learning: The empirical effect for increasing students’ achievement. Journal of Technical Education and Training, 14(1), 54–62. https://doi.org/10.30880/jtet.2022.14.01.005
  35. Widyawati, N. K. (2020). Pentingnya penguasaan konsep keselamatan dan kesehatan kerja (K3) dalam mendukung kinerja calon lulusan pendidikan kejuruan di dunia kerja [The importance of mastering occupational health and safety (OHS) concepts in supporting the workplace performance of prospective vocational graduates]. Jurnal Bosaparis: Pendidikan Kesejahteraan Keluarga, 11(3), 87–93.
  36. Wu, J., Tlili, A., Salha, S., Mizza, D., Saqr, M., López-Pernas, S., & Huang, R. (2025). Unlocking the potential of artificial intelligence in improving learning achievement in blended learning: A meta-analysis. Frontiers in Psychology, 16, 1691414. https://doi.org/10.3389/fpsyg.2025.1691414
  37. Yin, J., Goh, T.-T., & Hu, Y. (2024). Using a chatbot to provide formative feedback: A longitudinal study of intrinsic motivation, cognitive load, and learning performance. IEEE Transactions on Learning Technologies, 17, 1378–1389. https://doi.org/10.1109/TLT.2024.3364015

Downloads

PDF

Statistic

Abstract Views
: 53 times
PDF
: 38 times