Fine-Grained Classroom Activity Recognition via Detection, Head-Pose, and Appearance Fusion
DOI:
https://doi.org/10.47709/brilliance.v6i3.9453Keywords:
classroom analytics, fine-grained classroom activity recognition, head-pose estimation, multi-cue fusion, smartphone useAbstract
Automatic analysis of student behavior in classroom environments is an important prerequisite for smart learning systems that move beyond coarse engagement indicators toward behavior-specific and pedagogically interpretable feedback. Fine-grained classroom activities, however, remain difficult to recognize because several target behaviors exhibit similar visual patterns, are partially occluded, or involve subtle head and upper-body motion. This study addresses five classroom activities commonly observed in authentic instructional settings: nodding, hand-raising, smartphone use, head-supporting, and looking downward. We propose a student-centered framework that first constructs person-consistent 16-frame tracklets and then integrates three complementary sources of evidence: RGB appearance from person crops, head-pose estimation, and an explicit smartphone-related cue. This design addresses two major ambiguity clusters in classroom scenes, namely downward-looking versus smartphone use and downward-looking versus head-supporting, while preserving temporal sensitivity for nodding. The dataset comprises labeled student-centered clips extracted from multi-view classroom recordings. Evaluation follows a subject-separated and session-separated protocol using different acquisition sessions, classroom settings, and participant groups. Under this protocol, the proposed framework achieves 91.13% accuracy and 91.14% macro F1. Compared with a visual-only baseline, the full fusion model improves macro F1 by 6.21 percentage points, while outperforming the strongest non-fusion baseline by 2.83 percentage points. The confusion analysis further indicates that head-pose information and explicit smartphone cues effectively separate visually adjacent downward-oriented behaviors. These findings support multi-cue fusion as an effective strategy for fine-grained classroom activity recognition and classroom analytics under the present held-out evaluation protocol, while broader deployment and cross-session generalization should remain bounded by the current evidence.
References
Abozeid, A., Alrashdi, I., & Al-Makhlasawy, R. M. (2026). Intelligent recognition of students’ behavior for smart learning environments. Scientific Reports 2026 16:1, 16(1), 5674-. https://doi.org/10.1038/s41598-026-36633-9
Algabri, R., Abdu, A., & Lee, S. (2024). Deep learning and machine learning techniques for head pose estimation: a survey. Artificial Intelligence Review 2024 57:10, 57(10), 288-. https://doi.org/10.1007/S10462-024-10936-7
Alkabbany, I., Ali, A. M., Foreman, C., Tretter, T., Hindy, N., & Farag, A. (2023). An Experimental Platform for Real-Time Students Engagement Measurements from Video in STEM Classrooms. Sensors 2023, Vol. 23, Page 1614, 23(3), 1614. https://doi.org/10.3390/S23031614
Alruwais, N. M., & Zakariah, M. (2024). Student Recognition and Activity Monitoring in E-Classes Using Deep Learning in Higher Education. IEEE Access, 12, 66110–66128. https://doi.org/10.1109/ACCESS.2024.3354981
Asperti, A., & Filippini, D. (2023). Deep Learning for Head Pose Estimation: A Survey. SN Computer Science, 4(4), 349-. https://doi.org/10.1007/S42979-023-01796-Z/FIGS/17
Cao, Y., Cao, Q., Qian, C., & Chen, D. (2025). YOLO-AMM: A Real-Time Classroom Behavior Detection Algorithm Based on Multi-Dimensional Feature Optimization. Sensors 2025, Vol. 25, Page 1142, 25(4), 1142. https://doi.org/10.3390/S25041142
Chen, H., Zhou, G., & Jiang, H. (2023). Student Behavior Detection in the Classroom Based on Improved YOLOv8. Sensors 2023, Vol. 23, Page 8385, 23(20), 8385. https://doi.org/10.3390/S23208385
Consoli, T., Schmitz, M. L., Antonietti, C., Gonon, P., Cattaneo, A., & Petko, D. (2024). Quality of technology integration matters: Positive associations with students’ behavioral engagement and digital competencies for learning. Education and Information Technologies 2024 30:6, 30(6), 7719–7752. https://doi.org/10.1007/S10639-024-13118-8
Dimitriadou, E., & Lanitis, A. (2023). A critical evaluation, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms. Smart Learning Environments 2023 10:1, 10(1), 12-. https://doi.org/10.1186/S40561-023-00231-3
Ding, J., Niu, S., Nie, Z., & Zhu, W. (2024). Research on Human Posture Estimation Algorithm Based on YOLO-Pose. Sensors 2024, Vol. 24, Page 3036, 24(10), 3036. https://doi.org/10.3390/S24103036
Edozie, E., Shuaibu, A. N., John, U. K., & Sadiq, B. O. (2025). Comprehensive review of recent developments in visual object detection based on deep learning. Artificial Intelligence Review 2025 58:9, 58(9), 277-. https://doi.org/10.1007/S10462-025-11284-W
Elbawab, M., & Henriques, R. (2023). Machine Learning applied to student attentiveness detection: Using emotional and non-emotional measures. Education and Information Technologies 2023 28:12, 28(12), 15717–15737. https://doi.org/10.1007/S10639-023-11814-5
Elnady, M., & Abdelmunim, H. E. (2025). A novel YOLO LSTM approach for enhanced human action recognition in video sequences. Scientific Reports 2025 15:1, 15(1), 17036-. https://doi.org/10.1038/s41598-025-01898-z
Hempel, T., Abdelrahman, A. A., & Al-Hamadi, A. (2022). 6D ROTATION REPRESENTATION FOR UNCONSTRAINED HEAD POSE ESTIMATION. Proceedings - International Conference on Image Processing, ICIP, 2496–2500. https://doi.org/10.1109/ICIP46576.2022.9897219
Hu, J., Huang, Z., Li, J., Xu, L., & Zou, Y. (2024). Real-Time Classroom Behavior Analysis for Enhanced Engineering Education: An AI-Assisted Approach. International Journal of Computational Intelligence Systems 2024 17:1, 17(1), 167-. https://doi.org/10.1007/S44196-024-00572-Y
Ikram, S., Ahmad, H., Mahmood, N., Faisal, C. M. N., Abbas, Q., Qureshi, I., & Hussain, A. (2023). Recognition of Student Engagement State in a Classroom Environment Using Deep and Efficient Transfer Learning Algorithm. Applied Sciences 2023, Vol. 13, Page 8637, 13(15), 8637. https://doi.org/10.3390/APP13158637
Jiang, H., & Fu, W. (2024). Computer vision recognition in the teaching classroom: A Review. EAI Endorsed Transactions on AI and Robotics, 3. https://doi.org/10.4108/AIRO.4079
Jocher, G., Qiu, J., Liu, M., Lyu, S., Akyon, F. C., & Kalfaoglu, M. E. (2026). Ultralytics YOLO26: Unified Real-Time End-to-End Vision Models. https://doi.org/10.48550/arXiv.2606.03748
Kassab, S. E., Al-Eraky, M., El-Sayed, W., Hamdy, H., & Schmidt, H. (2023). Measurement of student engagement in health professions education: a review of literature. BMC Medical Education 2023 23:1, 23(1), 354-. https://doi.org/10.1186/S12909-023-04344-8
Lamaakal, I., Yahyati, C., Maleh, Y., El Makkaoui, K., Ouahbi, I., El-Latif, A. A. A., Zomorodi, M., & El-Rahiem, B. A. (2025). A tiny inertial transformer for human activity recognition via multimodal knowledge distillation and explainable AI. Scientific Reports 2025 15:1, 15(1), 42335-. https://doi.org/10.1038/s41598-025-26297-2
Liu, J., Mu, X., Liu, Z., & Li, H. (2023). Human skeleton behavior recognition model based on multi-object pose estimation with spatiotemporal semantics. Machine Vision and Applications 2023 34:3, 34(3), 44-. https://doi.org/10.1007/S00138-023-01396-0
Liu, Q., Jiang, R., Xu, Q., Wang, D., Sang, Z., Jiang, X., & Wu, L. (2024). YOLOv8n-BT: Research on Classroom Learning Behavior Recognition Algorithm Based on Improved YOLOv8n. IEEE Access, 12, 36391–36403. https://doi.org/10.1109/ACCESS.2024.3373536
Liu, Q., Jiang, X., & Jiang, R. (2025). Classroom Behavior Recognition Using Computer Vision: A Systematic Review. Sensors 2025, Vol. 25, Page 373, 25(2), 373. https://doi.org/10.3390/S25020373
Loshchilov, I., & Hutter, F. (n.d.). Decoupled Weight Decay Regularization. Retrieved July 28, 2026, from https://github.com/loshchil/AdamW-and-SGDW
Nadeem, M., Elbasi, E., Zreikat, A. I., & Sharsheer, M. (2024). Sitting Posture Recognition Systems: Comprehensive Literature Review and Analysis. Applied Sciences 2024, Vol. 14, Page 8557, 14(18), 8557. https://doi.org/10.3390/APP14188557
Qureshi, T. S., Shahid, M. H., Farhan, A. A., & Alamri, S. (2025). A systematic literature review on human activity recognition using smart devices: advances, challenges, and future directions. Artificial Intelligence Review 2025 58:9, 58(9), 276-. https://doi.org/10.1007/S10462-025-11275-X
Sharma, V., Gupta, M., Kumar, A., & Mishra, D. (2024). STAR-3D: A Holistic Approach for Human Activity Recognition in the Classroom Environment. Information 2024, Vol. 15, Page 179, 15(4), 179. https://doi.org/10.3390/INFO15040179
Sheng, X., Li, S., & Chan, S. (2025). Real-time classroom student behavior detection based on improved YOLOv8s. Scientific Reports 2025 15:1, 15(1), 14470-. https://doi.org/10.1038/s41598-025-99243-x
Shou, Z.?;, Yuan, X.?;, Li, D.?;, Mo, J.?;, Zhang, H.?;, Zhang, J.?;, Wu, Z. A., Shou, Z., Yuan, X., Li, D., Mo, J., Zhang, H., Zhang, J., & Wu, Z. (2024). A Dynamic Position Embedding-Based Model for Student Classroom Complete Meta-Action Recognition. Sensors 2024, Vol. 24, Page 5371, 24(16), 5371. https://doi.org/10.3390/S24165371
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (pp. 6105–6114). PMLR. https://proceedings.mlr.press/v97/tan19a.html
Trabelsi, Z., Alnajjar, F., Parambil, M. M. A., Gochoo, M., & Ali, L. (2023). Real-Time Attention Monitoring System for Classroom: A Deep Learning Approach for Student’s Behavior Recognition. Big Data and Cognitive Computing 2023, Vol. 7, Page 48, 7(1), 48. https://doi.org/10.3390/BDCC7010048
Vaswani, A., Brain, G., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ?., & Polosukhin, I. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, 30.
Vermander, P., Mancisidor, A., Cabanes, I., & Perez, N. (2024). Intelligent systems for sitting posture monitoring and anomaly detection: an overview. Journal of NeuroEngineering and Rehabilitation 2024 21:1, 21(1), 28-. https://doi.org/10.1186/S12984-024-01322-Z
Wang, Z., Wang, M., Zeng, C., & Li, L. (2024). SBD-Net: Incorporating Multi-Level Features for an Efficient Detection Network of Student Behavior in Smart Classrooms. Applied Sciences 2024, Vol. 14, Page 8357, 14(18), 8357. https://doi.org/10.3390/APP14188357
Wang, Z.?;, Yao, J.?;, Zeng, C.?;, Fu, S., Yang, B., Zarifis, A., Wang, Z., Yao, J., Zeng, C., Li, L., & Tan, C. (2023). Students’ Classroom Behavior Detection System Incorporating Deformable DETR with Swin Transformer and Light-Weight Feature Pyramid Network. Systems 2023, Vol. 11, Page 372, 11(7), 372. https://doi.org/10.3390/SYSTEMS11070372
Xu, L., Li, Z., Gan, Y., & Xia, H. (2025). Soft-label guided stacked dual attention network for head pose estimation and its application to classroom gaze analysis. Scientific Reports 2025 16:1, 16(1), 405-. https://doi.org/10.1038/s41598-025-29814-5
Xu, Q., Wei, Y., Gao, J., Yao, H., & Liu, Q. (2023). ICAPD Framework and simAM-YOLOv8n for Student Cognitive Engagement Detection in Classroom. IEEE Access, 11, 136063–136076. https://doi.org/10.1109/ACCESS.2023.3337435
Zhang, H., Zhou, G., He, W., & Deng, H. (2025). Classroom Behavior Detection Method Based on PLA-YOLO11n. Sensors 2025, Vol. 25, Page 5386, 25(17), 5386. https://doi.org/10.3390/S25175386
Zhang, J., Guo, L., & Wang, X. (2025). Student Classroom Behavior Recognition Based on YOLOv8 and Attention Mechanism. Information 2025, Vol. 16, Page 934, 16(11), 934. https://doi.org/10.3390/INFO16110934
Zhang, X., Nie, J., Wei, S., Zhu, G., Dai, W., & Yang, C. (2024). A Study of Classroom Behavior Recognition Incorporating Super-Resolution and Target Detection. Sensors 2024, Vol. 24, Page 5640, 24(17), 5640. https://doi.org/10.3390/S24175640
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., & Wang, X. (2022). ByteTrack: Multi-object Tracking by Associating Every Detection Box. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13682 LNCS, 1–21. https://doi.org/10.1007/978-3-031-20047-2_1/SAVE-RESEARCH
Zhang, Y., & Wang, Y. (2025). A comprehensive survey on RGB-D-based human action recognition: algorithms, datasets, and popular applications. EURASIP Journal on Image and Video Processing 2025 2025:1, 2025(1), 15-. https://doi.org/10.1186/S13640-025-00677-0
Zhou, L., Liu, X., Guan, X., & Cheng, Y. (2025). CSSA-YOLO: Cross-Scale Spatiotemporal Attention Network for Fine-Grained Behavior Recognition in Classroom Environments. Sensors 2025, Vol. 25, Page 3132, 25(10), 3132. https://doi.org/10.3390/S25103132
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Yaya Wihardi, Raffi Ardhi Naufal, Meutia Jasmine Annisa Herawan, Rani Megasari
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
info@itscience.org, web: 
