A Survey on Emotion Recognition with Human Centred Software Engineering

Section: Research Paper
Issue
Vol. 23 No. 1 (2026): Volume 23 Issue 1
Published
May 31, 2026
Pages
131-140

Abstract

The survey explores the connection between emotion recognition technology and Human-Centered Software Engineering (HCSE) and, in particular, how the two aspects can be integrated to enhance user experiences, especially in customer services.  The big picture is to research how emotional intelligence can be built into software system, and how this can lead to more empathic response with human beings.  The work is grounded on the critical review of the current developments in the field of emotion identification, particularly on how deep learning models, namely Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks are being utilized. such models are evaluated in terms of their role in real-time detection of emotional states and ratio of enhanced classification. Also, the research takes into account the multimodal recognition approaches, which involve facial expression, voice, and physiological signals to perform a thorough analysis of emotions. The survey also covers such significant implementation issues as technical constraints, ethical concerns, and algorithmic biases that exist. The results of this effort have broad implications in many fields, such as customer support, healthcare, education, and entertainment, and provide clues on how to create emotionally intelligent systems that can make the human-computer interactions much more personalized and productive.

References

  1. Abdulmajeed and N. N. Saleem, "Intelligent tool for detecting Covid-19 using convolutional neural network based on both CT and x-ray lung images," AIP Conference Proceedings, vol. 2404, no. 1, p. 030002, Oct. 2021. [Online]. Available: https://www.semanticscholar.org/paper/Intelligent-tool-for-detecting-Covid-19-using-based-Abdulmajeed-Saleem/ed08fcbba80868b24262605040606791569c3868. DOI: 10.1063/5.0068889
  2. AL thanoon, A., Altaie, A., & Alsarraj, R. (2024). Principles of Requirements Management and Analysis for Supporting Software Engineering Development: A Literature Review.. IRAQI JOURNAL OF STATISTICAL SCIENCES, 21(2), 181-190. doi: 10.33899/iqjoss.2024.185257
  3. Alahi, M. E. E., Sukkuea, A., Tina, F. W., Nag, A., Kurdthongmee, W., Suwannarat, K., & Mukhopadhyay, S. C. (2021). A novel vision-based smart sensor IoT system for real-time monitoring of shrimp water quality. Sensors, 21(15), 5015. https://doi.org/10.3390/s21155015
  4. Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., Santamaría, J., Ouyang, C., Gupta, A., & Deveci, M. (2022). A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion. Information Fusion, 96, 156-191. https://doi.org/10.1016/j.inffus.2022.11.024
  5. Albahri, A. S., Duhaim, A. M., Fadhel, M. A., Alnoor, A., Baqer, N. S., Alzubaidi, L., Albahri, O. S., Alamoodi, A. H., Bai, J., Salhi, A., Santamaría, J., Ouyang, C., Gupta, A., & Deveci, M. (2022). A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion. Information Fusion, 96, 156-191. https://doi.org/10.1016/j.inffus.2022.11.024
  6. Ali, J. M. (2023). Introduction for human-centric software engineering. Advances in Engineering Innovation, 5(1). https://doi.org/10.54254/2977-3903/5/2023032. DOI: 10.54254/2977-3903/5/2023032
  7. Alom, M. Z., Taha, T. M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M. S., Hasan, M., Van Essen, B. C., Awwal, A. A. S., & Asari, V. K. (2018). The history began from alexnet: A comprehensive survey on deep learning approaches. 2018 International Conference on Intelligent Systems and Computer Vision (ISCV), 1–6. https://doi.org/10.1109/ISACV.2018.8354116
  8. Altalabani, D., & Erdogan, F. (2024). Classification of Diabetes Data Set from Iraq via Different Machine Learning Techniques. IRAQI JOURNAL OF STATISTICAL SCIENCES, 21(1), 170-189. doi: 10.33899/iqjoss.2024.183258
  9. Barros, P., Wermter, S., & Weber, C. (2019). Multimodal affective interaction with a humanoid robot using deep neural architectures. Philosophical Transactions of the Royal Society B, 374(1771), 20180036. https://doi.org/10.1098/rstb.2018.0036
  10. Brun, A., Liu, R., Shukla, A., Watson, F., & Gratch, J. (2025). Exploring Emotion-Sensitive LLM-Based Conversational AI. arXiv preprint arXiv:2502.08920.
  11. Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, 81, 77–91. DOI 10.57702/6jsb3bqm
  12. Calvo, R. A., & D’Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18–37. https://doi.org/10.1109/T-AFFC.2010.1
  13. F. A. Mohamed, M. S. A. Aziz, and M. Mahmud, "Cues to Deception in Islamic Websites Text and Design: A Case Study Approach," Int. J. Percept. Cogn. Comput., vol. 9, no. 1, pp. 14–23, Jan. 2023. [Online]. DOI: https://doi.org/10.31436/ijpcc.v9i1.299
  14. Feng, Y., & Devillers, L. (2023, September). End-to-end continuous speech emotion recognition in real-life customer service call center conversations. In 2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) (pp. 1-8). IEEE. DOI: 10.1109/ACIIW59127.2023.10388120
  15. Ferguson, C. J. (2019). Everything in moderation: Moderate use of screens unassociated with child behavior problems. Psychological Science in the Public Interest, 20(1), 1–15. https://doi.org/10.1177/1529100619832930
  16. George, G., Haas, M. R., McGahan, A. M., Schillebeeckx, S. J. D., & Tihanyi, L. (2021). Purpose in the for-profit firm: A review and framework for management research. Academy of Management Review, 46(3), 406-430. https://doi.org/10.5465/amr.2018.0072
  17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27. Available in : https://proceedings.neurips.cc/paper_files/paper/2014/file/f033ed80deb0234979a61f95710dbe25-Paper.pdf
  18. Khan, M. A., Alkaabi, N., & Al-Rakhami, M. S. (2023). Artificial intelligence and blockchain for cybersecurity in cyber-physical systems: Recent advances and future research directions. *Internet of Things and Cyber-Physical Systems, 3*, 100-115. https://doi.org/10.1016/j.iotcps.2023.07.001
  19. Kollias, D., Zafeiriou, S., & Shen, J. (2020). Deep neural network architectures for real-time facial expression recognition. IEEE Transactions on Affective Computing, 13(3), 1190–1203. DOI: 10.1109/TAFFC.2019.2912644
  20. M. V. Ramana and K. Srinivasan, "Speech emotion recognition: A review," International Journal of Speech Technology, vol. 20, pp. 1-12, 2017. ISSN: 2278-3075, Volume-2, Issue-4.
  21. Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., ... & Ni, S. (2024). Generative technology for human emotion recognition: A scoping review. Information Fusion, 102753. DOI: 10.1016/j.inffus.2024.102753.
  22. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT) (pp. 220-229). ACM. https://doi.org/10.1145/3442188.3445918
  23. Mohamed, A. (2024). Innovative Cloud Security Solutions: Hybrid RNN and CNN Models for Intrusion Detection. IRAQI JOURNAL OF STATISTICAL SCIENCES, 21(2), 24-35. doi: 10.33899/iqjoss.2024.185233
  24. Moufaq, A., Jader, M., & O. Abdul-Majjed, I. (2011). Sound Encryption Using Feature Extraction and Neural Network. IRAQI JOURNAL OF STATISTICAL SCIENCES, 11(2), 291-308. doi: 10.33899/iqjoss.2011.027926
  25. N. M. Edan and S. M. Hadeed, "Using Software Engineering to Design and Implement Mixer of Multiple Cameras, Microphone, and Screens," MINAR International Journal of Applied Sciences and Technology, vol. 9, no. 1, pp. 1–10, Jul. 2022. [Online]. Available: https://journals.iium.edu.my/kict/index.php/ijminar/article/view/503
  26. P. Ekman and W. V. Friesen, "The facial action coding system: A technique for the measurement of facial movement," CNS Spectrums, vol. 16, no. 2, pp. 27-44, 2011. DOI: 10.1017/S1092852900018754
  27. Plaza, M., Kazała, R., Koruba, Z., Kozłowski, M., Lucińska, M., Sitek, K., & Spyrka, J. (2022). Emotion recognition method for call/contact centre systems. Applied Sciences, 12(21), 10951. DOI: 10.3390/app122110951.
  28. Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125. https://doi.org/10.1016/j.inffus.2017.02.003.
  29. S. I. Khaleel and L. F. Salih, "A survey of predicting software reliability using machine learning methods," IAES International Journal of Artificial Intelligence (IJ-AI), vol. 13, no. 1, pp. 35–44, Mar. 2024. [Online]. Available: https://pdfs.semanticscholar.org/c7e8/eafaead07e9e8dbcf1529ed6eb45058df4d8.pdf
  30. S. Mohammad and H. K. M. Al-Khateeb, "Speech emotion recognition: A review," International Journal of Advanced Computer Science and Applications, vol. 10, no. 7, pp. 50-56, 2019. https://doi.org/10.1007/978-981-15-5341-7_61
  31. Saffaryazdi, N., Gunasekaran, T. S., Laveys, K., Broadbent, E., & Billinghurst, M. (2025). Empathetic Conversational Agents: Utilizing Neural and Physiological Signals for Enhanced Empathetic Interactions. arXiv preprint arXiv:2501.08393.
  32. Shukur, O., & Malaa, O. (2023). Comparison of Logistic regression, Convolution Neural Network, and Kernel Approaches for Classifying the Caenorhabditis Elegans Motion. IRAQI JOURNAL OF STATISTICAL SCIENCES, 20(2), 175-187. doi: 10.33899/iqjoss.2023.181225
  33. Song, H., Meng, D., Sun, L., & Ma, Z. (2020). Two-stream generative adversarial network for weakly supervised facial expression recognition. IEEE Transactions on Image Processing, 29, 6725–6738. DOI: 10.1109/TIP.2020.2980258.
  34. Sultan, D. A., & Ibrahim, L. M. (2024, November). Video_audio multimodal for deepfake detection. In AIP Conference Proceedings (Vol. 3229, No. 1). AIP Publishing. https://doi.org/10.1063/5.0236010
  35. V. Govindaraju and D. Thangam, "Emotion Recognition in Human-Machine Interaction and a Review in Interpersonal Communication Perspective," in Human-Machine Collaboration and Emotional Intelligence in Industry 5.0, J. Editor, Ed. Hershey, PA, USA: IGI Global, Jun. 2024, pp. 329–343. doi: 10.4018/979-8-3693-6806-0.ch018.
  36. Xu, Y., Lin, Y. S., Zhou, X., & Shan, X. (2024). Utilizing emotion recognition technology to enhance user experience in real-time. Computing and Artificial Intelligence, 2(1), 1388-1388. https://doi.org/10.59400/cai.v2i1.1388
  37. Y. Wang et al., "Speech emotion recognition using deep learning: A review," Artificial Intelligence Review, vol. 52, pp. 2419-2433, 2019. DOI: 10.1007/s10462-018-9640-y
  38. Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2017). A 3D facial expression database for facial behavior research. International Conference on Automatic Face and Gesture Recognition (FG). https://doi.org/10.1109/FG.2006.1699552.
  39. Yurtay, Y., Demirci, H., Tiryaki, H., & Altun, T. (2024). Emotion Recognition on Call Center Voice Data. Applied Sciences, 14(20), 9458. DOI: 10.3390/app14209458.
  40. Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2018). A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39-58. DOI: 10.1109/TPAMI.2008.52
  41. Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2022). Conditional GAN for data augmentation in facial emotion recognition. Pattern Recognition Letters, 153, 12–19. DOI: 10.1016/j.patrec.2021.05.008.
  42. Zhao, X., Liang, X., Liu, Y., & Meng, D. (2021). Robust facial expression recognition under occlusion via guided attention and discriminative feature learning. Pattern Recognition, 119, 108067. https://doi.org/10.1016/j.patcog.2021.108067
  43. Zhao, X., Mao, X., & Chen, L. (2021). Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control, 68, 102741. https://doi.org/10.1016/j.bspc.2021.102741

Authors

Zainab D. Altaiee
Laheeb M. Ibrahim

Identifiers

https://doi.org/10.33899/iqjoss.v23i1.62131
Download this PDF file
PDF

Statistics

How to Cite

Altaiee, Z. D. ., & Ibrahim, L. M. . (2026). A Survey on Emotion Recognition with Human Centred Software Engineering. IRAQI JOURNAL OF STATISTICAL SCIENCES, 23(1), 131–140. https://doi.org/10.33899/iqjoss.v23i1.62131