Research
Multimodal Assessment of Neonatal Pain Using Computer Vision
Abstract
Infants receiving care in the Neonatal Intensive Care Unit (NICU) experience several painful procedures during their hospitalization. Assessing neonatal pain is difficult because the current standard for assessment is subjective, inconsistent, and discontinuous. The intermittent and inconsistent assessment can induce poor treatment and, therefore, cause serious longterm outcomes. The main aim of this project is to develop a robust and comprehensive automatic system that generates a standardized pain assessment comparable to those obtained by conventional nurse-derived pain scores. The continuous monitoring of pain, using affordable, non-invasive, and easily integrable devices, provides immediate pain detection and intervention, and therefore, contribute to improved long term outcomes; i.e., reduce the outcomes of under- and over-treatment. It can also decrease caregivers’ bias and assessment burden.
While further research is needed, the preliminary results of our research showed that the automatic assessment of neonatal pain is a viable and more efficient alternative to the manual assessment.
Engineering Team:
Dr. Yu Sun, Professor, Computer Science and Engineering, USF
Dr. Dmitry Goldgof, Professor, Computer Science and Engineering, USF
Jacqueline Hausmann, PhD Student, Computer Science and Engineering, USF
Md Imran Hossain, PhD Student, Computer Science and Engineering, USF
Jiayi Wang, PhD Student, Computer Science and Engineering, USF
Medical Team:
Dr. Thao Ho, Assistant Professor, College of Medicine Pediatrics, USF Health, USF
Dr. Stephanie Prescott, Assistant Professor, College of Nursing, USF Health, USF
Dr. Denise Maguire, Associate Professor, College of Nursing, USF Health, USF
Dr. Yangxin Huang, Professor, College of Public Health, USF Health, USF
Marcia Kneusel, RN, Clinical Research Nurse, College of Medicine Pediatrics, USF Health,
USF
Collaborators:
Dr. Md Sirajus Salekin, Applied Scientist, Amazon
Dr. Ghada Zamzmi, Research Fellow, NLM, NIH
Dr. Peter Mouton, Director and Chief Scientific Officer, SRC Biosciences
Dr. Mark Last, Professor and Director of Data Science Research Center, Ben-Gurion
University of the Negev, Israel
Dr. Kanwaljeet S. Anand, Professor of Pediatrics, Stanford University, CA, United
States
Previous Team Members:
Dr. Terri Ashmeade, Professor, College of Medicine Pediatrics, USF Health, USF
Dr. Rangachar Kasturi, Professor, CSE, USF
Chih-Yun Pai, Master's Student, CSE, USF
Patents
- US-11631280-B2, "System and Method for Multimodal Spatiotemporal Pain Assessment", April 2023.
- US-20230309915-A1, "System and Method for Attentional Multimodal Pain Estimation", October 2023.
- US-11202604-B2, "Comprehensive and Context-sensitive Neonatal Pain Assessment System and Methods Using Multiple Modalities", December 2021.
- US-20210052215-A1, "System and Method for Multimodal Spatiotemporal Pain Assessment", Feburary 2021.
- US-20210030354-A1, "Neonatal Pain Identification From Neonatal Facial Expressions", Feburary 2021.
- US-10827973-B1, "Machine-based Infants Pain Assessment Tool", November 2020.
- US-20190320974-A1, "Comprehensive and Context-sensitive Neonatal Pain Assessment System and Methods Using Multiple Modalities", October 2020.
Published Papers
Explainable Neonatal Pain Assessment via Influence Function ModificationMd Imran Hossain, Ghada Zamzmi, Peter R Mouton, Yu Sun, and Dmitry Goldgof
IEEE Engineering in Medicine and Biology Society (EMBC24), 2024
Accurate Neonatal Face Detection for Improved Pain Classification in the Challenging
NICU Setting
Jacqueline Hausmann, Md Sirajus Salekin, Ghada Zamzmi, Peter R Mouton, Stephanie M.Prescott,
Yu Sun,and Dmitry Goldgof
IEEE Access, 2024
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Md Sirajus Salekin, Yu Sun, and Dmitry
Goldgof
ACM Computing Surveys, 2023
Neonatal pain assessment: Do we have the right tools?
Amelia Llerena, Krystal Tran, Danyal Choudhary, Jacqueline Hausmann, Dmitry Goldgof,
Yu Sun, and Stephanie M.Prescott
Frontier in Pediatrics, 2023
Enhancing Neonatal Pain Assessment Transparency via Explanatory Training Examples
Identification
Md Imran Hossain, Ghada Zamzmi, Peter R Mouton, Yu Sun, and Dmitry Goldgof
2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS)
Attentional Generative Multimodal Network for Neonatal Postoperative Pain Estimation
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Peter R. Mouton, Kanwaljeet J. S.
Anand, Terri Ashmeade, Stephanie Prescott, Yangxin Huang, and Yu Sun
Medical Image Computing and Computer Assisted Intervention (MICCAI), 2022
Robust Neonatal Face Detection in Real-world Clinical Settings
Jacqueline Hausmann, Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, and Yu Sun
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshops, 2021
Pattern Recognition in Vital Signs Using Spectrograms
Sidharth Srivatsav Sribhashyam, Md Sirajus Salekin, Dmitry Goldgof, Ghada Zamzmi,
Mark Last, and Yu Sun
IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2021
Future Roles of Artificial Intelligence in Early Pain Management of Newborns
Md Sirajus Salekin, Peter R Mouton, Ghada Zamzmi, Raj Patel, Dmitry Goldgof, Marcia
Kneusel, Sammie L Elkins, Eileen Murray, Mary E Coughlin, Denise Maguire, Thao Ho,
and Yu Sun
Paediatric and Neonatal Pain, 2021
Multimodal Neonatal Procedural and Postoperative Pain Dataset
Md Sirajus Salekin, Ghada Zamzmi, Jacqueline Hausmann, Dmitry Goldgof, Rangachar Kasturi,
Marcia Kneusel, Terri Ashmeade, Thao Ho, Yu Sun
Data in Brief, 2021
Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain
Assessment
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
Computers in Biology and Medicine, 2021
First Investigation Into the Use of Deep Learning for Continuous Assessment of Neonatal
Postoperative Pain
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE International Conference on Automatic Face and Gesture Recognition (FG), 2020
Harnessing the Power of Deep Learning Methods in Healthcare: Neonatal Pain Assessment
from Crying Sound
Md Sirajus Salekin, Ghada Zamzmi, Rahul Paul, Dmitry Goldgof, Rangachar Kasturi, Thao
Ho, Yu Sun
IEEE Healthcare Innovations and Point of Care Technologies Conference (HI-POCT), 2019
Multi-Channel Neural Network for Assessing Neonatal Pain from Videos
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2019
Convolutional Neural Network for Neonatal Pain Assessment
Ghada Zamzmi, Rahul Paul, Md Sirajus Salekin, Dmitry Goldgof, Rangachar Kasturi, Thao
Ho, Yu Sun
IEEE Transactions on Biometrics, Behavior, and Identity Science, 2019
Pain Assessment From Facial Expression: Neonatal Convolutional Neural Network (N-CNN)
Ghada Zamzmi, Rahul Paul, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu Sun
International Joint Conference on Neural Networks (IJCNN), 2019
A Comprehensive and Context-Sensitive Neonatal Pain Assessment Using Computer Vision
Ghada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu
Sun
IEEE Transactions on Affective Computing, 2019
Toward Ubiquitous Assessment of Neonates' Health Condition
Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Yu Sun
UbiComp, 2018
Automatic Infants’ Pain Assessment by Dynamic Facial Representation: Effects of Profile
View, Gestational Age, Gender, and Race
Ruicong Zhi, Ghada Zamzmi, Dmitry Goldgof, Terri Ashmeade, Yu Sun
Journal of Clinical Medicine, 2018
Infants' Pain Recognition Based on Facial Expression: Dynamic Hybrid Descriptions
Ruicong Zhi, Ghada Zamzmi, Dmitry Goldgof, Terri Ashmeade, Tingting Li, Yu Sun
IEICE Transactions on Information and Systems, 2018
A Review of Automated Pain Assessment in Infants: Features, Classification Tasks,
and Databases
Ghada Zamzmi, Rangachar Kasturi, Dmitry Goldgof, Ruicong Zhi, Terri Ashmeade, Yu Sun
IEEE Reviews in Biomedical Engineering, 2018
Automated Pain Assessment in NeonatesGhada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Yu Sun, Terri Ashmeade
Scandinavian Conference on Image Analysis (SCIA), 2017
An Approach for Automated Multimodal Analysis of Infants' PainGhada Zamzmi, Chih-Yun Pai, Dmitry Goldgof, Rangachar Kasturi, Terri Ashmeade, Yu
Sun
International Conference on Pattern Recognition (ICPR), 2016
Pain Assessment in Infants: Towards Spotting Pain Expression Based on Infants' Facial
Strain
Ghada Zamzami, Gabriel Ruiz, Dmitry Goldgof, Rangachar Kasturi, Yu Sun, Terri Ashmeade
IEEE International Conference and Workshops on Automatic Face and Gesture Recognition
(FG), 2015
Media Coverage
- Jun 15, 2021 - USF Research and Innovation Newsroom 2021
- Jun 14, 2017 - USF Magazine Summer 2017
- Mar 24, 2017 - USF Health
- Feb 22, 2017 - FOX 13 News
Talk / Presentation
Aug 06, 2019 - Dr. Dmitry Goldgof, A joint seminar of the Data Science Research Center
and the Department of Biomedical Engineering at Ben-Gurion University of Negev, Israel
on "Healthcare in the Age of AI and Deep Learning: Automatic Assessment of Neonatal
Pain"
Jun 18, 2019 - Dr. Dmitry Goldgof, Weekly Idea/Research Collabaration Meeting at Johns
Hopkins All Children's Hospital, St. Petersburg, FL, USA, on "Automatic Neonatal Pain
Detection"
Mar 26, 2019 - Dr. Dmitry Goldgof, High Profile Talk at University of South Florida,
Tampa, FL, USA, on "Healthcare in the Age of AI and Deep Learning: Automatic Assessment
of Neonatal Pain"
Short Description
This pilot dataset (USF-MNPAD-I) is a part of a neonatal pain assessment project, which aims to develop new algorithms and techniques for the automatic monitoring of neonatal pain. We are currently involved in an ongoing effort to collect a large multimodal pain dataset from newborns and premature neonates. USF-MNPAD-I dataset contains recorded subjects in the Neonatal Intensive Care Unit (NICU) at Tampa General Hospital. The neonates were recorded during baseline and while undergoing procedural painful procedures (e.g., heel lancing and immunization). In addition to the recorded raw videos, this dataset contains vital signs data as well as ground truth labels documented by two trained nurses.
Release of Dataset
To advance the state-of-the-art in neonatal pain assessment, portion of the dataset
will be made available to researchers on a case-by-case basis. All requests for USF-MNPAD-I
must be submitted in writing and sent to the Principal Investigators of this project.
The requestor must sign this document and agree to observe the restrictions listed
below. In addition to other possible remedies, failure to observe these restrictions
may result in access being denied for any other portion of the database. Current installments
of the database may be made available to researchers via a protected internet site
or on CD. There will be no charge for imagery made available and downloaded via the
Internet.
* Please note that initially shareable subjects from the USF-MNPAD-I will be shared.
A larger dataset, will be made available, as per request, for research use.
Citations
Any manuscript developed completely or partially using USF-MNPAD-I dataset must include the following citations:
Multimodal Neonatal Procedural and Postoperative Pain Dataset
Md Sirajus Salekin, Ghada Zamzmi, Jacqueline Hausmann, Dmitry Goldgof, Rangachar Kasturi,
Marcia Kneusel, Terri Ashmeade, Thao Ho, Yu Sun
Data in Brief, 2021
Multimodal Spatio-Temporal Deep Learning Approach for Neonatal Postoperative Pain
Assessment
Md Sirajus Salekin, Ghada Zamzmi, Dmitry Goldgof, Rangachar Kasturi, Thao Ho, Yu Sun
Computers in Biology and Medicine, 2021
Contact
To get the datasets, you need to contact the following person with a full description of why you need it.
Jacqueline Hausmann
Department of CSE, University of South Florida, United States
hausmannj@usf.edu
Our Sponsors
