Nhat Truong Pham

(†) denotes equal contribution

(*) denotes correspondance

denotes journal

denotes conference

2025

1. JCheminf

   

   xBitterT5: an explainable transformer-based framework with multimodal inputs for identifying bitter-taste peptides

   Nguyen Doan Hieu Nguyen^† , Nhat Truong Pham^† , Duong Thanh Tran , Leyi Wei , Adeel Malik , and Balachandran Manavalan

   Journal of Cheminformatics, 2025

   Abs Bib HTML PDF Code Website

   Bitter peptides (BPs), derived from the hydrolysis of proteins in food, play a crucial role in both food science and biomedicine by influencing taste perception and participating in various physiological processes. Accurate identification of BPs is essential for understanding food quality and potential health impacts. Traditional machine learning approaches for BP identification have relied on conventional feature descriptors, achieving moderate success but struggling with the complexities of biological sequence data. Recent advances utilizing protein language model embedding and meta-learning approaches have improved the accuracy, but frequently neglect the molecular representations of peptides and lack interpretability. In this study, we propose xBitterT5, a novel multimodal and interpretable framework for BP identification that integrates pretrained transformer-based embeddings from BioT5+ with the combination of peptide sequence and its SELFIES molecular representation. Specifically, incorporating both peptide sequences and their molecular strings, xBitterT5 demonstrates superior performance compared to previous methods on the same benchmark datasets. Importantly, the model provides residue-level interpretability, highlighting chemically meaningful substructures that significantly contribute to its bitterness, thus offering mechanistic insights beyond black-box predictions. A user-friendly web server (https://balalab-skku.org/xBitterT5/) and a standalone version (https://github.com/cbbl-skku-org/xBitterT5/) are freely available to support both computational biologists and experimental researchers in peptide-based food and biomedicine.

   Scientific Contribution

   We propose xBitterT5, a novel multimodal transformer-based framework for the identification of BPs. By utilizing the pretrained BioT5+ model, xBitterT5 effectively extracts high-level representations from both the peptide sequences and their corresponding SELFIES molecular representation. This dual-modality approach enables a more comprehensive understanding of the peptide sequence by leveraging its molecular string, leading to substantial improvements in performance across two benchmark datasets. Additionally, xBitterT5 offers interpretability by identifying key molecular substructures that contribute to bitterness, thereby providing mechanistic insights essential for peptide-based food and drug applications.

   @article{khater2025generative,
     title = {xBitterT5: an explainable transformer-based framework with multimodal inputs for identifying bitter-taste peptides},
     author = {Nguyen, Nguyen Doan Hieu and Pham, Nhat Truong and Tran, Duong Thanh and Wei, Leyi and Malik, Adeel and Manavalan, Balachandran},
     journal = {Journal of Cheminformatics},
     volume = {17},
     number = {1},
     pages = {127},
     year = {2025},
     publisher = {Springer},
     doi = {10.1186/s13321-025-01078-1},
   }

2. JPA

   

   HyPepTox-Fuse: An interpretable hybrid framework for accurate peptide toxicity prediction fusing protein language model-based embeddings with conventional descriptors

   Duong Thanh Tran^† , Nhat Truong Pham^† , Nguyen Doan Hieu Nguyen , Leyi Wei , and Balachandran Manavalan

   Journal of Pharmaceutical Analysis, 2025

   Abs Bib HTML PDF Code Website

   Peptide-based therapeutics hold great promise for the treatment of various diseases; however, their clinical application is often hindered by toxicity challenges. The accurate prediction of peptide toxicity is crucial for designing safe peptide-based therapeutics. While traditional experimental approaches are time-consuming and expensive, computational methods have emerged as viable alternatives, including similarity-based and machine learning (ML)-/deep learning (DL)-based methods. However, existing methods often struggle with robustness and generalizability. To address these challenges, we propose HyPepTox-Fuse, a novel framework that fuses protein language model (PLM)-based embeddings with conventional descriptors. HyPepTox-Fuse integrates ensemble PLM-based embeddings to achieve richer peptide representations by leveraging a cross-modal multi-head attention mechanism and Transformer architecture. A robust feature ranking and selection pipeline further refines conventional descriptors, thus enhancing prediction performance. Our framework outperforms state-of-the-art methods in cross-validation and independent evaluations, offering a scalable and reliable tool for peptide toxicity prediction. Moreover, we conducted a case study to validate the robustness and generalizability of HyPepTox-Fuse, highlighting its effectiveness in enhancing model performance. Furthermore, the HyPepTox-Fuse server is freely accessible at https://balalab-skku.org/HyPepTox-Fuse/ and the source code is publicly available at https://github.com/cbbl-skku-org/HyPepTox-Fuse/. The study thus presents an intuitive platform for predicting peptide toxicity and supports reproducibility through openly available datasets.

   @article{tran2025hypeptox,
     title = {HyPepTox-Fuse: An interpretable hybrid framework for accurate peptide toxicity prediction fusing protein language model-based embeddings with conventional descriptors},
     author = {Tran, Duong Thanh and Pham, Nhat Truong and Nguyen, Nguyen Doan Hieu and Wei, Leyi and Manavalan, Balachandran},
     journal = {Journal of Pharmaceutical Analysis},
     volume = {15},
     number = {8},
     pages = {101410},
     year = {2025},
     publisher = {Elsevier},
     doi = {10.1016/j.jpha.2025.101410},
   }

3. JMB

   

   DOGpred: A Novel Deep Learning Framework for Accurate Identification of Human O-linked Threonine Glycosylation Sites

   Ki Wook Lee^† , Nhat Truong Pham^† , Hye Jung Min , Hyun Woo Park , Ji Won Lee , Han-En Lo , Na Young Kwon , Jimin Seo , Illia Shaginyan , Heeje Cho , Leyi Wei , Balachandran Manavalan , and Young-Jun Jeon

   Journal of Molecular Biology, 2025

   Abs Bib HTML Code

   O-linked glycosylation is a crucial post-transcriptional modification that regulates protein function and biological processes. Dysregulation of this process is associated with various diseases, underscoring the need to accurately identify O-linked glycosylation sites on proteins. Current experimental methods for identifying O-linked threonine glycosylation (OTG) sites are often complex and costly. Consequently, developing computational tools that predict these sites based on protein features is crucial. Such tools can complement experimental approaches, enhancing our understanding of the role of OTG dysregulation in diseases and uncovering potential therapeutic targets. In this study, we developed DOGpred, a deep learning-based predictor for precisely identifying human OTGs using high-latent feature representations. Initially, we extracted nine different conventional feature descriptors (CFDs) and nine pre-trained protein language model (PLM)-based embeddings. Notably, each feature was encoded as a 2D tensor, capturing both the sequential and inherent feature characteristics. Subsequently, we designed a stacked convolutional neural network (CNN) module to learn spatial feature representations from CFDs and a stacked recurrent neural network (RNN) module to learn temporal feature representations from PLM-based embeddings. These features were integrated using attention-based fusion mechanisms to generate high-level feature representations for final classification. Ablation analysis and independent tests demonstrated that the optimal model (DOGpred), employing a stacked 1D CNN and a stacked attention-based RNN module with cross-attention feature fusion, achieved the best performance on the training dataset and significantly outperformed machine learning-based single-feature models and state-of-the-art methods on independent datasets. Furthermore, DOGpred is publicly available at https://github.com/JeonRPM/DOGpred/ for free access and usage.

   @article{lee2025dogpred,
     title = {DOGpred: A Novel Deep Learning Framework for Accurate Identification of Human O-linked Threonine Glycosylation Sites},
     author = {Lee, Ki Wook and Pham, Nhat Truong and Min, Hye Jung and Park, Hyun Woo and Lee, Ji Won and Lo, Han-En and Kwon, Na Young and Seo, Jimin and Shaginyan, Illia and Cho, Heeje and Wei, Leyi and Manavalan, Balachandran and Jeon, Young-Jun},
     journal = {Journal of Molecular Biology},
     volume = {437},
     number = {6},
     pages = {168977},
     year = {2025},
     publisher = {Elsevier},
     doi = {10.1016/j.jmb.2025.168977},
   }

2024

1. LPM@ACL

   

   Mol2Lang-VLM: Vision- and Text-Guided Generative Pre-trained Language Models for Advancing Molecule Captioning through Multimodal Fusion

   Duong Thanh Tran^† , Nhat Truong Pham^† , Nguyen Doan Hieu Nguyen , and Balachandran Manavalan

   In Proceedings of the 1st Workshop on Language + Molecules (L+M 2024), 2024

   Abs Bib HTML PDF Code

   This paper introduces Mol2Lang-VLM, an enhanced method for refining generative pre-trained language models for molecule captioning using multimodal features to achieve more accurate caption generation. Our approach leverages the encoder and decoder blocks of the Transformer-based architecture by introducing third sub-layers into both. Specifically, we insert sub-layers in the encoder to fuse features from SELFIES strings and molecular images, while the decoder fuses features from SMILES strings and their corresponding descriptions. Moreover, cross multi-head attention is employed instead of common multi-head attention to enable the decoder to attend to the encoder’s output, thereby integrating the encoded contextual information for better and more accurate caption generation. Performance evaluation on the CheBI-20 and L+M-24 benchmark datasets demonstrates Mol2Lang-VLM’s superiority, achieving higher accuracy and quality in caption generation compared to existing methods. Our code and pre-processed data are available at https://github.com/nhattruongpham/mol-lang-bridge/tree/mol2lang/.

   @inproceedings{tran2024mollangvlm,
     title = {Mol2Lang-{VLM}: Vision- and Text-Guided Generative Pre-trained Language Models for Advancing Molecule Captioning through Multimodal Fusion},
     author = {Tran, Duong Thanh and Pham, Nhat Truong and Nguyen, Nguyen Doan Hieu and Manavalan, Balachandran},
     booktitle = {Proceedings of the 1st Workshop on Language + Molecules L+M 2024},
     pages = {97--102},
     year = {2024},
     doi = {10.18653/v1/2024.langmol-1.12},
   }

2. LPM@ACL

   

   Lang2Mol-Diff: A Diffusion-Based Generative Model for Language-to-Molecule Translation Leveraging SELFIES Molecular String Representation

   Nguyen Doan Hieu Nguyen^† , Nhat Truong Pham^† , Duong Thanh Tran , and Balachandran Manavalan

   In Proceedings of the 1st Workshop on Language + Molecules (L+M 2024), 2024

   Abs Bib HTML PDF Code

   Generating de novo molecules from textual descriptions is challenging due to potential issues with molecule validity in SMILES representation and limitations of autoregressive models. This work introduces Lang2Mol-Diff, a diffusion-based language-to-molecule generative model using the SELFIES representation. Specifically, Lang2Mol-Diff leverages the strengths of two state-of-the-art molecular generative models: BioT5 and TGM-DLM. By employing BioT5 to tokenize the SELFIES representation, Lang2Mol-Diff addresses the validity issues associated with SMILES strings. Additionally, it incorporates a text diffusion mechanism from TGM-DLM to overcome the limitations of autoregressive models in this domain. To the best of our knowledge, this is the first study to leverage the diffusion mechanism for text-based de novo molecule generation using the SELFIES molecular string representation. Performance evaluation on the L+M-24 benchmark dataset shows that Lang2Mol-Diff outperforms all existing methods for molecule generation in terms of validity. Our code and pre-processed data are available at https://github.com/nhattruongpham/mol-lang-bridge/tree/lang2mol/.

   @inproceedings{nguyen2024langmoldiff,
     title = {Lang2Mol-Diff: A Diffusion-Based Generative Model for Language-to-Molecule Translation Leveraging SELFIES Molecular String Representation},
     author = {Nguyen, Nguyen Doan Hieu and Pham, Nhat Truong and Tran, Duong Thanh and Manavalan, Balachandran},
     booktitle = {Proceedings of the 1st Workshop on Language + Molecules L+M 2024},
     pages = {128--134},
     year = {2024},
     doi = {10.18653/v1/2024.langmol-1.15},
   }

3. CIBM

   

   HOTGpred: Enhancing human O-linked threonine glycosylation prediction using integrated pretrained protein language model-based features and multi-stage feature selection approach

   Nhat Truong Pham^† , Ying Zhang^† , Rajan Rakkiyappan , and Balachandran Manavalan

   Computers in Biology and Medicine, 2024

   Abs Bib HTML Website

   O-linked glycosylation is a complex post-translational modification (PTM) in human proteins that plays a critical role in regulating various cellular metabolic and signaling pathways. In contrast to N-linked glycosylation, O-linked glycosylation lacks specific sequence features and maintains an unstable core structure. Identifying O-linked threonine glycosylation sites (OTGs) remains challenging, requiring extensive experimental tests. While bioinformatics tools have emerged for predicting OTGs, their reliance on limited conventional features and absence of well-defined feature selection strategies limit their effectiveness. To address these limitations, we introduced HOTGpred (Human O-linked Threonine Glycosylation predictor), employing a multi-stage feature selection process to identify the optimal feature set for accurately identifying OTGs. Initially, we assessed 25 different feature sets derived from various pretrained protein language model (PLM)-based embeddings and conventional feature descriptors using nine classifiers. Subsequently, we integrated the top five embeddings linearly and determined the most effective scoring function for ranking hybrid features, identifying the optimal feature set through a process of sequential forward search. Among the classifiers, the extreme gradient boosting (XGBT)-based model, using the optimal feature set (HOTGpred), achieved 92.03% accuracy on the training dataset and 88.25% on the balanced independent dataset. Notably, HOTGpred significantly outperformed the current state-of-the-art methods on both the balanced and imbalanced independent datasets, demonstrating its superior prediction capabilities. Additionally, SHapley Additive exPlanations (SHAP) and ablation analyses were conducted to identify the features contributing most significantly to HOTGpred. Finally, we developed an easy-to-navigate web server, accessible at https://balalab-skku.org/HOTGpred/, to support glycobiologists in their research on glycosylation structure and function.

   @article{pham2024hotgpred,
     title = {HOTGpred: Enhancing human O-linked threonine glycosylation prediction using integrated pretrained protein language model-based features and multi-stage feature selection approach},
     author = {Pham, Nhat Truong and Zhang, Ying and Rakkiyappan, Rajan and Manavalan, Balachandran},
     journal = {Computers in Biology and Medicine},
     volume = {179},
     pages = {108859},
     year = {2024},
     publisher = {Elsevier},
     doi = {10.1016/j.compbiomed.2024.108859},
   }

4. JMB

   

   mACPpred 2.0: Stacked Deep Learning for Anticancer Peptide Prediction with Integrated Spatial and Probabilistic Feature Representations

   Vinoth Kumar Sangaraju^† , Nhat Truong Pham^† , Leyi Wei , Xue Yu , and Balachandran Manavalan

   Journal of Molecular Biology, 2024

   Abs Bib HTML Code Website

   Anticancer peptides (ACPs), naturally occurring molecules with remarkable potential to target and kill cancer cells. However, identifying ACPs based solely from their primary amino acid sequences remains a major hurdle in immunoinformatics. In the past, several web-based machine learning (ML) tools have been proposed to assist researchers in identifying potential ACPs for further testing. Notably, our meta-approach method, mACPpred, introduced in 2019, has significantly advanced the field of ACP research. Given the exponential growth in the number of characterized ACPs, there is now a pressing need to create an updated version of mACPpred. To develop mACPpred 2.0, we constructed an up-to-date benchmarking dataset by integrating all publicly available ACP datasets. We employed a large-scale of feature descriptors, encompassing both conventional feature descriptors and advanced pre-trained natural language processing (NLP)-based embeddings. We evaluated their ability to discriminate between ACPs and non-ACPs using eleven different classifiers. Subsequently, we employed a stacked deep learning (SDL) approach, incorporating 1D convolutional neural network (1D CNN) blocks and hybrid features. These features included the top seven performing NLP-based features and 90 probabilistic features, allowing us to identify hidden patterns within these diverse features and improve the accuracy of our ACP prediction model. This is the first study to integrate spatial and probabilistic feature representations for predicting ACPs. Rigorous cross-validation and independent tests conclusively demonstrated that mACPpred 2.0 not only surpassed its predecessor (mACPpred) but also outperformed the existing state-of-the-art predictors, highlighting the importance of advanced feature representation capabilities attained through SDL. To facilitate widespread use and accessibility, we have developed a user-friendly for mACPpred 2.0, available at https://balalab-skku.org/mACPpred2/.

   @article{sangaraju2024macppred,
     title = {mACPpred 2.0: Stacked Deep Learning for Anticancer Peptide Prediction with Integrated Spatial and Probabilistic Feature Representations},
     author = {Sangaraju, Vinoth Kumar and Pham, Nhat Truong and Wei, Leyi and Yu, Xue and Manavalan, Balachandran},
     journal = {Journal of Molecular Biology},
     volume = {436},
     number = {17},
     pages = {168687},
     year = {2024},
     publisher = {Elsevier},
     doi = {10.1016/j.jmb.2024.168687},
   }

5. MTNA

   

   ac4C-AFL: A high-precision identification of human mRNA N4-acetylcytidine sites based on adaptive feature representation learning

   Nhat Truong Pham , Annie Terrina Terrance , Young-Jun Jeon , Rajan Rakkiyappan , and Balachandran Manavalan

   Molecular Therapy-Nucleic Acids, 2024

   Abs Bib HTML PDF Website

   RNA N4-acetylcytidine (ac4C) is a highly conserved RNA modification that plays a crucial role in controlling mRNA stability, processing, and translation. Consequently, accurate identification of ac4C sites across the genome is critical for understanding gene expression regulation mechanisms. In this study, we have developed ac4C-AFL, a bioinformatics tool that precisely identifies ac4C sites from primary RNA sequences. In ac4C-AFL, we identified the optimal sequence length for model building and implemented an adaptive feature representation strategy that is capable of extracting the most representative features from RNA. To identify the most relevant features, we proposed a novel ensemble feature importance scoring strategy to rank features effectively. We then used this information to conduct the sequential forward search, which individually determine the optimal feature set from the 16 sequence-derived feature descriptors. Utilizing these optimal feature descriptors, we constructed 176 baseline models using 11 popular classifiers. The most efficient baseline models were identified using the two-step feature selection approach, whose predicted scores were integrated and trained with the appropriate classifier to develop the final prediction model. Our rigorous cross-validations and independent tests demonstrate that ac4C-AFL surpasses contemporary tools in predicting ac4C sites. Moreover, we have developed a publicly accessible web server at https://balalab-skku.org/ac4C-AFL/.

   @article{pham2024ac4c,
     title = {ac4C-AFL: A high-precision identification of human mRNA N4-acetylcytidine sites based on adaptive feature representation learning},
     author = {Pham, Nhat Truong and Terrance, Annie Terrina and Jeon, Young-Jun and Rakkiyappan, Rajan and Manavalan, Balachandran},
     journal = {Molecular Therapy-Nucleic Acids},
     volume = {35},
     number = {2},
     pages = {102192},
     year = {2024},
     publisher = {Elsevier},
     doi = {10.1016/j.omtn.2024.102192},
   }

6. BiB

   

   H2Opred: a robust and efficient hybrid deep learning model for predicting 2’-O-methylation sites in human RNA

   Nhat Truong Pham , Rajan Rakkiyappan , Jongsun Park , Adeel Malik , and Balachandran Manavalan

   Briefings in Bioinformatics, 2024

   Abs Bib HTML PDF Website Media

   2’-O-methylation (2OM) is the most common post-transcriptional modification of RNA. It plays a crucial role in RNA splicing, RNA stability and innate immunity. Despite advances in high-throughput detection, the chemical stability of 2OM makes it difficult to detect and map in messenger RNA. Therefore, bioinformatics tools have been developed using machine learning (ML) algorithms to identify 2OM sites. These tools have made significant progress, but their performances remain unsatisfactory and need further improvement. In this study, we introduced H2Opred, a novel hybrid deep learning (HDL) model for accurately identifying 2OM sites in human RNA. Notably, this is the first application of HDL in developing four nucleotide-specific models [adenine (A2OM), cytosine (C2OM), guanine (G2OM) and uracil (U2OM)] as well as a generic model (N2OM). H2Opred incorporated both stacked 1D convolutional neural network (1D-CNN) blocks and stacked attention-based bidirectional gated recurrent unit (Bi-GRU-Att) blocks. 1D-CNN blocks learned effective feature representations from 14 conventional descriptors, while Bi-GRU-Att blocks learned feature representations from five natural language processing-based embeddings extracted from RNA sequences. H2Opred integrated these feature representations to make the final prediction. Rigorous cross-validation analysis demonstrated that H2Opred consistently outperforms conventional ML-based single-feature models on five different datasets. Moreover, the generic model of H2Opred demonstrated a remarkable performance on both training and testing datasets, significantly outperforming the existing predictor and other four nucleotide-specific H2Opred models. To enhance accessibility and usability, we have deployed a user-friendly web server for H2Opred, accessible at https://balalab-skku.org/H2Opred/. This platform will serve as an invaluable tool for accurately predicting 2OM sites within human RNA, thereby facilitating broader applications in relevant research endeavors.

   @article{pham2024h2opred,
     title = {H2Opred: a robust and efficient hybrid deep learning model for predicting 2'-O-methylation sites in human RNA},
     author = {Pham, Nhat Truong and Rakkiyappan, Rajan and Park, Jongsun and Malik, Adeel and Manavalan, Balachandran},
     journal = {Briefings in Bioinformatics},
     volume = {25},
     number = {1},
     pages = {bbad476},
     year = {2024},
     publisher = {OxfOxford University Pressord},
     doi = {10.1093/bib/bbad476},
   }

7. BiB

   

   Advancing the accuracy of SARS-CoV-2 phosphorylation site detection via meta-learning approach

   Nhat Truong Pham^† , Le Thi Phan^† , Jimin Seo , Yeonwoo Kim , Minkyung Song , Sukchan Lee , Young-Jun Jeon , and Balachandran Manavalan

   Briefings in Bioinformatics, 2024

   Abs Bib HTML PDF Website Media Interview

   The worldwide appearance of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has generated significant concern and posed a considerable challenge to global health. Phosphorylation is a common post-translational modification that affects many vital cellular functions and is closely associated with SARS-CoV-2 infection. Precise identification of phosphorylation sites could provide more in-depth insight into the processes underlying SARS-CoV-2 infection and help alleviate the continuing coronavirus disease 2019 (COVID-19) crisis. Currently, available computational tools for predicting these sites lack accuracy and effectiveness. In this study, we designed an innovative meta-learning model, Meta-Learning for Serine/Threonine Phosphorylation (MeL-STPhos), to precisely identify protein phosphorylation sites. We initially performed a comprehensive assessment of 29 unique sequence-derived features, establishing prediction models for each using 14 renowned machine learning methods, ranging from traditional classifiers to advanced deep learning algorithms. We then selected the most effective model for each feature by integrating the predicted values. Rigorous feature selection strategies were employed to identify the optimal base models and classifier(s) for each cell-specific dataset. To the best of our knowledge, this is the first study to report two cell-specific models and a generic model for phosphorylation site prediction by utilizing an extensive range of sequence-derived features and machine learning algorithms. Extensive cross-validation and independent testing revealed that MeL-STPhos surpasses existing state-of-the-art tools for phosphorylation site prediction. We also developed a publicly accessible platform at https://balalab-skku.org/MeL-STPhos/. We believe that MeL-STPhos will serve as a valuable tool for accelerating the discovery of serine/threonine phosphorylation sites and elucidating their role in post-translational regulation.

   @article{pham2024advancing,
     title = {Advancing the accuracy of SARS-CoV-2 phosphorylation site detection via meta-learning approach},
     author = {Pham, Nhat Truong and Phan, Le Thi and Seo, Jimin and Kim, Yeonwoo and Song, Minkyung and Lee, Sukchan and Jeon, Young-Jun and Manavalan, Balachandran},
     journal = {Briefings in Bioinformatics},
     volume = {25},
     number = {1},
     pages = {bbad433},
     year = {2024},
     publisher = {OxfOxford University Pressord},
     doi = {10.1093/bib/bbad433},
   }

2023

1. ESWA

   

   Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognition

   Nhat Truong Pham , Duc Ngoc Minh Dang , Ngoc Duy Nguyen , Thanh Thi Nguyen , Hai Nguyen , Balachandran Manavalan , Chee Peng Lim , and Sy Dzung Nguyen

   Expert Systems with Applications, 2023

   Abs arXiv Bib HTML PDF Code

   Recently, speech emotion recognition (SER) has become an active research area in speech processing, particularly with the advent of deep learning (DL). Numerous DL-based methods have been proposed for SER. However, most of the existing DL-based models are complex and require a large amounts of data to achieve a good performance. In this study, a new framework of deep attention-based dilated convolutional-recurrent neural networks coupled with a hybrid data augmentation method was proposed for addressing SER tasks. The hybrid data augmentation method constitutes an upsampling technique for generating more speech data samples based on the traditional and generative adversarial network approaches. By leveraging both convolutional and recurrent neural networks in a dilated form along with an attention mechanism, the proposed DL framework can extract high-level representations from three-dimensional log Mel spectrogram features. Dilated convolutional neural networks acquire larger receptive fields, whereas dilated recurrent neural networks overcome complex dependencies as well as the vanishing and exploding gradient issues. Furthermore, the loss functions are reconfigured by combining the SoftMax loss and the center-based losses to classify various emotional states. The proposed framework was implemented using the Python programming language and the TensorFlow deep learning library. To validate the proposed framework, the EmoDB and ERC benchmark datasets, which are imbalanced and/or small datasets, were employed. The experimental results indicate that the proposed framework outperforms other related state-of-the-art methods, yielding the highest unweighted recall rates of 88.03 ± 1.39 (%) and 66.56 ± 0.67 (%) for the EmoDB and ERC datasets, respectively.

   @article{pham2023hybrid,
     title = {Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognition},
     author = {Pham, Nhat Truong and Dang, Duc Ngoc Minh and Nguyen, Ngoc Duy and Nguyen, Thanh Thi and Nguyen, Hai and Manavalan, Balachandran and Lim, Chee Peng and Nguyen, Sy Dzung},
     journal = {Expert Systems with Applications},
     pages = {120608},
     year = {2023},
     publisher = {Elsevier},
     doi = {10.1016/j.eswa.2023.120608},
   }

2. ESWA

   

   Fruit-CoV: An efficient vision-based framework for speedy detection and diagnosis of SARS-CoV-2 infections through recorded cough sounds

   Long H. Nguyen^† , Nhat Truong Pham^†* , Van Huong Do , Liu Tai Nguyen , Thanh Tin Nguyen , Hai Nguyen , Ngoc Duy Nguyen , Thanh Thi Nguyen , Sy Dzung Nguyen , Asim Bhatti , and Chee Peng Lim

   Expert Systems with Applications, 2023

   Abs arXiv Bib HTML Code

   COVID-19 is an infectious disease caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). This deadly virus has spread worldwide, leading to a global pandemic since March 2020. A recent variant of SARS-CoV-2 named Delta is intractably contagious and responsible for more than four million deaths globally. Therefore, developing an efficient self-testing service for SARS-CoV-2 at home is vital. In this study, a two-stage vision-based framework, namely Fruit-CoV, is introduced for detecting SARS-CoV-2 infections through recorded cough sounds. Specifically, audio signals are converted into Log-Mel spectrograms, and the EfficientNet-V2 network is used to extract their visual features in the first stage. In the second stage, 14 convolutional layers extracted from the large-scale Pretrained Audio Neural Networks for audio pattern recognition (PANNs) and the Wavegram-Log-Mel-CNN are employed to aggregate feature representations of the Log-Mel spectrograms and the waveform. Finally, the combined features are used to train a binary classifier. In this study, a dataset provided by the AICovidVN 115M Challenge is employed for evaluation. It includes 7,371 recorded cough sounds collected throughout Vietnam, India, and Switzerland. Experimental results indicate that the proposed model achieves an Area Under the Receiver Operating Characteristic Curve (AUC) score of 92.8% and ranks first on the final leaderboard of the AICovidVN 115M Challenge. Our code is publicly available.

   @article{nguyen2023fruit,
     title = {Fruit-CoV: An efficient vision-based framework for speedy detection and diagnosis of SARS-CoV-2 infections through recorded cough sounds},
     author = {Nguyen, Long H. and Pham, Nhat Truong and Do, Van Huong and Nguyen, Liu Tai and Nguyen, Thanh Tin and Nguyen, Hai and Nguyen, Ngoc Duy and Nguyen, Thanh Thi and Nguyen, Sy Dzung and Bhatti, Asim and Lim, Chee Peng},
     journal = {Expert Systems with Applications},
     volume = {213},
     pages = {119212},
     year = {2023},
     publisher = {Elsevier},
     doi = {10.1016/j.eswa.2022.119212},
   }