  Applications of single-cell RNA sequencing in various biomedical research
areas have been blooming. This new technology provides unprecedented
opportunities to study disease heterogeneity at the cellular level. However,
unique characteristics of scRNA-seq data, including large dimensionality, high
dropout rates, and possibly batch effects, bring great difficulty into the
analysis of such data. Not appropriately addressing these issues obstructs true
scientific discovery. Herein, we propose a unified Regularized Zero-inflated
Mixture Model framework designed for scRNA-seq data (RZiMM-scRNA) to
simultaneously detect cell subgroups and identify gene differential expression
based on a developed importance score, accounting for both dropouts and batch
effects. We conduct extensive simulation studies in which we evaluate the
performance of RZiMM-scRNA and compare it with several popular methods,
including Seurat, SC3, K-Means, and Hierarchical Clustering. Simulation results
show that RZiMM-scRNA demonstrates superior clustering performance and enhanced
biomarker detection accuracy compared to alternative methods, especially when
cell subgroups are less distinct, verifying the robustness of our method. Our
empirical investigations focus on two brain tumor studies dealing with
astrocytoma of various grades, including the most malignant of all brain
tumors, glioblastoma multiforme (GBM). Our goal is to delineate cell
heterogeneity and identify driving biomarkers associated with these tumors.
Notably, RZiMM-scNRA successfully identifies a small group of oligodendrocyte
cells which has drawn much attention in biomedical literature on brain cancers.
