Author: Changjiang Xu, Delaram Pouyabahar, Veronique Voisin, and Gary D. Bader
Date: 2026-04-07

# Single-cell differential expression analysis with FLASHMM

#### Changjiang Xu, Delaram Pouyabahar, Veronique Voisin, and Gary D. Bader

#### April 07, 2026

Abstract

FLASHMM is a package for single-cell differential expression analysis using linear mixed-effects models (LMMs). Mixed-effects models have become a powerful tool in single-cell studies due to their ability to model intra-subject correlation and inter-subject variability. However, classic LMM estimation methods face limitations in computational speed and memory usage when applied to large-scale single-cell transcriptomics data. The FLASHMM package provides a fast and scalable approach to address scalability and memory efficiency in LMM fitting. This vignette describes the methods for LMM estimation and hypothesis testing, as well as the R functions for implementing these methods, and demonstrates the use of the package through an example.

# 1 FLASHMM

## 1.1 Linear mixed-effects model

Consider the linear mixed-effects model (LMM) as expressed below (Searle, Casella, and McCulloch 2006) \[\begin{equation}\label{eq:lmm} y = X\beta + Zb + \epsilon, \end{equation}\] where \(y\) is an \(n\times 1\) vector of observed responses, \(X\) is an \(n\times p\) design matrix for fixed effects \(\beta\), \(Z\) is an \(n\times q\) design matrix for random effects \(b\), and \(\epsilon\) is a vector of residual errors. The term of random effects may be a combination of various components, \[ Zb = Z_1 b_1 + \cdots + Z_K b_K, \] where \(Z=[Z_1,\ldots,Z_K]\), \(b=[b^T_1,\ldots,b^T_K]^T\), \(K\) is the number of random-effect components, and \(Z_k\) is an \(n\times q_k\) design matrix for the \(k\)-th random-effect component. The superscript \(T\) denotes a transpose of vector or matrix. The basic assumptions are as follows:

1. The design matrix \(X\) is of full rank, satisfying conditions of estimability for the parameters;
2. The random vectors \(b_k\) and \(\epsilon\) are independent and follow a normal distribution, \[b_k \sim N(\mathbf{0}, \sigma^2_k I_{q_k})\quad\text{and}\quad\epsilon \sim N(\mathbf{0}, \sigma^2I_n).\]

Here \(\mathbf{0}\) is a vector of zero elements, \(I_n\) is an \(n\times n\) identity matrix, and \(\sigma^2_k\) and \(\sigma^2\) are unknown parameters, called variance components. Assumption (1) implies \(p < n\). We also assume \(q_k < n\). If \(q_k\geq n\), we can use principal component analysis (PCA) to obtain an equivalent LMM with the number of random effects less than \(n\).

## 1.2 The fast and scalable algorithm

Hartley and Rao (1967) developed the maximum likelihood (ML) method for estimating the LMM parameters (fixed effects and variance components). Patterson and Thompson (1971) proposed a modified maximum likelihood procedure, known as restricted maximum likelihood (REML) method. Estimating the variance components by either ML or REML is a numerical optimization problem. The log-likelihood based gradient methods have been proposed (Searle, Casella, and McCulloch 2006).

We developed a fast and scalable algorithm for implementing the gradient methods based on summary statistics. Let \(XX\), \(XY\), \(ZX\), \(ZY\), and \(ZZ\) denote a matrix, respectively, which define the summary statistics that are computed from cell-level data \(X\), \(Y\) and \(Z\) as follows: \[\begin{equation} \label{eq:sdata} \begin{array}{l} XX = X^TX, ~XY = X^TY^T,\\ ZX = Z^TX, ~ZY = Z^TY^T, ~ZZ = Z^TZ,\\ Ynorm = [y_1y_1^T, \ldots, y_my_m^T], \end{array} \end{equation}\] where \(Y = [y_1^T, \ldots, y_m^T]^T\) is a \(m\)-by-\(n\) matrix of gene expression profile with each row \(y_i\) corresponding to the expression of gene \(i\), \(i=1,\ldots,m\), \(m\) is the number of genes and \(n\) is the number of cells (sample size). After the summary statistics are precomputed, the summary statistics based algorithm has a complexity of \(O(m(p^3 + q^3))\), which doesn’t depend on the number of cells (sample size \(n\)). In single-cell differential expression (DE) analysis, the numbers of fixed and random effects, \(p\) and \(q\), are relatively small. Therefore, the algorithm is fast and scalable, and requires less computer memory. See the Supplementary Material in Xu et al. (2026) for details.

## 1.3 Functions

The FLASHMM package provides three functions: lmm, lmmfit, and flashmm, for fitting LMMs. The lmm function takes summary statistics as input, whereas lmmfit and flashmm are a wrapper around lmm that directly processes cell-level data and computes the summary statistics internally. The lmmfit takes the model design matrices as input. Instead the flashmm takes the model formulas as input, where the design matrices are generated internally. While lmmfit and flashmm are easier to use, they have the limitation of higher memory consumption. For extremely large scale data, we can precompute the summary-level data by \(\eqref{eq:sdata}\), and then use lmm function to fit LMMs. FLASHMM provides lmmtest function to perform statistical test on the fixed effects and the contrasts of the fixed effects.

In summary, FLASHMM package provides the following main functions.

- lmm: fit LMM using summary-level data.
- lmmfit: fit LMM using model design matrices based on cell-level data.
- flashmm: fit LMM using model formulas based on cell-level data.
- lmmtest: perform statistical tests on the fixed effects and their contrasts.
- contrast.matrix: construct a contrast matrix of the fixed effects for various comparisons.
- simuRNAseq: simulate multi-sample multi-cell-type scRNA-seq data.

    ##Install FLASHMM from CRAN.
    # install.packages("FLASHMM")
    ##Install FLASHMM from Github.
    # devtools::install_github("https://github.com/Baderlab/FLASHMM")
    ##Load the package.
    library(FLASHMM)

# 2 Example

We use a simulated multi-sample multi-cell-type single-cell RNA-seq (scRNA-seq) dataset to illustrate how to utilize FLASHMM to perform single-cell differential expression analysis. In this example, we are interested in identifying the genes differentially expressed between two treatments (conditions) within a cell type.

## 2.1 Simulating scRNA-seq data

We simulate the multi-sample multi-cell-type scRNA-seq data by simuRNAseq function in FLASHMM package. The simuRNAseq function generates scRNA-seq data with or without a reference dataset. The reference dataset consists of count data and metadata. The count data is a genes-by-cells matrix. The metadata is a data frame containing three columns: samples (subjects), cell types (clusters), and treatments (experimental conditions), which are named as ‘sam’, ‘cls’, and ‘trt’, respectively.

The simulated expression count is generated by the negative binomial (NB) distribution with dispersion \(\phi_g\) and mean \(\mu_{g,ijk}\), \[\begin{equation}\label{nbinomsimu} y_{g, ijk}\sim NB(\mu_{g,ijk}, \phi_g), \end{equation}\] \[\log(\mu_{g,ijk}) = \log(\mu_g) + \alpha_{ijk} + \beta_{g, ij} + b_{g,k},\] where \(y_{g, ijk}\) be the count for gene \(g\) and the cell from subject \(k = 1,\ldots, n_s\) and cell type \(j = 1, \ldots, n_c\) with treatment \(i=1,2\). \(\phi_g\) and \(\mu_g\) are the dispersion and mean estimated from the reference data for gene \(g\), \(\alpha_{ijk}\) is a centered log-library-size for the cell \(c_{ijk}\), \(\beta_{g,ij}\) represents the effect of treatment \(i\) specific to cell type \(j\), \(b_{g,k} \sim N(0, \sigma^2_b)\) is a random effect of subject variation. For the non-DE genes, \(\beta_{g,1j}=\beta_{g,2j}=0\). For the DE genes, \(\beta_{g,1j} = 0\) and \(\beta_{g, 2j}\sim \pm Uniform(a_1, a_2)\), a uniform distribution with \(a_1a_2>0\), where \(a_1\) and \(a_2\) are the lower and upper bounds of the DE gene effect sizes. See the Supplementary Material in Xu et al. (2026) for details.

Next we simulate scRNA-seq data that contains 100 genes and 100,000 cells from 25 samples (subjects) and 10 clusters (cell types) with 2 treatments. The samples are nested in one treatment A or B. There are totally 10 DE genes specific to a cell type.

    ##Generate the scRNA-seq data by simuRNAseq function.
    set.seed(2503)
    dat <- simuRNAseq(nGenes = 100, nCells = 100000, nsam = 25, ncls = 10, ntrt = 2, nDEgenes = 10)

    names(dat)
    #> [1] "ref.mean.dispersion" "metadata"            "counts"
    #> [4] "DEgenes"             "treatment"
    dim(dat$counts)
    #> [1]    100 100000
    head(dat$metadata)
    #>       sam cls trt libsize
    #> Cell1  A4  10   A     209
    #> Cell2  A7   6   A     194
    #> Cell3  B9   6   B     175
    #> Cell4  B6   6   B     213
    #> Cell5 A11  10   A     190
    #> Cell6  B6   1   B     201

    ##Samples are nested in one treatment A or B.
    table(dat$metadata[, c("trt", "sam")])
    #>    sam
    #> trt   A1  A10  A11  A12  A13  A14  A15  A16   A2   A3   A4   A5   A6   A7   A8
    #>   A 3128 3194 3143 3152 3073 3104 3001 3107 3238 3166 3184 3151 3088 3123 3093
    #>   B    0    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    #>    sam
    #> trt   A9   B1   B2   B3   B4   B5   B6   B7   B8   B9
    #>   A 2968    0    0    0    0    0    0    0    0    0
    #>   B    0 5662 5441 5502 5589 5601 5590 5499 5618 5585

The simulated data contains

- counts: a genes-by-cells matrix of expression counts
- metadata: a data frame consisting of samples (sam), cell types (cls) and treatments (trt).
- DEgenes: differentially expressed (DE) genes.

## 2.2 DE analysis of scRNA-seq data

We perform differential expression (DE) analysis of the simulated data using FLASHMM package. We are to identify the significantly differentially expressed genes between two treatments in a cell type based on the LMM. The gene expression profile is taken as log-transformed count matrix, \[\begin{equation}\label{exprs} Y = \log(1+\text{counts}), \end{equation}\] in which each row represents the expression profile for a gene. The analyses involve following steps: LMM design, LMM fitting, and hypothesis testing.

### 2.2.1 LMM design

LMM provides a framework to represent the single-cell gene expression profile by incorporating fixed effects, which capture systematic differences across experimental conditions, and random effects, which model the correlations within subjects and the variations between subjects. Constructing design matrices for fixed and random effects is an important step in LMM-based DE analysis.

Fixed effects: As a general linear model design, the fixed effects can be specified by either model formula or design matrix as follows \[\begin{equation}\label{fixedeff} \begin{array}{l} fixed = log(library.size) + covariates + cell.type + cell.type:treatment,\\ X = model.matrix(fixed), \end{array} \end{equation}\] where

- \(library.size\), the total sum of counts across genes, is recommended including as a role of the normalization factor of the scRNA-seq counts.
- \(covariates\), e.g., age, gender, batch effects, or principal components, can be included as fixed effects to adjust for unwanted variability, if appropriate.
- \(cell.type\) is used to adjust the cell type effect.
- \(cell.type:treatment\) represents the treatment effect in a specific cell type.

Random effects: In multi-subject single-cell studies, cells are nested within subjects, and subjects are often nested within experimental conditions, meaning that cells from the same subject are correlated, and subjects within the same condition share common sources of variation. To account for this hierarchical structure, the mixed models treat the subjects (samples) as random effects, efficiently capturing both within-subject correlation and between-subject variability. In many scRNA-seq studies, samples are perfectly confounded with experimental conditions. In such cases, including subjects as a fixed effect may introduce collinearity.

The single-component random effects can be specified by \[\begin{equation}\label{randomeff1} \begin{array}{l} random = \sim 0 + subject,\\ Z = model.matrix(random), \end{array} \end{equation}\] with a dimension \(d = ncol(Z)\).

If appropriate, for example, we also take account of the measurement locations as a random effect, we may consider the LMM with two-component random effects specified by \[\begin{equation}\label{randomeff2} \begin{array}{l} randomS = \sim 0 + subject,\\ randomL = \sim 0 + location,\\ Z_1 = model.matrix(randomS),\\ Z_2 = model.matrix(randomL),\\ Z = [Z_1,~Z_2], \end{array} \end{equation}\] with two dimensions \(d = (ncol(Z_1), ncol(Z_2))\).

    ## Gene expression matrix, Y = log2(1 + counts)
    Y <- log2(1 + dat$counts)

    ## Since the simulated data contains no covariates other than libsize (library size) and
    ## cls (clsters or cell types), the fixed effects are specified as follows.
    fixed <- ~0 + log(libsize) + cls + cls:trt
    X <- model.matrix(fixed, data = dat$metadata)

    ## The single-component random effects are specified by
    randomS <- ~0 + as.factor(sam)
    Z1 <- model.matrix(randomS, data = dat$metadata)

    ## Two-component random effects: Suppose the data contains different measurement
    ## locations within a sample, denoted as 'location', which are randomly generated. To
    ## account for the variability of the locations, we may use the LMM with two-component
    ## random effects described below.
    set.seed(250801)
    dat$metadata$location <- sample(LETTERS[1:20], nrow(X), replace = TRUE)
    randomL <- ~0 + location
    Z2 <- model.matrix(randomL, data = dat$metadata)
    # Za <- cbind(Z, Z2) #two-component random-effects design matrix da <- c(ncol(Z),
    # ncol(Z2)) #dimension

### 2.2.2 LMM fitting

Given the cell-level transformed expression profile matrix \(Y\), fixed-effects formula (\(fixed\)), random-effects formula (\(random\)), and the metadata (\(metadata\)), the LMMs can be fit by \[ fit = flashmm(Y, fixed, random, metadata, is.counts = FALSE). \]

With the cell-level data matrices \(Y\), \(X\) and \(Z\), the LMMs can be fit by \[fit = lmmfit(Y, X, Z, d = d),\] where \(d\) is a vector of the numbers of random-effects in different components. For \(K\) components of random-effects, \(d = (q_1, \ldots, q_K)\), where \(q_k\) is the number of columns of the design matrix \(Z_k\) for the \(k\)-th random-effect component.

The lmmfit and flashmm functions have high memory usage. For extremely large-scale data, it is recommended to pre-compute and store the summary-level data: \(XX\), \(XY\), \(ZX\), \(ZY\), \(ZZ\), and \(Ynorm\), defined in \(\eqref{eq:sdata}\), and then use the lmm function to fit the LMMs as follows: \[fit = lmm(XX, XY, ZX, ZY, ZZ, Ynorm = Ynorm, n = n, d = d).\] The summary-level data doesn’t depend on the sample size \(n\). This makes lmm memory-efficient. By default, these functions use the restricted maximum likelihood (REML) method (method = ‘REML’). To fit the LMM using maximum likelihood (ML), set method = ‘ML’.

    ##Fit the LMM with one-component random effects.

    ##Option 1: Fit the LMM using model formulas based on cell-level data.
    fit1 <- flashmm(Y, fixed, random = randomS, metadata = dat$metadata, is.counts = FALSE)
    names(fit1)
    #>  [1] "method"       "epsilon"      "dlogL"        "logLik"       "niter"
    #>  [6] "coef"         "se"           "t"            "p"            "df"
    #> [11] "cov"          "theta"        "se.theta"     "FIM.beta"     "FIM.theta"
    #> [16] "RE"           "summary_data"

    ##Option 2: Fit the LMM using model design matrices based on cell-level data.
    fit2 <- lmmfit(Y, X, Z = Z1, d = ncol(Z1))
    identical(fit1, fit2)
    #> [1] TRUE

    ##Option 3: Fit the LMM based on summary-level data.
    ##Step 1: computing the summary-level data
    #n <- nrow(X); XX <- t(X)%*%X; XY <- t(Y%*%X)
    #ZX <- t(Z)%*%X; ZY <- t(Y%*%Z); ZZ <- t(Z)%*%Z
    #Ynorm <- rowSums(Y*Y)
    ##or computed by sslmm function:
    ss <- sslmm(X, Y, Z = Z1)
    XX <- ss$XX; XY <- ss$XY
    ZX <- ss$ZX; ZY <- ss$ZY; ZZ <- ss$ZZ
    n <- ss$n; Ynorm <- ss$Ynorm
    d1 <- ncol(Z1)

    ##Step 2: fitting the LMM using lmm function
    fit3 <- lmm(XX, XY, ZX, ZY, ZZ, Ynorm = Ynorm, n = n, d = d1)
    identical(fit2, fit3)
    #> [1] TRUE

    ##Fit the LMM with two-component random effects.
    fit2 <- flashmm(Y, fixed, random = list(randomS, randomL), metadata = dat$metadata)
    fit3 <- lmmfit(Y, X, Z = cbind(Z1, Z2), d = c(ncol(Z1), ncol(Z2)))

    identical(fit2, fit3)
    #> [1] TRUE

The outputs from LMM fitting contain:

- estimates of LMM parameters (coefficients and variance components), standard errors of the estimates;
- covariance matrix of the coefficients (fixed effects), and t-values and p-values for the hypothesis testing on the coefficients;
- maximum log-likelihood, the number of iterations, and the first partial derivatives of the log likelihood.
- Fisher information matrices (FIM) for fixed effects (FIM.beta) and variance components (FIM.theta) if output.FIM = TRUE.

Convergence of LMM fitting: The LMM fitting is considered converged if the absolute values of first partial derivatives of the log-likelihood are smaller than the given convergence tolerance (epsilon); otherwise, the fitting does not converge. The genes for which LMM fitting doesn’t converge should be excluded in the subsequent analysis because the estimated coefficients for these genes are not reliable.

    ##Check the genes for which LMM fitting converges.
    fit1$epsilon
    #> [1] 1e-05
    convg <- (apply(abs(fit1$dlogL), 2, max) < fit1$epsilon)
    sum(convg)
    #> [1] 100

### 2.2.3 Hypothesis testing

Testing of fixed effects using the lmmtest function

Although lmm and lmmfit functions return t-values and p-values for testing the fixed effects (coefficients), we can also use lmmtest function to conduct statistical tests on the fixed effects and their contrasts for various comparisons between different levels.

    ##Testing coefficients (fixed effects)
    test <- lmmtest(fit1)
    ##The t-value and p-values are identical with those provided in the LMM fit.
    range(test - cbind(t(fit1$coef), t(fit1$t), t(fit1$p)))
    #> [1] 0 0

    fit1$coef[, 1:4]
    #>                   Gene1      Gene2       Gene3       Gene4
    #> log(libsize)  0.6374934  0.7705731  0.64719938  0.81898403
    #> cls1         -2.5711912 -2.8468810 -2.55712144 -2.94868670
    #> cls10        -2.5916568 -2.8644930 -2.56851683 -2.98189917
    #> cls2         -2.5719214 -2.8165668 -2.57723359 -2.96349626
    #> cls3         -2.5876929 -2.8251454 -2.58092263 -2.94790559
    #> cls4         -2.5959205 -2.8300663 -2.58906911 -2.94215679
    #> cls5         -2.5692815 -2.8529653 -2.56874474 -2.97700998
    #> cls6         -2.5852827 -2.8170462 -2.57031717 -2.95099651
    #> cls7         -2.5907649 -2.8269791 -2.57041172 -2.98207635
    #> cls8         -2.5988346 -2.8391140 -2.55705273 -2.95808436
    #> cls9         -2.5900606 -2.8381226 -2.56955264 -2.97274041
    #> cls1:trtB     0.1558683  0.2260717  0.06743826 -0.04299325
    #> cls10:trtB    0.1727050  0.2461573  0.08426602 -0.05930840
    #> cls2:trtB     0.1645468  0.2090950  0.09020917 -0.04033431
    #> cls3:trtB     0.1667314  0.2205625  0.10256963 -0.07131573
    #> cls4:trtB     0.1879568  0.2257336  0.11139951 -0.06543985
    #> cls5:trtB     0.1356348  0.2529515  0.08288371 -0.04229847
    #> cls6:trtB     0.1646795  0.1724673  0.08022732 -0.06705943
    #> cls7:trtB     0.1740453  0.2091635  0.10025723 -0.04130775
    #> cls8:trtB     0.1687667  0.2193172  0.06931996 -0.04835126
    #> cls9:trtB     0.1692256  0.2303956  0.07198198 -0.05961417

    #fit1$t[, 1:4]
    #fit1$p[, 1:4]

Differentially expressed (DE) genes: The coefficients of the interaction term cls\(\ast\):trtB represent the effects of treatment B versus A in a cell type (cls\(\ast\)). The DE genes between two treatments in a cell type can be identified based on the false discovery rate (FDR) or family-wise error rate (Bonferroni correction). Additionally, we may exclude genes with small coefficients (effect size or log-fold change).

    ##The DE genes specific to a cell type
    ##Coefficients, t-values, and p-values for the genes specific to a cell type.
    index <- grep(":", rownames(fit1$coef))
    ce <- fit1$coef[index, ]
    tv <- fit1$t[index, ]
    pv <- fit1$p[index, ]

    out <- data.frame(
        gene = rep(colnames(ce), nrow(ce)),
        cluster = rep(rownames(ce), each = ncol(ce)),
        coef = c(t(ce)), t = c(t(tv)), p = c(t(pv)))

    ##FDR.
    out$FDR <- p.adjust(out$p, method = "fdr")

    ##The DE genes with FDR < 0.05 and abs(logFC) > 0.5
    out <- out[order(out$p), ]
    rownames(out) <- NULL
    out[(out$FDR < 0.05) & (abs(out$coef) > 0.5) , ]
    #>     gene   cluster       coef         t            p          FDR
    #> 1 Gene98 cls7:trtB  0.9093044 13.393506 7.157702e-41 7.157702e-38
    #> 2 Gene94 cls4:trtB  0.7051694  7.452634 9.222821e-14 4.611411e-11
    #> 3 Gene93 cls2:trtB -0.8459774 -7.230808 4.835463e-13 1.611821e-10
    #> 4 Gene91 cls1:trtB -0.7010988 -5.964488 2.462226e-09 6.155564e-07

Using contrast: We can make a comparison of the treatment B versus A across cell types via the contrast that sums the treatment effects across cell types as follows.

    ##Construct the contrast.
    contrast <- cbind("BvsA" = numeric(nrow(fit1$coef)))
    index <- grep(":", rownames(fit1$coef))
    contrast[index, ] <- 1/length(index)
    length(index)
    #> [1] 10

    ##Test the contrast.
    test <- lmmtest(fit1, contrast = contrast)
    head(test)
    #>         BvsA_coef     BvsA_t     BvsA_p
    #> Gene1  0.16601603  2.1371340 0.03258952
    #> Gene2  0.22119152  1.8670170 0.06190214
    #> Gene3  0.08605528  1.1752292 0.23990583
    #> Gene4 -0.05380226 -0.4663953 0.64093360
    #> Gene5 -0.04452584 -0.4541155 0.64974670
    #> Gene6 -0.15364508 -1.0628308 0.28786124

The contrast can also be constructed by contrast.matrix function as follows.

    ncls <- 10 #length(index)
    sumeff <- paste0(paste0("cls", 1:ncls, ":trtB"), collapse = "+")
    sumeff <- paste0("(", sumeff, ")/", ncls)
    #sumeff
    contrast <- contrast.matrix(
                contrast = c(BvsA = sumeff),
                model.matrix.names = rownames(fit1$coef))
    test <- lmmtest(fit1, contrast = contrast)
    head(test)
    #>         BvsA_coef     BvsA_t     BvsA_p
    #> Gene1  0.16601603  2.1371340 0.03258952
    #> Gene2  0.22119152  1.8670170 0.06190214
    #> Gene3  0.08605528  1.1752292 0.23990583
    #> Gene4 -0.05380226 -0.4663953 0.64093360
    #> Gene5 -0.04452584 -0.4541155 0.64974670
    #> Gene6 -0.15364508 -1.0628308 0.28786124

Testing of variance component in the LMM with single-component random effects

Since the simulated data is generated by the LMM with single-component random effects, see Simulating scRNA-seq data Section, the variance component of random effects should not be zero. Next we use z-statistic \(\eqref{zvarcomp}\) to test the variance component. Note that ‘fit1’ is the output of fitting the LMM with single-component random effects using lmmfit function, see LMM fitting Section.

    ##Using z-test for testing variance components
    i <- grep("var1", rownames(fit1$theta))  #The (first) variance component
    z <- fit1$theta[i, ]/fit1$se.theta[i, ]   #z-statistics

    ##One-sided z-test p-values for hypotheses:
    ##H0: theta <= 0 vs H1: theta > 0
    p <- pnorm(z, lower.tail = FALSE)

    ##Number of significant p-values
    sum(p < 0.05/length(p)) #Bonferroni correction
    #> [1] 100

Testing of variance component in the LMM with two-component random effects

As described in LMM design Section, the LMM with two-component random effects has two variance components, the first one for random effects of subjects and the second one for random effects of measurement locations within a subject. Since the simulated data was generated by the LMM with one-component random effects (i.e., subjects), the second variance component should be zero. Next we use both z-statistic \(\eqref{zvarcomp}\) and LRT statistic \(\eqref{lrtvarcomp}\) to test the second variance component of the random effects. Note that ‘fit2’ is the output of fitting the LMM with two-component random effects, see LMM fitting Section.

    ##(1) z-test for testing the second variance component
    ##Z-statistics for the second variance component
    i <- grep("var2", rownames(fit2$theta))
    z <- fit2$theta[i, ]/fit2$se.theta[i, ]
    ##One-sided z-test p-values for hypotheses:
    ##H0: theta <= 0 vs H1: theta > 0
    p <- pnorm(z, lower.tail = FALSE)

    ##number of significant p-values
    sum(p.adjust(p, method = "fdr") < 0.05)
    #> [1] 0
    sum(p < 0.05/length(p)) #Bonferroni correction
    #> [1] 0

    ##(2) LRT for testing the second variance component
    LRT <- 2*(fit2$logLik - fit1$logLik)
    pLRT <- pchisq(LRT, df = 1, lower.tail = FALSE)

    ##number of significant p-values
    sum(p.adjust(pLRT, method = "fdr") < 0.05)
    #> [1] 0
    sum(pLRT < 0.05/length(pLRT)) #Bonferroni correction
    #> [1] 0

    ##QQ-plot
    qqplot(runif(length(p)), p, xlab = "Uniform quantile", ylab = "Z-test p-value", col = "blue")
    abline(0, 1, col = "gray")

    qqplot(runif(length(pLRT)), pLRT, xlab = "Uniform quantile", ylab = "LRT p-value", col = "blue")
    abline(0, 1, col = "gray")

### 2.2.4 Using ML method

To use the maximum likelihood (ML) method to fit the LMM, set method = ‘ML’ in the lmm, lmmfit, and flashmm functions.

    ##Fitting LMM using ML method
    #fit3 <- lmmfit(Y, X, Z, d = d1, method = "ML")
    fit3 <- lmm(XX, XY, ZX, ZY, ZZ, Ynorm = Ynorm, n = n, d = d1, method = "ML")

    ##The DE genes specific to a cell type
    ##Coefficients, t-values, and p-values
    index <- grep(":", rownames(fit3$coef))
    ce <- fit3$coef[index, ]
    tv <- fit3$t[index, ]
    pv <- fit3$p[index, ]
    out <- data.frame(
        gene = rep(colnames(ce), nrow(ce)),
        cluster = rep(rownames(ce), each = ncol(ce)),
        coef = c(t(ce)), t = c(t(tv)), p = c(t(pv)))

    ##The DE genes with FDR < 0.05 and abs(logFC) > 0.5
    out$FDR <- p.adjust(out$p, method = "fdr") #FDR
    out <- out[order(out$p), ]
    rownames(out) <- NULL
    out[(out$FDR < 0.05) & (abs(out$coef) > 0.5) , ]
    #>     gene   cluster       coef         t            p          FDR
    #> 1 Gene98 cls7:trtB  0.9093058 13.912096 5.879097e-44 5.879097e-41
    #> 2 Gene94 cls4:trtB  0.7051668  7.759646 8.596761e-15 4.298380e-12
    #> 3 Gene93 cls2:trtB -0.8459774 -7.531629 5.053013e-14 1.684338e-11
    #> 4 Gene91 cls1:trtB -0.7010987 -6.213697 5.195542e-10 1.298886e-07

# 3 Methods

## 3.1 LMM parameter estimation

Hartley and Rao (1967) developed the maximum likelihood (ML) method for estimating the LMM parameters (fixed effects and variance components). Patterson and Thompson (1971) proposed a modified maximum likelihood procedure, known as restricted maximum likelihood (REML), which partitions the data into two mutually uncorrelated parts, one being free of the fixed effects used for estimating variance components. The REML estimator is unbiased, whereas the ML estimator of variance components is generally biased. With variance components, \(\theta\), estimated, the fixed effects estimated by either ML or REML are given as follows \[ \hat\beta = (X^TV_{\theta}^{-1}X )^{-1}X^TV_{\theta}^{-1}y, \] with covariance matrix \[var(\hat\beta) = (X^TV_{\theta}^{-1}X)^{-1},\] where \(\theta = (\theta_0, \theta_1,\ldots, \theta_K)\), \(\theta_0 = \sigma^2\), \(\theta_k = \sigma^2_k\), and \[V_{\theta} = \theta_0 I_n + \theta_1 Z_1Z_1^T + \ldots + \theta_K Z_KZ_K^T.\]

Estimating the variance components by either ML or REML is a numerical optimization problem. Various iterative methods based on log likelihood, called gradient methods, have been proposed (Searle, Casella, and McCulloch 2006). The gradient methods are represented by the iteration equation \[\begin{equation}\label{eq:gradient} \theta^{(i+1)} = \theta^{(i)} + \Gamma(\theta^{(i)})\frac{\partial l(\theta^{(i)})}{\partial\theta}, \end{equation}\] where \(\partial l(\theta)/\partial\theta\) is the gradient of the log likelihood function, and \(\Gamma(\theta)\) is a modifier matrix of the gradient direction, which can be specified by Newton–Raphson, Fisher scoring or average information. The Newton-Raphson method uses the inverse of the negative Hessian matrix (observed Fisher information) as a modifier, while the Fisher scoring method uses the inverse of Fisher information matrix (the expectation of negative Hessian matrix). The average information method uses the inverse of the average of the negative Hessian matrix and its expectation. See the Supplementary Material in Xu et al. (2026) for details. The Fisher scoring method is more stable than others because the negative Hessian matrix may not be positive definite but its expectation is positive definite.

## 3.2 Hypothesis testing

The hypotheses for testing fixed effects and variance components can be respectively defined as \[ H_{0, i}: \beta_i = 0 ~~\text{versus}~~H_{1,i}: \beta_i\ne 0, \] \[ H_{0, k}: \theta_k \leq 0 ~~\text{versus}~~H_{1,k}: \theta_k > 0, \] where \(\theta_k\), \(k=1, \ldots, K\), represent the parameters of variance components \(\sigma^2_k\) but are allowed to be negative. The lower boundary of the parameters, \(\theta\), can be the negative value such that the variance-covariance matrix, \(V_{\theta}\), remains definable (positive definite). The negative lower boundary exists and can be less than or equal to \(- \sigma^2/\lambda_{max}\), where \(\lambda_{max} > 0\) is the largest singular value of \(ZZ^T\) or \(Z^TZ\). If \(\theta_k > 0\), then \(\sigma_k^2 = \theta_k\) is definable and the mixed model is well-specified. Otherwise, if \(\theta_k \le 0\), the mixed model is miss-specified and the \(k\)-th random-effect component shouldn’t be included.

Allowing the parameters of variance components to take negative value avoids the zero boundary at the null hypothesis for the variance components. Consequently, the asymptotic normality of the maximum likelihood estimates at the null hypothesis holds under regularity conditions, which enables us to use z-statistics or t-statistics for hypothesis testing of the fixed effects and variance components. The t-statistics for fixed effects are given by \[\begin{equation} \label{eq:tcoef} T_i = \frac{\hat\beta_i}{\sqrt{var(\hat\beta_i)}} = \frac{\hat\beta_i}{\sqrt{var(\hat\beta)_{ii}}} ~\sim ~t(n - p). \end{equation}\] The t-statistic for a contrast, a linear combination of the estimated fixed effects, \(c^T\hat\beta\), is \[\begin{equation} \label{eq:tcontrast} T_c = \frac{c^T\hat\beta}{\sqrt{c^Tvar(\hat\beta) c}} \sim t(n-p). \end{equation}\] The z-statistics for the parameters of variance components are given by \[\begin{equation} \label{zvarcomp} Z_{\theta_k} = \frac{\hat\theta_k}{\sqrt{[I(\hat\theta)^{-1}]_{kk}}} \sim N(0, 1), \end{equation}\] where \(I(\theta)\) is the Fisher information matrix.

We can also use likelihood ratio test (LRT) statistics to test the variance components. Let \(\theta_{sub}\) be the sub-vector of \(\theta\) that is not equal to zero. Let \(L(\hat\theta_{sub})\) and \(L(\hat\theta)\) be the maximum likelihood for the sub-model and the full model, respectively. Note that \(L(\hat\theta_{sub})\) and \(L(\hat\theta)\) depend on \(\hat\beta\) for the ML method which is suppressed for simplicity of notation. The LRT statistic, \[\begin{equation}\label{lrtvarcomp} \lambda_{LR} = 2[\log(L(\hat\theta)) - \log(L(\hat\theta_{sub}))] \sim\chi^2_r, \end{equation}\] an asymptotic \(\chi^2\) distribution, where \(r\) is the number of zero variance components. The LRT requires that both sub and full models are fitted by the same method, either ML or REML, and the design matrix of fixed effects must be the same for both sub and full models when using REML method to fit the models.

# 4 Remarks

Building design matrices for fixed and random effects is a key step in LMM-based differential expression analysis. Including library size, a normalization factor for scRNA-seq, as a fixed effect can help reduce p-value inflation. If necessary, we can add principal components as fixed effects to further mitigate unknown technical effects.

In differential expression analysis, modeling samples (subjects) as random effects accounts for between-sample variability and within-sample correlation. If samples have different effects on gene expression, we can also model them as fixed effects. However, this can lead to the random-effects design matrix linearly depends on the fixed-effects design matrix. In this case, the LMM cannot be fit by REML method. Moreover, this can also result in overfitting when the number of samples (subjects) is large and collinearity when the samples are nested within an experimental condition.

# Session info

    sessionInfo()
    #> R version 4.4.3 (2025-02-28)
    #> Platform: x86_64-apple-darwin20
    #> Running under: macOS Monterey 12.7.6
    #>
    #> Matrix products: default
    #> BLAS:   /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRblas.0.dylib
    #> LAPACK: /Library/Frameworks/R.framework/Versions/4.4-x86_64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
    #>
    #> locale:
    #> [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
    #>
    #> time zone: America/Toronto
    #> tzcode source: internal
    #>
    #> attached base packages:
    #> [1] stats     graphics  grDevices utils     datasets  methods   base
    #>
    #> other attached packages:
    #> [1] FLASHMM_1.3.0
    #>
    #> loaded via a namespace (and not attached):
    #>  [1] digest_0.6.39     R6_2.6.1          fastmap_1.2.0     Matrix_1.7-4
    #>  [5] xfun_0.56         lattice_0.22-7    cachem_1.1.0      knitr_1.51
    #>  [9] htmltools_0.5.9   rmarkdown_2.30    lifecycle_1.0.5   cli_3.6.5
    #> [13] grid_4.4.3        sass_0.4.10       jquerylib_0.1.4   compiler_4.4.3
    #> [17] rstudioapi_0.18.0 tools_4.4.3       evaluate_1.0.5    bslib_0.10.0
    #> [21] yaml_2.3.12       formatR_1.14      otel_0.2.0        jsonlite_2.0.0
    #> [25] rlang_1.1.7       MASS_7.3-65

# References

Hartley, H. O., and J. N. K. Rao. 1967. “Maximum-Likelihood Estimation for the Mixed Analysis of Variance Model.” Biometrika 54 (1/2): 93–108. http://www.jstor.org/stable/2333854.
Patterson, H. D., and R. Thompson. 1971. “Recovery of Inter-Block Information When Block Sizes Are Unequal.” Biometrika 58 (3): 545–54. https://doi.org/10.2307/2334389.
Searle, Shayle R., George Casella, and Charles E. McCulloch. 2006. Variance Components. New Jersey: John Wiley & Sons, Inc.
Xu, Changjiang, Delaram Pouyabahar, Veronique Voisin, Hamed Heydari, and Gary D. Bader. 2026. “FLASH-MM: Fast and Scalable Single-Cell Differential Expression Analysis Using Linear Mixed-Effects Models.” Nature Communications 17: 2384. https://doi.org/10.1038/s41467-026-69063-2.