Author: Joshua F. Wiley
Date: 2025-09-06

# Model Performance and Tests

#### Joshua F. Wiley

#### 2025-09-06

To start, load the package.

    library(JWileymisc)

# Model Performance

`modelPerformance()` is a generic function that can be used to calculate performance metrics for a model. `JWileymisc` implements methods for `lm` class objects. The output is a named list, with a data table containing results. For linear models, current performance metrics include:

- AIC (Akaike Information Criterion)
- BIC (Bayesian Information Criterion)
- LL (Log Likelihood)
- LLDF (degrees of freedom for log likelihood)
- Sigma (residual standard deviation)
- R2 (\(R^2\) variance accounted for in the sample)
- F2 (Cohen’s \(f^2\) effect size, calculated as \(\frac{R^{2}}{1 - R^{2}}\))
- AdjR2 (Sample size adjusted \(R^2\), a better estimate of population variance accounted for)
- F (model F test)
- FNumDF (numerator degrees of freedom for model F test)
- FDenDF (denominator degrees of freedom for model F test)
- P (p value for model F test)

    mtcars$cyl <- factor(mtcars$cyl)
    m <- stats::lm(mpg ~ hp + cyl, data = mtcars)

    mp <- modelPerformance(m)
    print(mp)
    #> $Performance
    #>     Model N_Obs      AIC      BIC        LL  LLDF    Sigma        R2       F2
    #>    <char> <num>    <num>    <num>     <num> <num>    <num>     <num>    <num>
    #> 1:     lm    32 169.8964 177.2251 -79.94822     5 3.146243 0.7538578 3.062692
    #>        AdjR2        F FNumDF FDenDF           P
    #>        <num>    <num>  <num>  <num>       <num>
    #> 1: 0.7274854 28.58513      3     28 1.13969e-08
    #>
    #> attr(,"class")
    #> [1] "modelPerformance.lm" "modelPerformance"

If only certain metrics are desired, these can be found by extracting the “Performance” list element and then the correct column from the data table.

    ## Cohen's f^2 effect size
    mp$Performance[, F2]
    #> [1] 3.062692

Another function, `modelTest()` is a generic providing a comprehensive series of tests for a model. Currently methods are implemented for both `lm` class models and `vglm` class models from the `VGAM` package with a multinomial family.

`modelTest()`

    mt <- modelTest(m)
    print(mt)
    #> $FixedEffects
    #>           Term         Est           LL           UL         Pval
    #>         <char>       <num>        <num>        <num>        <num>
    #> 1: (Intercept) 28.65011816  25.39768395 31.902552374 5.921199e-17
    #> 2:          hp -0.02403883  -0.05560048  0.007522814 1.299540e-01
    #> 3:        cyl6 -5.96765508  -9.32556307 -2.609747083 1.092089e-03
    #> 4:        cyl8 -8.52085075 -13.28559928 -3.756102224 1.028617e-03
    #>
    #> $RandomEffects
    #> [1] NA
    #>
    #> $EffectSizes
    #>      Term N_Obs         AIC        BIC       LL  LLDF       Sigma         R2
    #>    <char> <num>       <num>      <num>    <num> <num>       <num>      <num>
    #> 1:     hp     0  -0.6675031  0.7982328 1.333752     1 -0.07685536 0.02139775
    #> 2:    cyl     0 -11.3421811 -8.4107093 7.671091     2 -0.71671885 0.15142046
    #>            F2     AdjR2        F FNumDF FDenDF           P   Type
    #>         <num>     <num>    <num>  <num>  <num>       <num> <char>
    #> 1: 0.08693246 0.0134764 2.434109      1     28 0.129954045  Fixed
    #> 2: 0.61517476 0.1383002 8.612447      2     28 0.001215981  Fixed
    #>
    #> $OverallModel
    #> $Performance
    #>     Model N_Obs      AIC      BIC        LL  LLDF    Sigma        R2       F2
    #>    <char> <num>    <num>    <num>     <num> <num>    <num>     <num>    <num>
    #> 1:     lm    32 169.8964 177.2251 -79.94822     5 3.146243 0.7538578 3.062692
    #>        AdjR2        F FNumDF FDenDF           P
    #>        <num>    <num>  <num>  <num>       <num>
    #> 1: 0.7274854 28.58513      3     28 1.13969e-08
    #>
    #> attr(,"class")
    #> [1] "modelPerformance.lm" "modelPerformance"
    #>
    #> attr(,"class")
    #> [1] "modelTest.lm" "modelTest"

    APAStyler(mt)
    #>                 Term                      Est          Type
    #>               <char>                   <char>        <char>
    #>  1:      (Intercept) 28.65*** [ 25.40, 31.90] Fixed Effects
    #>  2:               hp    -0.02 [ -0.06,  0.01] Fixed Effects
    #>  3:             cyl6  -5.97** [ -9.33, -2.61] Fixed Effects
    #>  4:             cyl8  -8.52** [-13.29, -3.76] Fixed Effects
    #>  5: N (Observations)                       32 Overall Model
    #>  6:        logLik DF                        5 Overall Model
    #>  7:           logLik                   -79.95 Overall Model
    #>  8:              AIC                   169.90 Overall Model
    #>  9:              BIC                   177.23 Overall Model
    #> 10:               F2                     3.06 Overall Model
    #> 11:               R2                     0.75 Overall Model
    #> 12:           Adj R2                     0.73 Overall Model
    #> 13:               hp      f2 = 0.09, p = .130  Effect Sizes
    #> 14:              cyl      f2 = 0.62, p = .001  Effect Sizes

The model tests can also be used with interactions.

    m2 <- stats::lm(mpg ~ hp * cyl, data = mtcars)

    APAStyler(modelTest(m2))
    #>                 Term                       Est          Type
    #>               <char>                    <char>        <char>
    #>  1:      (Intercept)  35.98*** [ 27.99, 43.98] Fixed Effects
    #>  2:               hp    -0.11* [ -0.21, -0.02] Fixed Effects
    #>  3:             cyl6   -15.31* [-30.59, -0.03] Fixed Effects
    #>  4:             cyl8  -17.90** [-28.71, -7.09] Fixed Effects
    #>  5:          hp:cyl6      0.11 [ -0.04,  0.25] Fixed Effects
    #>  6:          hp:cyl8      0.10 [  0.00,  0.20] Fixed Effects
    #>  7: N (Observations)                        32 Overall Model
    #>  8:        logLik DF                         7 Overall Model
    #>  9:           logLik                    -77.54 Overall Model
    #> 10:              AIC                    169.08 Overall Model
    #> 11:              BIC                    179.34 Overall Model
    #> 12:               F2                      3.72 Overall Model
    #> 13:               R2                      0.79 Overall Model
    #> 14:           Adj R2                      0.75 Overall Model
    #> 15:               hp       f2 = 0.23, p = .021  Effect Sizes
    #> 16:              cyl       f2 = 0.47, p = .007  Effect Sizes
    #> 17:           hp:cyl       f2 = 0.16, p = .142  Effect Sizes