5. Agreement and ICC for Wide Data

Scope

Agreement and reliability are related to correlation, but they are not the same problem. Correlation describes co-movement. Agreement describes similarity on the measurement scale itself. Reliability describes the proportion of variation attributable to stable differences among subjects rather than to measurement error or method disagreement.

This vignette covers the wide-data functions:

Pairwise concordance and Bland-Altman analysis

Lin’s concordance correlation coefficient combines precision and accuracy in a single number. Bland-Altman analysis separates that question into estimated bias and limits of agreement.

library(matrixCorr)

set.seed(40)
ref <- rnorm(50, mean = 100, sd = 10)
m1 <- ref + rnorm(50, sd = 2)
m2 <- ref + 1.2 + rnorm(50, sd = 3)

fit_ba <- ba(m1, m2)
fit_ccc <- ccc(data.frame(m1 = m1, m2 = m2), ci = TRUE)

print(fit_ba)
#> Bland-Altman preview:
#>   based_on    : 50
#>   loa_rule    : mean +/- 1.96 * SD
#>   ci          : 95%
#>   sd_diff     : 3.722
#>   width       : 14.589
#> 
#>  quantity        estimate lwr     upr   
#>  Mean difference -1.290   -2.347  -0.232
#>  Lower LoA       -8.584   -10.416 -6.752
#>  Upper LoA       6.005    4.173   7.837
summary(fit_ccc)
#> Lin's concordance summary
#>   dimensions  : 2 x 2
#>   pairs       : 1
#>   estimate    : 0.9299
#>   most_negative: m1-m2 (0.9299)
#>   most_positive: m1-m2 (0.9299)
#>   ci          : 95%
#>   ci_method   : delta_method
#>   ci_width    : 0.08
#> 
#> Strongest pairs by |estimate|
#> 
#>  item1 item2 estimate lwr  upr  n_complete
#>  m1    m2    0.9299   0.88 0.96 50

The two summaries are complementary rather than redundant. ccc() gives a single concordance coefficient, while ba() makes the scale of disagreement explicit.

Pairwise ICC

icc() extends the wide-data reliability workflow in two directions. It can return a pairwise matrix across method pairs, or it can return the overall classical ICC table for the full set of methods.

wide_methods <- data.frame(
  J1 = ref + rnorm(50, sd = 1.5),
  J2 = ref + 4.0 + rnorm(50, sd = 1.8),
  J3 = ref - 3.0 + rnorm(50, sd = 2.0),
  J4 = ref + rnorm(50, sd = 1.6)
)

fit_icc_pair <- icc(
  wide_methods,
  model = "twoway_random",
  type = "agreement",
  unit = "single",
  scope = "pairwise"
)

fit_icc_overall <- icc(
  wide_methods,
  model = "twoway_random",
  type = "agreement",
  unit = "single",
  scope = "overall",
  ci = TRUE
)

print(fit_icc_pair, digits = 2)
#> Intraclass correlation matrix
#>   method      : Intraclass correlation (two-way random, agreement, single)
#>   dimensions  : 4 x 4
#> 
#>      J1   J2   J3   J4
#> J1 1.00 0.91 0.95 0.98
#> J2 0.91 1.00 0.82 0.92
#> J3 0.95 0.82 1.00 0.94
#> J4 0.98 0.92 0.94 1.00
summary(fit_icc_pair)
#> Intraclass correlation summary
#>   method      : Intraclass correlation (two-way random, agreement, single)
#>   dimensions  : 4 x 4
#>   pairs       : 6
#>   n_complete  : 50
#>   estimate    : 0.8209 to 0.9813
#>   most_negative: J2-J3 (0.8209)
#>   most_positive: J1-J4 (0.9813)
#> 
#> Strongest pairs by |estimate|
#> 
#>  item1 item2 estimate n_complete
#>  J1    J4    0.9813   50        
#>  J1    J3    0.9487   50        
#>  J3    J4    0.9448   50        
#>  J2    J4    0.9168   50        
#>  J1    J2    0.9063   50        
#> ... 1 more rows not shown (omitted)
#> Use as.data.frame()/tidy()/as.matrix() to inspect the full result.
print(fit_icc_overall)
#> Overall intraclass correlation
#>   method      : Overall intraclass correlation table
#>   subjects    : 50
#>   raters      : 4
#>   selected    : ICC2
#> 
#> Coefficient table
#> 
#>  coefficient label            estimate ... upr    selected
#>  ICC1        Single absolute  0.9160   ... 0.9471 FALSE   
#>  ICC2        Single random    0.9173   ... 0.9679  TRUE   
#>  ICC3        Single fixed     0.9751   ... 0.9846 FALSE   
#>  ICC1k       Average absolute 0.9776   ... 0.9862 FALSE   
#>  ICC2k       Average random   0.9779   ... 0.9918 FALSE   
#>  ICC3k       Average fixed    0.9936   ... 0.9961 FALSE   
#> ... 5 more variables not shown (omitted)
#> Use as.data.frame()/tidy()/as.matrix() to inspect the full result.

Pairwise versus overall ICC

This is the most important distinction in the ICC interface.

scope = "pairwise" answers: “How reliable is each specific pair of methods?”

scope = "overall" answers: “How reliable is the full set of methods when analysed jointly?”

Those are different quantities. The overall ICC cannot, in general, be recovered by averaging the pairwise matrix.

Consistency versus agreement

This simulation also includes systematic method bias, so it is a natural place to contrast type = "consistency" with type = "agreement".

fit_icc_cons <- icc(
  wide_methods,
  model = "twoway_random",
  type = "consistency",
  unit = "single",
  scope = "overall",
  ci = FALSE
)

fit_icc_agr <- icc(
  wide_methods,
  model = "twoway_random",
  type = "agreement",
  unit = "single",
  scope = "overall",
  ci = FALSE
)

data.frame(
  type = c("consistency", "agreement"),
  selected_coefficient = c(
    attr(fit_icc_cons, "selected_coefficient"),
    attr(fit_icc_agr, "selected_coefficient")
  ),
  estimate = c(
    attr(fit_icc_cons, "selected_row")$estimate,
    attr(fit_icc_agr, "selected_row")$estimate
  )
)
#>          type selected_coefficient  estimate
#> 1 consistency                 ICC3 0.9750694
#> 2   agreement                 ICC2 0.9172615

Consistency discounts additive method shifts, whereas agreement penalises them. When methods differ mainly by a systematic offset, consistency can therefore look substantially better than agreement.

Model, type, and unit

The classical ICC family is controlled by three arguments.

For pairwise ICC, average-measure output uses k = 2 because each estimate is based on exactly two methods. For overall ICC, average-measure output uses the full number of analysed columns.

Choosing among CCC, BA, and ICC

In practice these methods answer different questions.

There is overlap in interpretation, but these are not interchangeable estimators.