Agreement and reliability are related to correlation, but they are not the same problem. Correlation describes co-movement. Agreement describes similarity on the measurement scale itself. Reliability describes the proportion of variation attributable to stable differences among subjects rather than to measurement error or method disagreement.
This vignette covers the wide-data functions:
ccc()ba()icc()Lin’s concordance correlation coefficient combines precision and accuracy in a single number. Bland-Altman analysis separates that question into estimated bias and limits of agreement.
library(matrixCorr)
set.seed(40)
ref <- rnorm(50, mean = 100, sd = 10)
m1 <- ref + rnorm(50, sd = 2)
m2 <- ref + 1.2 + rnorm(50, sd = 3)
fit_ba <- ba(m1, m2)
fit_ccc <- ccc(data.frame(m1 = m1, m2 = m2), ci = TRUE)
print(fit_ba)
#> Bland-Altman preview:
#> based_on : 50
#> loa_rule : mean +/- 1.96 * SD
#> ci : 95%
#> sd_diff : 3.722
#> width : 14.589
#>
#> quantity estimate lwr upr
#> Mean difference -1.290 -2.347 -0.232
#> Lower LoA -8.584 -10.416 -6.752
#> Upper LoA 6.005 4.173 7.837
summary(fit_ccc)
#> Lin's concordance summary
#> dimensions : 2 x 2
#> pairs : 1
#> estimate : 0.9299
#> most_negative: m1-m2 (0.9299)
#> most_positive: m1-m2 (0.9299)
#> ci : 95%
#> ci_method : delta_method
#> ci_width : 0.08
#>
#> Strongest pairs by |estimate|
#>
#> item1 item2 estimate lwr upr n_complete
#> m1 m2 0.9299 0.88 0.96 50The two summaries are complementary rather than redundant.
ccc() gives a single concordance coefficient, while
ba() makes the scale of disagreement explicit.
icc() extends the wide-data reliability workflow in two
directions. It can return a pairwise matrix across method pairs, or it
can return the overall classical ICC table for the full set of
methods.
wide_methods <- data.frame(
J1 = ref + rnorm(50, sd = 1.5),
J2 = ref + 4.0 + rnorm(50, sd = 1.8),
J3 = ref - 3.0 + rnorm(50, sd = 2.0),
J4 = ref + rnorm(50, sd = 1.6)
)
fit_icc_pair <- icc(
wide_methods,
model = "twoway_random",
type = "agreement",
unit = "single",
scope = "pairwise"
)
fit_icc_overall <- icc(
wide_methods,
model = "twoway_random",
type = "agreement",
unit = "single",
scope = "overall",
ci = TRUE
)
print(fit_icc_pair, digits = 2)
#> Intraclass correlation matrix
#> method : Intraclass correlation (two-way random, agreement, single)
#> dimensions : 4 x 4
#>
#> J1 J2 J3 J4
#> J1 1.00 0.91 0.95 0.98
#> J2 0.91 1.00 0.82 0.92
#> J3 0.95 0.82 1.00 0.94
#> J4 0.98 0.92 0.94 1.00
summary(fit_icc_pair)
#> Intraclass correlation summary
#> method : Intraclass correlation (two-way random, agreement, single)
#> dimensions : 4 x 4
#> pairs : 6
#> n_complete : 50
#> estimate : 0.8209 to 0.9813
#> most_negative: J2-J3 (0.8209)
#> most_positive: J1-J4 (0.9813)
#>
#> Strongest pairs by |estimate|
#>
#> item1 item2 estimate n_complete
#> J1 J4 0.9813 50
#> J1 J3 0.9487 50
#> J3 J4 0.9448 50
#> J2 J4 0.9168 50
#> J1 J2 0.9063 50
#> ... 1 more rows not shown (omitted)
#> Use as.data.frame()/tidy()/as.matrix() to inspect the full result.
print(fit_icc_overall)
#> Overall intraclass correlation
#> method : Overall intraclass correlation table
#> subjects : 50
#> raters : 4
#> selected : ICC2
#>
#> Coefficient table
#>
#> coefficient label estimate ... upr selected
#> ICC1 Single absolute 0.9160 ... 0.9471 FALSE
#> ICC2 Single random 0.9173 ... 0.9679 TRUE
#> ICC3 Single fixed 0.9751 ... 0.9846 FALSE
#> ICC1k Average absolute 0.9776 ... 0.9862 FALSE
#> ICC2k Average random 0.9779 ... 0.9918 FALSE
#> ICC3k Average fixed 0.9936 ... 0.9961 FALSE
#> ... 5 more variables not shown (omitted)
#> Use as.data.frame()/tidy()/as.matrix() to inspect the full result.This is the most important distinction in the ICC interface.
scope = "pairwise" answers: “How reliable is each
specific pair of methods?”
scope = "overall" answers: “How reliable is the full set
of methods when analysed jointly?”
Those are different quantities. The overall ICC cannot, in general, be recovered by averaging the pairwise matrix.
This simulation also includes systematic method bias, so it is a
natural place to contrast type = "consistency" with
type = "agreement".
fit_icc_cons <- icc(
wide_methods,
model = "twoway_random",
type = "consistency",
unit = "single",
scope = "overall",
ci = FALSE
)
fit_icc_agr <- icc(
wide_methods,
model = "twoway_random",
type = "agreement",
unit = "single",
scope = "overall",
ci = FALSE
)
data.frame(
type = c("consistency", "agreement"),
selected_coefficient = c(
attr(fit_icc_cons, "selected_coefficient"),
attr(fit_icc_agr, "selected_coefficient")
),
estimate = c(
attr(fit_icc_cons, "selected_row")$estimate,
attr(fit_icc_agr, "selected_row")$estimate
)
)
#> type selected_coefficient estimate
#> 1 consistency ICC3 0.9750694
#> 2 agreement ICC2 0.9172615Consistency discounts additive method shifts, whereas agreement penalises them. When methods differ mainly by a systematic offset, consistency can therefore look substantially better than agreement.
The classical ICC family is controlled by three arguments.
model selects the one-way, two-way random, or two-way
mixed formulation.type selects consistency or agreement.unit selects single-measure or average-measure
reliability.For pairwise ICC, average-measure output uses k = 2
because each estimate is based on exactly two methods. For overall ICC,
average-measure output uses the full number of analysed columns.
In practice these methods answer different questions.
ccc() when one concordance coefficient per pair is
the main target.ba() when the size and direction of disagreement
should be visible on the original measurement scale.icc() when the target is reliability under a
classical variance components interpretation.There is overlap in interpretation, but these are not interchangeable estimators.