# Evidence-based Bayesian disaggregation

## What this package does

Given an observed aggregate price index \(\mathrm{cpi}_t\) and a matrix of (known) sectoral aggregation weights \(W_{t,k}\) — value-added (VAB) shares — the goal is to recover the \(K\) latent sectoral price indices \(\varphi_{t,k}\) that the aggregate is made of. The sectoral indices then feed a downstream nested Ornstein–Uhlenbeck model (`bayesianOU`) as the market price \(\varphi\).

The disaggregation is genuinely Bayesian: the aggregate enters as evidence (an observation density), and the sectoral indices come out as a posterior with credible intervals, not as a single deterministic re-weighting.

## Why “evidence-based”: the F1–F6 history

The 0.1.x family advertised “MCMC-free Bayesian disaggregation”, but the aggregate CPI never entered the computation (F1): the “posterior” was derived from the prior weight matrix alone, the Dirichlet concentration cancelled on renormalization (F2), the temporal pattern cancelled too (F3), an “efficiency” term was a fixed constant (F4), there were no recovery tests (F5), and a correlation helper opportunistically picked whichever of Pearson/Spearman was larger (F6). That foundational defect — not using the data — cannot be patched within a deterministic re-weighting; the fix is a model that conditions on the aggregate. The deterministic family has been removed; two honest Bayesian engines replace it.

## The model (state-space, “Model A”)

Latent state in logs, with a random walk plus drift and partial pooling: \[ \log \varphi_{t,k} = \log \varphi_{t-1,k} + \delta_k + \tau_k\,\eta_{t,k}, \qquad \eta_{t,k}\sim\mathcal N(0,1), \] with \(\delta_k \sim \mathcal N(\delta_\mu,\delta_\sigma)\) and \(\log\tau_k \sim \mathcal N(\mu_{\log\tau}, \sigma_{\log\tau})\) (the drift and the innovation scale are pooled across sectors). The cross-section at \(t=1\) is anchored at the aggregate level with an estimable dispersion \(\omega_{\text{struct}}\) (the real concentration the old Dirichlet \(\gamma\) failed to be): \[ \log\varphi_{1,k} = \log(\text{phi1\_center}) + \omega_{\text{struct}}\,z_k . \] The aggregate is the genuine observation: \[ \mathrm{cpi}_t \sim \mathrm{Student\text{-}t}\!\left(\nu,\ \textstyle\sum_k W_{t,k}\,\varphi_{t,k},\ \sigma\right), \] (Gaussian if `student_obs = FALSE`).

### Identification, honestly (rigour by layers)

The aggregate \(\sum_k W\varphi\) is strongly identified by the observation density. The per-sector split is only weakly identified: at each period one linear combination of the \(K\) sectors is pinned by the CPI, and the remaining \(K-1\) directions are governed by the cross-sectional prior plus temporal smoothness. So the per-sector intervals are honestly wide and prior-influenced. This is not a defect to hide — it is the correct uncertainty, and it is precisely why we feed the full posterior draws (not a point estimate) to the OU by multiple imputation: the sectoral uncertainty is propagated, not faked away.

## Two engines, one trade-off

- Closed-form (conjugate) — `disaggregate_conjugate()`. A linear-Gaussian random walk in levels with the same aggregate observation; its exact posterior is the Kalman filter + RTS smoother, with no MCMC. Joint posterior draws come from the Durbin–Koopman simulation smoother. This is the correct realization of the original “MCMC-free posterior” idea.
- MCMC — `disaggregate_statespace()`. The richer model above (log scale ⇒ positivity, Student-t ⇒ robustness to aggregate outliers, hierarchical pooling), which is not conjugate and therefore needs HMC.

Both are Bayesian. Closed form buys speed and exactness at the cost of a simpler (Gaussian, linear) model; MCMC buys richness at the cost of sampling.

## A runnable example (closed-form engine)

    library(BayesianDisaggregation)

    sim <- simulate_disagg(T = 30, K = 4, seed = 1)   # synthetic CPI + VAB weights
    bl  <- disaggregate_conjugate(sim$cpi, sim$W, n_draws = 100, seed = 1)
    bl
    #> <disagg_conjugate>  closed-form linear-Gaussian baseline (Kalman/RTS)
    #>   periods T = 30, sectors K = 4, joint draws = 100
    #>   aggregate Gaussian log-likelihood = -64.56

    ## the smoothed aggregate tracks the CPI tightly (aggregate is well identified)
    round(cor(bl$agg_summary[, "median"], sim$cpi), 4)
    #> [1] 0.999

    ## joint posterior draws: the [T, K, draws] contract consumed by the nested OU
    dim(bl$phi_draws)
    #> [1]  30   4 100

## The MCMC engine (sketch, not evaluated here)

    fit <- disaggregate_statespace(sim$cpi, sim$W, chains = 4, iter = 2000, warmup = 1000)
    fit$diagnostics                 # rhat_max, divergences
    dim(fit$phi_draws)              # T x K x draws
    str(fit$phi_summary)            # median, q2.5, q97.5 (T x K each)

    ## couple to the nested OU (uncertainty propagated by Rubin's rule):
    ## bayesianOU::fit_ou_nested_mi(phi_draws = fit$phi_draws, X = Phi_index, ...)

From Excel directly, reusing the bundled readers:

    cpi_file <- system.file("extdata", "CPI.xlsx", package = "BayesianDisaggregation")
    w_file   <- system.file("extdata", "WEIGHTS.xlsx", package = "BayesianDisaggregation")
    fit <- disaggregate_from_files(cpi_file, w_file, chains = 2, iter = 1000)

## Data note: comparing index vs index

The model is about index levels, so the CPI must be a level series (FRED units “Index source base”, aggregation “Average” for annual data — never a rate-of-change), re-indexed to the same base as the production prices it will be compared against (e.g. 1982–1984 = 100 via the project’s `convert_to_index`). Feeding a percent-change series here is a category error: the aggregate would not be on the same scale as \(\sum_k W\varphi\).

## Coupling to the nested OU

`disaggregate_statespace()$phi_draws` (or `disaggregate_conjugate(..., n_draws = M)$phi_draws`) is a `[T, K, M]` array — exactly the multiple-imputation input of `bayesianOU::fit_ou_nested_mi()`. The OU refits once per imputation and combines the analyses by Rubin’s rule, so the disaggregation uncertainty becomes part of the OU posterior. ```