Author: Jorge Cabral

# Further considerations

#### Jorge Cabral

## Normalized Entropy

Golan et al. [1] defined normalized entropy for the signal, \(\mathbf{X}\boldsymbol{\beta}\), in the GME framework, as

\[\begin{align} \qquad \qquad \qquad \qquad \qquad S(\mathbf{\widehat{p}})=\frac{-\mathbf{\widehat{p}}' \text{ln}\mathbf{\widehat{p}}}{(K+1)\text{ln} M} \qquad \qquad \qquad \qquad \qquad (3) \end{align}\] where \(S(\mathbf{\widehat{p}})\in[0,1]\) and \(S(\mathbf{\widehat{p}})=1\) indicates perfect uncertainty, and \(S(\mathbf{\widehat{p}})=0\) indicates no uncertainty.

In the GCE framework it can be defined as

\[\begin{align} \qquad \qquad \qquad \qquad \qquad S(\mathbf{\widehat{p}})=\frac{-\mathbf{\widehat{p}}' \text{ln}\mathbf{\widehat{p}}}{-\mathbf{\widehat{q}}' \text{ln}\mathbf{\widehat{q}}} \qquad \qquad \qquad \qquad \qquad (4) \end{align}\]

but in this case the we can no longer state that \(S(\mathbf{\widehat{p}})\in[0,1]\).

`GCEstim` package reports normalized entropies but it uses always the definition in (3) independently of the framework used.

Consider `dataGCE` (see “Generalized Maximum Entropy framework” and Generalized Cross Entropy framework”).

The GME estimation can be obtained, for instance, with

    res.lmgce.100.GME <-
      GCEstim::lmgce(
        y ~ .,
        data = dataGCE,
        cv = TRUE,
        cv.nfolds = 5,
        support.signal = c(-100, 100),
        support.signal.points = 5,
        twosteps.n = 0,
        seed = 230676
      )

and the GCE estimation with

    res.lmgce.100.GCE <-
      GCEstim::lmgce(
        y ~ .,
        data = dataGCE,
        cv = TRUE,
        cv.nfolds = 5,
        support.signal = c(-100, 100),
        support.signal.points =
          matrix(
            c(
              rep(1 / 5, 5),
              c(0.1, 0.1, 0.6, 0.1, 0.1),
              c(0.1, 0.1, 0.6, 0.1, 0.1),
              rep(1 / 5, 5),
              rep(1 / 5, 5),
              rep(1 / 5, 5)
            ),
            ncol = 5,
            byrow = TRUE
          ),
        twosteps.n = 0,
        seed = 230676
      )

The `NormEnt` extracts the normalized entropy from the models by default (`model=TRUE`).

    NormEnt(res.lmgce.100.GME)
    #> [1] 0.9977441
    NormEnt(res.lmgce.100.GCE)
    #> [1] 0.9187735

Each estimate has its own normalized entropy associated (`model=FALSE`)

    NormEnt(res.lmgce.100.GME, model = FALSE)
    #> (Intercept)        X001        X002        X003        X004        X005
    #>   0.9999346   0.9999985   0.9997937   0.9993152   0.9956162   0.9918064

    NormEnt(res.lmgce.100.GCE, model = FALSE)
    #> (Intercept)        X001        X002        X003        X004        X005
    #>   0.9999345   0.7627071   0.7627323   0.9993525   0.9958272   0.9920873

## References

1.
Golan A, Judge GG, Miller D. Maximum Entropy Econometrics : Robust Estimation with Limited Data. Wiley; 1996.

## Acknowledgements

This work was supported by Fundação para a Ciência e Tecnologia (FCT) through CIDMA and projects https://doi.org/10.54499/UIDB/04106/2020 and https://doi.org/10.54499/UIDP/04106/2020.