kth‐order Markov chain‐based approximation of the Shannon entropy of Gaussian photon‐counting processes

Abstract

A method for approximation of the Shannon entropy of Gaussian photon‐counting processes with infinite history was constructed on the memory function of these processes, described by autoregressive‐integrated moving average (ARIMA) models. Most frequently, photon‐counting processes are stationary or nonstationary multidimensional Gaussian discrete‐time stochastic ones which justify the use of the ARIMA models. Starting from the memory function, a memory time‐equivalent finite autoregressive representation of a given process with infinite history, i.e. a stationary finite‐order Gaussian Markov chain, was determined, then corresponding autocorrelation matrices were calculated from the truncated memory function using the Yule‐Walker equations, and an autocorrelation‐based formula for approximation of the entropy of the process through the entropy of its stationary Markovian representation was given. An ARMA(1,1) process together with its stationary (MA(1)) or nonstationary (IMA(0,1,1)) boundary cases were considered to demonstrate opposite changes in the entropy as the memory time increases at a fixed variance of the process: the entropy was found to decrease for stationary processes and increase for nonstationary ones. It was also found on experimental examples (perturbed human neutrophils and yeast cells) that those changes can be reversed by opposite changes in the process variance. The method allows us to determine, at any desired accuracy, the Shannon entropy of time‐discrete stochastic processes, and reveals new aspects of the relationship between the process' stationarity, memory, entropy and heteroskedasticity.

Citation

Kochel, B. (2004), "kth‐order Markov chain‐based approximation of the Shannon entropy of Gaussian photon‐counting processes", Kybernetes, Vol. 33 No. 9/10, pp. 1491-1509. https://doi.org/10.1108/03684920410546701