Discussion

    EVI is an efficient and easy to implement early-warning tool for an upcoming rise in the number of new cases. The performance of EVI, as expressed by its overall \(Se\) and \(Sp\), was, in all instances, high. A more important aspect lies in the fact that repetitive issuance of early warnings indicates the beginning of an epidemic wave. This is a consistent and remarkably stable finding across all countries and each of the United States (Fig. 1, 2 and http://83.212.174.99:3838/). In a similar manner, the absence of a series of early warnings implies that the number of new cases will remain stable or drop. The latter was also a consistent finding. Additionally, false early warnings (i.e. false positives) were isolated instances and did not occur in a consecutive series. There were few occasions with a consecutive absence of early warnings despite a continuing rise in the number of cases (i.e. false negatives). Nevertheless, such series of false negatives were always close to the peak of a wave. This finding is reasonable and could be interpreted as an early sign of reaching the peak because EVI depends on volatility and the increase in the number of new cases decelerates when approaching the peak of an epidemic wave. Positive and negative predictive values that are calculated at each time point can also be used to assess the probability that an early warning, or its absence, is true. In all instances, predictive values were high with the exception of few instances at the beginning of the time series due to the absence of enough data. 
    Work on the SIR and SIS models has revealed that moving-window estimates of the variance increase while approaching the emergence of a pathogen as well as during the elimination phase and that it can be used as an early warning tool \cite{O_Regan_2013}. EVI is based on the relative rather than the absolute change of the standard deviation because the latter depends on the underlying prevalence at each time point of the epidemic. Hence, a low threshold would be efficient in detecting a surge in the new cases at the beginning of an epidemic, when the baseline prevalence is low, but would have failed to do so for subsequent epidemic waves that commence from a higher baseline prevalence.  On the other hand, a high absolute threshold would have failed to capture waves at the beginning of the epidemic. EVI is based on the relative increase in volatility, which implicitly adjusts for the baseline prevalence at each point of the time series.
    In general, the ability of EVI to provide valid predictions does not seem to be affected by the fact that sampling and testing schemes for COVID-19 are mainly based on passive surveillance systems. EVI performed equally well among different countries with different control strategies, testing intensity and reporting accuracy and despite the fact that even within countries sampling and testing has changed over time and/or differs between regions \cite{brynildsrud2020covid,middelburg2020covid}. Restriction of the maximum window size \(\left(m_{\max}\right)\) to one month plays a key role, because reporting bias is expected to remain similar over short time periods. This form of non-differential misclassification leads to reporting rates that, though biased, do not have a significant impact on volatility, EVI and its predictive ability. Crucially, it is important that the data do not exhibit strong artifacts of recording bias, as there is no way for the method to distinguish between a trend due to underlying epidemic patterns and an observed trend due to changes in reporting practices or increased testing capacity or effort\cite{Halasa_2020}. This could for instance happen when a country changes its general testing regime, experiences local outbreaks and focus testing in a specific area, or targets other subgroups of the population than previously. Thus, EVI should preferably be evaluated for use in smaller geographical regions, such as counties or municipalities, if sufficient, high quality data are available. Undoubtedly,  all models are prone to limitations due to imperfect data \cite{Vespignani_2020} but  the continuing enhancement of active and passive surveillance systems - as the testing regimes and methods also improve - will lead to improved data quality. 
    The performance of EVI depends on the specified case definition and \(r\), parameters which are epidemic-specific and country-specific. Modifications to allow for different case definition and \(r\), for the different periods of an epidemic, are rather straightforward to implement. Parameters \(c\) and \(m\) are allowed to vary and take values that would satisfy the conditions set by the defined case and the desired accuracy. A point of concern is the selection of \(m_{\max}\). For an ongoing epidemic with multiple waves, as is the case with COVID-19, \(m_{\max}\) should be limited to a period shorter than the entire observation period. This prevents excess volatility of past epidemic waves from affecting the most recent volatility estimates and the ability of EVI to warn for upcoming waves that may be smaller and of lower volatility than previous ones. In our example, we limited \(m_{\max}\) to one month. EVI also depends on data intensity. Detailed data at the lowest time unit (i.e., days rather than weeks) is preferable in order to detect changes rapidly. In the COVID-19 example the 7-day moving average was analyzed instead of the daily reported cases because daily data had unnatural variability due to reporting variations between working days and weekends. Nevertheless, analysis based on the daily reported cases provided similar results (data not shown here). 
    Beyond the case of epidemics or rare events, like the COVID-19 pandemic, an important application of EVI can be in the context of syndromic surveillance \cite{henning2004syndromic}, not limited to outbreaks from terrorist attacks, but in its broader sense: the detection of temporal and spatial aberrations in the expected number of cases for signs and symptoms. Such systems already exist and utilize state-of-the-art information technologies within the context of public health \cite{heffernan2004syndromic} as well as one health \cite{Beltrán-Alcrudo2009,Gilbert2014}. EVI can provide an additional early warning tool for these systems.

Appendix