Appendix 3.A: Low-Pass Filters and Linear Trends

The time series used in this report have undergone diverse quality controls that have, for example, led to removal of outliers, thereby building in some smoothing. In order to highlight decadal and longer time-scale variations and trends, it is often desirable to apply some kind of low-pass filter to the monthly, seasonal or annual data. In the literature cited for the many indices used in this chapter, a wide variety of schemes was employed. In this chapter, the same filter was used wherever it was reasonable to do so. The desirable characteristics of such filters are 1) they should be easily understood and transparent; 2) they should avoid introducing spurious effects such as ripples and ringing (Duchon, 1979); 3) they should remove the high frequencies; and 4) they should involve as few weighting coefficients as possible, in order to minimise end effects. The classic low-pass filters widely used have been the binomial set of coefficients that remove 2∆t fluctuations, where ∆t is the sampling interval. However, combinations of binomial filters are usually more efficient, and those have been chosen for use here, for their simplicity and ease of use. Mann (2004) discusses smoothing time series and especially how to treat the ends. This chapter uses the ‘minimum slope’ constraint at the beginning and end of all time series, which effectively reflects the time series about the boundary. If there is a trend, it will be conservative in the sense that this method will underestimate the anomalies at the end.

The first filter (e.g., Figure 3.5) is used in situations where only the smoothed series is shown, and it is designed to remove interannual fluctuations and those on El Niño time scales. It has 5 weights 1/12 [1-3-4-3-1] and its response function (ratio of amplitude after to before) is 0.0 at 2 and 3∆t, 0.5 at 6∆t, 0.69 at 8∆t, 0.79 at 10∆t, 0.91 at 16∆t, and 1 for zero frequency, so for yearly data (∆t = 1) the half-amplitude point is for a 6-year period, and the half-power point is near 8.4 years.

The second filter used in conjunction with annual values (∆t =1) or for comparisons of multiple curves (e.g., Figure 3.8) is designed to remove fluctuations on less than decadal time scales. It has 13 weights 1/576 [1-6-19-42-71-96-106-96-71-42-19-6-1]. Its response function is 0.0 at 2, 3 and 4∆t, 0.06 at 6∆t, 0.24 at 8∆t, 0.41 at 10∆t, 0.54 at 12∆t, 0.71 at 16∆t, 0.81 at 20∆t, and 1 for zero frequency, so for yearly data the half-amplitude point is about a 12-year period, and the half-power point is 16 years. This filter has a very similar response function to the 21-term binomial filter used in the TAR.

Another low-pass filter, widely used and easily understood, is to fit a linear trend to the time series although there is generally no physical reason why trends should be linear, especially over long periods. The overall change in the time series is often inferred from the linear trend over the given time period, but can be quite misleading. Such measures are typically not stable and are sensitive to beginning and end points, so that adding or subtracting a few points can result in marked differences in the estimated trend. Furthermore, as the climate system exhibits highly nonlinear behaviour, alternative perspectives of overall change are provided by comparing low-pass-filtered values (see above) near the beginning and end of the major series.

The linear trends are estimated by Restricted Maximum Likelihood regression (REML, Diggle et al., 1999), and the estimates of statistical significance assume that the terms have serially uncorrelated errors and that the residuals have an AR1 structure. Brohan et al. (2006) and Rayner et al. (2006) provide annual uncertainties, incorporating effects of measurement and sampling error and uncertainties regarding biases due to urbanisation and earlier methods of measuring SST. These are taken into account, although ignoring their serial correlation. The error bars on the trends, shown as 5 to 95% ranges, are wider and more realistic than those provided by the standard ordinary least squares technique. If, for example, a century-long series has multi-decadal variability as well as a trend, the deviations from the fitted linear trend will be autocorrelated. This will cause the REML technique to widen the error bars, reflecting the greater difficulty in distinguishing a trend when it is superimposed on other long-term variations and the sensitivity of estimated trends to the period of analysis in such circumstances. Clearly, however, even the REML technique cannot widen its error estimates to take account of variations outside the sample period of record. Robust methods for the estimation of linear and nonlinear trends in the presence of episodic components became available recently (Grieser et al., 2002).

As some components of the climate system respond slowly to change, the climate system naturally contains persistence. Hence, the statistical significances of REML AR1-based linear trends could be overestimated (Zheng and Basher, 1999; Cohn and Lins, 2005). Nevertheless, the results depend on the statistical model used, and more complex models are not as transparent and often lack physical realism. Indeed, long-term persistence models (Cohn and Lins, 2005) have not been shown to provide a better fit to the data than simpler models.