8.3.5 Changes in Model Performance

Standard experiments, agreed upon by the climate modelling community to facilitate model intercomparison (see Section 8.1.2.2), have produced archives of model output that make it easier to track historical changes in model performance. Most of the modelling groups that contributed output to the current MMD at PCMDI also archived simulations from their earlier models (circa 2000) as part of the Coupled Model Intercomparison Project (CMIP1&2). The TAR largely relied on the earlier generation of models in its assessment.

Based on the archived model output, it is possible to quantify changes in performance of evolving models.^[4] This can be done most straightforwardly by only considering the 14 modelling groups that contributed output from both their earlier and more recent models. One important aspect of model skill is how well the models simulate the seasonally varying global pattern of climatically important fields. The only monthly mean fields available in the CMIP1&2 archive are surface air temperature, precipitation and mean sea level pressure, so these are the focus of this analysis. Although the simulation conditions in the MMD 20th-century simulations were not identical to those in the CMIP1&2 control runs, the differences do not alter the conclusions summarised below because the large-scale climatological features dominate, not the relatively small perturbations resulting from climate change.

A summary of the ability of AOGCMs to simulate the seasonally varying climate state is provided by Figure 8.11, which displays error measures that gauge how well recent models simulate precipitation, sea level pressure and surface temperature, compared with their predecessors. The normalised RMS error shown is a so-called space-time statistic, computed from squared errors, summed over all 12 climatological months and over the entire globe, with grid cell values weighted by the corresponding grid cell area. This statistic can be used to assess the combined contributions of both spatial pattern errors and seasonal cycle errors. The RMS error is divided by the corresponding observed standard deviation of the field to provide a relative measure of the error. In Figure 8.11 this scaling implies that pressure is better simulated than precipitation, and that surface temperature is simulated best of all.

The models in Figure 8.11 are categorised based on whether or not flux adjustments were applied (see Section 8.2.7). Of the earlier generation models, 8 of the 14 models were flux adjusted, but only two of these groups continue this practice. Several conclusions can be drawn from the figure: 1) although flux-adjusted models on average have smaller errors than those without (in both generations), the smallest errors in simulating sea level pressure and surface temperature are found in models without flux adjustment; 2) despite the elimination of flux adjustment in all but two of the recent models, the mean error obtained from the recent suite of 14 models is smaller than errors found in the corresponding earlier suite of models; and 3) models without flux adjustment have improved on average, as have the flux-adjusted models. An exception to this last statement is the slight increase in mean RMS error for sea level pressure found in non-flux-adjusted models. Despite no apparent improvement in the mean in this case, three of the recent generation models have smaller sea level pressure errors than any of the earlier models.

Figure 8.11. Normalised RMS error in simulation of climatological patterns of monthly precipitation, mean sea level pressure and surface air temperature. Recent AOGCMs (circa 2005) are compared to their predecessors (circa 2000 and earlier). Models are categorised based on whether or not any flux adjustments were applied. The models are gauged against the following observation-based datasets: Climate Prediction Center Merged Analysis of Precipitation (CMAP; Xie and Arkin, 1997) for precipitation (1980–1999), European Centre for Medium Range Weather Forecasts 40-year reanalysis (ERA40; Uppala et al., 2005) for sea level pressure (1980–1999) and Climatic Research Unit (CRU; Jones et al., 1999) for surface temperature (1961–1990). Before computing the errors, both the observed and simulated fields were mapped to a uniform 4° x 5° latitude-longitude grid. For the earlier generation of models, results are based on the archived output from control runs (specifically, the first 30 years, in the case of temperature, and the first 20 years for the other fields), and for the recent generation models, results are based on the 20th-century simulations with climatological periods selected to correspond with observations. (In both groups of models, results are insensitive to the period selected.)

These results demonstrate that the models now being used in applications by major climate modelling groups better simulate seasonally varying patterns of precipitation, mean sea level pressure and surface air temperature than the models relied on by these same groups at the time of the TAR.