Implications of regional improvement in global climate models for agricultural impact research

Here we have used various sets of observed data for all calculations. We have also used reanalysis data as it is considered to be in the upper bound of climate model skill [33]. We preferred to use various sources of observational data (of different nature) as this allows a more robust treatment of observational uncertainties. Each dataset used here would have biases and/or different degrees of suitability when an assessment of climate models is intended (e.g. station data may not be compared directly with coarse GCM grid cells, while reanalysis data for precipitation is expected to have large biases). For this reason, most studies assessing GCM predictions would attempt to use various different sets of observations, and so we have adopted a similar approach (see e.g. [23]).

For assessing mean climates, we have used four different sources of observation-based data:

• (1)  
  CL-WST: Following [30], data were gathered from the Global Historical Climatology Network [34] (GHCN), the World Meteorological Organization Climatology Normals (WMO CLINO), FAOCLIM 2.0 (Food and Agriculture Organization of the United Nations Agro-Climatic database) [35], and a number of other minor sources (see [36]). The final dataset contained data for 35 608 locations (precipitation), 16 875 locations (mean temperature), and 12 458 locations (diurnal temperature range) at the global level. Wet-day frequency data were not available in this dataset.
• (2)  
  CL-WCL: Global interpolated surfaces were downloaded from WorldClim [30] representative for the period 1950–2000. Global gridded data were downloaded at the resolution of 10 arcmin. Monthly maximum and minimum temperatures were used to compute diurnal temperature range. The final dataset comprised monthly climatological means of precipitation, mean temperature and diurnal temperature range for the 12 months of the year). As in CL-WST, wet-day frequency data were not available in CL-WCL.
• (3)  
  CL-CRU: The University of East Anglia Climatic Research Unit (CRU) high resolution interpolated climatology version 2.0 [29]. This dataset holds significant similarities to WorldClim in terms of input data, methods and gridded data [30], but is acknowledged to be more robust, as input data quality checking is reported to be much more rigorous [29]. We downloaded data at the only available resolution (10 arcmin) for monthly total precipitation, wet-day frequency, mean temperature, and diurnal temperature range.
• (4)  
  CL-E40: Finally, the open-access version (i.e. 2.5° × 2.5°) of the European Centre for Medium-Range Weather Forecasts (ECMWF) 40+ Reanalysis (ERA-40) [31] was used as it is a fair intermediate between observations and climate model outputs [37]. We used ERA-40 as it has shown better skill than other available reanalysis products [23], and the alternative use of bias-corrected reanalysis was deemed unnecessary owing to the sometimes unexpected effects of bias correction [25], particularly in areas where moist convection is the main driver of regional precipitation as well as the similarity in temperature data in reanalysis and bias-corrected reanalysis datasets. Daily mean, maximum and minimum temperatures and total precipitation were retrieved from the ECMWF archive (at http://data-portal.ecmwf.int/data/d/era40_daily/) for the period 1961–2000. Wet-day frequency was calculated as for all other datasets as the number of days in a month with precipitation above 0.1 mm, whereas diurnal temperature range was calculated as the difference between maximum and minimum temperatures. Daily data were then aggregated to the monthly level from which mean monthly climatology was calculated. This yielded gridded (2.5° × 2.5°) global datasets of total precipitation, wet-day frequency, mean temperature and diurnal temperature range for each month, as averages of all years in the period 1961–2000.

Data for assessing interannual variability were compiled from three sources:

• (1)  
  TS-CRU: the CRU time series (CRU-TS3.0 at www.cru.uea.ac.uk/cru/data/hrg) of monthly precipitation, wet-day frequency, mean temperature and diurnal temperature range for the period 1961–2000 were downloaded at the resolution of 0.5°;
• (2)  
  TS-WST: monthly time series of precipitation, mean, maximum and minimum temperature were downloaded from the GHCN version 2 [34] dataset and were then combined with precipitation series that were previously assembled by researchers at CIAT [36]. The CIAT weather station database contained data only for precipitation, and so this was the only variable for which the two sources (i.e. GHCN and CIAT) accounted data (i.e. temperature data consisted only of GHCN stations). The final dataset, at the global level, comprised 29 736 rainfall locations (CIAT and GHCN), 7198 mean temperature locations (only GHCN), and 4959 diurnal temperature locations (only GHCN); although not all locations had data for all months and years and many had data for less than 10 years. Wet-day frequency data were not available in TS-WST.
• (3)  
  TS-E40: the ERA-40 daily data were again used but this time only aggregated to the monthly level for each year, which produced monthly gridded (2.5° × 2.5°) datasets of total precipitation, wet-day frequency, mean temperature and diurnal temperature range for every year between 1961 and 2000.

All these datasets can be grouped in three categories: (1) point-based data (CL-WST, TS-WST), (2) gridded observed data (CL-CRU, CL-WCL, TS-CRU), and (3) gridded reanalysis data (CL-E40, TS-E40). By including these three types of data we expect to provide a broader picture of climate model skill than if only one type of data were used. Data were in all cases gridded to climate model resolution by averaging onto the model grid and these further re-gridded to 1° × 1° for presenting all results.

GCM data were downloaded from the CMIP3 [38] and the CMIP5 archives [20]. For CMIP3, only monthly precipitation, maximum, minimum and mean temperature data were available for the 20th century simulation (i.e. 20C3M). Wet-day frequency was thus not analyzed in CMIP3. Data were downloaded for 24 GCMs (supplementary table S1 available at stacks.iop.org/ERL/8/024018/mmedia), with only three models having missing data for maximum and minimum temperatures. Time series of monthly total precipitation and means of mean monthly temperature and diurnal temperature range were computed ('TS-C3'). From these, climatological means were further calculated ('CL-C3').

In relation to CMIP3, CMIP5 has a wider range of numerical experiments [20], and data for a larger number of models and model ensembles. CMIP3 historical ('20C3M') simulations included 24 CGCMs, whereas CMIP5's included more than 35 CGCMs [20]. Here, 26 of these presented data for the historical simulation at the time queried (February 2012), although the total number of simulations is 70 (supplementary table S2 available at stacks.iop.org/ERL/8/024018/mmedia), owing to individual ensemble members. CMIP5's experimental design includes individual perturbed physics and initial conditions ensemble members for a number of models and higher resolution models (supplementary figure S1 available at stacks.iop.org/ERL/8/024018/mmedia). Similarly, models have increased their complexity by including atmospheric chemistry, aerosols, the carbon cycle, and experiments at decadal timescales [20]. Available daily outputs of historical simulations (a total of 70, supplementary table S2) for the variables of interest were downloaded and processed for the years 1961–2000 in order to produce time series of total monthly precipitation, the number of wet days, and means for mean temperature and diurnal temperature range ('TS-C5'). Using these, climatological means for the same variables ('CL-C5') were produced. For both CMIP3 and CMIP5 ensembles multi-model means (MMM) were calculated using the climatological means and monthly time series of all GCM simulations.

Climate models and the MMM were assessed for their ability to represent mean climates for each of the four variables. Performance was assessed for five regions (figure 1). For each region and season we compared the climate model predictions and the observed (or reanalysis) data using all pixels in that particular geographic domain, thus assessing the time-mean geographic pattern. We used four metrics: (1) the Pearson product-moment correlation coefficient (R); (2) the root mean squared error (RMSE, equation (1)); (3) the RMSE normalized by the observed mean times 100 (RMSE_M, equation (1)); and (4) the RMSE normalized by the observed standard deviation times 100 (RMSE_SD).

$$\begin{eqnarray} &&{\mathrm{RMSE}}_{\mathrm{M}}=\frac{\mathrm{RMSE}}{\bar {X}}=\frac{\sqrt{\frac{\sum _{i=1}^{n}\left ({X}_{i}-{Y}_{i}\right )^{2}}{n}}}{\frac{\sum _{i=1}^{n}{X}_{i}}{n}} \end{eqnarray} \tag{ 1 }$$

where X and Y are the observed and GCM values (respectively) for a grid cell (i) in a given season (e.g. the JJA season) over a domain with n grid cells. X-bar is the average of observed values. Throughout the paper, we mostly report and conclude based on the RMSE_M because (1) it was generally a better indicator of model skill as it allowed better comparisons across climate models, regions and seasons than the RMSE; (2) high values of r were not always associated with high similarity between predicted and observed values; (3) errors generally exceeded observed spatial variability by a factor of 1.5 or more, and (4) the alternative use of the standard deviation of the time series as normalizing factor was not possible due to the unavailability of these data for all mean climate datasets.

Interannual variability in climate models was assessed following [32, 39], in which an interannual variability index (VI) is calculated for each climate model run as the differences between the ratios of model (M, termed TS-C3 and TS-C5 here) and observed (O, consisting of three different datasets: TS-CRU, TS-WST, and TS-E40) standard deviations (equation (2))

$$\begin{eqnarray} &&{\mathrm{VI}}_{v}^{i}=\left (\frac{{\sigma }_{\mathrm{M}v}^{i}}{{\sigma }_{\mathrm{O}v}^{i}}-\frac{{\sigma }_{\mathrm{O}v}^{i}}{{\sigma }_{\mathrm{M}v}^{i}}\right )^{2} \end{eqnarray} \tag{ 2 }$$

where σ is the standard deviation of the time series (1961–2000 in this study) of a variable (v) for a given grid point (i). The index is always positive and with no upper limit. As opposed to all mean climate calculations, VI calculations were performed individually for each grid cell using all years in the period 1961–2000.

In order to compare the two model ensembles, we first present some advantages of CMIP5 in terms of experimental design and model resolution. We then draw probability density functions (PDFs) using country and season values of each climate model of each ensemble. Finally, we select thresholds to classify the values of RMSE_M,RMSE_SD, and VI. These thresholds were then used to separate individual countries and seasons. By counting the per cent of country-season combinations above or below these thresholds out of the total (total being 22 countries times 4 seasons = 88), a more objective comparison of the differences between regions, model ensembles, and variables was achieved. For RMSE_M and RMSE_SD we chose a value of 40% (chosen to be in agreement with the value of VI chosen). For VI, we focused on the work of [39], who defined the upper bound of skilled models to be at 0.5. A value of VI below 0.5 ensures that simulated interannual variability is between −25 and +40% of the observed value. Albeit somewhat arbitrary, the thresholds chosen are consistent between them, and are likely to be representative of boundaries beyond which impact models would be severely constrained [40]. The use of other (albeit close to the present ones) thresholds showed no effect on our conclusions.

The ability of CMIP GCMs to represent geographical variation in mean climate over a year is neither uniform nor consistent. The lowest performance is observed for precipitation in the DJF period. Conversely, seasonal mean temperatures show the highest correlations and overall lowest RMSE_M and RMSE_SD values (figure 2, supplementary figures S2–S7 available at stacks.iop.org/ERL/8/024018/mmedia). Seasonal mean temperatures in the CMIP3 model ensemble depict RMSE between 1 and 18 ° C (RMSE_M between 2 and 130%), whereas those of CMIP5 show RMSE in the range 1–16 ° C (5–100% of the mean, 5–600% of the standard deviation) (figure 2, supplementary figures S2–S7). However, the majority of models depict RMSE < 5 °C in both ensembles (figure 2, supplementary figures S2 and S5). Only in Nepal the majority of models show RMSE > 10 °C in all seasons, owing to the temperature variations across the Himalayas [32]. In Africa, errors are lower, with CMIP3 model RMSE values generally between 2 and 10 ° C (5–40% of the mean, supplementary figure S6). CMIP5 models show lower values for all metrics. In particular for CMIP5, South African countries show RMSE values generally below 2 ° C (10% of the mean, supplementary figure S3). Results are consistent between observed and reanalysis datasets, hence we focus on observed datasets.

Zoom In Zoom Out Reset image size

Diurnal temperature range shows higher values for all skill metrics, particularly for the Andes, where RMSE reached 15 ° C in the Andes (Ecuador, Peru, and Bolivia). Most models in both ensembles depict errors above 40% with respect to the mean for all seasons and regions (figure 2, supplementary figures S2–S3), but RMSE values are very large in relation to spatial variability, and typically exceed 100%. Across Africa, night-day temperature differences are simulated more accurately as compared to Asia and the Andes (figure 2, supplementary figures S2–S7). Errors in precipitation are closely tied to errors in the wet-day frequency. Models overestimate the number of days, while underestimating the total rainfall [32]. Particularly in monsoon-driven seasonal climates such as those of West Africa and South Asia, RMSE values for seasonal precipitation and wet-day frequency (the latter only for CMIP5) are large both in absolute terms and in relation to the mean and variability (figure 2, supplementary figures S2–S7). In both ensembles, precipitation errors are the largest in South Asia (in Bangladesh RMSE values reach 2000 mm season⁻¹), followed by the Andes, where RMSE values typically exceed 500 mm season⁻¹ (50% of the mean) (figure 1, supplementary figures S2–S7). Seasonal precipitation RMSE is typically below 500 mm season⁻¹ across African countries. Wet-day frequency RMSE varies between 150 and 200 days yr⁻¹.

We find interannual variability significantly misrepresented in both model ensembles. Some models, however, show strengths in some of the regions: at all grid cells, there is always at least one GCM with VI < 0.5, but in all cases the ensemble maxima are above this threshold. This indicated that models' skill in simulated interannual variability is generally not geographically consistent. Accurate simulations of interannual variability (i.e. VI < 0.5) are only achieved for mean temperature across West Africa and Southern Africa, in addition to precipitation in Southern Africa (only for CMIP5) (supplementary figure S8). The wet-day frequency, only analyzed for CMIP5, is the most poorly simulated of all variables, owing the its limited predictability [11].

Our comparison of CMIP5 with its predecessor indicates that gains in skill have occurred mainly in simulated mean temperatures and diurnal range. We report an increase in the frequency of the left-hand side of the PDFs of RMSE_M. Increases are in the range 5–15% for climatological mean temperatures and 3–5% for climatological diurnal range (figure 3, supplementary figure S9). This result is consistent throughout the seasons. Conversely, gains in skill in reproducing seasonal precipitation are limited to the JJA season, with gains being limited to 1–2% increases in frequency of low RMSE_M values (supplementary figure S10). Improvements in interannual variability occur to a lesser extent, but are consistent for the three variables (supplementary figures S11 and S12). Improvements in interannual variability are the largest in the JJA season, where increases in the frequency of values of VI below 0.5 are 1–5%, with little difference across variables. Importantly, the spread of the PDFs (i.e. shading in figure 3, and supplementary figures S9–S11) is lower in CMIP5 in relation to CMIP3, although the differences are not large.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We report significant regional differences in the increases in model skill from one ensemble to the other (figure 4, supplementary figures S12–S13). Gains in mean climate model skill are larger in South Asia for all variables. Conversely, interannual variability improved more significantly over West Africa, where skill is already relatively high (figure 4, bottom).

In CMIP5, model complexity has increased [19, 20]. This implies that while newly incorporated processes have increased the physical plausibility of the models, skill has either maintained or increased (figure 3). This is likely to be an important step forward, but still only one of many needed if the final aim is to facilitate impact assessment and adaptation. Two comparisons were hereby performed to investigate the potential effect of GCM errors on agricultural impact modeling. First, we compared temperature thresholds in 12 major crops (table 1) with the mean RMSE in CMIP5's mean climatological temperatures. We find that, if GCM outputs are used with no bias treatment into crop models, threshold exceedance could be increased by 6–69% for all crops as a cause of climate model error. Although this risk has decreased by roughly 15% from CMIP3 to CMIP5 (owing to an average reduction of 15% in RMSE), this comparison suggests that raw CMIP5 simulated regional temperatures are of limited use for agricultural impact research.

Zoom In Zoom Out Reset image size

Second, we used recent literature to illustrate the degree of sensitivity of cropping systems to variation in prevailing climate conditions. Although differences between future and current yields are expected to arise primarily from the climate change signal [26], model errors could bias such signal [41], hence biasing estimates of future cropping system sensitivity to climate. We find large sensitivities in cropping systems to +2 ° C increases in mean temperature and 20% decreases in seasonal rainfall (figure 5). Median sensitivity to temperature ranges from −23% (wheat) to −12% (sorghum), whereas that related to 20% increase in precipitation was −2.5% for maize, −7.5% for sorghum and −6% for wheat. In our analyses, GCMs exceeded 20% RMSE_M in the majority of cases (supplementary figure S3) and 2 ° C in the majority of seasons, particularly in the Andes, East Africa and South Asia (figure 2).

Temperature is the primary driver of crop phenology, but also affects fertility, canopy growth, plant senescence and nutrient absorption. On the other hand, total seasonal rainfall as well as the number of days with rainfall are major drivers in the world's 50% of agricultural lands that are rainfed [42]. Especially for regions where crops suffer from limited water availability, biased seasonal rainfall or wrongly timed rainfall can have large effect in model simulations [40]. Water-induced crop failures are to a large extent dependent on seasonally timed stresses that trigger death of either photosynthetic or reproductive organs in the plant. Thus, lack of soil moisture due to biased input rainfall can constrain the predictability of future crop failures (see [43]). Although the absolute effects of model errors over impact estimates are difficult to determine without crop model simulations being carried out, we argue that the use of raw CMIP5 data into impact models could significantly under- or overestimate cropping system sensitivity by 2.5–7.5% for precipitation-driven areas and 1.3–23% for temperature-driven areas.