Dear Tim,
Thanks for the "heads up". As Phil mentioned, I was already aware of
this. The Douglass et al. paper was rejected twice before it was finally
accepted by IJC. I think this paper is a real embarrassment for the IJC.
It has serious scientific flaws. I'm already working on a response.
Phil can tell you about some of the other sordid details of Douglass et
al. These guys ignored information from radiosonde datasets that did not
support their "models are wrong" argument (even though they had these
datasets in their possession). Pretty deplorable behaviour...
Douglass is the guy who famously concluded (after examining the
temperature response to Pinatubo) that the climate system has negative
sensitivity. Amazingly, he managed to publish that crap in GRL. Christy
sure does manage to pick some brilliant scientific collaborators...
With best regards,
Ben
Tim Osborn wrote:
> Hi Ben,
>
> I guess it's likely that you're aware of the Douglass paper that's just
> come out in IJC, but in case you aren't then a reprint is attached.
> They are somewhat critical of your 2005 paper, though I recall that some
> (most?) of Douglass' previous papers -- and papers that he's tried to
> get through the review process -- appear to have serious problems.
>
> cc Phil & Keith for your interest too!
>
> Cheers
>
> Tim
> Dr Timothy J Osborn, Academic Fellow
> Climatic Research Unit
> School of Environmental Sciences
> University of East Anglia
> Norwich  NR4 7TJ, UK
>
> e-mail:   t.osborn@uea.ac.uk
> phone:    +44 1603 592089
> fax:      +44 1603 507784
> web:      ~timo/
> sunclock: ~timo/sunclock.htm
>
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------


Dear folks,
I've been doing some calculations to address one of the statistical
issues raised by the Douglass et al. paper in the International Journal
of Climatology. Here are some of my results.
Recall that Douglass et al. calculated synthetic T2LT and T2
temperatures from the CMIP-3 archive of 20th century simulations
("20c3m" runs). They used a total of 67 20c3m realizations, performed
with 22 different models. In calculating the statistical uncertainty of
the model trends, they introduced sigma{SE}, an "estimate of the
uncertainty of the mean of the predictions of the trends". They defined
sigma{SE} as follows:
sigma{SE} = sigma / sqrt(N - 1), where
"N = 22 is the number of independent models".
As we've discussed in our previous correspondence, this definition has
serious problems (see comments from Carl and Steve below), and allows
Douglass et al. to reach the erroneous conclusion that modeled T2LT and
T2 trends are significantly different from the observed T2LT and T2
trends in both the RSS and UAH datasets. This comparison of simulated
and observed T2LT and T2 trends is given in Table III of Douglass et al.
[As an amusing aside, I note that the RSS datasets are referred to as
"RSS" in this table, while UAH results are designated as "MSU". I guess
there's only one true "MSU" dataset...]
I decided to take a quick look at the issue of the statistical
significance of differences between simulated and observed tropospheric
temperature trends. My first cut at this "quick look" involves only UAH
and RSS observational data - I have not yet done any tests with
radiosonde datas, UMD T2 data, or satellite results from Zou et al.
I operated on the same 49 realizations of the 20c3m experiment that we
used in Chapter 5 of CCSP 1.1. As in our previous work, all model
results are synthetic T2LT and T2 temperatures that I calculated using a
static weighting function approach. I have not yet implemented Carl's
more sophisticated method of estimating synthetic MSU temperatures from
model data (which accounts for effects of topography and land/ocean
differences). However, for the current application, the simple static
weighting function approach is more than adequate, since we are focusing
on T2LT and T2 changes over tropical oceans only - so topographic and
land-ocean differences are unimportant. Note that I still need to
calculate synthetic MSU temperatures from about 18-20 20c3m realizations
which were not in the CMIP-3 database at the time we were working on the
CCSP report. For the full response to Douglass et al., we should use the
same 67 20c3m realizations that they employed.
For each of the 49 realizations that I processed, I first masked out all
tropical land areas, and then calculated the spatial averages of
monthly-mean, gridded T2LT and T2 data over tropical oceans (20N-20S).
All model and observational results are for the common 252-month period
from January 1979 to December 1999 - the longest period of overlap
between the RSS and UAH MSU data and the bulk of the 20c3m runs. The
simulated trends given by Douglass et al. are calculated over the same
1979 to 1999 period; however, they use a longer period (1979 to 2004)
for calculating observational trends - so there is an inconsistency
between their model and observational analysis periods, which they do
not explain. This difference in analysis periods is a little puzzling
given that we are dealing with relatively short observational record
lengths, resulting in some sensitivity to end-point effects.
I then calculated anomalies of the spatially-averaged T2LT and T2 data
(w.r.t. climatological monthly-means over 1979-1999), and fit
least-squares linear trends to model and observational time series. The
standard errors of the trends were adjusted for temporal autocorrelation
of the regression residuals, as described in Santer et al. (2000)
["Statistical significance of trends and trend differences in
layer-average atmospheric temperature time series"; JGR 105, 7337-7356.]
Consider first panel A of the attached plot. This shows the simulated
and observed T2LT trends over 1979 to 1999 (again, over 20N-20S, oceans
only) with their adjusted 1-sigma confidence intervals). For the UAH and
RSS data, it was possible to check against the adjusted confidence
intervals independently calculated by Dian during the course of work on
the CCSP report. Our adjusted confidence intervals are in good
agreement. The grey shaded envelope in panel A denotes the 1-sigma
standard error for the RSS T2LT trend.
There are 49 pairs of UAH-minus-model trend differences and 49 pairs of
RSS-minus-model trend differences. We can therefore test - for each
model and each 20c3m realization - whether there is a statistically
significant difference between the observed and simulated trends.
Let bx and by represent any single pair of modeled and observed trends,
with adjusted standard errors s{bx} and s{by}. As in our previous work
(and as in related work by John Lanzante), we define the normalized
trend difference d as:
d = (bx - by) / sqrt[ (s{bx})**2 + (s{by})**2 ]
Under the assumption that d is normally distributed, values of d > +1.96
or
Hi Steve
>
> I'd say it's the equivalent of rolling a 6-sided die a hundred times, and
> finding a mean value of ~3.5 and a standard deviation of ~1.7, and
> calculating the standard error of the mean to be ~0.17 (so far so
> good).  An then rolling the die one more time, getting a 2, and
> claiming that the die is no longer 6 sided because the new measurement
> is more than 2 standard errors from the mean.
>
> In my view, this problem trumps the other problems in the paper.
> I can't believe Douglas is a fellow of the American Physical Society.
>
> -Carl
>
>
> At 02:07 AM 12/6/2007, you wrote:
>> If I understand correctly, what Douglass et al. did makes the stronger
>> assumption that unforced variability is *insignificant*.  Their
>> statistical test is logically equivalent to falsifying a climate model
>> because it did not consistently predict a particular storm on a
>> particular day two years from now.
>
>
> Dr. Carl Mears
> Remote Sensing Systems
> 438 First Street, Suite 200, Santa Rosa, CA 95401
> mears@remss.com
> 707-545-2904 x21
> 707-545-2906 (fax))
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------
Attachment Converted: "c:\eudora\attach\douglass_reply1.pdf"


Dear folks,
I've been doing some calculations to address one of the statistical
issues raised by the Douglass et al. paper in the International Journal
of Climatology. Here are some of my results.
Recall that Douglass et al. calculated synthetic T2LT and T2
temperatures from the CMIP-3 archive of 20th century simulations
("20c3m" runs). They used a total of 67 20c3m realizations, performed
with 22 different models. In calculating the statistical uncertainty of
the model trends, they introduced sigma{SE}, an "estimate of the
uncertainty of the mean of the predictions of the trends". They defined
sigma{SE} as follows:
sigma{SE} = sigma / sqrt(N - 1), where
"N = 22 is the number of independent models".
As we've discussed in our previous correspondence, this definition has
serious problems (see comments from Carl and Steve below), and allows
Douglass et al. to reach the erroneous conclusion that modeled T2LT and
T2 trends are significantly different from the observed T2LT and T2
trends in both the RSS and UAH datasets. This comparison of simulated
and observed T2LT and T2 trends is given in Table III of Douglass et al.
[As an amusing aside, I note that the RSS datasets are referred to as
"RSS" in this table, while UAH results are designated as "MSU". I guess
there's only one true "MSU" dataset...]
I decided to take a quick look at the issue of the statistical
significance of differences between simulated and observed tropospheric
temperature trends. My first cut at this "quick look" involves only UAH
and RSS observational data - I have not yet done any tests with
radiosonde datas, UMD T2 data, or satellite results from Zou et al.
I operated on the same 49 realizations of the 20c3m experiment that we
used in Chapter 5 of CCSP 1.1. As in our previous work, all model
results are synthetic T2LT and T2 temperatures that I calculated using a
static weighting function approach. I have not yet implemented Carl's
more sophisticated method of estimating synthetic MSU temperatures from
model data (which accounts for effects of topography and land/ocean
differences). However, for the current application, the simple static
weighting function approach is more than adequate, since we are focusing
on T2LT and T2 changes over tropical oceans only - so topographic and
land-ocean differences are unimportant. Note that I still need to
calculate synthetic MSU temperatures from about 18-20 20c3m realizations
which were not in the CMIP-3 database at the time we were working on the
CCSP report. For the full response to Douglass et al., we should use the
same 67 20c3m realizations that they employed.
For each of the 49 realizations that I processed, I first masked out all
tropical land areas, and then calculated the spatial averages of
monthly-mean, gridded T2LT and T2 data over tropical oceans (20N-20S).
All model and observational results are for the common 252-month period
from January 1979 to December 1999 - the longest period of overlap
between the RSS and UAH MSU data and the bulk of the 20c3m runs. The
simulated trends given by Douglass et al. are calculated over the same
1979 to 1999 period; however, they use a longer period (1979 to 2004)
for calculating observational trends - so there is an inconsistency
between their model and observational analysis periods, which they do
not explain. This difference in analysis periods is a little puzzling
given that we are dealing with relatively short observational record
lengths, resulting in some sensitivity to end-point effects.
I then calculated anomalies of the spatially-averaged T2LT and T2 data
(w.r.t. climatological monthly-means over 1979-1999), and fit
least-squares linear trends to model and observational time series. The
standard errors of the trends were adjusted for temporal autocorrelation
of the regression residuals, as described in Santer et al. (2000)
["Statistical significance of trends and trend differences in
layer-average atmospheric temperature time series"; JGR 105, 7337-7356.]
Consider first panel A of the attached plot. This shows the simulated
and observed T2LT trends over 1979 to 1999 (again, over 20N-20S, oceans
only) with their adjusted 1-sigma confidence intervals). For the UAH and
RSS data, it was possible to check against the adjusted confidence
intervals independently calculated by Dian during the course of work on
the CCSP report. Our adjusted confidence intervals are in good
agreement. The grey shaded envelope in panel A denotes the 1-sigma
standard error for the RSS T2LT trend.
There are 49 pairs of UAH-minus-model trend differences and 49 pairs of
RSS-minus-model trend differences. We can therefore test - for each
model and each 20c3m realization - whether there is a statistically
significant difference between the observed and simulated trends.
Let bx and by represent any single pair of modeled and observed trends,
with adjusted standard errors s{bx} and s{by}. As in our previous work
(and as in related work by John Lanzante), we define the normalized
trend difference d as:
d = (bx - by) / sqrt[ (s{bx})**2 + (s{by})**2 ]
Under the assumption that d is normally distributed, values of d > +1.96
or
Hi Steve
>
> I'd say it's the equivalent of rolling a 6-sided die a hundred times, and
> finding a mean value of ~3.5 and a standard deviation of ~1.7, and
> calculating the standard error of the mean to be ~0.17 (so far so
> good).  An then rolling the die one more time, getting a 2, and
> claiming that the die is no longer 6 sided because the new measurement
> is more than 2 standard errors from the mean.
>
> In my view, this problem trumps the other problems in the paper.
> I can't believe Douglas is a fellow of the American Physical Society.
>
> -Carl
>
>
> At 02:07 AM 12/6/2007, you wrote:
>> If I understand correctly, what Douglass et al. did makes the stronger
>> assumption that unforced variability is *insignificant*.  Their
>> statistical test is logically equivalent to falsifying a climate model
>> because it did not consistently predict a particular storm on a
>> particular day two years from now.
>
>
> Dr. Carl Mears
> Remote Sensing Systems
> 438 First Street, Suite 200, Santa Rosa, CA 95401
> mears@remss.com
> 707-545-2904 x21
> 707-545-2906 (fax))
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\douglass_reply1.pdf"


Dear Tom,
As promised, I've now repeated all of the significance testing involving
model-versus-observed trend differences, but this time using
spatially-averaged T2 and T2LT changes that are not "masked out" over
tropical land areas. As I mentioned this morning, the use of non-masked
data facilitates a direct comparison with Douglass et al.
The results for combined changes over tropical land and ocean are very
similar to those I sent out yesterday, which were for T2 and T2LT
changes over tropical oceans only:
COMBINED LAND/OCEAN RESULTS (WITH STANDARD ERRORS ADJUSTED FOR TEMPORAL
AUTOCORRELATION EFFECTS; SPATIAL AVERAGES OVER 20N-20S; ANALYSIS PERIOD
1979 TO 1999)
T2LT tests, RSS observational data: 0 out of 49 model-versus-observed
trend differences are significant at the 5% level.
T2LT tests, UAH observational data: 1 out of 49 model-versus-observed
trend differences are significant at the 5% level.
T2 tests, RSS observational data: 1 out of 49 model-versus-observed
trend differences are significant at the 5% level.
T2 tests, UAH observational data: 1 out of 49 model-versus-observed
trend differences are significant at the 5% level.
So our conclusion - that model tropical T2 and T2LT trends are, in
virtually all realizations and models, not significantly different from
either RSS or UAH trends - is not sensitive to whether we do the
significance testing with "ocean only" or combined "land+ocean"
temperature changes.
With best regards, and happy holidays to all!
Ben
Thomas.R.Karl wrote:
> Ben,
>
> This is very informative.  One question I raise is whether the results
> would have been at all different if you had not masked the land.  I
> doubt it, but it would be nice to know.
>
> Tom
>
> Ben Santer said the following on 12/13/2007 9:58 PM:
>> Dear folks,
>>
>> I've been doing some calculations to address one of the statistical
>> issues raised by the Douglass et al. paper in the International
>> Journal of Climatology. Here are some of my results.
>>
>> Recall that Douglass et al. calculated synthetic T2LT and T2
>> temperatures from the CMIP-3 archive of 20th century simulations
>> ("20c3m" runs). They used a total of 67 20c3m realizations, performed
>> with 22 different models. In calculating the statistical uncertainty
>> of the model trends, they introduced sigma{SE}, an "estimate of the
>> uncertainty of the mean of the predictions of the trends". They defined
>> sigma{SE} as follows:
>>
>> sigma{SE} = sigma / sqrt(N - 1), where
>>
>> "N = 22 is the number of independent models".
>>
>> As we've discussed in our previous correspondence, this definition has
>> serious problems (see comments from Carl and Steve below), and allows
>> Douglass et al. to reach the erroneous conclusion that modeled T2LT
>> and T2 trends are significantly different from the observed T2LT and
>> T2 trends in both the RSS and UAH datasets. This comparison of
>> simulated and observed T2LT and T2 trends is given in Table III of
>> Douglass et al.
>> [As an amusing aside, I note that the RSS datasets are referred to as
>> "RSS" in this table, while UAH results are designated as "MSU". I
>> guess there's only one true "MSU" dataset...]
>>
>> I decided to take a quick look at the issue of the statistical
>> significance of differences between simulated and observed
>> tropospheric temperature trends. My first cut at this "quick look"
>> involves only UAH and RSS observational data - I have not yet done any
>> tests with radiosonde datas, UMD T2 data, or satellite results from
>> Zou et al.
>>
>> I operated on the same 49 realizations of the 20c3m experiment that we
>> used in Chapter 5 of CCSP 1.1. As in our previous work, all model
>> results are synthetic T2LT and T2 temperatures that I calculated using
>> a static weighting function approach. I have not yet implemented
>> Carl's more sophisticated method of estimating synthetic MSU
>> temperatures from model data (which accounts for effects of topography
>> and land/ocean differences). However, for the current application, the
>> simple static weighting function approach is more than adequate, since
>> we are focusing on T2LT and T2 changes over tropical oceans only - so
>> topographic and land-ocean differences are unimportant. Note that I
>> still need to calculate synthetic MSU temperatures from about 18-20
>> 20c3m realizations which were not in the CMIP-3 database at the time
>> we were working on the CCSP report. For the full response to Douglass
>> et al., we should use the same 67 20c3m realizations that they employed.
>>
>> For each of the 49 realizations that I processed, I first masked out
>> all tropical land areas, and then calculated the spatial averages of
>> monthly-mean, gridded T2LT and T2 data over tropical oceans (20N-20S).
>> All model and observational results are for the common 252-month
>> period from January 1979 to December 1999 - the longest period of
>> overlap between the RSS and UAH MSU data and the bulk of the 20c3m
>> runs. The simulated trends given by Douglass et al. are calculated
>> over the same 1979 to 1999 period; however, they use a longer period
>> (1979 to 2004) for calculating observational trends - so there is an
>> inconsistency between their model and observational analysis periods,
>> which they do not explain. This difference in analysis periods is a
>> little puzzling given that we are dealing with relatively short
>> observational record lengths, resulting in some sensitivity to
>> end-point effects.
>>
>> I then calculated anomalies of the spatially-averaged T2LT and T2 data
>> (w.r.t. climatological monthly-means over 1979-1999), and fit
>> least-squares linear trends to model and observational time series.
>> The standard errors of the trends were adjusted for temporal
>> autocorrelation of the regression residuals, as described in Santer et
>> al. (2000) ["Statistical significance of trends and trend differences
>> in layer-average atmospheric temperature time series"; JGR 105,
>> 7337-7356.]
>>
>> Consider first panel A of the attached plot. This shows the simulated
>> and observed T2LT trends over 1979 to 1999 (again, over 20N-20S,
>> oceans only) with their adjusted 1-sigma confidence intervals). For
>> the UAH and RSS data, it was possible to check against the adjusted
>> confidence intervals independently calculated by Dian during the
>> course of work on the CCSP report. Our adjusted confidence intervals
>> are in good agreement. The grey shaded envelope in panel A denotes the
>> 1-sigma standard error for the RSS T2LT trend.
>>
>> There are 49 pairs of UAH-minus-model trend differences and 49 pairs
>> of RSS-minus-model trend differences. We can therefore test - for each
>> model and each 20c3m realization - whether there is a statistically
>> significant difference between the observed and simulated trends.
>>
>> Let bx and by represent any single pair of modeled and observed
>> trends, with adjusted standard errors s{bx} and s{by}. As in our
>> previous work (and as in related work by John Lanzante), we define the
>> normalized trend difference d as:
>>
>> d = (bx - by) / sqrt[ (s{bx})**2 + (s{by})**2 ]
>>
>> Under the assumption that d is normally distributed, values of d >
>> +1.96 or
> are significant at the 5% level. We are performing a two-tailed test
>> here, since we have no information a priori about the "direction" of
>> the model trend (i.e., whether we expect the simulated trend to be
>> significantly larger or smaller than observed).
>>
>> Panel c shows values of the normalized trend difference for T2LT trends.
>> the grey shaded area spans the range +1.96 to -1.96, and identifies
>> the region where we fail to reject the null hypothesis (H0) of no
>> significant difference between observed and simulated trends.
>>
>> Consider the solid symbols first, which give results for tests
>> involving RSS data. We would reject H0 in only one out of 49 cases
>> (for the CCCma-CGCM3.1(T47) model). The open symbols indicate results
>> for tests involving UAH data. Somewhat surprisingly, we get the same
>> qualitative outcome that we obtained for tests involving RSS data:
>> only one of the UAH-model trend pairs yields a difference that is
>> statistically significant at the 5% level.
>>
>> Panels b and d provide results for T2 trends. Results are very similar
>> to those achieved with T2LT trends. Irrespective of whether RSS or UAH
>> T2 data are used, significant trend differences occur in only one of
>> 49 cases.
>>
>> Bottom line: Douglass et al. claim that "In all cases UAH and RSS
>> satellite trends are inconsistent with model trends." (page 6, lines
>> 61-62). This claim is categorically wrong. In fact, based on our
>> results, one could justifiably claim that THERE IS ONLY ONE CASE in
>> which model T2LT and T2 trends are inconsistent with UAH and RSS
>> results! These guys screwed up big time.
>>
>> SENSITIVITY TESTS
>>
>> QUESTION 1: Some of the model-data trend comparisons made by Douglass
>> et al. used temperatures averaged over 30N-30S rather than 20N-20S.
>> What happens if we repeat our simple trend significance analysis using
>> T2LT and T2 data averaged over ocean areas between 30N-30S?
>>
>> ANSWER 1: Very little. The results described above for oceans areas
>> between 20N-20S are virtually unchanged.
>>
>> QUESTION 2: Even though it's clearly inappropriate to estimate the
>> standard errors of the linear trends WITHOUT accounting for temporal
>> autocorrelation effects (the 252 time sample are clearly not
>> independent; effective sample sizes typically range from 6 to 56),
>> someone is bound to ask what the outcome is when one repeats the
>> paired trend tests with non-adjusted standard errors. So here are the
>> results:
>>
>> T2LT tests, RSS observational data: 19 out of 49 trend differences are
>> significant at the 5% level.
>> T2LT tests, UAH observational data: 34 out of 49 trend differences are
>> significant at the 5% level.
>>
>> T2 tests, RSS observational data: 16 out of 49 trend differences are
>> significant at the 5% level.
>> T2 tests, UAH observational data: 35 out of 49 trend differences are
>> significant at the 5% level.
>>
>> So even under the naive (and incorrect) assumption that each model and
>> observational time series contains 252 independent time samples, we
>> STILL find no support for Douglass et al.'s assertion that: "In all
>> cases UAH and RSS satellite trends are inconsistent with model trends."
>> Q.E.D.
>>
>> If Leo is agreeable, I'm hopeful that we'll be able to perform a
>> similar trend comparison using synthetic MSU T2LT and T2 temperatures
>> calculated from the RAOBCORE radiosonde data - all versions, not just
>> v1.2!
>>
>> As you can see from the email list, I've expanded our "focus group" a
>> little bit, since a number of you have written to me about this issue.
>>
>> I am leaving for Miami on Monday, Dec. 17th. My Mom is having cataract
>> surgery, and I'd like to be around to provide her with moral and
>> practical support. I'm not exactly sure when I'll be returning to
>> PCMDI - although I hope I won't be gone longer than a week. As soon as
>> I get back, I'll try to make some more progress with this stuff. Any
>> suggestions or comments on what I've done so far would be greatly
>> appreciated. And for the time being, I think we should not alert
>> Douglass et al. to our results.
>>
>> With best regards, and happy holidays! May all your "Singers" be carol
>> singers, and not of the S. Fred variety...
>>
>> Ben
>>
>> (P.S.: I noticed one unfortunate typo in Table II of Douglass et al.
>> The MIROC3.2 (medres) model is referred to as "MIROC3.2_Merdes"....)
>>
>> carl mears wrote:
>>> Hi Steve
>>>
>>> I'd say it's the equivalent of rolling a 6-sided die a hundred times,
>>> and
>>> finding a mean value of ~3.5 and a standard deviation of ~1.7, and
>>> calculating the standard error of the mean to be ~0.17 (so far so
>>> good).  An then rolling the die one more time, getting a 2, and
>>> claiming that the die is no longer 6 sided because the new measurement
>>> is more than 2 standard errors from the mean.
>>>
>>> In my view, this problem trumps the other problems in the paper.
>>> I can't believe Douglas is a fellow of the American Physical Society.
>>>
>>> -Carl
>>>
>>>
>>> At 02:07 AM 12/6/2007, you wrote:
>>>> If I understand correctly, what Douglass et al. did makes the
>>>> stronger assumption that unforced variability is *insignificant*.
>>>> Their statistical test is logically equivalent to falsifying a
>>>> climate model because it did not consistently predict a particular
>>>> storm on a particular day two years from now.
>>>
>>>
>>> Dr. Carl Mears
>>> Remote Sensing Systems
>>> 438 First Street, Suite 200, Santa Rosa, CA 95401
>>> mears@remss.com
>>> 707-545-2904 x21
>>> 707-545-2906 (fax))
>>
>>
>
> --
>
> *Dr. Thomas R. Karl, L.H.D.*
>
> */Director/*//
>
> NOAA’s National Climatic Data Center
>
> Veach-Baley Federal Building
>
> 151 Patton Avenue
>
> Asheville, NC 28801-5001
>
> Tel:  (828) 271-4476
>
> Fax:  (828) 271-4246
>
> Thomas.R.Karl@noaa.gov
>
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------


Thanks Ben,
You have the makings of a nice article.
I note that we would expect to 10 cases that are significantly different by chance (based
on the 196 tests at the .05 sig level).  You found 3.  With appropriately corrected Leopold
I suspect you will find there is indeed stat sig. similar trends incl. amplification.
Setting up the statistical testing should be interesting with this many combinations.
Regards, Tom
Ben Santer said the following on 12/14/2007 5:31 PM:
Dear Tom,
As promised, I've now repeated all of the significance testing involving
model-versus-observed trend differences, but this time using spatially-averaged T2 and
T2LT changes that are not "masked out" over tropical land areas. As I mentioned this
morning, the use of non-masked data facilitates a direct comparison with Douglass et al.
The results for combined changes over tropical land and ocean are very similar to those
I sent out yesterday, which were for T2 and T2LT changes over tropical oceans only:
COMBINED LAND/OCEAN RESULTS (WITH STANDARD ERRORS ADJUSTED FOR TEMPORAL AUTOCORRELATION
EFFECTS; SPATIAL AVERAGES OVER 20N-20S; ANALYSIS PERIOD 1979 TO 1999)
T2LT tests, RSS observational data: 0 out of 49 model-versus-observed trend differences
are significant at the 5% level.
T2LT tests, UAH observational data: 1 out of 49 model-versus-observed trend differences
are significant at the 5% level.
T2 tests, RSS observational data: 1 out of 49 model-versus-observed trend differences
are significant at the 5% level.
T2 tests, UAH observational data: 1 out of 49 model-versus-observed trend differences
are significant at the 5% level.
So our conclusion - that model tropical T2 and T2LT trends are, in virtually all
realizations and models, not significantly different from either RSS or UAH trends - is
not sensitive to whether we do the significance testing with "ocean only" or combined
"land+ocean" temperature changes.
With best regards, and happy holidays to all!
Ben
Thomas.R.Karl wrote:
Ben,
This is very informative.  One question I raise is whether the results would have been
at all different if you had not masked the land.  I doubt it, but it would be nice to
know.
Tom
Ben Santer said the following on 12/13/2007 9:58 PM:
Dear folks,
I've been doing some calculations to address one of the statistical issues raised by the
Douglass et al. paper in the International Journal of Climatology. Here are some of my
results.
Recall that Douglass et al. calculated synthetic T2LT and T2 temperatures from the
CMIP-3 archive of 20th century simulations ("20c3m" runs). They used a total of 67 20c3m
realizations, performed with 22 different models. In calculating the statistical
uncertainty of the model trends, they introduced sigma{SE}, an "estimate of the
uncertainty of the mean of the predictions of the trends". They defined
sigma{SE} as follows:
sigma{SE} = sigma / sqrt(N - 1), where
"N = 22 is the number of independent models".
As we've discussed in our previous correspondence, this definition has serious problems
(see comments from Carl and Steve below), and allows Douglass et al. to reach the
erroneous conclusion that modeled T2LT and T2 trends are significantly different from
the observed T2LT and T2 trends in both the RSS and UAH datasets. This comparison of
simulated and observed T2LT and T2 trends is given in Table III of Douglass et al.
[As an amusing aside, I note that the RSS datasets are referred to as "RSS" in this
table, while UAH results are designated as "MSU". I guess there's only one true "MSU"
dataset...]
I decided to take a quick look at the issue of the statistical significance of
differences between simulated and observed tropospheric temperature trends. My first cut
at this "quick look" involves only UAH and RSS observational data - I have not yet done
any tests with radiosonde datas, UMD T2 data, or satellite results from Zou et al.
I operated on the same 49 realizations of the 20c3m experiment that we used in Chapter 5
of CCSP 1.1. As in our previous work, all model results are synthetic T2LT and T2
temperatures that I calculated using a static weighting function approach. I have not
yet implemented Carl's more sophisticated method of estimating synthetic MSU
temperatures from model data (which accounts for effects of topography and land/ocean
differences). However, for the current application, the simple static weighting function
approach is more than adequate, since we are focusing on T2LT and T2 changes over
tropical oceans only - so topographic and land-ocean differences are unimportant. Note
that I still need to calculate synthetic MSU temperatures from about 18-20 20c3m
realizations which were not in the CMIP-3 database at the time we were working on the
CCSP report. For the full response to Douglass et al., we should use the same 67 20c3m
realizations that they employed.
For each of the 49 realizations that I processed, I first masked out all tropical land
areas, and then calculated the spatial averages of monthly-mean, gridded T2LT and T2
data over tropical oceans (20N-20S). All model and observational results are for the
common 252-month period from January 1979 to December 1999 - the longest period of
overlap between the RSS and UAH MSU data and the bulk of the 20c3m runs. The simulated
trends given by Douglass et al. are calculated over the same 1979 to 1999 period;
however, they use a longer period (1979 to 2004) for calculating observational trends -
so there is an inconsistency between their model and observational analysis periods,
which they do not explain. This difference in analysis periods is a little puzzling
given that we are dealing with relatively short observational record lengths, resulting
in some sensitivity to end-point effects.
I then calculated anomalies of the spatially-averaged T2LT and T2 data (w.r.t.
climatological monthly-means over 1979-1999), and fit least-squares linear trends to
model and observational time series. The standard errors of the trends were adjusted for
temporal autocorrelation of the regression residuals, as described in Santer et al.
(2000) ["Statistical significance of trends and trend differences in layer-average
atmospheric temperature time series"; JGR 105, 7337-7356.]
Consider first panel A of the attached plot. This shows the simulated and observed T2LT
trends over 1979 to 1999 (again, over 20N-20S, oceans only) with their adjusted 1-sigma
confidence intervals). For the UAH and RSS data, it was possible to check against the
adjusted confidence intervals independently calculated by Dian during the course of work
on the CCSP report. Our adjusted confidence intervals are in good agreement. The grey
shaded envelope in panel A denotes the 1-sigma standard error for the RSS T2LT trend.
There are 49 pairs of UAH-minus-model trend differences and 49 pairs of RSS-minus-model
trend differences. We can therefore test - for each model and each 20c3m realization -
whether there is a statistically significant difference between the observed and
simulated trends.
Let bx and by represent any single pair of modeled and observed trends, with adjusted
standard errors s{bx} and s{by}. As in our previous work (and as in related work by John
Lanzante), we define the normalized trend difference d as:
d = (bx - by) / sqrt[ (s{bx})**2 + (s{by})**2 ]
Under the assumption that d is normally distributed, values of d > +1.96 or
--
Dr. Thomas R. Karl, L.H.D.
Director
NOAA's National Climatic Data Center
Veach-Baley Federal Building
151 Patton Avenue
Asheville, NC 28801-5001
Tel:  (828) 271-4476
Fax:  (828) 271-4246
[4]Thomas.R.Karl@noaa.gov
References
1. mailto:mears@remss.com
2. mailto:Thomas.R.Karl@noaa.gov
3. mailto:Thomas.R.Karl@noaa.gov
4. mailto:Thomas.R.Karl@noaa.gov


Dear all,
I have attached a plot which summarizes the recent developments
concerning tropical  radiosonde temperature datasets and which could be
a candidate to be included in a reply to Douglass et al.
It contains trend profiles from unadjusted radiosondes, HadAT2-adjusted
radiosondes, RAOBCORE (versions 1.2-1.4) adjusted radiosondes
and from radiosondes adjusted with a neighbor composite method (RICH)
that uses the break dates detected with RAOBCORE (v1.4) as metadata.
RAOBCORE v1.2,v1.3 are documented in Haimberger (2007), RAOBCORE v1.4
and RICH are discussed in the manuscript I mentioned in my previous email.
Latitude range is 20S-20N, only time series with less than 24 months of
missing data are included. Spatial sampling of all curves is the same
except HadAT which contains less stations that meet the 24month
criterion. Sampling uncertainty of the trend curves is ca.
+/-0.1K/decade (95% percentiles estimated with bootstrap method).
RAOBCORE v1.3,1.4 and RICH are results from ongoing research and warming
trends from radiosondes may still be underestimated.
The upper tropospheric warming maxima from RICH are even larger (up to
0.35K/decade, not shown), if only radiosondes within the tropics
(20N-20S) are allowed as reference for adjustment of tropical radiosonde
temperatures. The pink/blue curves in the attached plot should therefore
not be regarded as upper bound of what may be achieved with  plausible
choices of reference series for homogenization.
Please let me know your comments.
I wish you a merry Christmas.
With best regards
Leo
John Lanzante wrote:
> Ben,
>
> Perhaps a resampling test would be appropriate. The tests you have performed
> consist of pairing an observed time series (UAH or RSS MSU) with each one
> of 49 GCM times series from your "ensemble of opportunity". Significance
> of the difference between each pair of obs/GCM trends yields a certain
> number of "hits".
>
> To determine a baseline for judging how likely it would be to obtain the
> given number of hits one could perform a set of resampling trials by
> treating one of the ensemble members as a surrogate observation. For each
> trial, select at random one of the 49 GCM members to be the "observation".
> From the remaining 48 members draw a bootstrap sample of 49, and perform
> 49 tests, yielding a certain number of "hits". Repeat this many times to
> generate a distribution of "hits".
>
> The actual number of hits, based on the real observations could then be
> referenced to the Monte Carlo distribution to yield a probability that this
> could have occurred by chance. The basic idea is to see if the observed
> trend is inconsistent with the GCM ensemble of trends.
>
> There are a couple of additional tweaks that could be applied to your method.
> You are currently computing trends for each of the two time series in the
> pair and assessing the significance of their differences. Why not first
> create a difference time series and assess the significance of it's trend?
> The advantage of this is that you would reduce somewhat the autocorrelation
> in the time series and hence the effect of the "degrees of freedom"
> adjustment. Since the GCM runs are based on coupled model runs this
> differencing would help remove the common externally forced variability,
> but not internally forced variability, so the adjustment would still be
> needed.
>
> Another tweak would be to alter the significance level used to assess
> differences in trends. Currently you are using the 5% level, which yields
> only a small number of hits. If you made this less stringent you would get
> potentially more weaker hits. But it would all come out in the wash so to
> speak since the number of hits in the Monte Carlo simulations would increase
> as well. I suspect that increasing the number of expected hits would make the
> whole procedure more powerful/efficient in a statistical sense since you
> would no longer be dealing with a "rare event". In the current scheme, using
> a 5% level with 49 pairings you have an expected hit rate of 0.05 X 49 = 2.45.
> For example, if instead you used a 20% significance level you would have an
> expected hit rate of 0.20 X 49 = 9.8.
>
> I hope this helps.
>
> On an unrelated matter, I'm wondering a bit about the different versions of
> Leo's new radiosonde dataset (RAOBCORE). I was surprised to see that the
> latest version has considerably more tropospheric warming than I recalled
> from an earlier version that was written up in JCLI in 2007. I have a
> couple of questions that I'd like to ask Leo. One concern is that if we use
> the latest version of RAOBCORE is there a paper that we can reference --
> if this is not in a peer-reviewed journal is there a paper in submission?
> The other question is: could you briefly comment on the differences in
> methodology used to generate the latest version of RAOBCORE as compared to
> the version used in JCLI 2007, and what/when/where did changes occur to
> yield a stronger warming trend?
>
> Best regards,
>
> ______John
>
>
>
> On Saturday 15 December 2007 12:21 pm, Thomas.R.Karl wrote:
>
>> Thanks Ben,
>>
>> You have the makings of a nice article.
>>
>> I note that we would expect to 10 cases that are significantly different
>> by chance (based on the 196 tests at the .05 sig level).  You found 3.
>> With appropriately corrected Leopold I suspect you will find there is
>> indeed stat sig. similar trends incl. amplification.  Setting up the
>> statistical testing should be interesting with this many combinations.
>>
>> Regards, Tom
>>
>
>
--
Ao. Univ. Prof. Dr. Leopold Haimberger
Institut für Meteorologie und Geophysik, Universität Wien
Althanstraße 14, A - 1090 Wien
Tel.: +43 1 4277 53712
Fax.: +43 1 4277 9537
~haimbel7/
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\t00_trendbeltbg_Tropics_1979-2004_1.4.eps"


Dear folks,
This email briefly summarizes the trend significance test results. As I
mentioned in yesterday's email, I've added a new case (referred to as
"TYPE3" below). I've also added results for tests with a stipulated 10%
significance level. Here is the explanation of the four different types
of trend test:
1. "OBS-vs-MODEL": Observed MSU trends in RSS and UAH are tested against
trends in synthetic MSU data in 49 realizations of the 20c3m experiment.
Results from RSS and UAH are pooled, yielding a total of 98 tests for T2
trends and 98 tests for T2LT trends.
2. "MODEL-vs-MODEL (TYPE1)": Involves model data only. Trend in
synthetic MSU data in each of 49 20c3m realizations is tested against
each trend in the remaining 48 realizations (i.e., no trend tests
involving identical data). Yields a total of 49 x 48 = 2352 tests. The
significance of trend differences is a function of BOTH inter-model
differences (in climate sensitivity, applied 20c3m forcings, and the
amplitude of variability) AND "within-model" effects (i.e., is related
to the different manifestations of natural internal variability
superimposed on the underlying forced response).
3. "MODEL-vs-MODEL (TYPE2)": Involves model data only. Limited to the M
models with multiple realizations of the 20c3m experiment. For each of
these M models, the number of unique combinations C of N 20c3m
realizations into R trend pairs is determined. For example, in the case
of N = 5, C = N! / [ R!(N-R)! ] = 10. The significance of trend
differences is solely a function of "within-model" effects (i.e., is
related to the different manifestations of natural internal variability
superimposed on the underlying forced response). There are a total of 62
tests (not 124, as I erroneously reported yesterday!)
4. "MODEL-vs-MODEL (TYPE3)": Involves model data only. For each of the
19 models, only the first 20c3m realization is used. The trend in each
model's first 20c3m realization is tested against each trend in the
first 20c3m realization of the remaining 18 models. Yields a total of 19
x 18 = 342 tests. The significance of trend differences is solely a
function of inter-model differences (in climate sensitivity, applied
20c3m forcings, and the amplitude of variability).
REJECTION RATES FOR STIPULATED  5% SIGNIFICANCE LEVEL
Test type                  No. of tests       T2 "Hits"     T2LT "Hits"
1. OBS-vs-MODEL            49 x 2    (98)     2  (2.04%)     1  (1.02%)
2. MODEL-vs-MODEL (TYPE1)  49 x 48 (2352)    58  (2.47%)    32  (1.36%)
3. MODEL-vs-MODEL (TYPE2)    ---     (62)     0  (0.00%)     0  (0.00%)
4. MODEL-vs-MODEL (TYPE3)  19 x 18  (342)    22  (6.43%)    14  (4.09%)
REJECTION RATES FOR STIPULATED 10% SIGNIFICANCE LEVEL
Test type                  No. of tests       T2 "Hits"     T2LT "Hits"
1. OBS-vs-MODEL            49 x 2    (98)     4  (4.08%)     2  (2.04%)
2. MODEL-vs-MODEL (TYPE1)  49 x 48 (2352)    80  (3.40%)    46  (1.96%)
3. MODEL-vs-MODEL (TYPE2)    ---     (62)     1  (1.61%)     0  (0.00%)
4. MODEL-vs-MODEL (TYPE3)  19 x 18  (342)    28  (8.19%)    20  (5.85%)
REJECTION RATES FOR STIPULATED 20% SIGNIFICANCE LEVEL
Test type                  No. of tests       T2 "Hits"     T2LT "Hits"
1. OBS-vs-MODEL            49 x 2    (98)     7  (7.14%)     5  (5.10%)
2. MODEL-vs-MODEL (TYPE1)  49 x 48 (2352)   176  (7.48%)   100  (4.25%)
3. MODEL-vs-MODEL (TYPE2)    ---     (62)     4  (6.45%)     3  (4.84%)
4. MODEL-vs-MODEL (TYPE3)  19 x 18  (342)    42 (12.28%)    28  (8.19%)
Features of interest:
A) As you might expect, for each of the three significance levels, TYPE3
tests yield the highest rejection rates of the null hypothesis of "No
significant difference in trend". TYPE2 tests yield the lowest rejection
rates. This is simply telling us that the inter-model differences in
trends tend to be larger than the "between-realization" differences in
trends in any individual model.
B) Rejection rates for the model-versus-observed trend tests are
consistently LOWER than for the model-versus-model (TYPE3) tests. On
average, therefore, the tropospheric trend differences between the
observational datasets used here (RSS and UAH) and the synthetic MSU
temperatures calculated from 19 CMIP-3 models are actually LESS
SIGNIFICANT than the inter-model trend differences arising from
differences in sensitivity, 20c3m forcings, and levels of variability.
I also thought that it would be fun to use the model data to explore the
implications of Douglass et al.'s flawed statistical procedure. Recall
that Douglass et al. compare (in their Table III) the observed T2 and
T2LT trends in RSS and UAH with the overall means of the multi-model
distributions of T2 and T2LT trends. Their standard error, sigma{SE}, is
meant to represent an "estimate of the uncertainty of the mean" (i.e.,
the mean trend). sigma{SE} is given as:
sigma{SE} = sigma / sqrt{N - 1}
where sigma is the standard deviation of the model trends, and N is "the
number of independent models" (22 in their case). Douglass et al.
apparently estimate sigma using ensemble-mean trends for each model (if
20c3m ensembles are available).
So what happens if we apply this procedure using model data only? This
is rather easy to do. As above (in the TYPE1, TYPE2, and TYPE3 tests), I
simply used the synthetic MSU trends from the 19 CMIP-3 models employed
in our CCSP Report and in Santer et al. 2005 (so N = 19). For each
model, I calculated the ensemble-mean 20c3m trend over 1979 to 1999
(where multiple 20c3m realizations were available). Let's call these
mean trends b{j}, where j (the index over models) = 1, 2, .. 19.
Further, let's regard b{1} as the surrogate observations, and then use
Douglass et al.'s approach to test whether b{1} is significantly
different from the overall mean of the remaining 18 members of b{j}.
Then repeat with b{2} as surrogate observations, etc. For each
layer-averaged temperature series, this yields 19 tests of the
significance of differences in mean trends.
To give you a feel for this stuff, I've reproduced below the results for
tests involving T2LT trends. The "OBS" column is the ensemble-mean T2LT
trend in the surrogate observations. "MODAVE" is the overall mean trend
in the 18 remaining members of the distribution, and "SIGMA" is the
1-sigma standard deviation of these trends. "SIGMA{SE}" is 1 x
SIGMA{SE} (note that Douglass et al. give 2 x SIGMA{SE} in their Table
III; multiplying our SIGMA{SE} results by two gives values similar to
theirs). "NORMD" is simply the normalized difference (OBS-MODAVE) /
SIGMA{SE}, and "P-VALUE" is the p-value for the normalized difference,
assuming that this difference is approximately normally distributed.
MODEL          "OBS"     MODAVE    SIGMA   SIGMA{SE}   NORMD     P-VALUE
CCSM3.0        0.1580    0.2179    0.0910    0.0215    2.7918    0.0052
GFDL2.0        0.2576    0.2124    0.0915    0.0216    2.0977    0.0359
GFDL2.1        0.3567    0.2069    0.0854    0.0201    7.4404    0.0000
GISS_EH        0.1477    0.2185    0.0906    0.0214    3.3153    0.0009
GISS_ER        0.1938    0.2159    0.0919    0.0217    1.0205    0.3075
MIROC3.2_T42   0.1285    0.2196    0.0897    0.0211    4.3094    0.0000
MIROC3.2_T106  0.2298    0.2139    0.0920    0.0217    0.7305    0.4651
MRI2.3.2a      0.2800    0.2111    0.0907    0.0214    3.2196    0.0013
PCM            0.1496    0.2184    0.0907    0.0214    3.2170    0.0013
HADCM3         0.1936    0.2159    0.0919    0.0217    1.0327    0.3018
HADGEM1        0.3099    0.2095    0.0891    0.0210    4.7784    0.0000
CCCMA3.1       0.4236    0.2032    0.0769    0.0181   12.1591    0.0000
CNRM3.0        0.2409    0.2133    0.0918    0.0216    1.2762    0.2019
CSIRO3.0       0.2780    0.2113    0.0908    0.0214    3.1195    0.0018
ECHAM5         0.1252    0.2197    0.0895    0.0211    4.4815    0.0000
IAP_FGOALS1.0  0.1834    0.2165    0.0917    0.0216    1.5314    0.1257
GISS_AOM       0.1788    0.2168    0.0916    0.0216    1.7579    0.0788
INMCM3.0       0.0197    0.2256    0.0790    0.0186   11.0541    0.0000
IPSL_CM4       0.2258    0.2142    0.0920    0.0217    0.5359    0.5920
T2LT: No. of p-values .le. 0.05: 12.  Rejection rate:  63.16%
T2LT: No. of p-values .le. 0.10: 13.  Rejection rate:  68.42%
T2LT: No. of p-values .le. 0.20: 14.  Rejection rate:  73.68%
The corresponding rejection rates for the tests involving T2 data are:
T2:   No. of p-values .le. 0.05: 12.  Rejection rate:  63.16%
T2:   No. of p-values .le. 0.10: 13.  Rejection rate:  68.42%
T2:   No. of p-values .le. 0.20: 15.  Rejection rate:  78.95%
Bottom line: If we applied Douglass et al.'s ridiculous test of
difference in mean trends to model data only - in fact, to virtually the
same model data they used in their paper - one would conclude that
nearly two-thirds of the individual models had trends that were
significantly different from the multi-model mean trend! To follow
Douglass et al.'s flawed logic, this would mean that two-thirds of the
models really aren't models after all...
Happy New Year to all of you!
With best regards,
Ben
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------


Ben,
I have attached the tropical mean trend profiles, now for the period
1979-1999.
RAOBCORE  versions show much more upper tropospheric heating for this
period, RICH shows slightly more heating.
Note also stronger cooling of unadjusted radiosondes in stratospheric
layers compared to 1999-2004.
Just for information I have included also zonal mean trend plots for the
unadjusted radiosondes (tm), RAOBCORE v1.4 (tmcorr) and RICH (rgmra)
I do not suggest that these plots should be included but some of you
maybe want to know about the spatial coherence
of the zonal mean trends. It is interesting to see the lower
tropospheric warming minimum in the tropics in all three plots,
which I cannot explain. I believe it is spurious but it is remarkably
robust against my adjustment efforts.
Meridional resolution is 10 degrees.
As you can imagine, the tropical upper tropospheric heating maximum at
5S and the cooling in the unadjusted radiosondes at 5N are
based on very few long records  in these belts. 2-3 in 5S, about 5 in 5N.
Best regards and I wish you all a happy new year.
Leo
Ben Santer wrote:
> Dear Leo,
>
> The Figure that you sent is extremely informative, and would be great
> to include in a response to Douglass et al. The Figure clearly
> illustrates that the "structural uncertainties" inherent in
> radiosonde-based estimates of tropospheric temperature change are much
> larger than Douglass et al. have claimed. This is an important point
> to make.
>
> Would it be possible to produce a version of this Figure showing
> results for the period 1979 to 1999 (the period that I've used for
> testing the significance of model-versus-observed trend differences)
> instead of 1979 to 2004?
>
> With best regards, and frohes Neues Jahr!
>
> Ben
> Leopold Haimberger wrote:
>> Dear all,
>>
>> I have attached a plot which summarizes the recent developments
>> concerning tropical  radiosonde temperature datasets and which could
>> be a candidate to be included in a reply to Douglass et al.
>> It contains trend profiles from unadjusted radiosondes,
>> HadAT2-adjusted radiosondes, RAOBCORE (versions 1.2-1.4) adjusted
>> radiosondes
>> and from radiosondes adjusted with a neighbor composite method (RICH)
>> that uses the break dates detected with RAOBCORE (v1.4) as metadata.
>> RAOBCORE v1.2,v1.3 are documented in Haimberger (2007), RAOBCORE v1.4
>> and RICH are discussed in the manuscript I mentioned in my previous
>> email.
>> Latitude range is 20S-20N, only time series with less than 24 months
>> of missing data are included. Spatial sampling of all curves is the
>> same except HadAT which contains less stations that meet the 24month
>> criterion. Sampling uncertainty of the trend curves is ca.
>> +/-0.1K/decade (95% percentiles estimated with bootstrap method).
>>
>> RAOBCORE v1.3,1.4 and RICH are results from ongoing research and
>> warming trends from radiosondes may still be underestimated.
>> The upper tropospheric warming maxima from RICH are even larger (up
>> to 0.35K/decade, not shown), if only radiosondes within the tropics
>> (20N-20S) are allowed as reference for adjustment of tropical
>> radiosonde temperatures. The pink/blue curves in the attached plot
>> should therefore not be regarded as upper bound of what may be
>> achieved with  plausible choices of reference series for homogenization.
>> Please let me know your comments.
>>
>> I wish you a merry Christmas.
>>
>> With best regards
>>
>> Leo
>>
>> John Lanzante wrote:
>>> Ben,
>>>
>>> Perhaps a resampling test would be appropriate. The tests you have
>>> performed
>>> consist of pairing an observed time series (UAH or RSS MSU) with
>>> each one
>>> of 49 GCM times series from your "ensemble of opportunity".
>>> Significance
>>> of the difference between each pair of obs/GCM trends yields a certain
>>> number of "hits".
>>>
>>> To determine a baseline for judging how likely it would be to obtain
>>> the
>>> given number of hits one could perform a set of resampling trials by
>>> treating one of the ensemble members as a surrogate observation. For
>>> each
>>> trial, select at random one of the 49 GCM members to be the
>>> "observation".
>>> From the remaining 48 members draw a bootstrap sample of 49, and
>>> perform
>>> 49 tests, yielding a certain number of "hits". Repeat this many
>>> times to
>>> generate a distribution of "hits".
>>>
>>> The actual number of hits, based on the real observations could then be
>>> referenced to the Monte Carlo distribution to yield a probability
>>> that this
>>> could have occurred by chance. The basic idea is to see if the observed
>>> trend is inconsistent with the GCM ensemble of trends.
>>>
>>> There are a couple of additional tweaks that could be applied to
>>> your method.
>>> You are currently computing trends for each of the two time series
>>> in the
>>> pair and assessing the significance of their differences. Why not first
>>> create a difference time series and assess the significance of it's
>>> trend?
>>> The advantage of this is that you would reduce somewhat the
>>> autocorrelation
>>> in the time series and hence the effect of the "degrees of freedom"
>>> adjustment. Since the GCM runs are based on coupled model runs this
>>> differencing would help remove the common externally forced
>>> variability,
>>> but not internally forced variability, so the adjustment would still be
>>> needed.
>>>
>>> Another tweak would be to alter the significance level used to assess
>>> differences in trends. Currently you are using the 5% level, which
>>> yields
>>> only a small number of hits. If you made this less stringent you
>>> would get
>>> potentially more weaker hits. But it would all come out in the wash
>>> so to
>>> speak since the number of hits in the Monte Carlo simulations would
>>> increase
>>> as well. I suspect that increasing the number of expected hits would
>>> make the
>>> whole procedure more powerful/efficient in a statistical sense since
>>> you
>>> would no longer be dealing with a "rare event". In the current
>>> scheme, using
>>> a 5% level with 49 pairings you have an expected hit rate of 0.05 X
>>> 49 = 2.45.
>>> For example, if instead you used a 20% significance level you would
>>> have an
>>> expected hit rate of 0.20 X 49 = 9.8.
>>>
>>> I hope this helps.
>>>
>>> On an unrelated matter, I'm wondering a bit about the different
>>> versions of
>>> Leo's new radiosonde dataset (RAOBCORE). I was surprised to see that
>>> the
>>> latest version has considerably more tropospheric warming than I
>>> recalled
>>> from an earlier version that was written up in JCLI in 2007. I have a
>>> couple of questions that I'd like to ask Leo. One concern is that if
>>> we use
>>> the latest version of RAOBCORE is there a paper that we can
>>> reference --
>>> if this is not in a peer-reviewed journal is there a paper in
>>> submission?
>>> The other question is: could you briefly comment on the differences
>>> in methodology used to generate the latest version of RAOBCORE as
>>> compared to the version used in JCLI 2007, and what/when/where did
>>> changes occur to
>>> yield a stronger warming trend?
>>>
>>> Best regards,
>>>
>>> ______John
>>>
>>>
>>>
>>> On Saturday 15 December 2007 12:21 pm, Thomas.R.Karl wrote:
>>>
>>>> Thanks Ben,
>>>>
>>>> You have the makings of a nice article.
>>>>
>>>> I note that we would expect to 10 cases that are significantly
>>>> different by chance (based on the 196 tests at the .05 sig level).
>>>> You found 3.  With appropriately corrected Leopold I suspect you
>>>> will find there is indeed stat sig. similar trends incl.
>>>> amplification.  Setting up the statistical testing should be
>>>> interesting with this many combinations.
>>>>
>>>> Regards, Tom
>>>>
>>>
>>>
>>
>
>
--
Ao. Univ. Prof. Dr. Leopold Haimberger
Institut für Meteorologie und Geophysik, Universität Wien
Althanstraße 14, A - 1090 Wien
Tel.: +43 1 4277 53712
Fax.: +43 1 4277 9537
~haimbel7/
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\t00_trendbeltbg_Tropics_1979-1999_v1_4.eps"
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\t00_trendzonalGlobe_tmcorr_1979-1999.ps"
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\t00_trendzonalGlobe_rgmra_1979-1999.ps"
Attachment Converted: "c:\documents and settings\tim osborn\my documents\eudora\attach\t00_trendzonalGlobe_tm_1979-1999.ps"


Dear All,
Thanks very much for the helpful discussion on these issues.
I write to make a point that may not be well
recognized regarding the character of the
temperature trends in the lowermost
stratosphere/upper troposphere.  I have already
discussed this with Ben but want to share with
others since I believe it is relevant to this
controversy at least at some altitudes.   The
question I want to raise is not related to the
very important dialogue on how to handle the
errors and the statistics, but rather how to
think about the models.
The attached paper by Forster et al. appeared
recently in GRL.   It taught me something I
didn't realize, namely that ozone losses and
accompanying temperature trends at higher
altitudes can strongly affect lower altitudes,
through the influence of downwelling longwave.
There is now much evidence that ozone has
decreased significantly in the tropics near 70
mbar.    What we show in the attached paper by
Forster et al is that ozone depletion near 70
mbar affects temperatures not only at that level,
but also down to lower altitudes.  I think this
is bound to be important to the tropical
temperature trends at least in the 100-50 mbar
height range, possibly lower down as well,
depending upon the degree to which there is a
'substratosphere' that is more radiatively
influenced than the rest of the troposphere.
Whether it can have an influence as low as 200
mbar - I don't know.    But note that having an
influence could mean reducing the warming there,
not necessarily flipping it over to a net
cooling.    This 'long-distance' physics, whereby
ozone depletion and associated cooling up high
can affect the thermal structure lower down, is
not a point I had understood despite many years
of studying the problem so I thought it
worthwhile to point it out to you here.  It has
often been said (I probably said it myself five
years ago) that ozone losses and associated
cooling can't happen or aren't important in this
region - but that is wrong.
Further, the fundamental point made in the paper
of Thompson and Solomon a few years back remains
worth noting, and is, I believe, now resolved in
the more recent Forster et al paper:   that the
broad structure of the temperature trends, with
quite large cooing in the lowermost stratosphere
in the tropics, comparable to that seen at higher
latitudes, is a feature NOT explained by e.g. CO2
cooling, but now can be explained by the observed
ozone losses.   Exactly how big the tropical
cooling is, and exactly how low down it goes,
remains open to quantitative question and
improvement of radiosonde datasets.  But I
believe the fundamental point we made in 2005
remains true:  the temperature trends in the
lower stratosphere in the tropics are, even with
corrections, quite comparable to that seen at
other latitudes.     We can now say it is surely
linked to the now-well-observed trends in ozone
there.     The new paper further shows that you
don't have to have ozone trends at 100 mbar to
have a cooling there, due to down-welling
longwave, possibly lower down still.      Whether
enhanced upwelling is a factor is a central
question.
No global general circulation model can possibly
be expected to simulate this correctly unless it
has interactive ozone, or prescribes an observed
tropical ozone trend.   The AR4 models did not
include this, and any 'discrepancies' are not
relevant at all to the issue of the fidelity of
those models for global warming.    So in closing
let me just say that just how low down this
effect goes needs more study, but that it does
happen and is relevant to the key problem of
tropical temperature trends is one that I hope
this email has clarified.
Happy new year,
Susan
At 6:13 PM -0700 12/29/07, Tom Wigley wrote:
>Tom,
>
>Yes -- I had this in an earlier version, but I did not want to
>overwhelm people with the myriad errors in the D et al. paper.
>
>I liked the attached item -- also in an earlier version.
>
>Tom.
>
>+++++++++++++
>
>Thomas.R.Karl wrote:
>
>>Tom,
>>
>>This is a very nice set of slides clearly
>>showing the problem with the Douglass et al
>>paper.  One other aspect of this issue that
>>John L has mentioned and we discussed when we
>>were doing SAP 1.1 relates to difference
>>series.  I am not sure whether Ben was
>>calculating the significance of the difference
>>series between sets of observations and model
>>simulations (annually).  This would help offset
>>the effects of El-Nino and Volcanoes on the
>>trends.
>>
>>Tom K.
>>
>>Tom Wigley said the following on 12/29/2007 1:05 PM:
>>
>>>Dear all,
>>>
>>>I was recently at a meeting in Rome where Fred Singer was a participant.
>>>He was not on the speaker list, but, in
>>>advance of the meeting, I had thought
>>>he might raise the issue of the Douglass et
>>>al. paper. I therefore prepared the
>>>attached power point -- modified slightly since returning from Rome. As it
>>>happened, Singer did not raise the Douglass et al. issue, so I did not use
>>>the ppt. Still, it may be useful for members
>>>of this group so I am sending it
>>>to you all.
>>>
>>>Please keep this in confidence. I do not want
>>>it to get back to Singer or any
>>>of the Douglass et al. co-authors -- at least
>>>not at this stage while Ben is still
>>>working on a paper to rebut the Douglass et al. claims.
>>>
>>>On slide 6 I have attributed the die tossing
>>>argument to Carl Mears -- but, in
>>>looking back at my emails I can't find the
>>>original. If I've got this attribution
>>>wrong, please let me know.
>>>
>>>Other comments are welcome. Mike MacCracken and Ben helped in putting
>>>this together -- thanks to both.
>>>
>>>Tom.
>>>
>>>++++++++++++++++++++++++++++++++++++++++
>>
>>
>>--
>>
>>*Dr. Thomas R. Karl, L.H.D.*
>>
>>*/Director/*//
>>
>>NOAA's National Climatic Data Center
>>
>>Veach-Baley Federal Building
>>
>>151 Patton Avenue
>>
>>Asheville, NC 28801-5001
>>
>>Tel:  (828) 271-4476
>>
>>Fax:  (828) 271-4246
>>
>>Thomas.R.Karl@noaa.gov
>>
>
>
>
>Attachment converted: Junior:Comment on Douglass.ppt (SLD3/«IC») (0022CEF5)
Attachment Converted: "c:\eudora\attach\ForsterOzoneCooling.pdf"
Attachment Converted: "c:\eudora\attach\ThompsonSolomon2005.pdf"


Susan et al.,
I had also seen the Forster et al paper and was glad to see he had
followed up on work and ideas we had discussed some years ago when he
was at Reading and from the Exeter workshop. At the time I had done some
simple research on whether the stratosphere could affect the tropical
troposphere - possibly through convection modification or radiative
cooling. I'd done a simple timeseries regression of T2LT=a*Tsurf+b*T4+c
and got some regression coefficients out that suggested an influence.
Now, this was with old and now discredited data and the Fu et al.
technique has since superseded it to some extent (or at least cast
considerable doubt upon its efficacy) ... it would certainly be hard to
prove in a regression what was cause and effect with such broad
weighting functions even using T2LT which still isn't *really*
independent from T4.
But one thing I did do to try to "prove" the regression result was real
is take the composite differences between QBO phases on 45 years of
detrended (can't remember exactly how but I think I took differences
from decadally filtered data) data from radiosondes (HadAT1 at the
time). This showed a really very interesting result and suggested that
this communication if it was real went quite far down in to the
troposphere and was statistically significant, particularly in those
seasons when the ITCZ and QBO were geographically coincident. I attach
the slide for interest. I think this is the only scientifically valid
part of the analysis that I would stand by today given the rather
massive developments since. I doubt that raobs inhomogeneities could
explain the plot result as they project much more onto the trend than
they would onto this type of analysis.
The cooling stratosphere may really have an influence even quite low
down if this QBO composite technique is a good analogue for a cooling
startosphere's impact, and timeseries regression analysis supports it in
some obs (it would be interesting to repeat such an analysis with the
newer obs but I don't have time). A counter, however, is that surely the
models do radiation so those with ozone loss should do a good job of
this effect. This could be checked in Ben's ensemble in a poor man's
sense at least because some have ozone depletion and some don't.
The only way this could be a real factor not picked by the models, I
concluded at the time, is if models are far too keen to trigger
convection and that any real-world increased radiative cooling
efficiency effect is masked in the models because they convect far too
often and regain CAPE closure as a condition.
On another matter, we seem to be concentrating entirely on layer-average
temperatures. This is fine, but we know from CCSP these show little in
the way of differences. The key, and much harder test is to capture the
differences in behaviour between layers / levels - the "amplification"
behaviour. This was the focus of Santer et al. and I still believe is
the key scientific question given that each model realisation is
inherently so different but that we believe the physics determining the
temperature profile to be the key test that has to be answered. Maybe we
need to step back and rephrase the question in terms of the physics
rather than aiming solely to rebutt Douglass et al? In this case the key
physical questions in my view would be:
1. Why is there such strong evidence from sondes for a minima at c. 500?
Is this because it is near the triple point of water in the tropics? Or
at the top of the shallow convection? Or simply an artefact? [I don't
have any good ideas how we would answer the first two of these
questions]
2. Is there really a stratospheric radiative influence? If so, how low
does it go? What is the cause? Are the numbers consistent with the
underlying governing physics or simply an artefact of residual obs
errors?
3. Can any models show trend behaviour that deviates from a SALR on
multi-decadal timescales? If so, what is it about the model that causes
this effect? Physics? Forcings? Phasing of natural variability? Is it
also true on shorter timescales in this model?
It seems to me that trying to do an analysis based upon such physical
understanding / questions will clarify things far better than simply
doing another set of statistical analysis. I'm still particularly
interested if #2 is really true in the raobs (its not possible to do
with satellites I suspect, but if it is true it means we need to
massively rethink Fu et al. type analysis at least in the tropics) and
would be interested in helping someone follow up on that ... I think in
the future the Forster et al paper may be seen as the more
scientifically significant result when Douglass et al is no longer cared
about ...
Happy new year to you all.
Peter
--
Peter Thorne   Climate Research Scientist
Met Office Hadley Centre, FitzRoy Road, Exeter, EX1 3PB
tel. +44 1392 886552 fax +44 1392 885681
www.metoffice.gov.uk/hadobs
Attachment Converted: "c:\eudora\attach\qbo_slide.ppt"


Phil
Thanks for the Benestad reference, which I hadn't seen and will read
with interest.
Please keep me in the loop on your reprints.
I'm aware of the work with Dave Thompson, which is very interesting.
Happy new year to you too.
We can all look back on 2007 as a year in which we, the scientists,
did a fantastic job.
best
Susan
At 8:59 PM +0000 1/2/08, P.Jones@uea.ac.uk wrote:
>  Kevin, Susan,
>     Working on several things at the moment, so won't
>  have much time for a few weeks. Rasmus Benestad of
>  the Norwegian Met Service wrote a paper on a very similar
>  earlier verion of this McKittrick/Michaels paper (both
>  were in Climate Research). There is nothing new in this
>  paper in JGR.
>     The only thing new in both this JGR paper and the
>  Douglass et al one in IJC is the awful reviewing!!!!
>  Rebuttals help, but often the damage is done once the
>  paper comes out. The MM paper is bad, but the reviewing
>  is even worse. Why did MM refer to an erratum on their
>  paper which is essentially the same? Any reviewer worth
>  any salt should have spotted that and then they would have
>  seen the Benestad comment, which MM surprisingly don't refer to.
>
>      I'm hoping to submit a paper on urbanization soon -
>  based on work with Chinese series - this relates to the
>  fraud allegation against Wei-Chyung Wang that Kevin knows
>  about.
>
>      Also should be a press release tomorrow or Friday about
>  the forecast for 2008 temperatures. La Nina looks like making
>  it coolish - cooler just than all years since 2001 (including
>  2001) and 1998.  Pointing out that 2001-2007 is 0.21 warmer
>  than 1991-2000 which is exactly as it should be with ghg-related
>  warming of 0.2 per decade.
>
>     [Also working on something with Dave Thompson (Dave's laeding)
>  that will have an ENSO-factored out (and COWL) global T series.]
>
>
>      We're (with the Met Office) extending the press release
>  due to the silly coverage in mid-December about global warming
>  ending, as all years since 1998 are cooler than it. Mostly this
>  was by people just parrotting the same message from the same
>  people. It is a case of people who should know better (and check
>  their sources) just copying from people who don't know any
>  better.
>
>    Oh  - forgot - Happy New Year!
>
>  Any pictures on the IPCC web site of Oslo on Dec 10 !
>
>  Patchy is on the front cover of the last issue of the 2007 in Nature.
>
>  Cheers
>  Phil
>
>
>  Susan
>>  Not me.  Phil has been involved in various stuff related to this but I
>>  am not up to speed.  I'll cc him.
>>  I recall some exchanges a while ago now.
>>  Kevin
>>
>>  Susan Solomon wrote:
>>>  Kevin
>>>  Happy new year to you.   All's well here.   Have you or other
>>>  colleagues organized a rebuttal to the McKitrick and Michaels JGR 2007
>>>  material on urbanization?   It's getting exposure, along with the
>>>  Douglass et al. paper.  On the latter, you probably know Ben Santer is
>>>  preparing one.
>>>  best
>>>  Susan
>>
>>  --
>>  ****************
>>  Kevin E. Trenberth                  e-mail: trenbert@ucar.edu
>>  Climate Analysis Section,           www.cgd.ucar.edu/cas/trenbert.html
>>  NCAR
>>  P. O. Box 3000,                     (303) 497 1318
>>  Boulder, CO 80307                   (303) 497 1333 (fax)
>>
>>  Street address: 1850 Table Mesa Drive, Boulder, CO  80305
>>
>>
>>


Dear Tom,
In the end, I decided to test the significance of trends in the O(t)
minus M(t) difference time series, as you and John Lanzante have
suggested. I still think that this "difference series test" is more
appropriate when one is operating on a pair of time series with
correlated variability (for example, if you wished to test whether an
observed tropical T2LT trend was significantly different from the T2LT
trend simulated in an AMIP experiment). But you and John convinced me
that our response to Douglass et al. would be strengthened by using
several different approaches to address the statistical significance of
differences between modeled and observed temperature trends.
The Tables given below show the results from two different types of
test. You've already seen the "TYPE1" or "PAIRED TREND" results. These
involve b{O} and b{M}, which represent any single pair of Observed and
Modeled trends, with standard errors s{bO} and s{bM} (which are adjusted
for temporal autocorrelation effects). As in our previous work (and as
in related work by John Lanzante), we define the normalized trend
difference d as:
d1 = (b{O} - b{M}) / sqrt[ (s{bO})**2 + (s{bM})**2 ]
Under the assumption that d1 is normally distributed, values of d1 >
+1.96 or
Thanks Ben,
>
> You have been busy! I sent Tom an email before reading the last
> paragraph of this note.  Recognizing the "random" placement of ENSO in
> the models and volcanic effects (in a few) and the known impact of the
> occurrence of these events on the trends, I think it is appropriate that
> the noise and related uncertainty about the trend differences be
> increased.  Amplifying the noise could be argued as an appropriate
> conservative approach, since we know that these events are confounding
> our efforts to see differences between models and obs w/r to greenhouse
> forcing.
>
> I know it is more work, but  I think it does make sense to calculate
> O(1)-M(1), O(2)-M(2) .... O(n)-M(n)  for all combinations of observed
> data sets and model simulations.  You could test for significance by
> using a Monte Carlo bootstrap approach by randomizing the years for both
> models and data.
>
> Regards, Tom
>
>
> Ben Santer said the following on 12/26/2007 9:50 PM:
>> Dear John,
>>
>> Thanks for your email. As usual, your comments were constructive and
>> thought-provoking. I've tried to do some of the additional tests that
>> you suggested, and will report on the results below.
>>
>> But first, let's have a brief recap. As discussed in my previous
>> emails, I've tested the significance of differences between trends in
>> observed MSU time series and the trends in synthetic MSU temperatures
>> in a multi-model "ensemble of opportunity". The "ensemble of
>> opportunity" comprises results from 49 realizations of the CMIP-3
>> "20c3m" experiment, performed with 19 different A/OGCMs. This is the
>> same ensemble that was analyzed in Chapter 5 of the CCSP Synthesis and
>> Assessment Product 1.1.
>> I've used observational results from two different groups (RSS and
>> UAH). From each group, we have results for both T2 and T2LT. This
>> yields a total of 196 different tests of the significance of
>> observed-versus-model trend differences (2 observational datasets x 2
>> layer-averaged temperatures x 49 realizations of the 20c3m
>> experiment). Thus far, I've tested the significance of trend
>> differences using T2 and T2LT data spatially averaged over oceans only
>> (both 20N-20S and 30N-30S), as well as over land and ocean (20N-20S).
>> All results described below focus on the land and ocean results, which
>> facilitates a direct comparison with Douglass et al.
>>
>> Here was the information that I sent you on Dec. 14th:
>>
>> COMBINED LAND/OCEAN RESULTS (WITH STANDARD ERRORS ADJUSTED FOR
>> TEMPORAL AUTOCORRELATION EFFECTS; SPATIAL AVERAGES OVER 20N-20S;
>> ANALYSIS PERIOD 1979 TO 1999)
>>
>> T2LT tests, RSS observational data: 0 out of 49 model-versus-observed
>> trend differences are significant at the 5% level.
>> T2LT tests, UAH observational data: 1 out of 49 model-versus-observed
>> trend differences are significant at the 5% level.
>>
>> T2 tests, RSS observational data: 1 out of 49 model-versus-observed
>> trend differences are significant at the 5% level.
>> T2 tests, UAH observational data: 1 out of 49 model-versus-observed
>> trend differences are significant at the 5% level.
>>
>> In other words, at a stipulated significance level of 5% (for a
>> two-tailed test), we rejected the null hypothesis of "No significant
>> difference between observed and simulated tropospheric temperature
>> trends" in only 1 out of 98 cases (1.02%) for T2LT and 2 out of 98
>> cases (2.04%) for T2.
>>
>> You asked, John, how we might determine a baseline for judging the
>> likelihood of obtaining the 'observed' rejection rate by chance alone.
>> You suggested use of a bootstrap procedure involving the model data
>> only. In this procedure, one of the 49 20c3m realizations would be
>> selected at random, and would constitute the "surrogate observations".
>> The remaining 48 members would be randomly sampled (with replacement)
>> 49 times. The significance of the difference between the surrogate
>> "observed" trend and the 49 simulated trends would then be assessed.
>> This procedure would be repeated many times, yielding a distribution
>> of rejection rates of the null hypothesis.
>>
>> As you stated in your email, "The actual number of hits, based on the
>> real observations could then be referenced to the Monte Carlo
>> distribution to yield a probability that this could have occurred by
>> chance."
>>
>> One slight problem with your suggested bootstrap approach is that it
>> convolves the trend differences due to internally-generated
>> variability with trend differences arising from inter-model
>> differences in both climate sensitivity and in the forcings applied in
>> the 20c3m experiment. So the distribution of "hits" (as you call it;
>> or "rejection rates" in my terminology) is not the distribution that
>> one might expect due to chance alone.
>>
>> Nevertheless, I thought it would be interesting to generate a
>> distribution of "rejection rates" based on model data only. Rather
>> than implementing the resampling approach that you suggested, I
>> considered all possible combinations of trend pairs involving model
>> data, and performed the paired difference test between the trend in
>> each 20c3m realization and in each of the other 48 realizations. This
>> yields a total of 2352 (49 x 48) non-identical pairs of trend tests
>> (for each layer-averaged temperature time series).
>>
>> Here are the results:
>>
>> T2: At a stipulated 5% significance level, 58 out of 2352 tests
>> involving model data only (2.47%) yielded rejection of the null
>> hypothesis of no significant difference in trend.
>>
>> T2LT: At a stipulated 5% significance level, 32 out of 2352 tests
>> involving model data only (1.36%) yielded rejection of the null
>> hypothesis of no significant difference in trend.
>>
>> For both layer-averaged temperatures, these numbers are slightly
>> larger than the "observed" rejection rates (2.04% for T2 and 1.02% for
>> T2LT). I would conclude from this that the statistical significance of
>> the differences between the observed and simulated MSU tropospheric
>> temperature trends is comparable to the significance of the
>> differences between the simulated 20c3m trends from any two CMIP-3
>> models (with the proviso that the simulated trend differences arise
>> not only from internal variability, but also from inter-model
>> differences in sensitivity and 20th century forcings).
>>
>> Since I was curious, I thought it would be fun to do something a
>> little closer to what you were advocating, John - i.e., to use model
>> data to look at the statistical significance of trend differences that
>> are NOT related to inter-model differences in the 20c3m forcings or in
>> climate sensitivity. I did this in the following way. For each model
>> with multiple 20c3m realizations, I tested each realization against
>> all other (non-identical) realizations of that model - e.g., for a
>> model with an 20c3m ensemble size of 5, there are 20 paired trend
>> tests involving non-identical data. I repeated this procedure for the
>> next model with multiple 20c3m realizations, etc., and accumulated
>> results. In our CCSP report, we had access to 11 models with multiple
>> 20c3m realizations. This yields a total of 124 paired trend tests for
>> each layer-averaged temperature time series of interest.
>>
>> For both T2 and T2LT, NONE of the 124 paired trend tests yielded
>> rejection of the null hypothesis of no significant difference in trend
>> (at a stipulated 5% significance level).
>>
>> You wanted to know, John, whether these rejection rates are sensitive
>> to the stipulated significance level. As per your suggestion, I also
>> calculated rejection rates for a 20% significance level. Below, I've
>> tabulated a comparison of the rejection rates for tests with 5% and
>> 20% significance levels. The two "rows" of "MODEL-vs-MODEL" results
>> correspond to the two cases I've considered above - i.e., tests
>> involving 2352 trend pairs (Row 2) and 124 trend pairs (Row 3). Note
>> that the "OBSERVED-vs-MODEL" row (Row 1) is the combined number of
>> "hits" for 49 tests involving RSS data and 49 tests involving UAH data:
>>
>> REJECTION RATES FOR STIPULATED 5% SIGNIFICANCE LEVEL:
>>    Test type              No. of tests     T2 "Hits"       T2LT "Hits"
>>
>> Row 1. OBSERVED-vs-MODEL     49 x 2         2  (2.04%)     1  (1.02%)
>> Row 2. MODEL-vs-MODEL        2352          58  (2.47%)    32  (1.36%)
>> Row 3. MODEL-vs-MODEL         124           0  (0.00%)     0  (0.00%)
>>
>> REJECTION RATES FOR STIPULATED 20% SIGNIFICANCE LEVEL:
>>    Test type              No. of tests     T2 "Hits"       T2LT "Hits"
>>
>> Row 1. OBSERVED-vs-MODEL     49 x 2         7  (7.14%)     5  (5.10%)
>> Row 2. MODEL-vs-MODEL        2352         176  (7.48%)   100  (4.25%)
>> Row 3. MODEL-vs-MODEL         124           8  (6.45%)     6  (4.84%)
>>
>> So what can we conclude from this?
>>
>> 1) Irrespective of the stipulated significance level (5% or 20%), the
>> differences between the observed and simulated MSU trends are, on
>> average, substantially smaller than we might expect if we were
>> conducting these tests with trends selected from a purely random
>> distribution (i.e., for the "Row 1" results, 2.04 and 1.02%
> 7.14% and 5.10%
>
>> 2) Why are the rejection rates for the "Row 3" results substantially
>> lower than 5% and 20%? Shouldn't we expect - if we are only testing
>> trend differences between multiple realizations of the same model,
>> rather than trend differences between models - to obtain rejection
>> rates of roughly 5% for the 5% significance tests and 20% for the 20%
>> tests? The answer is clearly "no". The "Row 3" results do not involve
>> tests between samples drawn from a population of randomly-distributed
>> trends! If we were conducting this paired test using randomly-sampled
>> trends from a long control simulation, we would expect (given a
>> sufficiently large sample size) to eventually obtain rejection rates
>> of 5% and 20%. But our "Row 3" results are based on paired samples
>> from individual members of a given model's 20c3m experiment, and thus
>> represent both signal (response to the imposed forcing changes) and
>> noise - not noise alone. The common signal component makes it more
>> difficult to reject the null hypothesis of no significant difference
>> in trend.
>>
>> 3) Your point about sensitivity to the choice of stipulated
>> significance level was well-taken. This is obvious by comparing "Row
>> 3" results in the 5% and 20% test cases.
>>
>> 4) In both the 5% and 20% cases, the rejection rate for paired tests
>> involving model-versus-observed trend differences ("Row 1") is
>> comparable to the rejection rate for tests involving inter-model trend
>> differences ("Row 2") arising from the combined effects of differences
>> in internal variability, sensitivity, and applied forcings. On
>> average, therefore, model versus observed trend differences are not
>> noticeably more significant than the trends between any given pair of
>> CMIP-3 models. [N.B.: This inference is not entirely justified, since,
>> "Row 2" convolves the effects of both inter-model differences and
>> "within model" differences arising from the different manifestations
>> of natural variability superimposed on the signal. We would need a
>> "Row 4", which involves 19 x 18 paired tests of model results, using
>> only one 20c3m realization from each model. I'll generate "Row 4"
>> tomorrow.]
>>
>> John, you also suggested that we might want to look at the statistical
>> significance of trends in time series of differences - e.g., in O(t)
>> minus M(t), or in M1(t) minus M2(t), where "O" denotes observations,
>> and "M" denotes model, and t is an index of time in months. While I've
>> done this in previous work (for example in the Santer et al. 2000 JGR
>> paper, where we were looking at the statistical significance of trend
>> differences between multiple observational upper air temperature
>> datasets), I don't think it's advisable in this particular case. As
>> your email notes, we are dealing here with A/OGCM results in which the
>> phasing of El Ninos and La Ninas (and the effects of ENSO variability
>> on T2 and T2LT) differs from the phasing in the real world. So
>> differencing M(t) from O(t), or M2(t) from M1(t), probably actually
>> amplifies rather than damps noise, particularly in the tropics, where
>> the externally-forced component of M(t) or O(t) over 1979 to 1999 is
>> only a relatively small fraction of the overall variance of the time
>> series. I think this amplification of noise is a disadvantage in
>> assessing whether trends in O(t) and M(t) are significantly different.
>>
>> Anyway, thanks again for your comments and suggestions, John. They
>> gave me a great opportunity to ignore the hundreds of emails that
>> accumulated in my absence, and instead do some science!
>>
>> With best regards,
>>
>> Ben
>>
>> John Lanzante wrote:
>>> Ben,
>>>
>>> Perhaps a resampling test would be appropriate. The tests you have
>>> performed
>>> consist of pairing an observed time series (UAH or RSS MSU) with each
>>> one
>>> of 49 GCM times series from your "ensemble of opportunity". Significance
>>> of the difference between each pair of obs/GCM trends yields a certain
>>> number of "hits".
>>>
>>> To determine a baseline for judging how likely it would be to obtain the
>>> given number of hits one could perform a set of resampling trials by
>>> treating one of the ensemble members as a surrogate observation. For
>>> each
>>> trial, select at random one of the 49 GCM members to be the
>>> "observation".
>>> >From the remaining 48 members draw a bootstrap sample of 49, and
>>> perform
>>> 49 tests, yielding a certain number of "hits". Repeat this many times to
>>> generate a distribution of "hits".
>>>
>>> The actual number of hits, based on the real observations could then be
>>> referenced to the Monte Carlo distribution to yield a probability
>>> that this
>>> could have occurred by chance. The basic idea is to see if the observed
>>> trend is inconsistent with the GCM ensemble of trends.
>>>
>>> There are a couple of additional tweaks that could be applied to your
>>> method.
>>> You are currently computing trends for each of the two time series in
>>> the
>>> pair and assessing the significance of their differences. Why not first
>>> create a difference time series and assess the significance of it's
>>> trend?
>>> The advantage of this is that you would reduce somewhat the
>>> autocorrelation
>>> in the time series and hence the effect of the "degrees of freedom"
>>> adjustment. Since the GCM runs are based on coupled model runs this
>>> differencing would help remove the common externally forced variability,
>>> but not internally forced variability, so the adjustment would still be
>>> needed.
>>>
>>> Another tweak would be to alter the significance level used to assess
>>> differences in trends. Currently you are using the 5% level, which
>>> yields
>>> only a small number of hits. If you made this less stringent you
>>> would get
>>> potentially more weaker hits. But it would all come out in the wash
>>> so to
>>> speak since the number of hits in the Monte Carlo simulations would
>>> increase
>>> as well. I suspect that increasing the number of expected hits would
>>> make the
>>> whole procedure more powerful/efficient in a statistical sense since you
>>> would no longer be dealing with a "rare event". In the current
>>> scheme, using
>>> a 5% level with 49 pairings you have an expected hit rate of 0.05 X
>>> 49 = 2.45.
>>> For example, if instead you used a 20% significance level you would
>>> have an
>>> expected hit rate of 0.20 X 49 = 9.8.
>>>
>>> I hope this helps.
>>>
>>> On an unrelated matter, I'm wondering a bit about the different
>>> versions of
>>> Leo's new radiosonde dataset (RAOBCORE). I was surprised to see that the
>>> latest version has considerably more tropospheric warming than I
>>> recalled
>>> from an earlier version that was written up in JCLI in 2007. I have a
>>> couple of questions that I'd like to ask Leo. One concern is that if
>>> we use
>>> the latest version of RAOBCORE is there a paper that we can reference --
>>> if this is not in a peer-reviewed journal is there a paper in
>>> submission?
>>> The other question is: could you briefly comment on the differences
>>> in methodology used to generate the latest version of RAOBCORE as
>>> compared to the version used in JCLI 2007, and what/when/where did
>>> changes occur to
>>> yield a stronger warming trend?
>>>
>>> Best regards,
>>>
>>> ______John
>>>
>>>
>>>
>>> On Saturday 15 December 2007 12:21 pm, Thomas.R.Karl wrote:
>>>> Thanks Ben,
>>>>
>>>> You have the makings of a nice article.
>>>>
>>>> I note that we would expect to 10 cases that are significantly
>>>> different by chance (based on the 196 tests at the .05 sig level).
>>>> You found 3.  With appropriately corrected Leopold I suspect you
>>>> will find there is indeed stat sig. similar trends incl.
>>>> amplification.  Setting up the statistical testing should be
>>>> interesting with this many combinations.
>>>>
>>>> Regards, Tom
>>>
>>
>>
>
> --
>
> *Dr. Thomas R. Karl, L.H.D.*
>
> */Director/*//
>
> NOAAâ€™s National Climatic Data Center
>
> Veach-Baley Federal Building
>
> 151 Patton Avenue
>
> Asheville, NC 28801-5001
>
> Tel:  (828) 271-4476
>
> Fax:  (828) 271-4246
>
> Thomas.R.Karl@noaa.gov
>
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------


Dear Phil,
I was very sorry to hear of Hannah's health problems. I hope she makes a
speedy recovery. Please give her my best wishes, and tell her that there
is life and love after divorce!
My Mom's cataract surgery did not go very well, and it looks like she
won't be able to drive any longer. Nick and I are best placed to take
care of her, so I'm trying to persuade her to move to California. So
there could be some big changes in our lives in 2008.
Nick has turned into a fine young man. It's going to be tough to see him
leave for college in three and a half years.
I share your frustration about having to devote valuable time to the
rebuttal of crappy papers. Douglass et al. is truly awful. It should
never have been published. Any residual respect I might have had for
John Christy has now vanished. I can't believe that he's a coauthor on
this garbage.
Best wishes to all of you from rainy Livermore,
Ben
Phil Jones wrote:
>
>>  Ben,
>      Thanks for the card and photos of Nick and your caving exploits
>  with Tom and Karl !
>      Had a quiet Christmas and New Year. We did get to see Poppy
>  at Hannah's house in Deal in Kent. Matthew and Miranda came as well
>  along with Ruth's mum - so she saw her great granddaughter.
>      We were there as Hannah had to have another cyst removed from around
>  her ovary - all is well and she's recovering. Ruth has been with her since
>  mid-December. Hannah had an earlier cyst when she was 12, but this time
>  they managed to save the ovary. She still needs to see a gynaecologist to
>  see if the ovary is still working OK.
>      2007 hasn't been a great year for Hannah, as she has started divorce
>  proceedings from her husband (Gordon). They only married in 2005. He
>  seemed fine initially, but has had at least 2 affairs.
>
>       Keep up the good work on the Douglass et al comment. I'm trying to
> finish
>  a few things in the next couple of months. I will comment on drafts if
> you want.
>  Susan Solomon is trying to encourage me to respond to this piece of
>  rubbish. I'll try and encourage Rasmus Benestad of DNMI to respond. He did
>  so last time to a very similar paper in Climate Research. MM don't
> refer to
>  that and MM don't use RSS data! Their analysis is flawed anyway, but it
> would
>  all go away if they had used RSS instead of UAH!
>
>    What gets me is who are the reviewers of these two awful papers. I know
>  editors have a hard time finding reviewers, but they must have known that
>  both papers were likely awful.  It seems that editors (even of these
> two used-to-be OK
>  journals) just want more papers.
>
>      Sad day - coming in to hear of Bert Bolin's death.
>
>  Cheers
>  Phil
>
>
>
>
> Prof. Phil Jones
> Climatic Research Unit        Telephone +44 (0) 1603 592090
> School of Environmental Sciences    Fax +44 (0) 1603 507784
> University of East Anglia
> Norwich                          Email    p.jones@uea.ac.uk
> NR4 7TJ
> UK
> ----------------------------------------------------------------------------
>
--
----------------------------------------------------------------------------
Benjamin D. Santer
Program for Climate Model Diagnosis and Intercomparison
Lawrence Livermore National Laboratory
P.O. Box 808, Mail Stop L-103
Livermore, CA 94550, U.S.A.
Tel:   (925) 422-2486
FAX:   (925) 422-7675
email: santer1@llnl.gov
----------------------------------------------------------------------------