"The American Enterprise Institute is launching a major project to produce a review and policy critique of the forthcoming Fourth Assessment Report (FAR)." Thus begins a letter recently sent to my colleagues Jerry North and Steve Schroeder. This letter invites them to write a "well-supported but accessible discussion of which elements of climate modeling have demonstrated predictive value that might make them policy-relevant and which elements of climate modeling have less levels of predictive untility, and hence, less utility in developing climate policy."
I fully support the idea that people should dive into the AR/4 (that's what everyone else calls the Fourth Assessment Report) and take a look at the science themselves. While one might be skeptical that the AEI will give the AR/4 a fair hearing, the fact that they have solicited input from a credible and mainstream scientist like Jerry North suggests to me that I should not prejudge their effort.
That said, I think they've set themselves up to fail by their choice of question. "What's the policy value of climate models?" can be split into two parts:
(1) What are the scientific uncertainties in the models?
(2) Given this level of uncertainty, can model predictions then be used as a basis for action?
Question 1 is a scientific one, which fills thousands of journal pages every year. This could indeed be a valid subject for a scientific review paper. Quesiton 2, however, is not a scientific one. Two people can agree on the level of uncertainty in the model but disagree on whether that level of uncertainty precludes or demands action --- it just depends on a value judgment about how risk averse we should be.
For example: one person can say, "if there's a 1% chance that climate change might lead to catastrophic impacts, then we need to act now." Clearly the models provide that level of certainty. Another person might say, "I think we should be 99% certain that climate change is a significant risk before we take action." The models do not provide that level of certainty, so to that person the models are too uncertain. The choice of 1% vs. 99% is a moral judgment, not a scientific one.
This type of tangling between positive (scientific) and normative (moral) questions is one of the major reasons that the climate change debate can be so confusing to the general public. We discuss this in chapter 2 of our book.
Also note: they're willing to pay $10,000 to the authors. That's A LOT of money for this type of activity. It was enough that it made me think, "maybe I should get involved with this." Then I snapped back to reality.
[note added 7/31: My wife read this blog, saw the figure of $10,000, and asked me sweetly, "Are you SURE that climate change is real? We could really use the money."]
17 comments:
"the fact that they have solicited input from a credible and mainstream scientist like Jerry North suggests to me that I should not prejudge their effort."
An old chinese (or was it japanse?) proverb says: Know thine enemy.
In order to effectively argue against something, one must know both the strengths and the weaknesses of the aruments given by one's opponent.
A lawyer (read CEI) has no interest in actual truth, only in winning his case.
"While one might be skeptical that the AEI will give the AR/4 a fair hearing, the fact that they have solicited input from a credible and mainstream scientist like Jerry North suggests to me that I should not prejudge their effort."
AEI's record on the Iraq war alone would justify not trusting them. They were involved in the last round of Social Security lies, as well. And that's just what I can think of, off of the top of my head.
I concur with the suggestion that they're just trolling for possibly useful stuff. If they get something good, they'll use it. If not, they trash it. If they can quote mine it, they'll do that.
I understand the reflexive suspicion in responding to a conservative think tank; however in this instance is likely that their interests parallel your own.
Conservatives (the non-theological) have both an interest and generational faith in modernity - the modern (vs. the post-modern 'left') belief in objective truth and scholarly judgment. Ironically, the 'old time' liberal issues of university free speech, academic freedom, modernity, and the primacy of reason are important cultural battlegrounds, this time with the likes of AEI joining some older liberals against politicized scholarship. The old stereotypes of repressive backwoods yahoos are not relevant.
Simply put, they would like to know what mainstream scientists are confident and less confident about in this issue - without blog hysterics, conspiracy mongering, sinister speculations, etc.
On this issue, this is simply good sense - no matter who one thinks it helps.
Good luck on the new blog. It looks promising so far (from what little I can see).
Some suggestions and comments:
1) I would like to see some good discussion of the GCMs. They are quite opaque to me (I don't even understand what is gained by making them so complicated) and I haven't been able to find any reasonably dispassionate discussion of them.
2) A good comment section can be very interesting reading. I don't know how you actually do this without expending enormous effort, but if the comments become just political name-calling and nyah-nyah sessions, it isn't worth plodding through the dross. RealClimate often has very knowledgeable people talking serious science in the comments, and ClimateAudit has a surprising number of threads with serious development of ideas. (Of course, both RealClimate and ClimateAudit have large numbers of junk comments.)
3) Your should really link to ClimateAudit alongside RealClimate -- RealClimate is usually censored and gives only one side of the debate, and it is hard to see this unless you access ClimateAudit
I'm curious about who owns the resulting paper - AEI or the authors. If AEI doesn't like the analysis, can it say "thanks very much" and the paper goes down the rabbit hole, never to be seen by anyone else?
Brian-
That's an interesting question. I'm not a lawyer, but I'm quite certain that they are under no obligation to publish the paper if it doesn't meet their expectations. The real question is whether you get your $10K if they don't publish it ...
Regards.
Terry-
Thanks for the suggestions. I definitely plan to blog on GCMs, particularly where the uncertaities lie. I also agree that maintaining civility in the comment section is important. Finally, I've added a link to climateaudit.
Regards.
I have attempted to get discussions started about the AOLBGCM codes, and papers based on results from the codes, a couple of times around the blogsphere. The discussions also include editorial policies at the major 'science' journals. I have not had much success and I would like to try again here.
Within the “climate change” community computer software Verification, Validation (V&V), Software Quality Assurance (SQA), and user qualifications seem to be issues that are not of uppermost importance. I am an outsider and open to correction on this observation. I use the words Verification, Validation and Quality Assurance in the technical sense as used in the literature associated with these subjects. Citations to the literature are given below.
For purposes of the discussion here a brief summary of the technical definitions of Verification and Validation are given.
Verification is the process of demonstrating that the mathematical equations used in the computer program are solved correctly.
Validation is the process of demonstrating that the correct mathematical equations are used in the computer program.
Generally speaking, Verification is a mathematical process and Validation is an engineering/scientific processes.
The ultimate objective of applying the processes to computer software, and its users, is to ensure that calculated results accurately reflect the intended application of the software. Absence of application of these processes to computer software should be a very strong indication that the calculated results are suspect.
All calculations from programs in which inherently complex physical phenomena and processes occurring within complex geometries are the focus are generally considered to be suspect and usually taken with a grain of salt. The modeling and calculation of climate change over the spatial and temporal scales for a planet present major challenges to all aspects of mathematical modeling and computer calculations. The number of important systems involved along with the inherent complexity of the phenomena and processes, interacting on multiple time scales over extreme spatial and temporal extents, represent possibly unprecedented challenges. V&V and SQA are absolutely necessary under these situations and are standard operating procedures (SOP) for all major software development projects. They are also absolutely necessary and should be SOP for software the calculated results of which are submitted for publication in archival journals. For calculated results that form the basis of policies that affect the health and safety of the public, the requirements for application of these processes are codified in the laws of the country.
Excellent starting references include:
(1) Patrick Roache, “Verification and Validation in Computational Science and Engineering,” published by Hermosa Press. http://kumo.swcp.com/hermosa/index.html.
(2) William L. Oberkampf, Timothy G. Trucano, and Charles Hirsch, “Verification, Validation, and Predictive Capability in Computational Engineering and Physics,” Sandia National Laboratories Report SAND 2003-3769, 2003. http://www.csar.uiuc.edu/F_viz/gallery/VnV/SAND2003-3769.pdf. See also, https://www.dmso.mil/public/library/projects/vva/found_04/oberkampf_et_al_vv_and_predictive_etc.pdf
These publications contain extensive reference citations to the published literature on these subjects. A Google search will find many more resources. In particular, the group at Sandia has remained very active in these areas for several years now. Other National Laboratories, especially those associated with the ASCI program, also have groups actively working in these areas.
With this very brief introduction, let’s next look at publications in the technical journals of professional societies.
Climate change paper submittals the basis of which are calculations with computer software continue to be important sources of publications in journals such as Science, Nature, several technical journals from the American Geophysical Union (AGU), the American Meteorological Society (AMS), and many others. Some of these papers might at some time be used as audit/checkpoints/benchmarks for other calculations, and other of the papers might become part of the basis for public policy decisions. The software used for these papers needs to be Verified and Validated for the applications to which they are applied. Papers for which the software has not been Verified should not be accepted for publication in an archival journal.
The first crucial aspect of such papers should be the status of the Verification of the software. Several engineering societies and their journal editorial boards have recently put into place technical requirements on the Verification of the software before the paper can be considered for publication. If the requirements have not been met the paper will not be published; in some cases the paper will be rejected out-of-hand and not be sent out for review. Papers for which the basis is a single calculation on a single grid with no investigations of convergence and other stopping criteria are typically sent back to the authors.
Some of these professional organizations and associated journals include: The American Society of Mechanical Engineers (ASME) Journal of Heat Transfer and Journal of Fluids Engineering; The American Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft and Rockets; and the International Journal of Numerical Methods in Fluids. Other professional societies and journals are sure to follow the lead of these.
References for the editorial polices for these journals are as follows.
The ASME Journal of Heat Transfer:
Editorial Board, “Journal of Heat Transfer Editorial Policy Statement on Numerical Accuracy,” ASME Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.
The ASME Journal of Fluids Engineering:
A detailed discussion of the present status of the Journal of Fluids Engineering policy is available here: http://www.divisions.asme.org/fed/divisionadmin/cfdindex.html.
The existing policy is given as a part of the preceding discussion here: http://www.divisions.asme.org/fed/divisionadmin/Current%20JFE%20Publication%20Policy.pdf. (The %20 are blanks) The existing policy was first stated here: C. J. Freitas, “Editorial Policy Statement on the Control of Numerical Accuracy,” ASME Journal of Fluids Engineering, Vol. 117, No. 1, p. 9, 1995.
The Fluids Engineering policy can be accessed directly online and provides a good summary of the issues.
The AIAA Journal of Spacecraft and Rockets:
AIAA, Editorial Policy Statement on Numerical Accuracy and Experimental Uncertainty, AIAA Journal, Vol. 32, No. 1, p. 3, 1994.
The International Journal of Numerical Methods in Fluids:
P. M. Gresho and C. Taylor, “Editorial,” International Journal of Numerical Methods in Fluids, Vol. 19, Issue 7 p. iii, October 1994, DOI: 10.1002/fld.1650190702
I have been unsuccessful in locating editorial policies on these important issues for any of the science journals listed above. I consider this to be a major and extremely important failing. As discussed above, many engineering journals would reject out-of-hand all submittals for publication of papers that do not address these issues.
It is my understanding that the calculated results from most large complex AOLBGCM codes cannot be demonstrated to be independent of the discrete representations of the continuous equations. That is, the results are functions of the size of the discrete representations (or truncated series) of the spatial and temporal scales used in the calculations. I think this situation is without precedence in all of science and engineering. This is the most fundamental concept taught in every numerical methods textbook that I am familiar with. If this is the correct situation, the calculated results are not solutions to the continuous equations and at the very best represent some kind of approximate “solution”. However, “solution to the continuous equations” is not a phrase that I would apply to these calculations.
Focus on The Global Average Temperature calculated by the codes gives a false indication of the robustness of the modeling and calculations because this is a solution meta-functional result that maps everything calculated by the code into a single number. A process that has a high potential to hide an enormous number of problems.
In the absence of established, formal V&V and SQA processes and procedures, calculated results are very much less certain to contain true information and thus do not in fact provide knowledge. Even when these processes are rigorously applied to computer software, mistakes and errors still survive, although to a very much less degree than in the absence of the procedures. For the case of the very complex codes and applications associated with climate change, the chances of mistakes and errors are much larger than for less complex analyses. Consider situations that have occurred within the climate change community that sometimes surface in the literature. Data reduction software is an example.
The potential for mistakes and numerical errors in software in the absence of independent, formal, V&V and SQA procedures are among the reasons that engineering journals have implemented editorial policies. Apparently the organizations have decided that publication of a paper that has not been demonstrated to correctly solve the equations and to be based on the correct equations for its intended applications, has a high potential to not represent physical reality. Additionally, apparently they consider that the consequences of publishing such results will not contribute to advances in understanding and knowledge.
Regulatory agencies that are responsible decisions that affect the health and safety of the public will never, and I’m aware that I should never use always and never, make policy decisions based on computer software that has not been Verified and Validated and maintained and applied under SQA procedures. Observational data will always, of course, be more important than computer calculations alone.
If the science community does not begin to implement formal procedures, and at the same time base information on software that is not maintained and applied formally, regulatory agencies will ignore the calculations. And it is not just the large, complex AOLBGCM codes that will eventually be required to implement the processes and procedures. All software, and software users, will have to be demonstrated to be Qualified for applications to the analyses for which the software has been designed. Additionally, it is not just the ‘main’ routines in the large codes that will require these, but also the initial and boundary conditions, the pre- and post- processing routines ( run-time options, grid generation, processing of the calculated results, etc.), the qualifications of the users, and the procedures for installing the software onto a user’s computer system. That is, all aspects of the codes, users, applications, and results.
Can anyone point me to the editorial policies that address Verification and Validation issues for the professional science organizations and the associated technical journals?
Dan Hughes-
I do not believe that any of the journals that I'm familiar with (GRL, JGR, JAS, J. Clim, Science, Nature, etc.) have any policy requiring software validation prior to publication. I am not a GCM modeler, so I cannot really discuss these issues any further, but I hope that if anyone else has comments they will leave them on this thread.
Regards.
The long URLs in my post above got truncated. Let's try this.
SAND2003a
SAND2003b
ASME_FED_NEW
ASME_ORIGINAL
Dan Hughes said: "They (Software Verification and validation) are also absolutely necessary and should be SOP for software the calculated results of which are submitted for publication in archival journals."
I think if one is to make such a statement, it is important to specify exactly what one means by verification and validation.
For example, when you refer to software verification are you talking about "Formal mathematical proofs"?
If so, I'd say that such verification is not practical and does not even guarantee what it purports to "prove." (because there is no gurantee that the verification will be done without errors).
Peter Van Der Linden summed it up succintly in his book "expert C programming:
The problem with engineers is that they cheat in order to get results.
The problem with mathematicians is that they work on toy problems in order to get results.
The problem with program verifiers is that they cheat on toy problems in order to get results. -- Anonymous
As a software engineer who has worked on scientific software for a decade, I agree that it is important to have practices in place to check that software does what it is purported to do, but I would simply have to say that "formal verification" ain't the way to go.
Even if scientists were all trained to do formal verification on their computer programs, they certainly do not possess the considerable time required nor is there any guarantee that the verification will be any more "error free" than the software that it was meant to verify.
Anonymous. I provided references to recent Verification and Validation literature as related to science and engineering software. Some are online and easily accessible. None propose that formal mathematical proof methods be used. And I provided brief summaries of the definitions directly in the post.
The Method of Manufactured Solutions (MMS) [1], [2] [3] has proven to be very effective relative to proving the verification of the code. Use of a manufactured solution, in contrast to an analytical solution, has the advantage that the solution can be designed to make all the terms in the equations important. As shown in [3], for example, the MMS can be used to determine the presence of many common coding mistakes in the code. The MMS is used to demonstrate the order of accuracy of the numerical solution methods used in the code and the order of convergence for the discrete approximations employed.
Comparison of the order of accuracy and order of convergence of the discrete approximations
produced by the code with the theoretical values ensures verification of the coding of the
numerical methods.
The Method of Exact Solutions and Method of Manufactured Solutions have the advantage that
the results are quantitative in nature so that concrete numbers are determined by these methods.
1. P. J. Roache, Verification and Validation in Computational Science and Engineering,
Hermosa Publishers, Albuquerque, NM, 1998.
2. P. J. Roache, “Code Verification by the Method of Manufactured Solutions,” Journal of Fluids Engineering, Vol. 114, No. 1, pp. 4-10, 2002.
3. K. Salari and P. Knupp, “Code Verification by the Method of Manufactured Solutions,” Sandia National Laboratory Report SAND2000-1444, http://
www.osti.gov/bridge/servlets/purl/759450-wLI4Ux/native/, 2000.
Dr. Dressler, while I understand what you are getting at in your example of the two people who agree on the uncertainty in the model but disagree on the need for action, question 2 becomes a scientific question for the calculus of pure choice if you quantify and objectify the dimensions of the impacts by employing detailed economic analysis.
Eg, we know what it's costing to rebuild NOLA, what will it cost over the timeline of a specific model to relocate NYC? What will it cost over the timeline of a specific model to re-locate every city and every port on the three US coastlines?
The predictive utility of the models doesn't mean anything in the abstract.
But a 50% level of certainty takes on a whole new meaning when the catastrophic impact has a price tag of trillions of dollars attached to it.
I am also familiar with Verification and Validation methods. Problem is that you have to verify and validate the physics first, and V&V methods will not work very well there.
Indeed, the situation is very asymmetric. Get the physics wrong and all the good coding in the world will not help. Get the physics right and serious coding errors will show up as aphysical results, e.g. significant differences with simple models. At that point you go back in and check the coding.
Essentially that is what happened with the MSU measurements, except that Christy and Spencer were so wedded to their denialist positions that they resisted for years.
Eli Rabett said: "Problem is that you have to verify and validate the physics first, and V&V methods will not work very well there."
I agree, but I thought that was what scientific peer review was supposed to be about.
There are clearly some who seem to think that peer review is over-rated, but I don't happen to be one of them.
In fact, I think that anyone who has not subjected their work to the peer review/publication process has no business "playing" scientist.
Scientists have enough things to deal with without having to debunk every pseudo-scientific paper "published" on the website of some Exxon-Mobil-underwritten "think tank".
The web is great for exchange of information among scientists, but it is certainly no substitute for the traditional scientific processs of peer reviewed publication. Some (mainly non-scientists) clearly think otherwise.
Do you think that Sir Nicolas Sterns made do with a mere $10,000 (£5.800) while researching & writing his report? I suspect not. I suspect his typist earned more & neither of them are "credibl" scientists.
This is merely a small attempt to lessen the slope that the government's wallet has brought to the catastrophist side.
Post a Comment