Table of contents

The Guide to the Expression of Uncertainty in Measurement (GUM) provided for the first time an international consensus on how to approach the widespread difficulties associated with conveying information about how reliable the value resulting from a measurement is thought to be.

This paper examines the evolution in thinking and its impact on the terminology that accompanied the development of the GUM. Particular emphasis is put on the very clear distinction in the GUM between measurement uncertainty and measurement error, and on the reasons that even though 'true value' and 'error' are considered in the GUM to be 'unknowable' and, sometimes by implication, of little (or even no) use in measurement analysis, they remain as key concepts, especially when considering the objective of measurement.

While probability theory in measurement analysis from a frequentist perspective was in widespread use prior to the publication of the GUM, a key underpinning principle of the GUM was to instead consider probability as a 'degree of belief.' The terminological changes necessary to make this transition are also covered.

Even twenty years after the publication of the GUM, the scientific and metrology literatures sometimes contain uncertainty analyses, or discussions of measurement uncertainty, that are not terminologically consistent with the GUM, leading to the inability of readers to fully understand what has been done and what is intended in the associated measurements. This paper concludes with a discussion of the importance of using proper methodology and terminology for reporting measurement results.

The 'Guide to the Expression of Uncertainty in Measurement' has now served for more than twenty years. In this communication, after attempting a balance over this period, the logical reasons are given that, on the one hand, led to the decision to update such a successful document, and, on the other hand, dictated the modifications that are being carried out with respect to the current 2008 edition.

The author is convener of the Joint Committee for Guides in Metrology (JCGM) Working Group 1 (Guide to the expression of uncertainty in measurement, or GUM). The opinion expressed in this paper does not necessarily represent the view of this Working Group.

The Guide to the Expression of Uncertainty in Measurement (GUM) has proven to be a major step towards the harmonization of uncertainty evaluation in metrology. Its procedures contain elements from both classical and Bayesian statistics. The recent supplements 1 and 2 to the GUM appear to move the guidelines towards the Bayesian point of view, and they produce a probability distribution that shall encode one's state of knowledge about the measurand. In contrast to a Bayesian uncertainty analysis, however, Bayes' theorem is not applied explicitly. Instead, a distribution is assigned for the input quantities which is then 'propagated' through a model that relates the input quantities to the measurand. The resulting distribution for the measurand may coincide with a distribution obtained by the application of Bayes' theorem, but this is not true in general.

The relation between a Bayesian uncertainty analysis and the application of the GUM and its supplements is investigated. In terms of a simple example, similarities and differences in the approaches are illustrated. Then a general class of models is considered and conditions are specified for which the distribution obtained by supplement 1 to the GUM is equivalent to a posterior distribution resulting from the application of Bayes' theorem. The corresponding prior distribution is identified and assessed. Finally, we briefly compare the GUM approach with a Bayesian uncertainty analysis in the context of regression problems.

This paper is directed at practitioners seeking a degree of assurance in the quality of the results of an uncertainty evaluation when using the procedure in the Guide to the Expression of Uncertainty in Measurement (GUM) (JCGM 100 : 2008). Such assurance is required in adhering to general standards such as International Standard ISO/IEC 17025 or other sector-specific standards. We investigate the extent to which such assurance can be given. For many practical cases, a measurement result incorporating an evaluated uncertainty that is correct to one significant decimal digit would be acceptable. Any quantification of the numerical precision of an uncertainty statement is naturally relative to the adequacy of the measurement model and the knowledge used of the quantities in that model.

For general univariate and multivariate measurement models, we emphasize the use of a Monte Carlo method, as recommended in GUM Supplements 1 and 2. One use of this method is as a benchmark in terms of which measurement results provided by the GUM can be assessed in any particular instance. We mainly consider measurement models that are linear in the input quantities, or have been linearized and the linearization process is deemed to be adequate. When the probability distributions for those quantities are independent, we indicate the use of other approaches such as convolution methods based on the fast Fourier transform and, particularly, Chebyshev polynomials as benchmarks.

The 'Guide to the Expression of Uncertainty in Measurement' (GUM) provides a framework and procedure for evaluating and expressing measurement uncertainty. The procedure has two main limitations. Firstly, the way a coverage interval is constructed to contain values of the measurand with a stipulated coverage probability is approximate. Secondly, insufficient guidance is given for the multivariate case in which there is more than one measurand. In order to address these limitations, two specific guidance documents (or 'Supplements to the GUM') on, respectively, a Monte Carlo method for uncertainty evaluation (Supplement 1) and extensions to any number of measurands (Supplement 2) have been published. A further document on developing and using measurement models in the context of uncertainty evaluation (Supplement 3) is also planned, but not considered in this paper.

An overview is given of these guidance documents. In particular, a Monte Carlo method, which is the focus of Supplements 1 and 2, is described as a numerical approach to implement the 'propagation of distributions' formulated using the 'change of variables formula'. Although applying a Monte Carlo method is conceptually straightforward, some of the practical aspects of using the method are considered, such as the choice of the number of trials and ensuring an implementation is memory-efficient. General comments about the implications of using the method in measurement and calibration services, such as the need to achieve transferability of measurement results, are made.

When the mass laboratory of DFM was created in 1989, harmonized principles for the evaluation of measurement uncertainty were being created based on a very simple principle endorsed by the CIPM: all uncertainties should be interpreted as the square root of variances (standard deviations) and should be combined and expressed as such. This paper describes how DFM has interpreted this principle for the calibration of mass standards and for the measurement of mass in general. The importance of reporting calibration results in a compact way that is easily propagated down the traceability chain is also discussed.

This paper provides a selected review of topics relating to evaluating and expressing uncertainty for some measurands that occur in high-frequency electromagnetic metrology. Specific emphasis is given to complex-valued quantities (i.e. vector measurands having both an associated magnitude and phase component), such as scattering parameters (i.e. S-parameters) used at radio, microwave, millimetre-wave and terahertz frequencies.

The developments that led to the third edition of the Eurachem Guide 'Quantifying Uncertainty in Analytical Measurement' are reviewed. Particular attention is given to the rationale for early use of spreadsheet methods, the incorporation of method performance data in the second edition, and the third edition's provisions on uncertainties near zero and Monte Carlo methods. The development of uncertainty concepts in chemistry is reviewed briefly and some of the challenges found in early implementation of measurement uncertainty in chemistry are recalled. Problems arising from uncertainty evaluation for reference measurements with limited data are discussed.

Measurements often provide an objective basis for making decisions, perhaps when assessing whether a product conforms to requirements or whether one set of measurements differs significantly from another. There is increasing appreciation of the need to account for the role of measurement uncertainty when making decisions, so that a 'fit-for-purpose' level of measurement effort can be set prior to performing a given task. Better mutual understanding between the metrologist and those ordering such tasks about the significance and limitations of the measurements when making decisions of conformance will be especially useful. Decisions of conformity are, however, currently made in many important application areas, such as when addressing the grand challenges (energy, health, etc), without a clear and harmonized basis for sharing the risks that arise from measurement uncertainty between the consumer, supplier and third parties.

In reviewing, in this paper, the state of the art of the use of uncertainty evaluation in conformity assessment and decision-making, two aspects in particular—the handling of qualitative observations and of impact—are considered key to bringing more order to the present diverse rules of thumb of more or less arbitrary limits on measurement uncertainty and percentage risk in the field. (i) Decisions of conformity can be made on a more or less quantitative basis—referred in statistical acceptance sampling as by 'variable' or by 'attribute' (i.e. go/no-go decisions)—depending on the resources available or indeed whether a full quantitative judgment is needed or not. There is, therefore, an intimate relation between decision-making, relating objects to each other in terms of comparative or merely qualitative concepts, and nominal and ordinal properties. (ii) Adding measures of impact, such as the costs of incorrect decisions, can give more objective and more readily appreciated bases for decisions for all parties concerned. Such costs are associated with a variety of consequences, such as unnecessary re-manufacturing by the supplier as well as various consequences for the customer, arising from incorrect measures of quantity, poor product performance and so on.

Measurements underpin the engineering decisions that allow products to be designed, manufactured, operated, and maintained. Therefore, the quality of measured data needs to be systematically assured to allow decision makers to proceed with confidence. The use of standards is one way of achieving this. This paper explores the relevance of international documentary standards to the assessment of measurement system capability in High Value Manufacturing (HVM) Industry. An internal measurement standard is presented which supplements these standards and recommendations are made for a cohesive effort to develop the international standards to provide consistency in such industrial applications.

In the course of the twenty years since the publication of the Guide to the Expression of Uncertainty in Measurement (GUM), the recognition has been steadily growing of the value that statistical models and statistical computing bring to the evaluation of measurement uncertainty, and of how they enable its probabilistic interpretation. These models and computational methods can address all the problems originally discussed and illustrated in the GUM, and enable addressing other, more challenging problems, that measurement science is facing today and that it is expected to face in the years ahead.

These problems that lie beyond the reach of the techniques in the GUM include (i) characterizing the uncertainty associated with the assignment of value to measurands of greater complexity than, or altogether different in nature from, the scalar or vectorial measurands entertained in the GUM: for example, sequences of nucleotides in DNA, calibration functions and optical and other spectra, spatial distribution of radioactivity over a geographical region, shape of polymeric scaffolds for bioengineering applications, etc; (ii) incorporating relevant information about the measurand that predates or is otherwise external to the measurement experiment; (iii) combining results from measurements of the same measurand that are mutually independent, obtained by different methods or produced by different laboratories.

This review of several of these statistical models and computational methods illustrates some of the advances that they have enabled, and in the process invites a reflection on the interesting historical fact that these very same models and methods, by and large, were already available twenty years ago, when the GUM was first published—but then the dialogue between metrologists, statisticians and mathematicians was still in bud. It is in full bloom today, much to the benefit of all.

The expression of uncertainty has hitherto been seen as an add-on—first an estimate is obtained and then uncertainty in that estimate is evaluated. We argue that quantification of uncertainty should be an intrinsic part of measurement and that the measurement result should be a probability distribution for the measurand.

Full quantification of uncertainties in measurement, recognizing and quantifying all sources of uncertainty, is rarely simple. Many potential sources of uncertainty can effectively only be quantified by the application of expert judgement. Scepticism about the validity or reliability of expert judgement has meant that these sources of uncertainty have often been overlooked, ignored or treated in a qualitative, narrative way. But the consequence of this is that reported expressions of uncertainty regularly understate the true degree of uncertainty in measurements.

This article first discusses the concept of quantifying uncertainty in measurement, and then considers some of the areas where expert judgement is needed in order to quantify fully the uncertainties in measurement. The remainder of the article is devoted to describing methodology for eliciting expert knowledge.