Published November 25, 2025 | Version v1
Conference paper Open

Measures of Estimates Quality

Authors/Creators

Description

In evaluating population estimates, it is necessary to define measures of their quality. Population estimates can be characterized by four aspects of quality: bias, scale, outliers and extreme errors. This paper axiomatically examines each of these quality aspects and describes and criticizes statistics commonly used to measure them. An important distinction is drawn between extreme errors and outliers: the former are simply the largest errors, while the latter, normally large, indicate possible problems with their data generation processes. The entire framework is based on the assumption of nonidentically distributed errors, the implications of which are discussed.

A previous version was presented to the 2002 meetings of the Population Association of America, Atlanta, GA.

Files

Measures of Estimates Quality.pdf

Files (337.0 kB)

Name Size
md5:99ed5de9638b4e44d7500f9f409ba855
337.0 kB Preview Download

Additional details

References

  • Anselin, Luc (1999), "Spatial Econometrics," http://geog55.gis.uiuc.edu/~luc/papers/baltchap.pdf.
  • Armstrong, J. Scott (1985), Long-Range Forecasting: From Crystal Ball to Computer, 2nd edition, New York: Wiley.
  • Arrow, Kenneth J. (1950), "A Difficulty in the Concept of Social Welfare," The Journal of Political Economy 58, 328-346.
  • Balinski and Young (1982), Fair Representation, New Haven: Yale University Press.
  • Beaumont, Paul M. and Andrew M. Isserman (1987), "Tests of Accuracy and Bias for County Population Projections: Comment" Journal of the American Statistical Association 82, 1004-1009.
  • Box, G. E. P. and D. R. Cox (1964), "An Analysis of Transformations," Journal of the Royal Statistical Society Series B 26, 281-290.
  • Cohen, Michael L. and Xiao Di Zhang (1988), "Aggregate and Proportional Loss Functions in Adjustment using Artificial Populations," SRD Research Report CENSUS/SRD/RR-88/11, U.S. Census Bureau. Available at http://www.census.gov/srd/papers/pdf/rr88-11.pdf.
  • Coleman, Charles D. (1999), "Nonparametric Tests for Bias in Estimates and Forecasts," in American Statistical Association: 1999 Proceedings of the Business and Economic Statistics Section. Alexandria, VA: American Statistical Association, 251-256.
  • Coleman, Charles D. (2000a), "Loss Functions for Evaluating and Optimizing Population Projections," in Federal Forecasters Conference 2000: Papers and Proceedings, 27-32.
  • Coleman, Charles D. (2000b), "Tests for Bias and Differential Bias in Estimates and Forecasts," presented to the 2000 meetings of the Southern Demographic Association, New Orleans, LA, October.
  • Coleman, Charles D. (2001a), "Non-i.i.d. Generalizations of the Matched Pairs t-Test," in American Statistical Association: 2001 Proceedings of the Business and Economic Statistics Section. Alexandria, VA: American Statistical Association.
  • Coleman, Charles D. (2001b), "Total Loss Functions for Assessing the Accuracy of Cross-Sectional Estimates and Forecasts," manuscript, U.S. Census Bureau.
  • Coleman, Charles D. (2001c), "The Asymptotic Equivalence of Level-Based and Share-Based Loss Functions," manuscript, U.S. Census Bureau.
  • Coleman, Charles D. (2002), "Loss Functions for Measuring the Accuracy of Cross-Sectional Predictions," manuscript, U.S. Census Bureau. Previous version, entitled "Metrics for Assessing the Accuracy of Cross- Sectional Estimates and Forecasts," presented to 1999 meetings of the Southern Demographic Association, San Antonio, TX.
  • Coleman, Charles D., Thomas Bryan and Jason Devine (2002), "Loss Functions for Detecting Outliers in Panel Data," manuscript, U.S. Census Bureau. Previous version presented to the Spring 2000 meetings of the Federal-State Cooperative Program on Population Estimates, March, 2000, Los Angeles, CA and to the Statistics Colloquium, Department of Applied and Engineering Statistics, George Mason University, 2000.
  • Coleman, Charles D. with Melanie Martindale (2000), "Proposed Outlier Measure: The Anti-Beta Trimmed-Mean," Proposal to the Post 2000 Estimates Methods Committee by the Outlier Subcommittee, October 10.
  • Conover, W.J. (1999), Practical Nonparametric Statistics, third edition, Wiley: New York. Davis, Sam T. (1994), "Evaluation of Postcensal County Estimates for the 1980s," Population Division Working Paper No. 5, U.S. Census Bureau, Washington, DC.
  • Davis, Sam T. (1994), "Evaluation of Postcensal County Estimates for the 1980s," Population Division Working Paper No. 5, U.S. Census Bureau. Available at http://www.census.gov/population/www/documentation/twps0005/twps0005.html.
  • Ernst, Lawrence R. (1992), "Apportionment Methods for the House of Representatives and the Court Challenges," SRD Research Report 92/06, U.S. Census Bureau. Available at http://www.census.gov/srd/papers/pdf/rr92-6.pdf.
  • Falk, M. and W. Kohne (1984), "A Robustification of the Sign Test Under Mixing Conditions," Annals of Statistics 12, 716-729.
  • Gibbons, Jean Dickinson (1982), "Sign Tests," in Kotz, Samuel and Normal L. Johnson [eds.], Encyclopedia of Statistical Sciences, v. 8, New York: Wiley, 471-474.
  • McCloskey, Deirdre N. and Stephen T. Ziliak (1996), "The Standard Error of Regression," Journal of Economic Literature 34, 97-114.
  • National Research Council (1980), Estimating Population and Income of Small Areas, Washington, D.C.: National Academy Press.
  • von Neumann, John and Morgenstern, Oskar (1944), Theory of Games and Behavior, Princeton University Press, Princeton. Second edition, 1947; third edition 1953: Princeton University Press, Princeton. Reprinted 1963 and 1967: John Wiley and Sons, New York.
  • Rousseeuw, Peter J. and Annick M. Leroy (1989), Robust Regression and Outlier Detection, New York: Wiley.
  • Smith, Stanley K. (1987), "Tests of Accuracy and Bias for County Population Projections" Journal of the American Statistical Association 82, 991-1003.
  • Smith, Stanley K. and Terry Sincich (1992), "Evaluating the Forecast Accuracy and Bias of Alternative Population Projections for States," International Journal of Forecasting 8, 495-508.
  • Spencer, Bruce (1986), "Conceptual Issues in Measuring Improvement in Population Estimates," in Bureau of the Census, Second Annual Research Conference: Proceedings, March 23-26, 1986, 393-407.
  • Staudte, Robert J. and Simon J. Sheather (1990), Robust Estimation and Testing, New York: Wiley.
  • Swanson, David A., Jeff Tayman, and Charles F. Barr (2000), "A Note on the Measurement of Accuracy for Subnational Demographic Forecasts," Demography 37, 193-201.
  • Tayman, Jeff, Edward Schafer and Lawrence Carter (1998), "The Role of Population Size in the Determination and Prediction of Population Forecast Errors: An Evaluation using Confidence Intervals for Subcounty Areas," Population Research and Policy Review 17, 1-20.
  • Tayman, Jeff and David A. Swanson (1999), "On the Validity of MAPE as a Measure of Population Forecast Accuracy," Population Research and Policy Review 18, 299-322.
  • Tiku, M.L., W.Y. Tan and N. Balakrishnan (1986), Robust Inference, New York: Dekker.
  • Wald, Abraham (1950), Statistical Decision Functions, New York: Wiley.