Skip to main content

Decimal Places

How many decimal places? This is a subject where science and custom conflict. Consider a routine calibration on a 300 meter calibration course:


Average = 3001.25
With 1.001, constant = 10014.2 counts per kilometer

We see a variation of two counts for 300 meters. This would imply a variation of 6.67 counts per kilometer, or 67 counts in a 10 km course. Using our constant of 10014.17, this would amount to an uncertainty 0f 6.7 meters in a 10 km course or 28 meters in a marathon. This may be statistically twiddled to produce a different number, but the variation remains.

This suggests to me that under the above circumstances I might get an answer to great precision if I use enough decimal places, but I’d not be well advised to believe it to have the same degree of accuracy.

What we have historically done is to close our eyes to the above, and substitute custom for science.

Here is a suggested procedure:

A calibration course should be specified to no more than two decimal places i.e. 301.27 meters, 1000.07 feet.

The exact average of the four calibration rides should be used without rounding.

The calibration constant should be calculated using all the decimal places in the calibration course and the exact average of the four calibration rides. It should then be rounded off to six significant figures i.e 17067.3 counts per mile, 9992.44 counts per kilometer.

Course length should be calculated using the above constants. It should then be rounded to one decimal place. This will leave us with course lengths such as:


These figures reflect greater accuracy than really exists, but doing it this way at least preserves whatever accuracy we do have.

I recognize that opinions may differ, and welcome other views.
Last edited by peteriegel

Add Reply

Link copied to your clipboard.