As you reduce from six digits to five, the answer is rounded from six digits to five. This continues all the way down to one digit of accuracy. This behavior is characteristic of a perfectly stable computation: the length of the floating point numbers affects the number of digits that are produced, but doesn't affect what those digits are. Contrast this behavior with that of our original formula.

Return to lesson.

Joseph L. Zachary
Hamlet Project
Department of Computer Science
University of Utah