This issue of reproducibility arises with NMath users when writing and running unit tests; which is why it’s important when writing tests to compare floating point numbers only up to their designed precision, at an absolute maximum. With the IEEE 754 floating point representation which virtually all modern computers adhere to, the single precision float type uses 32 bits or 4 bytes and offers 24 bits of precision or about 7 decimal digits. While the double precision double type requires 64 bits or 8 bytes and offers 53 bits of precision or about 15 decimal digits. Few algorithms can achieve significant results to the 15th decimal place due to rounding, loss of precision due to subtraction and other sources of numerical precision degradation. NMath’s numerical results are tested, at a maximum, to the 14th decimal place.

A Precision Example

As an example, what does the following code output?

      double x = .050000000000000003;
      double y = .050000000000000000;
      if ( x == y )
        Console.WriteLine( "x is y" );
      else
        Console.WriteLine( "x is not y" );

I get “x is y”, which is clearly not the case, but the number x specified is beyond the precision of a double type.

Due to these limits on decimal number representation and the resulting rounding, the numerical results of some operations can be affected by the associative reordering of operations. For example, in some cases a*x + a*z may not equal a*(x + z) with floating point types. Although this can be difficult to test using modern optimizing compilers because the code you write and the code that runs can be organized in a very different way, but is mathematically equivalent if not numerically.

So reproducibility is impacted by precision via dynamic operation reorderings in the ALU and additionally by run-time processor dispatching, data-array alignment, and variation in thread number among other factors. These issues can create run-to-run differences in the least significant digits. Two runs, same code, two answers. This is by design and is not an issue of correctness. Subtle changes in the memory layout of the program’s data, differences in loading of the ALU registers and operation order, and differences in threading all due to unrelated processes running on the same machine cause these run-to-run differences.

Managing Reproducibility

Most importantly, one should test code’s numerical results only to the precision that can be expected by the algorithm, input data, and finally the limits of floating point arithmetic. To do this in unit tests, compare floating point numbers carefully only to a fixed number of digits. The code snippet below compares two double numbers and returns true only if the numbers match to a specified number of digits.

private static bool EqualToNumDigits( double expected, double actual, int numDigits )
    {
      double max = System.Math.Abs( expected ) > System.Math.Abs( actual ) ? System.Math.Abs( expected ) : System.Math.Abs( actual );
      double diff = System.Math.Abs( expected - actual );
      double relDiff = max > 1.0 ? diff / max : diff;
      if ( relDiff <= DOUBLE_EPSILON )
      {
        return true;
      }

      int numDigitsAgree = (int) ( -System.Math.Floor( Math.Log10( relDiff ) ) - 1 );
      return numDigitsAgree >= numDigits;
    }

This type of comparison should be used throughout unit testing code. The full code listing, which we use for our internal testing, is provided at the end of this article.

If it is essential to enforce binary run-to-run reproducibility to the limits of precision, NMath provides a flag in its configuration class to ensure this is the case. However this flag should be set for unit testing only because there can be a significant cost to performance. In general, expect a 10% to 20% reduction in performance with some common operations degrading far more than that. For example, some matrix multiplications will take twice the time with this flag set.

Note that the number of threads that Intel’s MKL library uses ( which NMath depends on ) must also be fixed before setting the reproducibility flag.

int numThreads = 2;  // This must be fixed for reproducibility.
NMathConfiguration.SetMKLNumThreads( numThreads );
NMathConfiguration.Reproducibility = true;

This reproducibility run configuration for NMath cannot be unset at a later point in the program. Note that both setting the number of threads and the reproducibility flag may be set in the AppConfig or in environmental variables. See the NMath User Guide for instructions on how to do this.

Paul

References

M. A. Cornea-Hasegan, B. Norin. IA-64 Floating-Point Operations and the IEEE Standard for Binary Floating-Point Arithmetic. Intel Technology Journal, Q4, 1999.
http://gec.di.uminho.pt/discip/minf/ac0203/icca03/ia64fpbf1.pdf

D. Goldberg, What Every Computer Scientist Should Know About Floating-Point Arithmetic. Computing Surveys. March 1991.
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

Full double Comparison Code

private static bool EqualToNumDigits( double expected, double actual, int numDigits )
    {
      bool xNaN = double.IsNaN( expected );
      bool yNaN = double.IsNaN( actual );
      if ( xNaN && yNaN )
      {
        return true;
      }
      if ( xNaN || yNaN )
      {
        return false;
      }
      if ( numDigits <= 0 )
      {
        throw new InvalidArgumentException( "numDigits is not positive in TestCase::EqualToNumDigits." );
      }

      double max = System.Math.Abs( expected ) > System.Math.Abs( actual ) ? System.Math.Abs( expected ) : System.Math.Abs( actual );
      double diff = System.Math.Abs( expected - actual );
      double relDiff = max > 1.0 ? diff / max : diff;
      if ( relDiff <= DOUBLE_EPSILON )
      {
        return true;
      }

      int numDigitsAgree = (int) ( -System.Math.Floor( Math.Log10( relDiff ) ) - 1 );
      //// Console.WriteLine( "x = {0}, y = {1}, rel diff = {2}, diff = {3}, num digits = {4}", x, y, relDiff, diff, numDigitsAgree );
      return numDigitsAgree >= numDigits;
    }

The post Precision and Reproducibility in Computing appeared first on CenterSpace.

I’m working on the primitives (NMathCoreShared.dll) and have found a rather odd ‘quirk’ with complex division by zero:

DoubleComplex aa = new DoubleComplex(0.0, 0.0);
DoubleComplex bb = new DoubleComplex(5.2, -9.1);
DoubleComplex cc = new DoubleComplex();

cc = bb/aa;
Console.WriteLine(cc);  // (NaN,NaN)

double g = -5.0 / 0.0;
Console.WriteLine(g);   // -Infinity

On my machine (Athlon FX), I get the value NaN back when I’m pretty sure it should be ‘Inf’?

The code implementing complex division in the NMath complex number class does not check to see if the divisor is zero. It just applies the formula for complex division in terms of the operands’ real and imaginary parts:

\frac{a + bi}{c + di} = \frac{ac + bd}{c^2 + d^2} + i\frac{bc - ad}{c^2 + d^2}     (1)

As you can see, when c = d = 0 both the real and imaginary components of the quotient take the form 0/0. Now the IEEE standard says that for real numbers a/0, a not equal to 0, should yield +∞ or -∞ where the sign depends on whether a > 0 or a < 0 and 0/0 should yield NaN (Not a Number). So a direct application of the formulas for division of complex number, without “special casing” the zero denominator case, will yield a result of (NaN, NaN).

One might be tempted to say “OK, that’s the end of that. (NaN, NaN) is obviously the correct behavior”. However, this does not set very well with intuition. If I have a fixed complex number w which is divided by another variable complex number z, it seems that as z gets smaller and smaller, the quotient should get bigger and bigger. There is a slight technicality here–there really is no notion of bigger and smaller for the complex number. That is, there is no ordering of the complex numbers that is consistent with its algebraic structure. In particular there is no notion of negative and positive complex numbers, so the concept of positive and negative infinity are not well defined. Let’s take a look at some theoretical stuff in the hopes it will help to guide our thinking.

Let’s start with something more familiar. What does it mean to divide a real number by zero? From a purely algebraic sense if a and b are two real numbers then the number a ÷ b is the solution to the equation b * x = a. When b = 0 this equation has no solution and so the quotient is not defined (unless we extend the real numbers to include infinity and define algebraic operations that include infinity, but let’s not go there now as it isn’t really germane to our conversation). Where does infinity come in? Well, one can also look at the question from an analysis point of view, where one deals in functions and limits. With this perspective, given a real number a one defines a division function f_a(x) = a ÷ x. This function is well defined for all x not equal to 0. But, analysis deals also with limits, and one would naturally ask what happens when x gets close to 0. The answer is as x → 0, f_a → ∞. Formally, this means the following: Given any real number M>0, there exists a corresponding real number δ > 0 such that |fa | > M whenever |x| < δ. Now, we can apply this definition to complex numbers since we have a notion of absolute value for complex numbers: if z = a + ib is a complex number, then the modulus of z, |z|, is defined as

|z|=\sqrt{a^2 + b^2}

That is it’s the euclidean distance from the point z = (a,b) to the origin in the complex plane. From this it is clear that as a complex number tends to zero, so must its real and imaginary parts. So as the denominator in equation (1) above tends to zero, so do its real and imaginary parts c and d. The fact that c and d are squared in the denominator of the real and imaginary parts of the quotient means that they will tend to zero faster than the their corresponding numerators and hence, using a L’Hôpital’s Rule based argument, or a straight δ, M based argument, we can see that both the real and imaginary parts of the quotient tend to infinity as the denominator tends towards zero. Whether it is positive or negative infinity is another question.

But enough theoretical talk. Let’s look at something concrete. Unlike C#, the standard C++ library includes a complex number class. What do they do when dividing by zero? Here’s the result I get with Microsoft’s C++ complex class:

Microsoft Visual C++ 15.0

(1,1)/(0,0) = (1.#QNAN,1.#QNAN)
(0,0)/(0,0) = (1.#QNAN,1.#QNAN)
1/0 = 1.#INF
0/0 = -1.#IND

Hmm. They seem to just apply the formulas and get 0/0 for the real and imaginary parts. How about the GNU C++ compiler?

g++ 4.3.2

(1,1)/(0,0) = (inf,inf)
(0,0)/(0,0) = (nan,nan)
1/0 = inf
0/0 = nan

Interesting. They return infinite real and imaginary parts. What’s more interesting is that the two compilers do not agree. And just for fun, here’s the result with the complex class included in Microsoft’s F# compiler:

F#

(1,1)/(0,0) = NaNr+NaNi
(0,0)/(0,0) = NaNr+NaNi
1/0 = Infinity
0/0 = NaN

No surprise here. At least Microsoft is consistent.

What does the GNU compiler do about positive and negative infinity in its results?

(1,1)/(0,0) = (inf,inf)
(-1,-1)/(0,0) = (-inf,-inf)
(1,-1)/(0,0) = (inf,-inf)
(-1,1)/(0,0) = (-inf,inf)

Apparently, they have adopted the convention that the sign of infinity in the real and imaginary parts of the quotient is the same as the signs of the real and imaginary parts of the numerator, respectively.

So, what should NMath do? On the one hand .NET is a Microsofty kind of thing, so maybe we should be consistent with their C++ implementation. On the other hand g++ is a widely distributed compiler and they seem to take the moral high road here. What would you do?

Steve

The post Complex division by zero appeared first on CenterSpace.