Numerical Precision

Standard ENDF-format files use an 11-column field to represent floating-point numbers. In order to maximize the number of significant digits possible within this limit, unnecessary characters are eliminated. Therefore, energies usually take on forms like " 1.234567+3", which provides for 7 significant digits. If larger exponents are needed, the format drops back to 6 or 5 significant digits.

One of the few places that this standard numerical format can be inadequate is for the representation of the cross sections reconstructed for very narrow resonances. Since narrow resonances only occur for a restricted range of energies (say, 1 eV to 1 MeV), 11-column forms such as " 123.456789" can provide an accuracy of 9 significant digits, which is always sufficient in practice.

However, the need for more than 7 sigificant digits is rare. Moreover, the smoothing effect of Doppler broadening usually assures that only 7 digits are needed at 300K, even if 11 are needed at 0K. In addition, we want to preserve as much similarity between PENDF tapes and ENDF tapes as possible. For these reasons, we have adopted a complex system for the precision of energy values used in NJOY97. The initial nodes, the values read from standard ENDF tapes, and values added by RECONR (such as those from the 1, 2, 5, 10, 20, ... sequence) will normally have only a few significant digits (typically 5 or fewer). This helps to identify them on listings, and BROADR will try not to thin out energy values with several zeros on the end. In subdividing energy panels to linearize resonance cross sections, RECONR will first try to fit the actual function with numbers limited to the 7-digit standard form. Only when this is impossible will RECONR resort to the 11-digit form. It is fairly easy to spot such numbers on a listing of the 0K PENDF tape.

In BROADR, the code will preferentially use the numbers with lower numbers of significant digits as nodes in its reconstruction process. Thus, if 11-digit numbers are not needed to represent the Doppler-broadened cross section, the PENDF tape from BROADR will contain only the ENDF-standard 7-digit form, and our goal of maximum compatibility with the appearance of standard ENDF tapes will have been achieved. However, if the more precise 11-digit values are needed to get the specified accuracy, they will be used.

All cross sections are adjusted to fit into the normal ENDF 7 significant-figure format.

NJOY97 binary ENDF tapes use 64-bit representations for all numbers in order to support the scheme described above. The number of significant digits in the binary versions of each number matches the number in the ASCII versions, which helps to preserve consistency between NJOY runs made in binary and ASCII modes.


23 January 2013 T-2 Nuclear Information Service