6.5. VADER: Trending Analysis for Code/Data Validation

S. Hart, J. Clarity

VADER (Validation Analysis Data Evaluation Resource) is a new module in SCALE 6.3 derived from the legacy USLSTATS program. VADER is a tool that allows the determination of bias and bias uncertainty for criticality safety computational methods. The older USLSTATS program, written in Java, existed outside of SCALE and provided tools to calculate only the confidence band with administrative margin (sometimes called USL-1) and the single-sided uniform width closed interval (USL-2). For normality testing it offered only a crude chi-squared normality test that had no user-configurable options and presented a simple pass/no-pass functionality.

VADER has been completely written in C++ using modern coding practices and standards. It has been moved to be part of the SCALE code package and is run through the normal SCALE executables. To support the VADER methods, large updates were made to the statistical capabilities of SCALE. Interfaces to known statistical distributions were added and many of the statistical calculations required for VADER were made available to all SCALE sequences. In addition to being able to process legacy USLSTATS inputs, VADER adds support for the single-sided tolerance band method (from NUREG/CR-6698) as well as two non-trending methods: the “historical” nonparametric and parametric methods, also as described in NUREG/CR-6698. VADER also includes several new tests for normality in addition to adding user control options on the original chi-squared normality test. A new t-test for the statistical significance of the trend is included.

VADER has also been integrated into the Fulcrum GUI used by SCALE. This allows VADER to benefit from the auto-complete, templating, and input validation features offered by Fulcrum. In addition, Fulcrum can plot the results given by VADER, and the plot files have been expanded to include the new methods available in VADER.

6.5.1. Theory

The methods and theory for VADER are largely taken from NUREG/CR-6698. That document contains equations and descriptions of the methods used in VADER. It also includes working examples, some of which data was used to verify the results and create example problems for VADER.

6.5.2. Input

The VADER input is based on the SON input syntax common in other modernized SCALE codes such as ORIGEN and ORIGAMI. The SON input format consists of key/value pairs where the value can be either a single item, an array of items, or other SON blocks.

The VADER input generally just has one level of depth. There are no SON blocks inside of SON blocks. All input is mostly arrays at the top level of the tree. A quick look at all the top level items is given in Table 6.5.1. Each top level input will then be described in detail in the next section.

Table 6.5.1 Summary of top level options for VADER
Keyword	Required	Description
data	Yes	Data entered in triplets in the form `X, Y, std. dev`.
title	No	Title to give the case.
parameters	No	Global parameters that apply to each test or trend.
trend_values	No	User desired trend values to run the tests on.
methods	Yes	Methods that the user wishes to apply to the data.
tests	No	Statistical checks that the user wishes to apply to the data.

6.5.3. Detailed Input

6.5.3.1. `data`

The data input array is a flat array consisting of data triplets. There is no punctuation between the triplets. The data is entered in the order $X_{1} Y_{1} σ_{1} X_{2} Y_{2} σ_{2} . . . X_{N} Y_{N} σ_{N}$ .

Where

X	The trended value, usually something like enrichment or moderator density.
Y	The y value, usually something like C/E or eigenvalue.
$σ$	The standard deviation of Y.

VADER will emit an error if the number of entries in this array is not divisible by three.

Example 6.5.1 Snippet of data input

data=[2.4 0.9976 0.00122 2.6 0.9978 0.00022
      ...
      1.9 1.0011 0.00213 1.99 1.013 0.00012]

6.5.3.2. `title`

The title entry allows the user to specify a title for the case. The title is cosmetic and doesn’t affect any of the results. The title is echoed to the output in several places and is included on some of the graphs generated by VADER.

Example 6.5.2 title

title="My Very First VADER Case"

6.5.3.3. `parameters`

The parameters section can exist outside of any test or method. If a parameter is listed in this global section, then it will apply to any method or test that has that parameter available instead of the default value. Values in this global parameters section can be overridden in the specific method or test block for which the user wishes to override the global parameter. For a list of all parameters, their default values, and which tests and methods they are applied to, see Sect. 6.5.6.

Example 6.5.3 parameters array

parameters {extrapolate=yes admin_margin=0.03}

6.5.3.4. `trend_values`

This array allows the user to insert values at which they wish the trend Y (typically the upper subcritical limit) values to be calculated. Extrapolation outside of the data range is only available if the method supports it and extrapolation is enabled.

Example 6.5.4 trend_values array

trend_values=[-0.1 0.2 2.4 10.0]

6.5.3.5. `methods`

This is where the user specifies the methods to enable and, optionally, gives method-specific parameters. Each method is followed by an array of the parameters the user wishes to set. Any omitted parameters are initialized with the default value (see Table 6.5.2). If the user wishes to enable a method with all default values, then an empty array can be given.

Example 6.5.5 methods block with some default methods and some specific parameters

methods {
  USL1 {} % uses default parameters
  CR6698 {weighted=yes admin_margin=0.03 chi_p=0.97}
  Parametric {confidence=0.9}
}

6.5.3.6. `tests`

This block is similar to the methods block, but it allows the user to specify which tests they wish to enable. These tests are run on the data and generally either pass or fail and give a few statistical values as their result. Most of the tests implemented for VADER for now are normality tests.

Example 6.5.6 tests block

tests {
  chisq {} % default parameters
  t-test {confidence=0.98}
}

6.5.4. Available Methods

The sections below indicate the input name to be used to activate the available method. Details on the methods can currently be found in NUREG/CR-6698.

6.5.4.1. `USL1` - Confidence Band with Administrative Margin

6.5.4.2. `USL2` - Single-Sided Uniform Width Closed Interval Approach

6.5.4.3. `CR6698` - Single-Sided Lower Tolerance Band

6.5.4.4. `NonParametric` - Historical Non-Parametric Method

6.5.4.5. `Parametric` - Historical Parametric Method

6.5.5. Available Tests

The sections below indicate the input name to be used to activate the available test. Details on the tests can currently be found in NUREG/CR-6698.

6.5.5.1. `chisq` - $χ^{2}$ Normality Test

6.5.5.2. `t-test` - Trend Significance t-test

6.5.5.3. `anderson_darling` - Anderson-Darling Normality Test

6.5.6. Available Parameters

Table 6.5.2 shows all of the available parameters, their default value, and the methods and tests to which they apply. Any parameter can be put into the global parameters block. Only parameters that apply to a specific method or test can be put into that method or tests parameters block. Following the table is a more detailed description of each parameter.

Table 6.5.2 Available Parameters for VADER
Keyword	Default Value	Available In
tsunami	Off	USL1 USL2
extrapolate	Off	USL1
fit_confidence	0.95	USL1 USL2
admin_margin	0.05	USL1 USL2 CR6698 NonParametric Parametric
xrange	All X Values	USL1 USL2 CR6698 NonParametric Parametric
proportion_confidence	0.95	USL2
proportion	0.95	USL2
f_p	0.95	CR6698
normal_p	0.95	CR6698
chi_p	0.95	CR6698
weighted	No	CR6698 NonParametric Parametric
confidence	0.95	NonParametric Parametric
proportion_of_pop	0.95	NonParametric Parametric
use_npm	No	NonParametric

6.5.6.1. `tsunami`

This enables a TSUNAMI-mode calculation for VADER. It requires that all X values be between -1.0 and 1.0.

6.5.6.2. `extrapolate`

This allows extrapolation past the data values entered by the user. This is generally only applicable when the user has input values in the trend_values block.

6.5.6.3. `fit_confidence`

Confidence in the statistical result. Sometimes referred to as $1 - γ$ . In USL1 this is used to calculated the t-statistic quantile.

6.5.6.4. `admin_margin`

Administrative margin subtracted from the result. This can be used to add an extra layer of confidence that the value reported is below criticality.

6.5.6.5. `xrange`

Allows the user to tailor X values for each method or test or change the values for all tests without having to change the data block. Data is entered as an array of two values. By default all of the X values are used.

Example 6.5.7 xrange

xrange=[-0.5 2.5]

6.5.6.6. `proportion_confidence`

In USL2 this controls the confidence on the proportion that is in the confidence band.

6.5.6.7. `proportion`

In USL2 this controls the proportion of the data points that will be inside the confidence band.

6.5.6.8. `f_p`

The percentile used for the F-distribution in the single-sided lower tolerance band method.

6.5.6.9. `normal_p`

The percentile used for the normal distribution in the single-sided lower tolerance band method.

6.5.6.10. `chi_p`

The percentile used for the $χ^{2}$ distribution in the single-sided lower tolerance band method.

6.5.6.11. `weighted`

Whether to use the weighted version of the mean and standard deviation equations when calculating the statistics for this method.

6.5.6.12. `confidence`

Statistical confidence that the proportion calculated to be above the cutoff is correct.

6.5.6.13. `proportion_of_pop`

Proportion of the population that will be above the cutoff value in the non-parametric method.

6.5.6.14. `use_npm`

Use (add) non-parametric margin to the non-parametric method.

6.5. VADER: Trending Analysis for Code/Data Validation

6.5.1. Theory

6.5.2. Input

6.5.3. Detailed Input

6.5.3.1. data

6.5.3.2. title

6.5.3.3. parameters

6.5.3.4. trend_values

6.5.3.5. methods

6.5.3.6. tests

6.5.4. Available Methods

6.5.4.1. USL1 - Confidence Band with Administrative Margin

6.5.4.2. USL2 - Single-Sided Uniform Width Closed Interval Approach

6.5.4.3. CR6698 - Single-Sided Lower Tolerance Band

6.5.4.4. NonParametric - Historical Non-Parametric Method

6.5.4.5. Parametric - Historical Parametric Method

6.5.5. Available Tests

6.5.5.1. chisq - χ2 Normality Test

6.5.5.2. t-test - Trend Significance t-test

6.5.5.3. anderson_darling - Anderson-Darling Normality Test

6.5.6. Available Parameters

6.5.6.1. tsunami

6.5.6.2. extrapolate

6.5.6.3. fit_confidence

6.5.6.4. admin_margin

6.5.6.5. xrange

6.5.6.6. proportion_confidence

6.5.6.7. proportion

6.5.6.8. f_p

6.5.6.9. normal_p

6.5.6.10. chi_p

6.5.6.11. weighted

6.5.6.12. confidence

6.5.6.13. proportion_of_pop

6.5.6.14. use_npm