Validation with Total Error method

Validation

The different regulations concerning to the good practices (GLP, GMP, GCP, and others) as well as the normative or regulatory documents (ISO, ICH, EMEA, and FDA) suggest that all procedures have to comply with acceptance criteria. This request imposes, therefore, that these proceduresmust be validated. There are several documents defining the validation criteria to be tested, but they do not propose experimental approaches and limit themselves, most often, to the general concepts. It is why the members o the SFSTP have contributed to the elaboration of consensus validation guides to help the pharmaceutical industry to validate their analytical procedures.

This approach allows to considerably minimize the risk to accept a procedure that would not be sufficiently accurate or, to the opposite, to reject a procedure that would be capable.

The accuracy profile method, uses ISO terminology and offers an experimental strategy for the validation of analytical procedures, independent of the industrial sector, to optimally use experiments performed, to extract a maximum of information from the results and to minimize in routine the risks to re-analyze samples.

Objectives of an analytical procedure

To describe the objectives of an analytical method, it is necessary to revisit its nature. Does its objective lie in demonstrating a linear response variation with respect to concentration? Is bias and fidelity below a certain percentage (x%), or does it aim to precisely quantify an unknown quantity? These questions appear pertinent. The objective of an analytical procedure is to be able to quantify an unknown quantity as accurately as possible, which the laboratory must determine. In other words, what the analyst seeks is for the difference between the measured value x and the true value $\mu_T$ (which will never be known) to be as small as possible, or at least below an acceptable limit. This requirement can be expressed by the following relationship:

$$ -\lambda < x - \mu_T < \lambda \Leftrightarrow | x - \mu_T | < \lambda $$

Where $\lambda$ represents the acceptance limit which will depend on the analyst's needs or the objectives of the analytical procedure. In fact, the acceptance limit can vary depending on the intended use of the analytical method (for example, 1% to 2% for pharmaceutical compounds, 5% for determining active substances, or 15% for bioanalyses).

Important concepts are thus introduced, not only acceptance limits for the performance of an analytical method but also the responsibility that the analyst has to take in the decision of accepting the performance of the method with respect to its intended use.

On the other hand, every analytical method can be characterized by two types of error :

• systematic error or bias, $\mu_M$
• random error assessed using a standard deviation $\sigma_M$

An estimate of bias and variance for the method can be obtained through experimentation during validation. The reliability of these estimates depends on the suitability of measurements performed on validation standards, the experimental design, and the number of repetitions of measurements during the validation phase. Based on these estimates of bias and variance, it is possible to determine if the method is good by comparing the method’s performance to an acceptance limit $\lambda$, defined for a given application (for example, a biopharmaceutical analysis).

The example below shows the implementation of the acceptance criteria for different configuration of bias and precision.

Aiming to develop a procedure without bias and without error has a considerable cost. This target is unrealistic for an analyst who has generally only little time to systematically and meticulously optimize all the analytical parameters in the development phase even if the use of experimental design is recommended and well described in the literature.

To overcome this dilemma, the analyst will have to take minimal risks (or all at least compatible with the analytical objectives). To control this risk, the reasoning can be reversed and one can fix as starting assumption that only an acceptable maximum proportion of future measures will be outside the acceptance limits, e.g. 5% of the measurements or 20% of the measurements to the maximum outside the acceptance limits. This proportion represents, therefore, the maximum risk that the analyst is ready to take.

A procedure can be considered validated if it is very probable that the difference between each measurement $x$ of a sample and its true value $\mu_T$ falls within the acceptance limits predefined by the analyst. This concept can be described by the following expression:

$$ P(|x − \mu_T | < \lambda) \geq \beta $$

with $\beta$ representing the proportion of measurements within acceptance limits and λ representing the acceptance limits set a priori by the analyst based on the method’s objectives. The proportion of expected values outside these acceptance limits evaluates the risk associated with the analytical procedure.

Objective of the validation

The objective of validation is to give to the laboratories as well as to regulatory bodies guarantees that every single measure that will be later performed in routine analysis will be close enough to the unknown true value of the sample to be analyzed or at least that the difference will be lower than an acceptable limit taking into account the intended use of the method.

Therefore, the objectives of validation are to minimize risk for the consumer, as well as the risk for the producer. Consequently, the objective of validation cannot simply be limited to obtaining estimates of bias and variance, but must focus on evaluating risks even if these estimators are necessary to assess risk. Concerning this objective, two key notions mentioned above should be explained: sufficiently close means, for example, that the measurement performed routinely will be within $x\%$ of the true value (unknown), which means that the measurement will be sufficiently close to the true value.

In that respect, trueness, precision, linearity, ... are no more statistics allowing to quantify these guarantees. In fact, one expects from an analytical procedure to be able to quantify and not to be precise, even if the precision itself unquestionably increases the likelihood to be successful. In this perspective, it is necessary to differentiate the statistics which allow to make a decision (e.g. the procedure can be considered as valid or not on the basis of its aptitude to quantify) and those which help to make a diagnosis (e.g. statistical tests evaluating the adequacy of the regression model or the homogeneity of the variances).

In fact, adapted decision tools are really needed to give guarantees that any future measurements will be reasonably inside the acceptance limits. If the guarantees offered by the decision rule are not satisfactory, then the diagnosis tools will help the analyst to identify the possible causes of the problem; but only if the guarantees are not satisfied.

Decision rules

The examination of the current situation with respect to the decision rules used in the validation phase shows that the most of them are based on the use of the null hypothesis as follows.

$$ H_0 : \text{bias} = 0 \leftrightarrow H_0 : \text{relative bias} = 0 \% \leftrightarrow H_0 : \text{Recouvrement} = 100 \% $$

with $\text{bias} = x-\mu_T$, $\text{relative bias} = \frac{x - \mu_T }{\mu_T} \cdot 100$ and $\text{recovery} = \frac{x}{\mu_T} \cdot 100$

On this basis, a procedure is wrongly declared adequate when the 95% confidence interval of the average bias includes the value of 0 (0% and 100% in the case of the relative bias and recovery, respectively). However, this test is inadequate in the validation context of analytical procedures because the decision is based on the computation of the rejection criterion of the Student's t-test. It does not take into account the purpose of the analytical method for determining the acceptance limit and secondly, we note that two conditions contribute to incorrectly validate a method: cases where the standard deviation is large (strong variability where bias will be non significant) and a small number of measurements of repetitions.

Working with limits $\pm \lambda$ allows us to avoid rejecting a method when we have low standard deviation and low bias, but also to accept a method with high standard deviation and significant bias. This demonstrates that the use of classical comparison tests is insufficient for validating analytical procedures. The acceptance criterion represents a compromise between bias and variability such that a given proportion of measurements, chosen a priori, falls within the $ \pm \lambda$ interval.

This latter decision rule appears clearly more sensible than the previous one based on the null hypothesis since all procedures having a small dispersion of the measurements are accepted, while the procedures having a large variance are rejected. In addition, if a procedure has a bias, it should have a small variance to be accepted. Symmetrically, a procedure with a high variance should have a small bias to be accepted.

Note that the use of the null hypothesis, is still widely used in many cases, such as the test of null intercept, of equality of slopes, lack of fit,etc. With all these statistical tests, the less the procedure is precise, the more chance to pass successfully these tests. This situation is certainly not the one expected by the analyst using the statistics to evaluate the capability of the method under investigation.

In this context, a simple and visual decision rule involves using an accuracy profile with acceptance limits relative to $\pm \lambda$.

Accuracy profile computed with SOSstat VPAQ

The accuracy profile, constructed from confidence intervals on expected measurements (or statistical dispersion interval), allows us to determine whether an analytical procedure is capable of delivering results within acceptance limits. The accuracy profile describes the range in which the procedure can quantify with known precision and a predefined risk set by the analyst. For example, for a presumed risk of 5%, the analyst can guarantee that 95 out of every 100 future measurements produced by their procedure will be included within the acceptance limits defined according to regulatory requirements (e.g., 1% or 2% in volume, 5% on pharmaceutical specialties, 15% in bioanalysis and environmental analysis, etc.). The profile is built from estimations of the bias and fidelity of the analytical procedure, as well as the confidence interval, at each concentration level during the validation phase. This confidence interval is also known as $\beta$ -expectation tolerance interval. It defines the range within which the expected proportion of future results will fall. This tolerance interval obeys the following property:

$$ E\{ P[|x_i - \mu_T| < \lambda ]/ \mu_M , \sigma_M \} \geq \beta $$

with E meaning expected value of the result.

Calculating the confidence interval or $\beta$ -expectation tolerance interval interval requires estimating the bias and intermediate fidelity standard deviation, respectively denoted as $\mu_M$ et $\sigma_M$.

The accuracy profile can simply be obtained by connecting the lower limits of tolerance or the upper limits of tolerance. If the tolerance interval is larger than the acceptance limits, new limits of quantification and a new dosage interval have to be defined : namely the upper limits of quantification (ULQ) , and the lower limit of quantification (LLQ). The latter is in perfect agreement with the definition of this criterion, i.e. the smallest quantity of the substance to analyze that can be measured with accuracy and a precision defined.

As can be seen, the use of the accuracy profile as single decision tool allows not only to reconcile the objectives of the procedure with those of the validation but also to visually grasp the capacity of the analytical procedure to fit its purpose.

Definitions of ICH

Analytical procedure

The analytical procedure refers to the way of performing the analysis. It should describe in detail the steps necessary to perform each analytical test. This may include but is not limited to: the sample, the reference standard and the reagents preparations, use of the apparatus, generation of the calibration curve, use of the formulae for the calculation, etc.

Specificity

Specificity is the ability to assess unequivocally the analyte in the presence of components which may be expected to be present. Typically these might include impurities, degradants, matrix, etc. Lack of specificity of an individual analytical procedure may be compensated by other supporting analytical procedure(s).

Accuracy

The accuracy of an analytical procedure expresses the closeness of agreement between the value which is accepted either as a conventional true value or an accepted reference value and the value found. This is sometimes termed trueness.

Precision

The precision of an analytical procedure expresses the closeness of agreement (degree of scatter) between a series of measurements obtained from multiple sampling of the same homogeneous sample under the prescribed conditions. Precision may be considered at three levels: repeatability, intermediate precision and reproducibility.

• Repeatability : Repeatability expresses the precision under the same operating conditions over a short interval of time. Repeatability is also termed intra-assay precision .
• Intermediate precision :Intermediate precision expresses within-laboratories variations: different days, different analysts, different equipment, etc.
• Reproducibility : Reproducibility expresses the precision between laboratories (collaborative studies, usually applied to standardization of methodology).

Detection limit

The detection limit of an individual analytical procedure is the lowest amount of analyte in a sample which can be detected but not necessarily quantitated as an exact value.

Quantitation limit

The quantitation limit of an individual analytical procedure is the lowest amount of analyte in a sample which can be quantitatively determined with suitable precision and accuracy. The quantitation limit is a parameter of quantitative assays for low levels of compounds in sample matrices, and is used particularly for the determination of impurities and/or degradation products.

Linearity

The linearity of an analytical procedure is its ability (within a given range) to obtain test results which are directly proportional to the concentration (amount) of analyte in the sample.

Range

The range of an analytical procedure is the interval between the upper and lower concentration (amounts) of analyte in the sample (including these concentrations) for which it has been demonstrated that the analytical procedure has a suitable level of precision, accuracy and linearity.

Robustess

The robustness of an analytical procedure is a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters and provides an indication of its reliability during normal usage.