021: Strategic & Quantitative Recommendations on Bioanalytical Cross-Validation Experiment Design from a Clinical Pharmacology Perspective

Statement of Purpose, Innovation or Hypothesis: With the development of drugs from first-in-human to late stage, it is inevitable that changes occur in the bioanalytical method or laboratory (lab). To evaluate whether the concentrations reported with a new method are consistent with the original method, a cross-validation experiment is performed. Although there is consensus across the industry and health authority that cross-validation is required when different labs/methods are used, there is limited guidance on experiment design and result evaluation. For experiment design, the most recent ICH M10 (2022)¹ recommends testing study samples, if available, that cover the concentration range of the study (with n≥30), in addition to the quality control samples. To evaluate differences, the guidance recommends Bland-Altman plots or Deming regression. However, this guidance does not provide details on sample selection, whether sample size is influenced by the magnitude of the difference and whether replicates are needed. The objective of this work is to evaluate the effect of sample size and number of replicates of the cross-validation experiment on the power to detect differences between two labs/methods.

Description of Methods & Materials: We simulated data from cross validation studies with sample sizes of 30, 60 or 90 samples and 1, 3 or 6 replicates per sample. True concentrations were assumed to follow a log-normal distribution, bounded between 0.1 and 100 ng/mL, reflecting a typical range (1000-fold) encountered in clinical study samples. The values from two assays were assumed to be related to the true concentrations through a log-linear relationship with assay variability ranging from 5 - 20%. The average difference between the two assays was assumed to be proportional (i.e. additive on the log-scale), ranging from no difference to 35%. To each simulated data set, a Deming regression was fitted and used to estimate the relationship between assays. For each combination of sample size, number of replicates, mean difference and magnitude of assay variability, we calculated the probability of detecting a significant difference between the assays, the average percent bias, and mean squared error.

Data & Results: As expected, the higher the sample size and replicate numbers, the higher the power in detecting the difference. For a 15% assay variability, a cross-validation study with 30 samples and one replicate provides 90% probability to detect a mean difference of 25%. To achieve the same power to detect a 12% difference, more samples or replicates are necessary. For example, a study with either 90 samples with three replicates or 30 samples with six replicates provides 90% power to detect a 12% difference. In all evaluated scenarios, the bias in the Deming regression parameters was less than 5%, and precision increased with increasing sample size or replicates. For sample selection, we recommend selecting samples mimicking the concentration distribution of clinical study samples.

Interpretation, Conclusion or Significance: Using simulation approach, this work provides guidance on bioanalytical cross-validation experiment design. It helps bioanalytical scientists to establish robust cross-validation methodologies and protocols. It also enables clinical pharmacologists to confidently assess PK data and draw meaningful conclusions across different datasets. A robust cross-validation experiment will ultimately advance drug development and promote evidence-based healthcare.

Disclosures:

Citations/References: 1. ICH M10 2022: https://database.ich.org/sites/default/files/M10_Guideline_Step4_2022_0524.pdf

Additional Information/Authors: Acknowledgements: This study was funded by Janssen Research & Development, LLC. We would like to thank bioanalytical colleagues for helpful discussions.