Interbull CoP  Appendix VIII  Interbull validation test for genomic evaluations – GEBV test
Document based on Mäntysaari, E., Liu, Z and VanRaden P. 2011. Interbull Validation Test for Genomic Evaluations. Interbull Bulletin 41, p. 1721.
Definitions:
 EBV – Estimated Breeding Value (conventional national evaluations of the trait, free of genomic information, which are submitted to Interbull to be used in MACE evaluations)
 DGV  Direct Estimated Genomic Value (genomic evaluations based on SNP prediction equations)
 GEBV – Genomically Enhanced Estimated Breeding Value (evaluations that combine EBV and DGV)
 EDC –Effective Daughter Contribution
 GEDC – Genomically Enhanced Effective Daughter Contribution (EDC plus the genomic contribution)
 GMACE  Multiple Trait Across Country Genomic Evaluation
 PA – Parent Average
 D_PGM – Deregressed Predicted Genetic Merit
 DD – Daughter Deviation
 NGEC  National Genetic Evaluation Centre
λ = (4h^{2})/h^{2}
r^{2} – Reliability of the bull’s evaluation
R^{2} – Accuracy of the test model
Motivation
The inclusion of genomic information in international comparisons for dairy breeds requires that the national genomic breeding values (GEBVs) get validated by Interbull in a similar fashion that conventional EBVs are validated as a precondition to participate in the MACE evaluations.
The GEBV test will be applied to validate national models used to compute GEBVs that the national genetic evaluation centers (NGEC) publish and will eventually submit to Interbull for international genetic evaluations including genomic information. The GEBV test can be considered also a quality assurance assessment for national genomic evaluations. GEBVs from models that have been tested can be referred to as breeding value estimates with appropriate reliability, and be converted to other country scale breeding values using conversion equations derived by Interbull.
Rationale
The GEBV test evaluates:
 the unbiasedness of the genomic evaluations through the evaluation of
 the consistency of the genetic trend captured by GEBV, and
 the consistency of the variation of GEBVs and EBVs;
 the improvement in accuracy from the use of GEBV instead of EBV.
The test for bias is done by verifying the ability of a model only including data from 4 years ago to predict current performances. NGEC have to exclude the last 4 years of data and rerun the analyses with the reduced data, with the same model that are being tested. However, in some cases the bull generation available for validation has not been genotyped in everything and all. Thus, bulls exist that will get more than 20 daughters in the full data, but that have no GEBVs. This is called selective genotyping, and it leads into systematic bias in the validation bull group. In the test, this bias needs to be corrected by accounting for the selection between the mean national EBV (current, conventional) of the bulls genotyped and the overall mean national EBV including all potential candidates. This selection differential can be used to derive the expected regression coefficient, which would be equal to unity as if no selective genotyping took place.
Testing the improvement in accuracy is done by comparing the coefficient of determination (R^{2}) of the reduced genomic model and the equivalent reduced conventional model (from 4 years ago) regressed to current performances. The R^{2} from the model including genomic information must be higher than the model including only parent average information.
Test data sets
Data formats are described at GEBVtest Software.
Full data sets
The full data sets include all animals present in the most recent Interbull MACE evaluation. They are of two types, one containing national official genetic merit values (EBVs) and another containing either deregressed predicted genetic merits (D_PGMs) or daughter deviations (DDs).
National official genetic evaluation file (fileCxxxf)
The files sent by the NGEC as input for the most recent Interbull MACE evaluation and will be used to identify the candidate bulls, estimate selection intensity and check bulls birth year and type of proof.
Daughter deviation file (fileDxxxf)
The NGEC needs to prepare either DD or D_PGM for the same animals included in fileCxxxf. These values represent the currently estimated performance of the animals and will be used as the dependent variable in the validation procedure. EDC and reliability estimates should be exactly the same as in fileCxxxf.
Reduced data sets
The reduced data sets should be prepared by truncating the phenotypes used as input for both the conventional and the genomic evaluations. The NGEC must exclude phenotypic information from the past 4 years and rerun the current models of genetic/genomic evaluation for the traits of interest, keeping the animals without progeny information after truncation (test bulls) in the data in order to obtain genetic merit estimates based solely on parent averages (EBVr) or on parent averages plus genomic prediction equations (GEBVr).
Reduced conventional genetic evaluation file (fileCxxxr)
The NGEC should carry out a conventional genetic evaluation using truncated data (only phenotypes up to 4 years prior to the date of analysis) but including in the analysis all animals present in the current official evaluations (fileCxxxf).
Reduced genomic evaluation file (fileGxxxr)
Similarly, new genomic evaluations should be carried out using exactly the same model being validated (current) but excluding phenotypic information up to four years ago (truncated data, fileCxxxr). All bulls that did not have a progeny test 4 years ago and that currently have at least 20 daughterequivalents in the national genetic evaluation (test bulls) need to have a genomically enhanced EBV (GEBVr) estimated and included in the output.
If a significant number of foreign animals are included in the reference population and estimation of genomic prediction equations uses deregressed MACE values for these animals as input, the reduced genomic evaluation can be achieved in two ways:
 the Interbull Centre can make historical files available upon request (e.g. information used four years ago) containing past MACE results and the correspondent national EDCs, as well as heritability and genetic correlations used in the respective evaluations – these data can then be used to estimate 4year old deregressed values; OR
the genomic prediction equations for the truncated data (only bulls with EDCr > 0) are obtained using current deregressed MACE values. This constitutes an exception and should only be used when the standard procedure is not practical.
Table 1 presents a comparison between the several types of data and the notation used to identify variables from different files.
Table 1 – Comparative specification of the data files needed for the GEBV test.
Test Data 
Type of information 
File types and format^{a} 
Specific variables^{b} (equivalent field in the fileCxxxf) 

EDC 
Reliability 
EBV 

Full data sets 
Conventional Genetic data 
C010f, C115f, C015f, C016f, C017f, C018f, C019f, C020f 
EDC 
r^{2}_{EBV} 
EBV 
Daughter deviation data 
D010f, D115f, D015f, D016f, D017f, D018f, D019f, D020f 
EDC 
r^{2}_{EBV} 
D_PGM (or DD, if available) 

Reduced data sets 
Conventional Genetic data 
C010r, C115r, C015r, C016r, C017r, C018r, C019r, C020r 
EDCr 
r^{2}_{EBVr} 
EBVr 
Genomic data 
G010r, G115r, G015r, G016r, G017r, G018r, G019r, G020r 
GEDCr 
r^{2}_{GEBVr} 
GEBVr 
^{a}The GEBVtest software (gebvtest.py) uses a traitindependent format (File300). Users can either prepare data in the new format or use the program gtconvert.py to convert the current format into the File300 format.
^{b}All other variables should be the same as in the Cxxxf files.
Specific instructions for data preparation:
The domestic bulls (type of proof ≠ 21 or 22) that have EDC≥20 and EDCr = 0 are called test bulls. Interbull recommends that number of test bulls would be about 0.25 *(number of bulls used as reference population).
If the number of bulls the country includes in the genomic evaluation is too small, then the accuracy of the GEBVs calculated using the truncated data becomes significantly smaller than with the full data. In that case, the country can use n < 4 years as the time difference between full and reduced data sets.
If the number of test bulls is too small (ntb < 50), the country may chose to consider foreign bulls (type of proof = 21 or 22) that have EDC≥20 and EDCr = 0 also as test bulls.
 In both exceptions above, the Interbull Centre must be communicated in detail about the criteria adopted to define the test bulls.
 Appropriate time windows (birth years of bulls) may vary depending of the trait to be validated, the speed of their progeny test program and other factors. A shift of the time window with one year will give a different set of bulls that qualify for the test. The standard adopted for the GEBV test is to include four years of candidate/test bulls, which corresponds to an age cutoff of (YYYY8). For instance, if the Cxxxf is from 2012, Cxxxr and Gxxxr should include performance records up to 2008 and test bulls would be those born from 2004 and 2008. Countries may include more birth years, but the Interbull Centre must be communicated about the reason.
 GEBV is the genomically enhanced breeding value. Correspondingly, the GEDC is a genomically enhanced EDC that combines the EDC from national nongenomic evaluation with the gain from genomic evaluation. This means that GEDC should be larger than EDC and GEDCr should be larger than EDCr.
 Include all the bulls having GEBVr in the data without data edits based on EDC, EDCr, GEDC or GEDCr.
If GEDCr is not available, then GEDCr = λ * r^{2}_{GEBVr} / (1 r^{2}_{GEBVr})
The method of estimation of GEDCr (and/or r^{2}_{GEBVr}) has to be reported in the Interbull GENO form.
 The GEBVr prediction equations also have to be based on the truncated data. If the GEBVr combines information of DGV and EBV (i.e. PA), the EBV (PA) information has to be also from the truncated data.
 Bulls with EBV in the full data sets that have no progeny information four years ago (EDCr=0), should be included in the reduced data set.
 If the EBVs from evaluations published four years ago are available, the country can use these values for the reduced data sets. However, if the evaluation model, trait definitions, etc. have changed from the estimation of EBVs in the reduced data sets and the estimation of EBVs in the full data sets, the GEBVr can be expected to have lower accuracy than GEBV. In this case, the country should report the expected correlation between the old (reduced) and the new (full) data EBVs (see Interbull Testing Method 3).
 In order to remove any change in scale of proof expression, EBVr and GEBVr should be rescaled to the same scale as EBVs, using bulls already proven in the reduced data sets.
Test description
Testing for bias
The bias in the national genomic evaluations will be tested using a regression model:
φ_{i}= b_{0} + b_{1}*GEBVr_{i} + e_{i} [1] ,
where φ_{i} is the D_PGM (or DD, if available) from the bulls that have EDC≥20 and EDCr =0. The EDCs from the full data set can be used as weights in the model if DDs are supplied, otherwise the accuracy of the D_PGMs (u_{i}=EDC/(EDC+λ)) will be used as weights.
This model is used to estimate b_{1} to compare with the expectation of b_{1} (H_{0}: b_{1} = E(b_{1})) and therefore test the bias on GEBVr. Item 4.3 describes how the expectation of b_{1} can be derived considering the impact of selective genotyping among test bulls.
The statistical significance will be tested using a ttest against H_{0} (C.I. = 0.95).
For larger populations the estimated standard error might become very small and then the ttest may become too restrictive. In those cases, a “biological significance” will be adopted to test H_{0} [P((E(b_{1})0.1) ≤ b_{1} ≤ (P(E(b_{1})+0.1))].
The countrytraitbreeds will pass the test, if b_{1} value is greater than the lower endpoint of the 95% confidence interval or its biological equivalent.
The accuracy of GEBVr will be estimated from the R^{2} of the model (accuracy of the model after selection for genotyping). This validation accuracy R^{2}_{validation} = R^{2}/ ū, where ū is the average weight of all the test bulls. It will be expected that the mean of published bull r^{2}_{GEBVr} is in agreement with R^{2}_{validation}.
Testing the improvement from conventional evaluation
The improvement of the added daughter information to the parental information will be estimated by comparing the R^{2} from model [1] with the R^{2} from model [2]:
φ_{i} = b_{0} + b_{1}*EBVr_{i}+ e_{i} [2] ,
where φ_{i} and the corresponding weight u_{i} are the same as in model [1]. The R^{2} from model [1] must be higher than the R^{2} from model [2].
Estimating the effect of selective genotyping on E(b1)
The expected value of b_{1 }is 1.0 only if the genotyped test bulls are a representative sample of the bulls in the corresponding age classes. The selection based on EBVs before genotyping will reduce the value of b_{1} and also the value of R^{2} for model [1]. The level of selective genotyping can be approximated from the difference between the mean EBV of the genotyped test bulls, µ_{EBVg}, and the mean EBV of all potential test bulls (i.e. bulls with EDC≥20 and EDC_{r}=0, genotyped or not), µ_{EBVall}, and the standard deviation of EBV of all potential test bulls (σ_{EBVall}).
i = (µ_{EBVg}  µ_{EBVall})/ σ_{EBVall }[3].
Using tables from quantitative genetics books, (e.g. page 379 from Falconer, D. S. & Mackay, T. F. C. Introduction to Quantitative Genetics, Longman, 4^{th} ed. 1996) the proportion of selected (genotyped) individuals (p) can be obtained for the selection differential (i) and the corresponding truncation point x that divides the standard normal density into selected proportion p and nonselected (1p).
Having the proportion of the selected individuals, the expected value of the b_{1} (E(b_{1})) and the effect of the selection on R^{2} of the test model can be estimated by approximation of the effect of selection on the variance of the selected trait and on the covariance between the independent (GEBVr) and the dependent (φ) variables. Having (i) as the mean deviation of the selected individuals from the total population in terms of standard deviation from the total population, and (x) as a selection truncation point from the overall mean:
k = i(i  x) [4]
v_{1} = 1 – k [5]
Calculating R^{2} before selection (R_{b}^{2}), which is the R^{2} for model [1], from R^{2} after selection
(R_{a}^{2}): R_{b}^{2} = R_{a}^{2} / (v_{1} + kR_{a}^{2}) [6]
v_{2} = 1 – kR_{b}^{2} [7]
E(b_{1}) = v_{1} / v_{2} [8]
Example: Assuming that: µ_{EBVg} = 16.00; µ_{EBVall} = 11.76; σ_{EBVall} = 10.00; R_{a}^{2} = 0.555. The selection differential (i) for the genotyped bulls equal to 0.424 standard deviations of EBVs (equation [3]), the proportion of genotyped bulls (p) would be 75 percent and the mean deviation of the truncation point from the overall mean (x) would be equals to 0.674 (from reference table). Applying equations [4], [5] and [6] it is possible to calculate R_{b}^{2} = 0.70. Using equations [7] and [8] E(b_{1}) = 0.793.
Table 2 – Examples of expected regression coefficients (E(b_{1})) as functions of the selection intensity (i) and the coefficient of determination before selection (R_{b}^{2}).
i 
p 
x 
E(b_{1}) 

R_{b}^{2} = 0.50 
R_{b}^{2} = 0.55 
R_{b}^{2} = 0.60 
R_{b}^{2} = 0.65 
R_{b}^{2} = 0.70 

0.644 
60 
0.253 
0.594 
0.619 
0.646 
0.676 
0.709 
0.570 
65 
0.385 
0.626 
0.650 
0.677 
0.705 
0.736 
0.497 
70 
0.524 
0.660 
0.683 
0.708 
0.735 
0.764 
0.424 
75 
0.674 
0.697 
0.718 
0.742 
0.766 
0.793 
0.350 
80 
0.842 
0.736 
0.756 
0.777 
0.800 
0.823 
0.274 
85 
1.036 
0.781 
0.799 
0.817 
0.836 
0.856 
0.195 
90 
1.282 
0.832 
0.846 
0.861 
0.876 
0.892 
0.109 
95 
1.645 
0.894 
0.904 
0.914 
0.924 
0.934 
0.000 
100 

1.000 
1.000 
1.000 
1.000 
1.000 
Interbull form GENO
The methodology for estimation of GEBV and its’ accuracy (r^{2}_{GEBV}) have to be reported by the NGEC in Interbull form GENO.