Appendix X - Interbull validation test for genomic evaluations – GEBV test

Document based on Mäntysaari, E., Liu, Z and VanRaden P. 2011. Interbull Validation Test for Genomic Evaluations. Interbull Bulletin 41, p. 17-21.

Motivation

The inclusion of genomic information in international comparisons for dairy breeds requires that the national genomic breeding values (GEBVs) get validated by Interbull in a similar fashion that conventional EBVs are validated as a pre-condition to participate in the MACE evaluations.

The GEBV test will be applied to validate national models used to compute GEBVs that the national genetic evaluation centers (NGEC) publish and will eventually submit to Interbull for international genetic evaluations including genomic information. The GEBV test can be considered also a quality assurance assessment for national genomic evaluations. GEBVs from models that have been tested can be referred to as breeding value estimates with appropriate reliability, and be converted to other country scale breeding values using conversion equations derived by Interbull.

Rationale

The GEBV test evaluates:

• the unbiasedness of the genomic evaluations through the evaluation of
  • the consistency of the genetic trend captured by GEBV, and
  • the consistency of the variation of GEBVs and EBVs;
• the improvement in accuracy from the use of GEBV instead of EBV.

The test for bias is done by verifying the ability of a model only including data from 4 years ago to predict current performances. NGEC have to exclude the last 4 years of data and re-run the analyses with the reduced data, with the same model that are being tested. However, in some cases the bull generation available for validation has not been genotyped in everything and all. Thus, bulls exist that will get more than 20 daughters in the full data, but that have no GEBVs. This is called selective genotyping, and it leads into systematic bias in the validation bull group. In the test, this bias needs to be corrected by accounting for the selection between the mean national EBV (current, conventional) of the bulls genotyped and the overall mean national EBV including all potential candidates. This selection differential can be used to derive the expected regression coefficient, which would be equal to unity as if no selective genotyping took place.
Testing the improvement in accuracy is done by comparing the coefficient of determination (R²) of the reduced genomic model and the equivalent reduced conventional model (from 4 years ago) regressed to current performances. The R² from the model including genomic information must be higher than the model including only parent average information.

Test data sets

For each trait group, five data sets will be necessary for the GEBV test: two full data sets, two reduced data sets and one additional file with genotyping information on test bulls. In order to facilitate data preparation and reading, Interbull file formats 01X will be used.

Full data sets

The full data sets include all animals present in the latest international evaluation before validation data is prepared (CURRENT), without editions. They are of two types, one containing national official genetic values (EBVs) and another containing either de-regressed predicted genetic merits (D_PGMs) or daughter deviations (DDs).

National official genetic evaluation file (fileCxxxf)

The files sent by the NGEC as input for the latest Interbull routine run (formats 010, 115, 015, 016, 017, 018, 019, 020) will be used by the Interbull Centre and the NGEC do not have to provide these files again. 3.1.2 Daughter deviation file (D01Xf) The NGEC needs to prepare either DD or D_PGM for the same animals included in 3.1.1. These values represent the currently estimated performance of the animals and will be used as the dependent variable in the validation procedure. EDC and reliability estimates should be exactly the same as the 01X file in 3.1.1. 3.2 Reduced data sets The reduced data sets should be prepared by truncating the phenotypes used as input for both the conventional and the genomic evaluations. The NGEC must exclude phenotypic information from the past 4 years and re-run the current models of genetic/genomic evaluation for the traits of interest, keeping the animals without progeny information after truncation (test bulls) in the data in order to obtain genetic merit estimates based solely on parent averages (EBVr) or on parent averages plus genomic prediction equations (GEBVr). 3.2.1. Reduced conventional genetic evaluation file (C01Xr) The NGEC should carry out a conventional genetic evaluation using truncated data (only phenotypes up to 4 years prior to the date of analysis) but including in the analysis all animals present in the current official evaluations (C01Xf). All animals in the C01Xf must be included in the C01Xr file sent to Interbull. 3.2.2. Reduced genomic evaluation file (G01Xr) Similarly, new genomic evaluations should be carried out using exactly the same model being validated (current) but excluding phenotypic information up to four years ago (truncated data C01Xr). (All bulls that did not have a progeny test 4 years ago and that currently have at least 20 daughter-equivalents in the national genetic evaluation (test bulls) need to have a genomically enhanced EBV (GEBVr) estimated and included in the output. All animals included in the C01Xf must be included in the G01Xr file sent to Interbull. If a significant number of foreign animals are included in the reference population and estimation of genomic prediction equations uses de-regressed MACE values for these animals as input, the reduced genomic evaluation can be achieved in two ways:

• the Interbull Centre can make historical files available upon request (e.g. information used four years ago) containing past MACE results and the correspondent national EDCs, as well as heritability and genetic correlations used in the respective evaluations – these data can then be used to estimate 4-year old de-regressed values; OR b. the genomic prediction equations for the truncated data (only bulls with EDCr > 0) are obtained using current de-regressed MACE values. This constitutes an exception and should only be used when other option is not practical. Table 1 presents a comparison between the several types of data and the notation used to identify variables from different files. Table 1 – Comparative specification of the data files needed for the GEBV test.

• ^aAll other variables should be the same as in the 01X files. 3.3 Genotyping information on test bulls (file fomat 732) In order to estimate the expectation of the regression coefficient of GEBVr on D_PGM (E[b1]), it is necessary to know which test bulls (EDC≥20 and EDCr = 0) have been genotyped. It consists of a simple list of test bulls with a flag indicating if the bull has been genotyped or not. The file format is given in Appendix II. 3.4 Specific instructions for data preparation:

• The domestic bulls (type of proof ≠ 21 or 22) that have EDC≥20 and EDCr = 0 are called test bulls. Interbull recommends that number of test bulls would be about 0.25 *(number of bulls used as reference population).

  1. If the number of bulls the country includes in the genomic evaluation is too small, then the accuracy of the GEBVs calculated using the truncated data becomes significantly smaller than with the full data. In that case, the country can use n < 4 years as the time difference between full and reduced data sets.

  2. ii. If the number of test bulls is too small (ntb < 50), the country may chose to consider foreign bulls (type of proof = 21 or 22) that have EDC≥20 and EDCr = 0 also as test bulls.

  3. iii. In both exceptions above, the Interbull Centre must be communicated in detail about the criteria adopted to define the test bulls.

  b. GEBV is the genomically enhanced breeding value. Correspondingly, the GEDC is a genomically enhanced EDC that combines the EDC from national non-genomic evaluation with the gain from genomic evaluation. This means that GEDC should be larger than EDC and GEDCr should be larger than EDCr. c. Include all the bulls having GEBVr in the data without data edits based on EDC, EDCr, GEDC or GEDCr. d. If GEDCr is not available, then GEDCr = λ * r²_GEBVr / (1- r²_GEBVr).

e. The method of estimation of GEDCr (and/or r²_GEBVr) has to be reported in the Interbull GENO form. f. The GEBVr prediction equations also have to be based on the truncated data. If the GEBVr combines information of DGV and EBV (i.e. PA), the EBV (PA) information has to be also from the truncated data. g. Bulls with EBV in the full data sets that have no progeny information four years ago (EDCr=0), should be included in the reduced data set.

• h. If the EBVs from evaluations published four years ago are available, the country can use these values for the reduced data sets. However, if the evaluation model, trait definitions, etc. have changed from the estimation of EBVs in the reduced data sets and the estimation of EBVs in the full data sets, the GEBVr can be expected to have lower accuracy than GEBV. In this case, the country should report the expected correlation between the old (reduced) and the new (full) data EBVs (see Interbull Testing Method 3).

• In order to remove any change in scale of proof expression, EBVr and GEBVr should be rescaled to the same scale as EBVs, using bulls already proven in the reduced data sets. 3.4. File names Submitted files should have the following name: file{FILETYPE}{FILEFORMAT}{DATASET}{BRD}.{COU}.yyyymmdd where file = prefix FILETYPE = C (conventional), G (genomic) or D (De-regressed/DD) (upper-case) FILEFORMAT = 010, 015, 115, 016, 017, 018, 019, 020 DATASET = f (full) or r(reduced) (lower-case) BRD = all, bsw, gue, hol, jer, rdc, sim (lower-case) COU = ARG, AUS, … (upper-case) yyyymmdd = year|month|day of the file creation Example: ‘fileG010rhol.AUS.20110509’ would be the genomic reduced data set for production traits from the Australian Hosltein population, prepared in May 09, 2011.

• Test description 4.1. Testing for bias The bias in the national genomic evaluations will be tested using a regression model: φ_i= b₀ + b₁*GEBVr_i + e_i [1] where φ_i is the D_PGM (or DD, if available) from the bulls that have EDC≥20 and EDCr =0. The EDCs from the full data set can be used as weights in the model if DDs are supplied, otherwise the accuracy of the D_PGMs (w_i=EDC/(EDC+λ)) will be used as weights.

• This model is used to estimate b₁ to compare with the expectation of b₁ (H₀: b₁ = E(b₁)) and therefore test the bias on GEBVr. Item 4.3 describes how the expectation of b₁ can be derived considering the impact of selective genotyping among test bulls.

  1. The statistical significance will be tested using a t-test against H₀ (C.I. = 0.95).

  2. ii. For larger populations the estimated standard error might become very small and then the t-test may become too restrictive. In those cases, a “biological significance” will be adopted to test H₀ [P((E(b₁)-0.1) ≤ b₁ ≤ (P(E(b₁)+0.1))].

  3. iii. Followed by ITC discussions in Cork 2012, the country-trait-breeds will pass the test, if b₁ value is greater than the lower endpoint of the 95% confidence interval or its biological equivalent.

  b. The accuracy of GEBVr will be estimated from the R² of the model (accuracy of the model after selection for genotyping). This validation accuracy R²_validation = R²/ , where is the average weight of all the test bulls. It will be expected that the mean of published bull r²_GEBVr is in agreement with R²_validation. 4.2. Testing the improvement from conventional evaluation The improvement of the added daughter information to the parental information will be estimated by comparing the R² from model [1] with the R² from model [2]: φ_i = b₀ + b₁*EBVr_i+ e_i [2] where φ_i and the corresponding weight w_i are the same as in model [1]. The R² from model [1] must be higher than the R² from model [2]. 4.3. Estimating the effect of selective genotyping on E(b₁) The expected value of b₁ is 1.0 only if the genotyped test bulls are a representative sample of the bulls in the corresponding age classes. The selection based on EBVs before genotyping will reduce the value of b₁ and also the value of R² for model [1]. The level of selective genotyping can be approximated from the difference between the mean EBV of the genotyped test bulls, µ_EBVg, and the mean EBV of all potential test bulls (i.e. bulls with EDC≥20 and EDC_r=0, genotyped or not), µ_EBVall, and the standard deviation of EBV of all potential test bulls (σ_EBVall).

  • [3]

  Using tables from quantitative genetics books, (e.g. page 379 from Falconer, D. S. & Mackay, T. F. C. Introduction to Quantitative Genetics, Longman, 4^th ed. 1996) the proportion of selected (genotyped) individuals (p) can be obtained for the selection differential (i) and the corresponding truncation point x that divides the standard normal density into selected proportion p and non-selected (1-p). Having the proportion of the selected individuals, the expected value of the b₁ (E(b₁)) and the effect of the selection on R² of the test model can be estimated by approximation of the effect of selection on the variance of the selected trait and on the covariance between the independent (GEBVr) and the dependent (φ) variables. Having (i) as the mean deviation of the selected individuals from the total population in terms of standard deviation from the total population, and (x) as a selection truncation point from the overall mean: k = i(i - x) [4] v₁ = 1 – k [5] Calculating R² before selection (R_b²), which is the R² for model [1], from R² after selection (R_a²): R_b² = R_a² / (v₁ + kR_a²) [6] v₂ = 1 – kR_b² [7] E(b₁) = v₁ / v₂ [8] Example: Assuming that: µ_EBVg = 16.00; µ_EBVall = 11.76; σ_EBVall = 10.00; R_a² = 0.555. The selection differential (i) for the genotyped bulls equal to 0.424 standard deviations of EBVs (equation [3]), the proportion of genotyped bulls (p) would be 75 percent and the mean deviation of the truncation point from the overall mean (x) would be equals to -0.674 (from reference table). Applying equations [4], [5] and [6] it is possible to calculate R_b² = 0.70. Using equations [7] and [8] E(b₁) = 0.793. Table 2 – Examples of expected regression coefficients (E(b₁)) as functions of the selection intensity (i) and the coefficient of determination before selection (R_b²).

1. Documents to be submitted by participating NGEC 5.1. Interbull GENO form

2. The methodology for estimation of GEBV and its’ accuracy (r²_GEBV) have to be reported by the NGEC in Interbull GENO form (Appendix IV). 5.2. GEBV test estimates (File format 731) The NGEC submitting genomic data for validation is required to provide also Form 731 (Appendix III), which contains the results from the validation test obtained by the applicant, as well as descriptive statistics needed for the correct treatment of the submitted data. Eventual discrepancies between values in file 731 and estimates obtained by ITBC will be discussed with the NGEC submitting data prior to publication of results. APPENDIX I – Definitions: · EBV – Estimated Breeding Value (conventional national evaluations of the trait, free of genomic information, which are submitted to Interbull to be used in MACE evaluations) · DGV - Direct Estimated Genomic Value (genomic evaluations based on SNP prediction equations) · GEBV – Genomically Enhanced Estimated Breeding Value (evaluations that combine EBV and DGV) · EDC –Effective Daughter Contribution · GEDC – Genomically Enhanced Effective Daughter Contribution (EDC plus the genomic contribution) · GMACE - Multiple Trait Across Country Genomic Evaluation · PA – Parent Average · D_PGM – De-regressed Predicted Genetic Merit · DD – Daughter Deviation · NGEC - National Genetic Evaluation Centre · λ = (4-h²)/h² · r² – Reliability of the bull’s evaluation · R² – Accuracy of the test model

   APPENDIX II - FILE FORMAT 732

3. File format for genotyping information on test bulls.

• APPENDIX III - FILE FORMAT 731 File format for description of the GEBV test results.

• Models [1] or [2], as proposed on the validation test description. APPENDIX IV – FORM GENO

• Standard Interbull form for description of national genomic evaluation systems. . Avaiable at:

  • http://www.interbull.org/index.php?option=com_content&view=article&id=55&Itemid=161

• [ii] http://www.interbull.org/index.php?option=com_content&view=article&id=55&Itemid=76 [iii] Current: latest official routine run before validation data is prepared.