Interbull CoP - Appendix VIII - Interbull validation test for genomic evaluations – GEBV test

Document based on Mäntysaari, E., Liu, Z and VanRaden P. 2011. Interbull Validation Test for Genomic Evaluations. Interbull Bulletin 41, p. 17-21.

Definitions:

Motivation

The inclusion of genomic information in international comparisons for dairy breeds requires that the national genomic breeding values (GEBVs) get validated by Interbull in a similar fashion that conventional EBVs are validated as a pre-condition to participate in the MACE evaluations.

The GEBV test will be applied to validate national models used to compute GEBVs that the national genetic evaluation centers (NGEC) publish and will eventually submit to Interbull for international genetic evaluations including genomic information. The GEBV test can be considered also a quality assurance assessment for national genomic evaluations. GEBVs from models that have been tested can be referred to as breeding value estimates with appropriate reliability, and be converted to other country scale breeding values using conversion equations derived by Interbull.

Rationale

The GEBV test evaluates:

The test for bias is done by verifying the ability of a model only including data from 4 years ago to predict current performances. NGEC have to exclude the last 4 years of data and re-run the analyses with the reduced data, with the same model that are being tested. However, in some cases the bull generation available for validation has not been genotyped in everything and all. Thus, bulls exist that will get more than 20 daughters in the full data, but that have no GEBVs. This is called selective genotyping, and it leads into systematic bias in the validation bull group. In the test, this bias needs to be corrected by accounting for the selection between the mean national EBV (current, conventional) of the bulls genotyped and the overall mean national EBV including all potential candidates. This selection differential can be used to derive the expected regression coefficient, which would be equal to unity as if no selective genotyping took place.
Testing the improvement in accuracy is done by comparing the coefficient of determination (R2) of the reduced genomic model and the equivalent reduced conventional model (from 4 years ago) regressed to current performances. The R2 from the model including genomic information must be higher than the model including only parent average information.

Test data sets

Data formats are described at GEBVtest Software.

Full data sets

The full data sets include all animals present in the most recent Interbull MACE evaluation. They are of two types, one containing national official genetic merit values (EBVs) and another containing either de-regressed predicted genetic merits (D_PGMs) or daughter deviations (DDs).

National official genetic evaluation file (fileCxxxf)

The files sent by the NGEC as input for the most recent Interbull MACE evaluation and will be used to identify the candidate bulls, estimate selection intensity and check bulls birth year and type of proof.

Daughter deviation file (fileDxxxf)

The NGEC needs to prepare either DD or D_PGM for the same animals included in fileCxxxf. These values represent the currently estimated performance of the animals and will be used as the dependent variable in the validation procedure. EDC and reliability estimates should be exactly the same as in fileCxxxf.

Reduced data sets

The reduced data sets should be prepared by truncating the phenotypes used as input for both the conventional and the genomic evaluations. The NGEC must exclude phenotypic information from the past 4 years and re-run the current models of genetic/genomic evaluation for the traits of interest, keeping the animals without progeny information after truncation (test bulls) in the data in order to obtain genetic merit estimates based solely on parent averages (EBVr) or on parent averages plus genomic prediction equations (GEBVr).

Reduced conventional genetic evaluation file (fileCxxxr)

The NGEC should carry out a conventional genetic evaluation using truncated data (only phenotypes up to 4 years prior to the date of analysis) but including in the analysis all animals present in the current official evaluations (fileCxxxf).

Reduced genomic evaluation file (fileGxxxr)

Similarly, new genomic evaluations should be carried out using exactly the same model being validated (current) but excluding phenotypic information up to four years ago (truncated data, fileCxxxr). All bulls that did not have a progeny test 4 years ago and that currently have at least 20 daughter-equivalents in the national genetic evaluation (test bulls) need to have a genomically enhanced EBV (GEBVr) estimated and included in the output.

If a significant number of foreign animals are included in the reference population and estimation of genomic prediction equations uses de-regressed MACE values for these animals as input, the reduced genomic evaluation can be achieved in two ways:

  1. the Interbull Centre can make historical files available upon request (e.g. information used four years ago) containing past MACE results and the correspondent national EDCs, as well as heritability and genetic correlations used in the respective evaluations – these data can then be used to estimate 4-year old de-regressed values; OR
  2. the genomic prediction equations for the truncated data (only bulls with EDCr > 0) are obtained using current de-regressed MACE values. This constitutes an exception and should only be used when the standard procedure is not practical.

Table 1 presents a comparison between the several types of data and the notation used to identify variables from different files.

Table 1 – Comparative specification of the data files needed for the GEBV test.

Test Data

Type of information

File types and formata

Specific variablesb (equivalent field in the fileCxxxf)

EDC

Reliability

EBV

Full data sets

Conventional Genetic data

C010f, C115f, C015f, C016f, C017f, C018f, C019f, C020f

EDC

r2EBV

EBV

Daughter deviation data

D010f, D115f, D015f, D016f, D017f, D018f, D019f, D020f

EDC

r2EBV

D_PGM (or DD, if available)

Reduced data sets

Conventional Genetic data

C010r, C115r, C015r, C016r, C017r, C018r, C019r, C020r

EDCr

r2EBVr

EBVr

Genomic data

G010r, G115r, G015r, G016r, G017r, G018r, G019r, G020r

GEDCr

r2GEBVr

GEBVr

aThe GEBVtest software (gebvtest.py) uses a trait-independent format (File300). Users can either prepare data in the new format or use the program gtconvert.py to convert the current format into the File300 format.
bAll other variables should be the same as in the Cxxxf files.

Specific instructions for data preparation:

  1. The domestic bulls (type of proof ≠ 21 or 22) that have EDC≥20 and EDCr = 0 are called test bulls. Interbull recommends that number of test bulls would be about 0.25 *(number of bulls used as reference population).

    1. If the number of bulls the country includes in the genomic evaluation is too small, then the accuracy of the GEBVs calculated using the truncated data becomes significantly smaller than with the full data. In that case, the country can use n < 4 years as the time difference between full and reduced data sets.

    2. If the number of test bulls is too small (ntb < 50), the country may chose to consider foreign bulls (type of proof = 21 or 22) that have EDC≥20 and EDCr = 0 also as test bulls.

    3. In both exceptions above, the Interbull Centre must be communicated in detail about the criteria adopted to define the test bulls.
  2. Appropriate time windows (birth years of bulls) may vary depending of the trait to be validated, the speed of their progeny test program and other factors. A shift of the time window with one year will give a different set of bulls that qualify for the test. The standard adopted for the GEBV test is to include four years of candidate/test bulls, which corresponds to an age cutoff of (YYYY-8). For instance, if the Cxxxf is from 2012, Cxxxr and Gxxxr should include performance records up to 2008 and test bulls would be those born from 2004 and 2008. Countries may include more birth years, but the Interbull Centre must be communicated about the reason.
  3. GEBV is the genomically enhanced breeding value. Correspondingly, the GEDC is a genomically enhanced EDC that combines the EDC from national non-genomic evaluation with the gain from genomic evaluation. This means that GEDC should be larger than EDC and GEDCr should be larger than EDCr.
  4. Include all the bulls having GEBVr in the data without data edits based on EDC, EDCr, GEDC or GEDCr.
  5. If GEDCr is not available, then GEDCr = λ * r2GEBVr / (1- r2GEBVr)

  6. The method of estimation of GEDCr (and/or r2GEBVr) has to be reported in the Interbull GENO form.

  7. The GEBVr prediction equations also have to be based on the truncated data. If the GEBVr combines information of DGV and EBV (i.e. PA), the EBV (PA) information has to be also from the truncated data.
  8. Bulls with EBV in the full data sets that have no progeny information four years ago (EDCr=0), should be included in the reduced data set.
  9. If the EBVs from evaluations published four years ago are available, the country can use these values for the reduced data sets. However, if the evaluation model, trait definitions, etc. have changed from the estimation of EBVs in the reduced data sets and the estimation of EBVs in the full data sets, the GEBVr can be expected to have lower accuracy than GEBV. In this case, the country should report the expected correlation between the old (reduced) and the new (full) data EBVs (see Interbull Testing Method 3).
  10. In order to remove any change in scale of proof expression, EBVr and GEBVr should be rescaled to the same scale as EBVs, using bulls already proven in the reduced data sets.

Test description

Testing for bias

The bias in the national genomic evaluations will be tested using a regression model:

φi= b0 + b1*GEBVri + ei [1] ,

where φi is the D_PGM (or DD, if available) from the bulls that have EDC≥20 and EDCr =0. The EDCs from the full data set can be used as weights in the model if DDs are supplied, otherwise the accuracy of the D_PGMs (ui=EDC/(EDC+λ)) will be used as weights.

  1. This model is used to estimate b1 to compare with the expectation of b1 (H0: b1 = E(b1)) and therefore test the bias on GEBVr. Item 4.3 describes how the expectation of b1 can be derived considering the impact of selective genotyping among test bulls.

    1. The statistical significance will be tested using a t-test against H0 (C.I. = 0.95).

    2. For larger populations the estimated standard error might become very small and then the t-test may become too restrictive. In those cases, a “biological significance” will be adopted to test H0 [P((E(b1)-0.1) ≤ b1 ≤ (P(E(b1)+0.1))].

    3. The country-trait-breeds will pass the test, if b1 value is greater than the lower endpoint of the 95% confidence interval or its biological equivalent.

  2. The accuracy of GEBVr will be estimated from the R2 of the model (accuracy of the model after selection for genotyping). This validation accuracy R2validation = R2/ ū, where ū is the average weight of all the test bulls. It will be expected that the mean of published bull r2GEBVr is in agreement with R2validation.

Testing the improvement from conventional evaluation

The improvement of the added daughter information to the parental information will be estimated by comparing the R2 from model [1] with the R2 from model [2]:

φi = b0 + b1*EBVri+ ei [2] ,

where φi and the corresponding weight ui are the same as in model [1]. The R2 from model [1] must be higher than the R2 from model [2].

Estimating the effect of selective genotyping on E(b1)

The expected value of b1 is 1.0 only if the genotyped test bulls are a representative sample of the bulls in the corresponding age classes. The selection based on EBVs before genotyping will reduce the value of b1 and also the value of R2 for model [1]. The level of selective genotyping can be approximated from the difference between the mean EBV of the genotyped test bulls, µEBVg, and the mean EBV of all potential test bulls (i.e. bulls with EDC≥20 and EDCr=0, genotyped or not), µEBVall, and the standard deviation of EBV of all potential test bulls (σEBVall).

Using tables from quantitative genetics books, (e.g. page 379 from Falconer, D. S. & Mackay, T. F. C. Introduction to Quantitative Genetics, Longman, 4th ed. 1996) the proportion of selected (genotyped) individuals (p) can be obtained for the selection differential (i) and the corresponding truncation point x that divides the standard normal density into selected proportion p and non-selected (1-p).

Having the proportion of the selected individuals, the expected value of the b1 (E(b1)) and the effect of the selection on R2 of the test model can be estimated by approximation of the effect of selection on the variance of the selected trait and on the covariance between the independent (GEBVr) and the dependent (φ) variables. Having (i) as the mean deviation of the selected individuals from the total population in terms of standard deviation from the total population, and (x) as a selection truncation point from the overall mean:

k = i(i - x) [4]

v1 = 1 – k [5]

Calculating R2 before selection (Rb2), which is the R2 for model [1], from R2 after selection

(Ra2): Rb2 = Ra2 / (v1 + kRa2) [6]

v2 = 1 – kRb2 [7]

E(b1) = v1 / v2 [8]

Example: Assuming that: µEBVg = 16.00; µEBVall = 11.76; σEBVall = 10.00; Ra2 = 0.555. The selection differential (i) for the genotyped bulls equal to 0.424 standard deviations of EBVs (equation [3]), the proportion of genotyped bulls (p) would be 75 percent and the mean deviation of the truncation point from the overall mean (x) would be equals to -0.674 (from reference table). Applying equations [4], [5] and [6] it is possible to calculate Rb2 = 0.70. Using equations [7] and [8] E(b1) = 0.793.

Table 2 – Examples of expected regression coefficients (E(b1)) as functions of the selection intensity (i) and the coefficient of determination before selection (Rb2).

i

p

x

E(b1)

Rb2 = 0.50

Rb2 = 0.55

Rb2 = 0.60

Rb2 = 0.65

Rb2 = 0.70

0.644

60

-0.253

0.594

0.619

0.646

0.676

0.709

0.570

65

-0.385

0.626

0.650

0.677

0.705

0.736

0.497

70

-0.524

0.660

0.683

0.708

0.735

0.764

0.424

75

-0.674

0.697

0.718

0.742

0.766

0.793

0.350

80

-0.842

0.736

0.756

0.777

0.800

0.823

0.274

85

-1.036

0.781

0.799

0.817

0.836

0.856

0.195

90

-1.282

0.832

0.846

0.861

0.876

0.892

0.109

95

-1.645

0.894

0.904

0.914

0.924

0.934

0.000

100

1.000

1.000

1.000

1.000

1.000

Interbull form GENO

The methodology for estimation of GEBV and its’ accuracy (r2GEBV) have to be reported by the NGEC in Interbull form GENO.

public/CoPAppendixVIII (last edited 2013-07-29 12:47:39 by Valentina)