Describe public/GDE_api_manual here.

GenoEx-GDE User’s manual v.1.2

part 3 - API Manual (including the use of the gxapi.py program).

See also part 1 in GDE_user_manual and part2 in GDE_gxprep_manual. This manual part assumes that those previous parts have already been read and understood.

The gxapi.py support program, maintained and distributed by the Interbull Centre, allows easy access to the API for upload/extract/download of 706 and 711 files associated with the GenoEx-GDE database and is provided as an easy way to get started with using the API. For those that can read the python code that it is written in, it also provides an additional source of documentation of the API.

This manual describes each of the calls of the API along with the usage of the gxapi program. These descriptions are organized into five sections to focus on the main aspects of the API where the first section provides an overview of, and some general information about, the API and the last section focus on the gxapi.py program. The remaining sections focus on different usage of the API.

Section 1, overview and general information

The API is provided as an alternative way to access the functionality provided via the https://genoex.org/ web site interface and is provided via POST calls on the same site. The operations have a basic structure where each call require arguments in JSON format split into parameters and auth, both of which are key/value mappings.
The auth part always contain keys username and pw where the respective values should be your registered email address and associated password.
The parameters part contain different keys depending on what call it is about, but always at least version.
An example call via the curl program looks like (in one long command):

• curl -s -F 'data={ "parameters": { "version": "220805" }, "auth": { "username": "username@company.com", "pw": "test" } }' https://genoex.org/genoex/api/gde_get_parameters

Naturally, the username@company.com and test strings need to be substituted with something more appropriate before running this. This shows the common basic structure of every call to the API, but most calls need additional parameters.

The details below are up-to-date with the 220805 version of the API (and gxapi.py).

The API is in large parts asynchronous, i.e. where an operation is first initiated and then the user need to periodically poll for the status of that operation until it terminates either successfully or with a failure. This mode of operation is needed to avoid the timeouts inherent in normal implementations of the HTTP protocol for long running operations.

The following three sections focus on, respectively, upload of 706/711 files, download of 706/711 files and finally other operations.

Section 2, upload of 706/711 files

This is a two step operation: a submit call (once) and then intermittently (once per minute or so) polling status of that submission until a terminating state is reached.

An example submit call via the curl program looks like:

• curl -s -F 'data={ "parameters": { "version": "220805" }, "auth": { "username": "username@company.com", "pw": "test" } }' \
   -F 'access_file=@system-data/711-access.zip;ţype=application/octet-stream' \
   -F 'dataset_file=@system-data/706-genotypes.zip;ţype=application/octet-stream' https://genoex.org:16905/genoex/api/gde_submission

As always, the username@company.com and test strings need to be substituted with something more appropriate before running this. In addition, the paths and filenames specified need to be adapted to your own situation.
Note that the use of a single backslash at the end of the lines is just a way to visualize that the single command continues on the next line.

This example specifies the aggregate upload of both a 706 file and the associated 711 file in one go, but if only one of these file types are to be uploaded then simply omit the other files -F switch and associated JSON string.

This submission call will return a JSON data structure containing, if successful, the job_id assigned to this submission:

 . {'return_values': {'access_file': '711-access.zip',
                   'access_status_message': 'Access file received',
                   'dataset_file': '706-genotypes.zip',
                   'dataset_status_message': 'Dataset file received',
                   'job_id': '9be6c0bf-de9f-4951-b9e1-27217ec1e0c4'},
 'status': True,
 'status_message': 'job started'}

Note that in all calls, if the key status has a False value, then the error message is found in status_message. Even if status is True, there may still be errors described inside the return_values data structure.

The second step, polling for status, is accomplished via a call like:

• curl -s -F 'data={ "parameters": { "version": "220805", "job_id": "9be6c0bf-de9f-4951-b9e1-27217ec1e0c4" }, "auth": { "username": "username@company.com", "pw": "test" } }' https://genoex.org/genoex/api/gde_job_status

The 9be6c0bf-de9f-4951-b9e1-27217ec1e0c4 string need to be replaced with the value of the job_id key provided in the return data structure of the submit call above.

This call is then intermittently repeated, with no change, until either a job_status of FINISHED or FAILED is reached and returned in a JSON data structure:

• {'return_values': {'error': '',
                     'error_list': [],
                     'job_id': '9be6c0bf-de9f-4951-b9e1-27217ec1e0c4',
                     'job_status': 'FINISHED',
                     'status_time': '2022-08-05 13:35:02'},
   'status': True,
   'status_message': 'GDE upload'}

Section 3, download of 706/711 files

Download operations are a bit different from upload as 711 files are downloaded in a synchronous mode and 706 files are downloaded in asynchronous mode similar to upload.

In addition, there is an optional preliminary step to retrieve the available values to choose from when selecting the parameters to provide in the download operation:

• curl -s -F 'data={ "parameters": { "version": "220805" }, "auth": { "username": "username@company.com", "pw": "test" } }' https://genoex.org/genoex/api/gde_get_parameters

Download 706 files

This is a three step operation: an extraction call (once) followed by intermittently (every 15 seconds or so) polling status of that extraction until a terminating state is reached and finally fetching the resulting assembled zip file.

Download 711 files

This is a single step operation.

1. GenoEx-GDE upload files’ formats

GenoEx-GDE allows upload of two types of files:

• data file (format 706 – section 1.1)

• sharing permissions file (format 711 – section 1.2)

Note: Both input files are prepared from the laboratory output files by the gxprep.py program (section 2). The program is also assigning an unique UUID identifier to each genotype allowing distinguishing between several records of one animal. Both file are delimited by semicolon.

1.1. File 706

This file contains the actual genomic data, as well as the information about the animal, genotyping laboratory and chip used for the genotyping.
Because typical genomic data contain a lot of information, data in this file is coded down to a single digit per SNP and single record per animal. Correctness of such coding requires SNPs to be written in certain order within the data stream, according to the SNP order list, where particular SNPs are recognized by name and given a position in the data stream. This coding, although allowing easy exchange of really large files, because of its dependency on the order, is unfortunately also prone to errors. Therefore, to ensure the highest data quality in the GenoEx-GDE database, we provide a program called gxprep that takes raw laboratory files with your data as input, fetches correct SNP order list from our servers and produces correctly ordered 706 file ready to be uploaded to the database. See section 2.

706 file format

1. Record type is always 706 for this File Format
2. ISO 3166-1 alpha-3 codes (3 characters, capital letters)
3. Breed of evaluation (3 characters, capital letters, BSW, GUE, HOL, JER, RDC, SIM)
4. Breed of animal (3 characters, capital letters)
5. Alpha-numerical, Interbull standard, always 12 characters long

6. UUID, one for every uploaded genotype sequence. Additional information about generation of UUID can be found at here

7. Genotyping laboratory, among the ones listed in the "Laboratories" table available in the GenoEx-GDE System Data page https://genoex.org/display. If the laboratory is not listed in the table, a request should be sent to GenoEx@slu.se

8. Array identifier, one of the listed in the "SNP Arrays" table ("Code" column) available in the GenoEx-GDE System Data page https://genoex.org/display.

9. n is equal to the number of SNPs reported in the stem of the Array identifier

10. coded SNP values written as a continuous string.
    Acceptable values depend on the Illumina coded allele values, according to the following:

BB→0
AB→1
AA→2
‘unknown’→5

706 example

706;BSW;ITA;M;000000A12345;ANARB;09c98b1e-6af8-4254-9768-58d7cd1ddafd;Weatherbys Ireland;S1234WI2001;;54609_a;021010…

1.2. File 711

• The function of 711 file is to set the permissions regarding who can have access to the data. Since the default sharing state for all the data uploaded to GenoEx-GDE database is ‘not shareable’, the User has to upload 711 file(s) to change it.

711 file format

1. Record type is always 711 for this File Format
2. Please see Interbull Bulletin 28. Each file can only contain any given animal in one row.
3. Breed of animal (3 characters, capital letters)
4. ISO 3166-1 alpha-3 codes (3 characters)

5. UUID, used as reference to every uploaded genotype sequence in the 706 file. Additional information about generation of UUID can be found here

6. Comma-separated list of zero or more organizations that should be allowed to download the associated genotype

711 example

711;BSW;ITA;M;000000A12345;09c98b1e-6af8-4254-9768-58d7cd1ddafd;BFRO,IBC

2. Upload preparation program – gxprep.py

gxprep.py support program, maintained and distributed by Interbull Centre, prepares a set of files: 706 and 711, ready to be uploaded to GenoEx-GDE database. Always check you are using the latest version of gxprep.py program downloading it from https://genoex.org/.

2.1. gxprep functions

The program has four main commands: parse, sharing, show and zip parse should always be run as first option, because the other commands run on the output files produced by this command. sharing and show can be run several times, allowing gradual fine adjustments of the sharing permissions. zip is to be run at the end of the preparation process, when both files are ready for the upload.

parse

reads the input files (see section 2.2) and produces file 706 and initial 711 file - with all the animal IDs and corresponding UUIDs - but only the default (if set) sharing permissions assigned Most of the standard laboratory output files can be used as input unmodified (see 2.2) but the User has to provide additional information while running this command. Note: All these values could be provided also as default by specifying them in an initialization file (see 2.3.)

• • -h [HELP]

• • -a [ARRAY] – alphanumerical value that defines the chip that was used for genotyping according to the Array List (see also section 3). Note: If the chip used for genotyping is not listed, the User is advised to contact Interbull Centre and provide the full list of SNPs and the name of the chip, which will then be added to the Array List. Using different array than the one assigned to the used chip is possible, but would raise multiple warnings. This is, however OK, if the actual chip and therefore laboratory output contains additional, custom SNPs, that the User does not wish to upload.

• • -d [DELIMITER] – character argument that defines the column delimiter in body section of input file (laboratory output file). Default delimiter is tab, thus this command line option should be used if another delimiter is used instead. • -l [LAB] – standardized name of the laboratory where genotyping was performed, according to the Laboratory List (see also section 3). If the laboratory is not listed there, the User should notify Interbull Centre before preparing the data. • -s [SAMPLEMAP] – file mapping sample ID from laboratory to the actual animal ID. Note: all animal IDs should be provided as International Interbull ID (see interbull.org/ib/interbull_guidelines for more information)

• • -l [LAB] – Genotyping laboratory, among the ones listed in the "Laboratories" table available in the GenoEx-GDE System Data page https://genoex.org/display. If the laboratory is not listed in the table, a request should be sent to GenoEx@slu.se

• • -s [SAMPLEMAP] – File mapping sample id to corresponding animal IDs.

• • -i [INPUTSPEC] – since the contents of the laboratory output files may differ between organizations, the User has to specify which columns in the file contain required information. With the assumption that column numbers start with 1, the order should be as follows: SNP name, sample ID, allele1, allele 2. This value should be provided as four space separated numbers in single or double quotation, e.g. ‘1 2 3 4’

• • -c [CACHEDIR] – specifies the directory where SNP order files will be downloaded from Interbull servers Note: Both output files names from this command will have a common stem, shared with input file, followed by the file type number (e.g. from input file I52690.txt, the stem is I52690, and the files produced by parse are I52690-706.csv and I52690-711.csv), which is used by the other commands in this program.

parse example

python gxprep.py parse -a 7931_a -l CIGENE -s SampleMap.txt -C ~/gde/gxprep/cachedir -i "1 2 3 4" Iexample.txt

This command first retrieves the SNP order file from Interbull server and saves it to /gde/gprep/cachedir folder. Then, it parses the Iexample.txt file retrieving animal IDs from SampleMap.txt file (see section 2.2. for description of input files). In Iexample.txt, it looks for SNP name in column 1, sample ID in column 2, allele1 in column 3 and allele2 in column 4. Allele 1 and allele 2 are then coded to one digit, accordingly to the formula: BB→0, AB→1, AA→2, ‘unknown’→5 and placed in the genotype string according to the SNP order. All samples listed in this file are get CIGENE as laboratory. Also, each newly created record gets assigned UUid identifier and the same number, along with corresponding animal ID is listed in 711 file. If you set up any defaults for sharing (see section 2.3) they will also be used in newly created 711 file, otherwise the last column in this file will remain empty.
The files created in this example will be named Iexample-706.csv and Iexample-711.csv

sharing

is used to add or remove organizations from the list of the organizations allowed to download given genotype. This command only operates on the 711 file and thus ignores the 706 file, if present.

Note: Newly created, by parse command 711 file, will normally have the last column (‘Shareable with organizations’) empty, unless the defaults are specified otherwise (see 2.3.). Therefore, in most of the cases it is necessary to run sharing to create the list of organizations each records can be shared with. This can be done either by assigning the same permissions to all the data within the file, by adding them according to pattern defined by breed, sex or country of origin or by providing a list of specific animals that should have the sharing permissions changed.

This command takes the following arguments:

• - a [ORGANIZATION] - adds organization(s) the list, the data should be shared with. If no further arguments are provided, this sharing will be assigned to all the records in given 711 file.

• - r [ORGANIZATION] - removes listed organization(s) from the sharing list

• - b [BREED] - assigns sharing permissions by breed(s)

• - g {M,m,F,f} - assigns sharing permissions by sex. Not defining this argument will assign defined sharing permissions to animals of both genders, given they fulfill the other conditions

• - f [COUNTRY] - assigns sharing permissions by country(s) of registration (part of animal ID)

• - i {filename} - with this command User can provide the list of specific animals (animal ID) that should be affected by sharing permissions change.

• INPUT FILE – the stem of (common part of the names of 706 and 711 files)

sharing examples

• python gxprep.py sharing -a ANARB –a BFRO –b BSW -g M -f ITA Iexample

This command adds sharing permissions for ANARB and BFRO to all the BSW males originating from Italy in Iexample-711.csv file

• python gxprep.py sharing -r ANARB –a IBC –i aidlist.txt Iexample

This command adds sharing permissions for IBC and removes it for ANARB for all the genotypes in Iexample-711.csv file according to the animal ID list in the file aidlist.txt

show

gives an overview of sharing patterns in given file, using the stem of input files as the only argument.

show example

• python gxprep.py show Iexample

This command shows the summary of the sharing settings, giving an output like this:

Content of sharing intermediate file Iexample-711.csv:
11 genotypes (all female) shared with IBC
9 genotypes (all male) shared with BFRO,IBC

zip

• prepares zip file to be uploaded to GenoEx-GDE database. It uses the stem of input files as the only argument.

zip example

• python gxprep.py zip Iexample

After running this command both 706 and 711 files will be zipped in two separate files ready for the upload to GenoEx-GDE database.

2.2. gxprep input files

gxprep is constructed to accept most typical laboratory output files and convert it to 706 data file adding also the initial version of 711 file for setting data sharing permissions. Of course, since both 706 and 711 files base on animal ID, whereas laboratory output files operate on sample IDs, the User also needs to provide a reference file mapping each sample ID to the corresponding animal ID.

The laboratory output file and ID reference file are expected to follow the formats as described below:

laboratory file

This file contains actual data as received from the laboratory, with Sample ID as a key. In the examples above this file is named Iexample.txt

[Header]

optional, general information regarding analysis, chip and number of samples

[Data]
please make sure that is specified, no matter if the header is included or not.
! [Data] marks the place where reading the information is started.

Note: the above columns are required for GenoEx. Laboratory file can however contain additional columns, or columns in different order. As described in point 1.2.1. under parse, the User can specify which columns are containing required information.

ID reference file

This file contains the key to identify which Sample ID belongs to which animal. In the examples above this file is named SampleMap.txt
Note: The only allowed delimiter allowed in this reference file is TAB

* International Interbull Animal ID consists of 18 characters as follows:

3 characters - breed code (capitals, according to ICAR breed coding),
3 characters – country code (capitals, according to Interbull country coding),
1 character – sex code capital M or F),
11 characters – registration ID (alphanumerical).

2.3. gxprep default settings

If the User is always using the same array, laboratory, the same columns in the laboratory file or always shares all the data with the same list of the organizations, they may want to pre set these values as default. This can be done by editing gxprep.ini file. Dependently on your local settings, this file is located in the current directory and/or in the users home directory and also the .gxpreprc file in the user’s home directory.

Recognized configuration options are the following:

• arrayspec - default for -a/--array switch of parse sub command
  labspec - default for -l/--lab switch of parse sub command
  samplemap - default for -s/--sample-map switch of parse sub command
  inputspec - default for -i/--input-spec switch of parse sub command
  delimchar - default for -d/--delimeter switch of parse sub command
  cachedir - default for -C/--cachedir switch of parse sub command sharing - default
  sharing, i.e. which organizations to share data with

• gxprep.ini example

  [gxprep]
  arrayspec=44887_a
  labspec=The Roslin Institute
  samplemap=~/referencefile.txt
  inputspec=1 2 5 6
  delimchar=,
  cachedir=~/gxprep
  sharing=ANARB BFRO

In the example above, if not specified otherwise in the command line:

• 706 file will be created accordingly to the SNP order 44887_a
• assigned laboratory is ‘The Roslin Institute’
• ID reference file is looked up under the name referencefile.txt.
• the information is read from input file as follows:
  • SNPname - column 1
  • sample ID - column 2
  • allele 1 and allele 2 - columns 5 and 6, respectively.
• the delimiter is expected to be comma.
• SNP order file is downloaded and stored in ~/gxprep
• all genotypes processed with these settings have sharing permitted for ANARB and BFRO

3. gxprep "Tips and Tricks"

Whereas all currently allowed values are available to view via GenoEx home page, one can also see it directly in the terminal with use of gxprep.py
Below, there is a list of commands to retrieve specific lists

list of supported arrays

• python gxprep.py parse -a xxxx xxxx

list of supported labs

• python gxprep.py parse -l xxxx xxxx

list of supported organizations

• python gxprep.py sharing -a xxxx xxxx

list of supported country codes

• python gxprep.py sharing -f xxxx xxxx

list of supported breed codes

• python gxprep.py sharing -b xxxx xxxx

download specific SNP order file - Windows

• python gxprep.py parse -C . -a 38275_a xxxx>g

download specific SNP order file - Linux

• python gxprep.py parse -C . -a 38275_a xxxx>g dev/null

Note: Trailing arguments ‘xxxx xxxx’ can be replaced by any other nonsense words at least 2 characters long.