Primal-Page

Tooling and schemas for the primerschemes index

Contents

Installation

Quick Start

Schemas

CLI

Installation

pip install primal-page

Quick Start

Each version of a primerscheme has three parts; {schemename}/{ampliconsize}/{version}, which when combined these form the schemes unique identifier.

For a scheme to be added to the repo it requires three essential files.

• primer.bed: Contains the primer information.
• reference.fasta: Contains the reference genomes.
• info.json: Contains key metadata for the scheme.

The primal-page create command will generates the info.json and parses a bed file and a fasta file into primer.bed and reference.fasta

Additional files are copied into the schemes work directory.

Example

Adding a minimal scheme

For a simple scheme, only fasta file (--reference) and a bed file (--primerbed) are required. Alongside metadata of --schemename, --algorithmversion, --species, --authors.

primal-page create \
    --schemename example-scheme \
    --algorithmversion ps:100 \
    --ampliconsize 500 \
    --species 123  \
    --schemeversion v1.0.0 \
    --primerbed ~/'minimal-scheme/minimal.bed' \
    --reference '~/minimal-scheme/ref.fasta' \
    --authors "me" \
    --authors "you" /
    '~/minimal-scheme' 

Adding a custom scheme to quick-lab/primerschemes

create a local fork of https://github.com/quick-lab/primerschemes

Add the files to the forks primerschemes folder

primal-page create primal-page create ... --output ~/primerschemes/primerschemes

Create a pull request

New Schemas

info.json

This is the main metadata file for each primerscheme.

• ampliconsize: PositiveInt
• schemeversion: Following format listed below
• schemename: Following format listed below
• primer_bed_md5: MD5 hash of the schemes primer.bed file
• reference_fasta_md5: MD5 hash of the schemes reference.fasta file
• status:
  • withdrawn: Removed due to major issue
  • deprecated: Newer scheme is recommended
  • autogenerated: Scheme has been autogenerated using species-agnostic pipelines
  • draft: Scheme has been inspected in silico
  • tested: Scheme has been tested in the laboratory
  • validated: Scheme has been validated and/or published
• citations: How the scheme should be cited if used. DOI links are recommended, however, tweets/blogs are all allowed
• authors: The person or organization who generated the scheme. It is recommended that only corresponding/primary authors are included, with all other contributors recognized in the citations field
• algorithmversion: The algorithm and the version used to generate the scheme
• species: A list of organisms targeted by this scheme. NCBI TaxIds are recommend
• license: The name of the license the primerscheme is offered under
• primerclass: The class of scheme. Only primerscheme at the moment
• infoschema: The version of the info.json
• articbedversion: The version of the primer.bed (See below)
• description: A free text description to describe the primerscheme
• derivedfrom: To show if this scheme has been based on another primerscheme.
• collections: A collection of vocabulary to provide filtering/grouping of schemes.
  • ARTIC: Developed with the ARTIC network
  • MODJADJI: Developed with MODJADJI
  • QUICK-LAB: Developed with QUICK-LAB
  • COMMUNITY: Developed by the COMMUNITY
  • WASTE-WATER: Scheme capable of recovering genomes from high Ct samples (~30) samples, like wastewater. Typically 400bp schemes
  • CLINICAL-ISOLATES: Scheme capable of recovering genomes from medium Ct samples (~25) samples. Typically ~1000bp schemes
  • WHOLE-GENOME: Scheme that can theoretically recover a full genome
  • PANEL: Scheme that can recover sections of a target genome
  • MULTI-TARGET: Scheme that contains more than one target

PrimerNames Versions

Expected primernames (col3) in the primer.bed file

V1

This is the first standard for primernames. It follows the general pattern of {scheme-name|uuid}_{amplicon-number}_{LEFT|RIGHT} and an optional {_alt} to denote spike in primers.

Regex:

V1_PRIMERNAME = r"^[a-zA-Z0-9\-]+_[0-9]+_(LEFT|RIGHT)(_ALT[0-9]*|_alt[0-9]*)*$"

Example:

SARS-CoV-2_10_LEFT
SARS-CoV-2_10_LEFT_alt1

V2

This follows the pattern of {scheme-name|uuid}_{amplicon-number}_{LEFT|RIGHT}_{primer-number}.

• This now enforces that splitting on _ will produce a consistent length, allowing the safe indexing of attributes.
• primer-number is not enforced to be continuous. Therefore, the _0 numbered primer should not be thought of as the original and all other numbers as alts.

Example:

SARS-CoV-2_10_LEFT_0
SARS-CoV-2_10_LEFT_1

Regex:

V2_PRIMERNAME = r"^[a-zA-Z0-9\-]+_[0-9]+_(LEFT|RIGHT)_[0-9]+$"

ARTIC-primer.bed Versions

These are rough guidelines for the format of the primer.bed file. The general format is based on the .bed file and maintains compatibility with other tools.

colnames and indexes:

• 0 - chrom: The name of the reference genome the primers are indexed to
• 1 - chromStart: 0-based inclusive start coordinate
• 2 - chromEnd: 0-based non-inclusive end coordinate
• 3 - primer-name: Name of each primer
• 4 - pool: The pool each primer should be added into. 1 based.
• 5 - strand: Either + (forward) or - (reverse) primer
• 6 - primer-sequence: The 5'-3' sequence of the primer

V1

Depreciated

This was the original 6-col bedfile used very early in PrimalSchemes development, which excluded primer-sequence.

V2

Legacy

This uses the 7 columns described above, alongside V1:primernames.

• Single chrom (reference) is expected
• No header lines

V3

Current

This uses the 7 columns described above, alongside V2:primernames.

• Multiple chrom (references)
• Circular primers are allowed. The start of x_LEFT can be greater than the end of x_RIGHT
• Header lines are used. Denoted with the # character

Scheme Version

In the form of v{Major}.{Minor}.{Patch}

• Major: New scheme being generated with different input params
• Minor: Change to primers. Either additional / removal of primers
• Patch: No change to primers. Often used for rebalancing or change in formatting

Regex:

VERSION_PATTERN = r"^v\d+\.\d+\.\d+$"

Scheme Name

Must only contain a-z, 0-9, and -. Cannot start or end with -

Regex:

SCHEMENAME_PATTERN = r"^[a-z0-9][a-z0-9-]*[a-z0-9]$"

CLI

primal-page

Usage:

$ primal-page [OPTIONS] COMMAND [ARGS]...

Options:

• --version
• --install-completion: Install completion for the current shell.
• --show-completion: Show completion for the current shell, to copy it or customize the installation.
• --help: Show this message and exit.

Commands:

• aliases: Manage aliases
• build-index: Build an index.json file from all schemes...
• create: Create a new scheme in the required format
• dev: Development commands
• download: Download schemes from the index.json
• modify: Modify an existing scheme's metadata...
• remove: Remove a scheme's version from the repo,...
• validate: Validate a scheme

primal-page aliases

Manage aliases

Usage:

$ primal-page aliases [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• add: Add an alias:schemename to the alias file
• remove: Remove an alias from the alias file.

primal-page aliases add

Add an alias:schemename to the alias file

Usage:

$ primal-page aliases add [OPTIONS] ALIASES_FILE ALIAS SCHEMENAME

Arguments:

• ALIASES_FILE: The path to the alias file to write to [required]
• ALIAS: The alias to add [required]
• SCHEMENAME: The schemename the alias refers to [required]

Options:

• --help: Show this message and exit.

primal-page aliases remove

Remove an alias from the alias file. Does nothing if the alias does not exist

Usage:

$ primal-page aliases remove [OPTIONS] ALIASES_FILE ALIAS

Arguments:

• ALIASES_FILE: The path to the alias file to write to [required]
• ALIAS: The alias to add [required]

Options:

• --help: Show this message and exit.

primal-page build-index

Build an index.json file from all schemes in the directory

Usage:

$ primal-page build-index [OPTIONS]

Options:

• --gitaccount TEXT: The name of the github account [default: quick-lab]
• --gitserver TEXT: The name of the github server [default: https://github.com/]
• --parentdir PATH: The parent directory [default: .]
• --git-commit-sha TEXT: The git commit
• --force / --no-force: Force the creation of the index.json [default: no-force]
• --help: Show this message and exit.

primal-page create

Create a new scheme in the required format

Usage:

$ primal-page create [OPTIONS]

Options:

• --schemename TEXT: The name of the scheme [required]
• --ampliconsize INTEGER RANGE: Amplicon size [x>=100; required]
• --schemeversion TEXT: Scheme version, default is parsed from config.json [required]
• --species INTEGER: The species this scheme targets. Please use NCBI taxonomy ids [required]
• --authors TEXT: Any authors. To provide multiple, use --authors '1' --authors '2' [required]
• --primerbed PATH: The path to the primer.bed file [required]
• --reference PATH: The path to the reference.fasta file [required]
• --schemestatus [withdrawn|deprecated|autogenerated|draft|tested|validated]: Scheme status [default: draft]
• --citations TEXT: Any associated citations. Please use DOI
• --output PATH: Where to output the scheme [default: primerschemes]
• --configpath PATH: Where the config.json file is located
• --algorithmversion TEXT: The version of primalscheme or other
• --description TEXT: A description of the scheme
• --derivedfrom TEXT: Which scheme has this scheme been derived from
• --primerclass [primerschemes]: The primer class [default: primerschemes]
• --collection [ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP]: The collection tags. To provide multiple, use --collection '1' --collection '2'
• --link-protocol TEXT: Optional link to protocol
• --link-validation TEXT: Optional link to validation data
• --links-homepage TEXT: Optional link to homepage
• --link-vendor TEXT: Optional link to vendors
• --link-misc TEXT: Optional miscellaneous link
• --fix / --no-fix: Attempt to fix the scheme [default: no-fix]
• --contact-info TEXT: Contact information
• --additional-files PATH: Additional files to include in the ./work directory
• --ref-select <TEXT PATH>...: Reference selection file. In the form 'chromosome file'
• --help: Show this message and exit.

primal-page download

Download schemes from the index.json

Usage:

$ primal-page download [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• all: Download all schemes from the index.json
• scheme: Download a scheme from the index.json

primal-page download all

Download all schemes from the index.json

Usage:

$ primal-page download all [OPTIONS]

Options:

• --output PATH: The directory the primerschemes dir will be created in [required]
• --index-url TEXT: The URL to the index.json [default: https://raw.githubusercontent.com/quick-lab/primerschemes/main/index.json]
• --help: Show this message and exit.

primal-page download scheme

Download a scheme from the index.json

Usage:

$ primal-page download scheme [OPTIONS] SCHEMENAME AMPLICONSIZE SCHEMEVERSION

Arguments:

• SCHEMENAME: The name of the scheme [required]
• AMPLICONSIZE: Amplicon size [required]
• SCHEMEVERSION: Scheme version [required]

Options:

• --output PATH: The directory the primerschemes dir will be created in [required]
• --index-url TEXT: The URL to the index.json [default: https://raw.githubusercontent.com/quick-lab/primerschemes/main/index.json]
• --help: Show this message and exit.

primal-page modify

Modify an existing scheme's metadata (info.json)

Usage:

$ primal-page modify [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• author-add: Append an author to the authors list in...
• author-remove: Remove an author from the authors list in...
• author-reorders: Reorder the authors in the info.json file
• citation-add: Append an citation to the authors list in...
• citation-remove: Remove an citation form the authors list...
• collection-add: Add a Collection to the Collection list in...
• collection-remove: Remove an Collection from the Collection...
• contactinfo: Change the contactinfo field in the info.json
• derivedfrom: Replaces the derivedfrom in the info.json...
• description: Replaces the description in the info.json...
• license: Replaces the license in the info.json file
• link-add: Add a link to the selected link field to...
• link-remove: Add a link to the selected link field to...
• primerclass: Change the primerclass field in the info.json
• refselect-add: Add a refselect file to the info.json
• refselect-remove: Removes a refselect file from the info.json
• regenerate: Validates the info.json and regenerate the...
• status: Change the status field in the info.json

primal-page modify author-add

Append an author to the authors list in the info.json file

Usage:

$ primal-page modify author-add [OPTIONS] SCHEMEINFO AUTHOR

Arguments:

• SCHEMEINFO: The path to info.json [required]
• AUTHOR: The author to add [required]

Options:

• --author-index INTEGER: The 0-based index to insert the author at. Default is the end [required]
• --help: Show this message and exit.

primal-page modify author-remove

Remove an author from the authors list in the info.json file

Usage:

$ primal-page modify author-remove [OPTIONS] SCHEMEINFO AUTHOR

Arguments:

• SCHEMEINFO: The path to info.json [required]
• AUTHOR: The author to remove [required]

Options:

• --help: Show this message and exit.

primal-page modify author-reorders

Reorder the authors in the info.json file

Usage:

$ primal-page modify author-reorders [OPTIONS] SCHEMEINFO [AUTHOR_INDEX]

Arguments:

• SCHEMEINFO: The path to info.json [required]
• [AUTHOR_INDEX]: The indexes in the new order, separated by spaces. e.g. 1 0 2. Any indexes not provided will be appended to the end

Options:

• --help: Show this message and exit.

primal-page modify citation-add

Append an citation to the authors list in the info.json file

Usage:

$ primal-page modify citation-add [OPTIONS] SCHEMEINFO CITATION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CITATION: The citation to add [required]

Options:

• --help: Show this message and exit.

primal-page modify citation-remove

Remove an citation form the authors list in the info.json file

Usage:

$ primal-page modify citation-remove [OPTIONS] SCHEMEINFO CITATION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CITATION: The citation to remove [required]

Options:

• --help: Show this message and exit.

primal-page modify collection-add

Add a Collection to the Collection list in the info.json file

Usage:

$ primal-page modify collection-add [OPTIONS] SCHEMEINFO COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}: The Collection to add [required]

Options:

• --help: Show this message and exit.

primal-page modify collection-remove

Remove an Collection from the Collection list in the info.json file

Usage:

$ primal-page modify collection-remove [OPTIONS] SCHEMEINFO COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• COLLECTION:{ARTIC|MODJADJI|QUICK-LAB|COMMUNITY|WASTE-WATER|CLINICAL-ISOLATES|WHOLE-GENOME|PANEL|MULTI-TARGET|VARVAMP}: The Collection to remove [required]

Options:

• --help: Show this message and exit.

primal-page modify contactinfo

Change the contactinfo field in the info.json

Usage:

$ primal-page modify contactinfo [OPTIONS] SCHEMEINFO CONTACTINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CONTACTINFO: The contact information for this scheme. Use 'None' to remove the contact info [required]

Options:

• --help: Show this message and exit.

primal-page modify derivedfrom

Replaces the derivedfrom in the info.json file

Usage:

$ primal-page modify derivedfrom [OPTIONS] SCHEMEINFO DERIVEDFROM

Arguments:

• SCHEMEINFO: The path to info.json [required]
• DERIVEDFROM: The new derivedfrom. Use 'None' to remove the derivedfrom [required]

Options:

• --help: Show this message and exit.

primal-page modify description

Replaces the description in the info.json file

Usage:

$ primal-page modify description [OPTIONS] SCHEMEINFO DESCRIPTION

Arguments:

• SCHEMEINFO: The path to info.json [required]
• DESCRIPTION: The new description. Use 'None' to remove the description [required]

Options:

• --help: Show this message and exit.

primal-page modify license

Replaces the license in the info.json file

Usage:

$ primal-page modify license [OPTIONS] SCHEMEINFO LICENSE

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LICENSE: The new license. Use 'None' show the work is not licensed (Not recommended) [required]

Options:

• --help: Show this message and exit.

primal-page modify link-add

Add a link to the selected link field to the info.json

Usage:

$ primal-page modify link-add [OPTIONS] SCHEMEINFO LINKFIELD LINK

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LINKFIELD: The link field to add to. protocols, validation, homepage, vendors, misc [required]
• LINK: The link to add. [required]

Options:

• --help: Show this message and exit.

primal-page modify link-remove

Add a link to the selected link field to the info.json

Usage:

$ primal-page modify link-remove [OPTIONS] SCHEMEINFO LINKFIELD LINK

Arguments:

• SCHEMEINFO: The path to info.json [required]
• LINKFIELD: The link field to remove from. protocols, validation, homepage, vendors, misc [required]
• LINK: The link to remove. [required]

Options:

• --help: Show this message and exit.

primal-page modify primerclass

Change the primerclass field in the info.json

Usage:

$ primal-page modify primerclass [OPTIONS] SCHEMEINFO PRIMERCLASS:{primerschemes}

Arguments:

• SCHEMEINFO: The path to info.json [required]
• PRIMERCLASS:{primerschemes}: The primerclass to change to [required]

Options:

• --help: Show this message and exit.

primal-page modify refselect-add

Add a refselect file to the info.json

Usage:

$ primal-page modify refselect-add [OPTIONS] SCHEMEINFO CHROM REFSELECT

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CHROM: The chromosome to add [required]
• REFSELECT: The reference selection file [required]

Options:

• --help: Show this message and exit.

primal-page modify refselect-remove

Removes a refselect file from the info.json

Usage:

$ primal-page modify refselect-remove [OPTIONS] SCHEMEINFO CHROM

Arguments:

• SCHEMEINFO: The path to info.json [required]
• CHROM: The chromosome to remove [required]

Options:

• --help: Show this message and exit.

primal-page modify regenerate

Validates the info.json and regenerate the README.md

Usage:

$ primal-page modify regenerate [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.

primal-page modify status

Change the status field in the info.json

Usage:

$ primal-page modify status [OPTIONS] SCHEMEINFO [SCHEMESTATUS]:[withdrawn|deprecated|autogenerated|draft|tested|validated]

Arguments:

• SCHEMEINFO: The path to info.json [required]
• [SCHEMESTATUS]:[withdrawn|deprecated|autogenerated|draft|tested|validated]: The scheme class [default: draft]

Options:

• --help: Show this message and exit.

primal-page remove

Remove a scheme's version from the repo, will also remove size and schemename directories if empty

Usage:

$ primal-page remove [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.

primal-page validate

Validate a scheme

Usage:

$ primal-page validate [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• all-schemes: Validate all schemes in a directory.
• scheme: Validate a single scheme

primal-page validate all-schemes

Validate all schemes in a directory. Calls the scheme command for each scheme and checks for final ref-select files.

Usage:

$ primal-page validate all-schemes [OPTIONS] DIRECTORY

Arguments:

• DIRECTORY: The path to the directory [required]

Options:

• --help: Show this message and exit.

primal-page validate scheme

Validate a single scheme

Usage:

$ primal-page validate scheme [OPTIONS] SCHEMEINFO

Arguments:

• SCHEMEINFO: The path to info.json [required]

Options:

• --help: Show this message and exit.