Add documentation and examples for configuring pipelines (#780)

* add documentation and examples for configuring pipelines * update useful commands to include updating pnpm * revised sentences and spelling
phac-nml · Oct 1, 2024 · 650af6e · 650af6e
1 parent c8ecd03
commit 650af6e
Show file tree

Hide file tree

Showing 2 changed files with 167 additions and 10 deletions.
diff --git a/docs-site/docs/configuration/pipelines.md b/docs-site/docs/configuration/pipelines.md
@@ -4,38 +4,55 @@ id: pipelines
 title: Registering Pipelines
 ---
 
-Pipelines must be registered in IRIDA Next to be able to launch workflows. They are registered when the server is started up and Future restarts of the server will only register new pipelines that are added to the configuration file, and schema files for a particular pipeline will update if newer versions are available in the repository.
+Pipelines are registered when the IRIDA Next server is started up.  Future restarts of the server will only register new pipelines that are added to the configuration file, and schema files for a particular pipeline will update if newer versions are available in the repository.
 
 ## Pipeline repository
 
-Currently only **Nextflow** pipelines are supported and they must have a Github repository. Each pipeline is required to have a `nextflow_schema.json` file at the top level of the repository, and a `schema_input.json` file under an `assets` directory within the repository.
+Currently, only **Nextflow** pipelines are supported and they must have a GitHub repository. Each pipeline is required to have a `nextflow_schema.json` file at the top level of the repository, and a `schema_input.json` file under an `assets` directory within the repository.
 
 ## Setup
 
+### Configuration
+
 The configuration file to register pipelines is in `json` format and stored in the `config/pipelines/` directory with the name `pipelines.json`.
 
-This `pipelines.json` file should be in the format below and must include the following:
+This `pipelines.json` file should be in the format below and can include the following:
+
+- **URL** *(Required)* of the pipeline GitHub repository
+- **name** *(Required)* of the pipeline
+- **description** *(Required)* of the pipeline
+- **versions** *(Required)* of the pipeline that should be available to launch.
+  - `name`: *(Required)* refers to the `-r` flag used by nextflow.
+  - `automatable`: *(Optional)* `true` or `false` to specify if the pipeline can be automated.
+  - `executable`: *(Optional)* `true` or `false` to specify if the pipeline is able to be executed. When set to `false`, the pipeline will not be listed to the user.
+- **overrides** *(Optional)* for the pipeline
 
-- **URL** of the pipeline Github repository
-- **name** of the pipeline
-- **description** of the pipeline
-- **versions** of the pipeline that should be available to launch. **Note:** The `name` in the versions refers to the `-r` flag used by nextflow.
+#### Example
 
 ```json
 [
   {
     "url": "https://github.com/phac-nml/iridanextexample",
     "name": "phac-nml/iridanextexample",
     "description": "IRIDA Next Example Pipeline",
+    "overrides": {
+      # SEE OVERRIDE SECTION BELOW
+    },
     "versions": [
       {
-        "name": "1.0.2"
+        "name": "1.0.2",
+        "automatable": true,
+        "executable": true
       },
       {
-        "name": "1.0.1"
+        "name": "1.0.1",
+        "automatable": true,
+        "executable": false
       },
       {
-        "name": "1.0.0"
+        "name": "1.0.0",
+        "automatable": false,
+        "executable": true
       }
     ]
   },
@@ -44,3 +61,142 @@ This `pipelines.json` file should be in the format below and must include the fo
   }
 ]
 ```
+
+### Schema Overrides
+
+The Overrides section can be used to change anything within the original nextflow pipeline schema. Anything within the `"overrides": {<json data>}` will overwrite the original schema with `<json data>` starting at the highest level.
+
+In the below example, we will override the database connection options so we can connect the pipeline to our custom database path. Note that only the overridden fields need to be provided, as everything else provided by the schema stays the same.
+
+#### Example schema
+
+```json
+{
+    "$schema": "http://example.com/schema",
+    "$id": "https://example.com/nextflow_schema.json",
+    "title": "My Example Schema",
+    "description": "Example Schema: for demonstrating overrides",
+    "type": "object",
+    "definitions": {
+        "input_output_options": {
+            "title": "Input/Output Options",
+            "type": "object",
+            "description": "Define which data to use with the pipeline.",
+            "required": ["input", "outdir"],
+            "properties": {
+                "input": {
+                    ...
+                },
+                "outdir": {
+                    ...
+                },
+                "database": {
+                  "type": "string",
+                  "description": "Kraken DB",
+                  "enum": [
+                    [
+                      "default_db",
+                      "PATH_TO_DB"
+                    ],
+                    [
+                      "organization db",
+                      "PATH_TO_ORG_DB"
+                    ]
+                  ]
+                }
+            }
+        },
+        "more options": {
+          ...
+        }
+    },
+    "more options": {
+      ...
+    }
+}
+```
+
+#### Example override
+
+```json
+[
+  {
+    "url": "https://github.com/phac-nml/iridanextexample",
+    "name": "phac-nml/iridanextexample",
+    "description": "IRIDA Next Example Pipeline",
+    "overrides": {
+      "definitions": {
+        "input_output_options": {
+          "properties": {
+            "database": {
+              "enum": [
+                [
+                  "custom_db",
+                  "PATH_TO_CUSTOM_DB"
+                ],
+                [
+                  "custom_db_2",
+                  "PATH_TO_CUSTOM_DB_2"
+                ]
+              ]
+            }
+          }
+        }
+      }
+    },
+    "versions": [...]
+  },
+  {
+    ........
+  }
+]
+```
+
+#### Effective Result
+
+
+```json
+{
+    "$schema": "http://example.com/schema",
+    "$id": "https://example.com/nextflow_schema.json",
+    "title": "My Example Schema",
+    "description": "Example Schema: for demonstrating overrides",
+    "type": "object",
+    "definitions": {
+        "input_output_options": {
+            "title": "Input/Output Options",
+            "type": "object",
+            "description": "Define which data to use with the pipeline.",
+            "required": ["input", "outdir"],
+            "properties": {
+                "input": {
+                    ...
+                },
+                "outdir": {
+                    ...
+                },
+                "database": {
+                  "type": "string",
+                  "description": "Kraken DB",
+                  "enum": [
+                    [
+                      "custom_db",
+                      "PATH_TO_CUSTOM_DB"
+                    ],
+                    [
+                      "custom_db_2",
+                      "PATH_TO_CUSTOM_DB_2"
+                    ]
+                  ]
+                }
+            }
+        },
+        "more options": {
+          ...
+        }
+    },
+    "more options": {
+      ...
+    }
+}
+```
diff --git a/docs-site/docs/development/useful_commands.md b/docs-site/docs/development/useful_commands.md
@@ -35,6 +35,7 @@ bin/rails graphql:dump_schema
 
 ```bash
 cd docs-site
+pnpm update
 pnpm build
 npm run serve
 ```