build based on 1c03d17

dev/command_dependency/index.html

@@ -64,4 +64,4 @@
 
     wrap_up = (inputs, outputs) -> run(`$SAMTOOLS index $(outputs["BAM"])`)
 )</code></pre><p>To run the program, we can decide whether to check dependencies with <code>run(..., check_dependencies=true)</code>:</p><pre><code class="language-julia">inputs = Dict(&quot;FASTQ&quot; =&gt; &quot;a.fastq&quot;, &quot;REF&quot; =&gt; &quot;ref.fasta&quot;)
-success, outputs = run(program_bowtie2, inputs; check_dependencies = true)</code></pre></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../julia_program/">« Julia Program</a><a class="docs-footer-nextpage" href="../API/">API »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <span class="colophon-date" title="Wednesday 14 June 2023 21:36">Wednesday 14 June 2023</span>. Using Julia version 1.7.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
+success, outputs = run(program_bowtie2, inputs; check_dependencies = true)</code></pre></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../julia_program/">« Julia Program</a><a class="docs-footer-nextpage" href="../API/">API »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <span class="colophon-date" title="Thursday 4 January 2024 16:36">Thursday 4 January 2024</span>. Using Julia version 1.7.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>

dev/command_program/index.html

@@ -122,4 +122,4 @@
     append::Bool=false
 ) -&gt; (success::Bool, outputs::Dict{String})</code></pre><ul><li><code>program_kwargs...</code> include elements in <code>p.inputs</code> and <code>p.outputs</code></li><li>Other keyword arguments are related to run. Details can be found at <a href="../API/#Base.run"><code>run</code></a>.</li></ul><div class="admonition is-compat"><header class="admonition-header">Compat</header><div class="admonition-body"><p>The old methods &lt; v0.8 still work, which put program&#39;s arguments in <code>inputs::Dict{String}</code> and <code>outputs::Dict{String}</code>:</p><pre><code class="language-julia">success, outputs = run(p::Program, inputs, outputs; run_kwargs...)
 # only usable when outputs have default values.
-success, outputs = run(p::Program, inputs; run_kwargs...)</code></pre></div></div><div class="admonition is-warning"><header class="admonition-header">Argument mod::Module</header><div class="admonition-body"><p><code>Expr</code>ressions will be evaluated to functions in <code>mod</code>. Please use <code>mod = @__MODULE__</code> to prevent precompilation fail when defining the program within a package.</p></div></div><div class="admonition is-warning"><header class="admonition-header">Warning</header><div class="admonition-body"><p>Redirecting and directory change in Julia are not thread safe, so unexpected redirection and directory change might be happen if you are running programs in different <code>Tasks</code> or multi-thread mode.</p></div></div><div class="admonition is-compat"><header class="admonition-header">Compatibility with JobSchedulers.jl</header><div class="admonition-body"><p>Pipelines.jl is fully compatible with <a href="https://github.com/cihga39871/JobSchedulers.jl">JobSchedulers.jl</a> which is a Julia-based job scheduler and workload manager inspired by Slurm and PBS.</p><p><code>run(::Program, ...)</code> can be replaced by <code>JobSchedulers.Job(::Program, ...)</code>. The latter creates a <code>Job</code>, and you can submit the job to queue by using <code>submit!(::Job)</code>.</p><p><code>arg_forward</code> (an argument of <a href="../API/#Pipelines.CmdProgram-Tuple{}"><code>CmdProgram</code></a>) is used to forward user-defined inputs/outputs to specific keyword arguments of <code>JobSchedulers.Job(::Program, ...)</code>, including <code>name::String</code>, <code>user::String</code>, <code>ncpu::Int</code>, <code>mem::Int</code>.</p></div></div><p>The explanation of arguments is in the next section.</p><h2 id="Workflow"><a class="docs-heading-anchor" href="#Workflow">Workflow</a><a id="Workflow-1"></a><a class="docs-heading-anchor-permalink" href="#Workflow" title="Permalink"></a></h2><ol><li><p>Go to the working directory. Establish redirection. (<code>dir</code>, <code>stdout</code>, <code>stderr</code>, <code>stdlog</code>, <code>append</code>).</p></li><li><p><strong>Check keywords consistency:</strong> Inputs/outputs keywords should be consistent in both <code>p::CmdProgram</code> and <code>run(p; inputs..., outputs...)</code>.</p><blockquote><p>For example, if inputs and outputs in <code>p::CmdProgram</code> is defined this way</p><pre><code class="language-julia">p = CmdProgram(..., inputs = [&quot;I&quot;, &quot;J&quot;], outputs = [&quot;K&quot;])</code></pre><p>You have to provide all I, J and K:</p><pre><code class="language-julia">run(p; I = something, J = something, K = something)</code></pre></blockquote></li><li><p>Print info about starting program.</p><blockquote><p>The content can set by <code>p.info_before::String</code>.</p><p>Disable: <code>run(..., verbose=false)</code></p></blockquote></li><li><p>Check whether the program ran successfully before. If so, return <code>(true, outputs::Dict{String})</code> without running it.</p><blockquote><p><strong>How does the program know it ran before?</strong></p><p>a. <code>run(..., skip_when_done=true)</code> skip running the program if it has been done before.</p><p>b. Run id file stores files information. File will be compared to determine re-run or not. You can use <code>run(..., touch_run_id_file=false)</code> to skip creating the run id file. Details of run id file can be found at <a href="../API/#Pipelines.create_run_id_file"><code>Pipelines.create_run_id_file</code></a></p><p>c. <code>p.validate_outputs(outputs)</code> run successfully without returning <code>false</code>.</p></blockquote></li><li><p>Check command dependencies (<code>CmdDependency</code>).</p><blockquote><p>Disable: <code>run(..., check_dependencies=false)</code></p><p>Read <strong>Command Dependency</strong> portion for details.</p></blockquote></li><li><p>Remove the run id file if exists.</p></li><li><p>Validate inputs. (<code>p.validate_inputs</code>)</p></li><li><p>Preparing the main command.</p><blockquote><p>If you specify <code>run(...; stdout=something, stderr=something, append::Bool)</code>, the command (<code>cmd</code>) will be wrapped with <code>pipeline(cmd; stdout=something, stderr=something, append::Bool)</code>. If <code>cmd</code> has its own file redirection, the outer wrapper may not work as you expect.</p></blockquote></li><li><p>Meet prerequisites. (<code>p.prerequisites</code>)</p><blockquote><p>It is the last code before running the main command.  For example, you can create a directory if  the main command cannot create itself.</p></blockquote></li><li><p>Run the main command.</p></li><li><p>Validate outputs. (<code>p.validate_outputs</code>)</p></li><li><p>Run the wrap up code. (<code>p.wrap_up</code>)</p><blockquote><p>It is the last code to do after-command jobs. For example, you can delete intermediate files if necessary.</p></blockquote></li><li><p>Create run id file if <code>run(..., touch_run_id_file=true)</code>. Read Step 4 for details.</p></li><li><p>Print info about finishing program.</p><blockquote><p>The content can set by <code>p.info_after::String</code>.</p><p>Disable: <code>run(..., verbose=false)</code></p><p>Simple info: <code>run(..., verbose=min)</code></p></blockquote></li><li><p>Return <code>(success::Bool, outputs{String})</code></p></li></ol><div class="admonition is-info"><header class="admonition-header">Dry Run</header><div class="admonition-body"><p><code>run(..., dry_run=true)</code> will return <code>(mature_command::AbstractCmd, run_id_file::String)</code> instead.</p></div></div></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../">« Home</a><a class="docs-footer-nextpage" href="../julia_program/">Julia Program »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <span class="colophon-date" title="Wednesday 14 June 2023 21:36">Wednesday 14 June 2023</span>. Using Julia version 1.7.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
+success, outputs = run(p::Program, inputs; run_kwargs...)</code></pre></div></div><div class="admonition is-warning"><header class="admonition-header">Argument mod::Module</header><div class="admonition-body"><p><code>Expr</code>ressions will be evaluated to functions in <code>mod</code>. Please use <code>mod = @__MODULE__</code> to prevent precompilation fail when defining the program within a package.</p></div></div><div class="admonition is-warning"><header class="admonition-header">Warning</header><div class="admonition-body"><p>Redirecting and directory change in Julia are not thread safe, so unexpected redirection and directory change might be happen if you are running programs in different <code>Tasks</code> or multi-thread mode.</p></div></div><div class="admonition is-compat"><header class="admonition-header">Compatibility with JobSchedulers.jl</header><div class="admonition-body"><p>Pipelines.jl is fully compatible with <a href="https://github.com/cihga39871/JobSchedulers.jl">JobSchedulers.jl</a> which is a Julia-based job scheduler and workload manager inspired by Slurm and PBS.</p><p><code>run(::Program, ...)</code> can be replaced by <code>JobSchedulers.Job(::Program, ...)</code>. The latter creates a <code>Job</code>, and you can submit the job to queue by using <code>submit!(::Job)</code>.</p><p><code>arg_forward</code> (an argument of <a href="../API/#Pipelines.CmdProgram-Tuple{}"><code>CmdProgram</code></a>) is used to forward user-defined inputs/outputs to specific keyword arguments of <code>JobSchedulers.Job(::Program, ...)</code>, including <code>name::String</code>, <code>user::String</code>, <code>ncpu::Int</code>, <code>mem::Int</code>.</p></div></div><p>The explanation of arguments is in the next section.</p><h2 id="Workflow"><a class="docs-heading-anchor" href="#Workflow">Workflow</a><a id="Workflow-1"></a><a class="docs-heading-anchor-permalink" href="#Workflow" title="Permalink"></a></h2><ol><li><p>Go to the working directory. Establish redirection. (<code>dir</code>, <code>stdout</code>, <code>stderr</code>, <code>stdlog</code>, <code>append</code>).</p></li><li><p><strong>Check keywords consistency:</strong> Inputs/outputs keywords should be consistent in both <code>p::CmdProgram</code> and <code>run(p; inputs..., outputs...)</code>.</p><blockquote><p>For example, if inputs and outputs in <code>p::CmdProgram</code> is defined this way</p><pre><code class="language-julia">p = CmdProgram(..., inputs = [&quot;I&quot;, &quot;J&quot;], outputs = [&quot;K&quot;])</code></pre><p>You have to provide all I, J and K:</p><pre><code class="language-julia">run(p; I = something, J = something, K = something)</code></pre></blockquote></li><li><p>Print info about starting program.</p><blockquote><p>The content can set by <code>p.info_before::String</code>.</p><p>Disable: <code>run(..., verbose=false)</code></p></blockquote></li><li><p>Check whether the program ran successfully before. If so, return <code>(true, outputs::Dict{String})</code> without running it.</p><blockquote><p><strong>How does the program know it ran before?</strong></p><p>a. <code>run(..., skip_when_done=true)</code> skip running the program if it has been done before.</p><p>b. Run id file stores files information. File will be compared to determine re-run or not. You can use <code>run(..., touch_run_id_file=false)</code> to skip creating the run id file. Details of run id file can be found at <a href="../API/#Pipelines.create_run_id_file"><code>Pipelines.create_run_id_file</code></a></p><p>c. <code>p.validate_outputs(outputs)</code> run successfully without returning <code>false</code>.</p></blockquote></li><li><p>Check command dependencies (<code>CmdDependency</code>).</p><blockquote><p>Disable: <code>run(..., check_dependencies=false)</code></p><p>Read <strong>Command Dependency</strong> portion for details.</p></blockquote></li><li><p>Remove the run id file if exists.</p></li><li><p>Validate inputs. (<code>p.validate_inputs</code>)</p></li><li><p>Preparing the main command.</p><blockquote><p>If you specify <code>run(...; stdout=something, stderr=something, append::Bool)</code>, the command (<code>cmd</code>) will be wrapped with <code>pipeline(cmd; stdout=something, stderr=something, append::Bool)</code>. If <code>cmd</code> has its own file redirection, the outer wrapper may not work as you expect.</p></blockquote></li><li><p>Meet prerequisites. (<code>p.prerequisites</code>)</p><blockquote><p>It is the last code before running the main command.  For example, you can create a directory if  the main command cannot create itself.</p></blockquote></li><li><p>Run the main command.</p></li><li><p>Validate outputs. (<code>p.validate_outputs</code>)</p></li><li><p>Run the wrap up code. (<code>p.wrap_up</code>)</p><blockquote><p>It is the last code to do after-command jobs. For example, you can delete intermediate files if necessary.</p></blockquote></li><li><p>Create run id file if <code>run(..., touch_run_id_file=true)</code>. Read Step 4 for details.</p></li><li><p>Print info about finishing program.</p><blockquote><p>The content can set by <code>p.info_after::String</code>.</p><p>Disable: <code>run(..., verbose=false)</code></p><p>Simple info: <code>run(..., verbose=min)</code></p></blockquote></li><li><p>Return <code>(success::Bool, outputs{String})</code></p></li></ol><div class="admonition is-info"><header class="admonition-header">Dry Run</header><div class="admonition-body"><p><code>run(..., dry_run=true)</code> will return <code>(mature_command::AbstractCmd, run_id_file::String)</code> instead.</p></div></div></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../">« Home</a><a class="docs-footer-nextpage" href="../julia_program/">Julia Program »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> on <span class="colophon-date" title="Thursday 4 January 2024 16:37">Thursday 4 January 2024</span>. Using Julia version 1.7.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>