[FEATURE] Add 'baketext' support where appropiate

[oliv3r]

Using variables with kicad is great, we can put in our schematic things such as $COMMENT1 or $GIT_VSC_TAG (which is generated in preflight.

However, these tags are 'replaced' with the actual text in the git repository that holds the kicad files, which is 'logical'. One (complex) way of dealing with this is a pre-commit script that would replace the value when you create a git tag; but if you update the code, you'd have to re-update this tag, becoming quickly cumbersome.

KiKit introduces a nice feature called baketext, that will expand variables when it creates new panel.kicad_pcb.

This is currently not done for other things kibot generates, such as present, navigate or kicanvas. Therefor I recommend that we get a flag (or even do it as default), where variables get expanded in these 'releases'.

Probably worth a different feature request, but we'd also could probably use to generate a simple 'dumb' output, that basically just copies (with baked text) the schematic and pcb files. One thing that's subtle here, other then 'why not just do sed in > outdir/pcb.sch, is that we don't know all files to copy, more specifically, not all schematics (with hierarchical schematics).E.g. when invoking kibot with --schematic root.kicad_sch --board board.kicad_pcb, kibot (kicad) will know exactly which schematic and board files to use, e.g. kicanvas etc all have the correct files included. Thus having a 'copy' output that does all preflight and other processing is definitely useful.

[set-soft]

Please explain it better.

1. The set_text_variables can change the variables in a permanent way, which means storing them in the project. What else do you want?
2. Usually you don't commit SCH/PCB files after running KiBot, so I fail to understand how any change will be permanent.

Please describe an step by step workflow for:

1. The "cumbersome" situation
2. The situation after the feature is implemented

BTW: You can already copy the schematic and or PCB using KiBot.

[oliv3r]

Please explain it better.

1. The _set_text_variables_ can change the variables in a permanent way, which means storing them in the project. What else do you want?

I actually do NOT want to store them in the project; the git repository is to be (at all times obviously) untouched. For the obvious 'dirty-ness' reason, but also we won't commit changes as part of the pipeline anyway.

2. Usually you don't commit SCH/PCB files after running KiBot, so I fail to understand how any change will be permanent.

That's an interesting point; but I also have 'restore' in my config set. So one work-around would indeed be, do not restore the files, but copy them after kibot as an output.

Please describe an step by step workflow for:

1. The "cumbersome" situation

Manually (or using kibot) change the text variables (but also the built-in kicad variables). e.g. whenever $COMMENT1 is used, we'd have to manually change that in the sch/pcb files.

I suppose when manually doing this, one would; parse project file (or root schematic), and then all linked schematics run sed 's|$VAR|content|g' parsedfile.kicad_ > outdir/release.kicad_.

2. The situation after the feature is implemented

We have a job running (build, deploy whatever) which will resolve all variables to static texts, those generated by kibot; but also those generated/populated by kicad. The generated files are now placed as an output into the output folder.

Again, pretty much what is done with kibot present, navigate or kicanvas.

BTW: You can already copy the schematic and or PCB using KiBot.
But not using just an output? I'll research this option :)

[set-soft]

I suppose when manually doing this, one would; parse project file (or root schematic), and then all linked schematics run sed 's|$VAR|content|g' parsedfile.kicad_ > outdir/release.kicad_.

But this equivalent to what the (deprecated) sch_replace and pcb_replace preflights do.

I still don't understand why is this needed. You should invest some time in creating a very simple example and KiBot configuration showing the use case and what is the exact problem with what we currently have.

[oliv3r]

So I tried the copy_from; and copy_files needs a list of files, which I don't know, but kibot does.

The simple example my pipeline template in itself. https://gitlab.com/ci-includes/kici. Here, the pipeline runs, and generates all sorts of outputs; see for example the output of the latest tag; https://gitlab.com/ci-includes/kici/-/jobs/7999345181/artifacts/browse where all generated outputs are shown. What's missing however, is the schematics and pcb files, with prefilled variables, because those are in the repository. However when sharing these files (output) you don't get the pcb files or schematics (well you do, because they are part of kicanvas, present or navigate, IF you use those outputs). So having an output that just does the pcb files would be enough to be able to 'release' them and offer them for easy sharing.

Actually kicanvas is more complete then present or navigate, as that also includes the project file. Granted, you do have to explicitly configure kicanvas to do so, so a generic 'kicad' output probably should just do that anyway ...

[set-soft]

So I tried the copy_from; and copy_files needs a list of files, which I don't know, but kibot does.

Look at the ExportProject template.

The simple example my pipeline template in itself. https://gitlab.com/ci-includes/kici.

Nope, this isn't a simple example.

Here, the pipeline runs, and generates all sorts of outputs;

And hence isn't a simple example

see for example the output of the latest tag; https://gitlab.com/ci-includes/kici/-/jobs/7999345181/artifacts/browse where all generated outputs are shown. What's missing however, is the schematics and pcb files, with prefilled variables, because those are in the repository.

The variables are in the project, as long as you copy the project you'll get the values.

However when sharing these files (output) you don't get the pcb files or schematics (well you do, because they are part of kicanvas, present or navigate, IF you use those outputs).

ExportProject is used to share your project.

So having an output that just does the pcb files would be enough to be able to 'release' them and offer them for easy sharing.

Ok, is already implemented, you also sch_variant and pcb_variant. But they are good only if the other person has the 3D models.

Actually kicanvas is more complete then present or navigate,

?!!
I can't understand how kicanvas is more complete than navigate, which can include kicanvas

as that also includes the project file.

Like the outputs I mentioned before.

Granted, you do have to explicitly configure kicanvas to do so, so a generic 'kicad' output probably should just do that anyway ...

[oliv3r]

So I tried the copy_from; and copy_files needs a list of files, which I don't know, but kibot does.

Look at the ExportProject template.

It was a bit hard to find, but yes, that's exactly what I want, the secret, which is undocumented, is

    options:
      files:
        - source_type: project

This seems to 'auto-detect' what files to upload.

The simple example my pipeline template in itself. https://gitlab.com/ci-includes/kici.

Nope, this isn't a simple example.

I guess we have to agree to disagree, while I can agree with that it's many examples, they are all each simple.

Here, the pipeline runs, and generates all sorts of outputs;

And hence isn't a simple example

it generates ONE output, per pipeline step, so one simple template is run. This is by design to make sure things try to stay simple, rather then having one major template with everything.

see for example the output of the latest tag; https://gitlab.com/ci-includes/kici/-/jobs/7999345181/artifacts/browse where all generated outputs are shown. What's missing however, is the schematics and pcb files, with prefilled variables, because those are in the repository.

The variables are in the project, as long as you copy the project you'll get the values.

Cool, my issue was in not knowing what files to supply; but the magic project file makes it work as I want :)

Just a little bit of 'concern' I have, is that the variables are stored in the project, they are not resolved into the silk-screen, like kikit's baketext. (btw kicanvas job also doesn't do that).

So then the feature request kind-of is still accurate, that during project export, we want to bake variables into the silk-screen. E.g. when I share ONLY the kicad_pcb file with someone else, without the project file, it should still be accurate and 'just work'. This is exactly what baketext afaik does. (for all layers/places, not just silk-screen of course, also worksheet etc).

However when sharing these files (output) you don't get the pcb files or schematics (well you do, because they are part of kicanvas, present or navigate, IF you use those outputs).

ExportProject is used to share your project.

👍

So having an output that just does the pcb files would be enough to be able to 'release' them and offer them for easy sharing.

Ok, is already implemented, you also sch_variant and pcb_variant. But they are good only if the other person has the 3D models.

They are part of the repo; which copy_files already takes care off :)

Kibot is actually really good in doing even more; it copies the symbol and footprints as well; impressive.

Just not sure why it's adding fp-lib-table and sym-lib-table as those are always generated anyway ..

ironically, they do not even work :p and throw an error (when generating the 3D render)

Expecting 'a symbol or number' in '/tmp/outputs/world/export/fp-lib-table', line 2, offset 31

Actually kicanvas is more complete then present or navigate,

?!! I can't understand how kicanvas is more complete than navigate, which can include kicanvas

Navigate doesn't include the kicad_pro file

as that also includes the project file.

Like the outputs I mentioned before.
yep indeed :)

Granted, you do have to explicitly configure kicanvas to do so, so a generic 'kicad' output probably should just do that anyway ...

[set-soft]

So I tried the copy_from; and copy_files needs a list of files, which I don't know, but kibot does.

Look at the ExportProject template.

It was a bit hard to find, but yes, that's exactly what I want, the secret, which is undocumented, is

Oops, the FileList class overlapped with other 2 classes and the 1.8.x docs made only one visible.
Now you can see the doc for the class used there.

    options:
      files:
        - source_type: project

This seems to 'auto-detect' what files to upload.

We have an internal template with this functionality (documented) and Quick Start uses it.

The simple example my pipeline template in itself. https://gitlab.com/ci-includes/kici.

Nope, this isn't a simple example.

I guess we have to agree to disagree, while I can agree with that it's many examples, they are all each simple.

But where we don't agree is in the time. You are looking for help, so you should make it easy to help you. Analyzing tons of simple files is time consuming.

Here, the pipeline runs, and generates all sorts of outputs;

And hence isn't a simple example

it generates ONE output, per pipeline step, so one simple template is run. This is by design to make sure things try to stay simple, rather then having one major template with everything.

And becomes really hard to track. When I say simple I mean just simple. I get the files, put them in a folder and run the command you say. Just one minute, otherwise I must spend 30 minutes trying to figure out what to do. If you want help make it simple for other to help you.

see for example the output of the latest tag; https://gitlab.com/ci-includes/kici/-/jobs/7999345181/artifacts/browse where all generated outputs are shown. What's missing however, is the schematics and pcb files, with prefilled variables, because those are in the repository.

The variables are in the project, as long as you copy the project you'll get the values.

Cool, my issue was in not knowing what files to supply; but the magic project file makes it work as I want :)

Just a little bit of 'concern' I have, is that the variables are stored in the project, they are not resolved into the silk-screen, like kikit's baketext. (btw kicanvas job also doesn't do that).

So then the feature request kind-of is still accurate, that during project export, we want to bake variables into the silk-screen. E.g. when I share ONLY the kicad_pcb file with someone else, without the project file, it should still be accurate and 'just work'. This is exactly what baketext afaik does. (for all layers/places, not just silk-screen of course, also worksheet etc).

The problem here is how can we know we are replacing everything ...
Also: KiCad has a mechanism for this without destroying the actual values, this is a cache to use when the variable is missing.

However when sharing these files (output) you don't get the pcb files or schematics (well you do, because they are part of kicanvas, present or navigate, IF you use those outputs).

ExportProject is used to share your project.

👍

So having an output that just does the pcb files would be enough to be able to 'release' them and offer them for easy sharing.

Ok, is already implemented, you also sch_variant and pcb_variant. But they are good only if the other person has the 3D models.

They are part of the repo; which copy_files already takes care off :)

Kibot is actually really good in doing even more; it copies the symbol and footprints as well; impressive.

Just not sure why it's adding fp-lib-table and sym-lib-table as those are always generated anyway ..

ironically, they do not even work :p and throw an error (when generating the 3D render)

Expecting 'a symbol or number' in '/tmp/outputs/world/export/fp-lib-table', line 2, offset 31

Actually kicanvas is more complete then present or navigate,

?!! I can't understand how kicanvas is more complete than navigate, which can include kicanvas

Navigate doesn't include the kicad_pro file

Are you really talking about the navigate_results output?

as that also includes the project file.

Like the outputs I mentioned before.
yep indeed :)

Granted, you do have to explicitly configure kicanvas to do so, so a generic 'kicad' output probably should just do that anyway ...

[oliv3r]

So I tried the copy_from; and copy_files needs a list of files, which I don't know, but kibot does.

Look at the ExportProject template.

It was a bit hard to find, but yes, that's exactly what I want, the secret, which is undocumented, is

Oops, the FileList class overlapped with other 2 classes and the 1.8.x docs made only one visible. Now you can see the doc for the class used there.

Ahh! That's why then :)

    options:
      files:
        - source_type: project

This seems to 'auto-detect' what files to upload.

We have an internal template with this functionality (documented) and Quick Start uses it.

The simple example my pipeline template in itself. https://gitlab.com/ci-includes/kici.

Nope, this isn't a simple example.

I guess we have to agree to disagree, while I can agree with that it's many examples, they are all each simple.

But where we don't agree is in the time. You are looking for help, so you should make it easy to help you. Analyzing tons of simple files is time consuming.

Fair enough; Next time I'll paste a 1 liner to reproduce things :)

Here, the pipeline runs, and generates all sorts of outputs;

And hence isn't a simple example

it generates ONE output, per pipeline step, so one simple template is run. This is by design to make sure things try to stay simple, rather then having one major template with everything.

And becomes really hard to track. When I say simple I mean just simple. I get the files, put them in a folder and run the command you say. Just one minute, otherwise I must spend 30 minutes trying to figure out what to do. If you want help make it simple for other to help you.

Obviously! I'll make it easy!

see for example the output of the latest tag; https://gitlab.com/ci-includes/kici/-/jobs/7999345181/artifacts/browse where all generated outputs are shown. What's missing however, is the schematics and pcb files, with prefilled variables, because those are in the repository.

The variables are in the project, as long as you copy the project you'll get the values.

Cool, my issue was in not knowing what files to supply; but the magic project file makes it work as I want :)
Just a little bit of 'concern' I have, is that the variables are stored in the project, they are not resolved into the silk-screen, like kikit's baketext. (btw kicanvas job also doesn't do that).
So then the feature request kind-of is still accurate, that during project export, we want to bake variables into the silk-screen. E.g. when I share ONLY the kicad_pcb file with someone else, without the project file, it should still be accurate and 'just work'. This is exactly what baketext afaik does. (for all layers/places, not just silk-screen of course, also worksheet etc).

The problem here is how can we know we are replacing everything ... Also: KiCad has a mechanism for this without destroying the actual values, this is a cache to use when the variable is missing.

I suppose 'just like kikit baketext does? for _file in $files; for _var in $vars; do sed -i 's|_var|$_var|g' $_file|g; done; done covers probably 99% of usecases?

However when sharing these files (output) you don't get the pcb files or schematics (well you do, because they are part of kicanvas, present or navigate, IF you use those outputs).

ExportProject is used to share your project.

👍

So having an output that just does the pcb files would be enough to be able to 'release' them and offer them for easy sharing.

Ok, is already implemented, you also sch_variant and pcb_variant. But they are good only if the other person has the 3D models.

They are part of the repo; which copy_files already takes care off :)
Kibot is actually really good in doing even more; it copies the symbol and footprints as well; impressive.
Just not sure why it's adding fp-lib-table and sym-lib-table as those are always generated anyway ..
ironically, they do not even work :p and throw an error (when generating the 3D render)

Expecting 'a symbol or number' in '/tmp/outputs/world/export/fp-lib-table', line 2, offset 31

Actually kicanvas is more complete then present or navigate,

?!! I can't understand how kicanvas is more complete than navigate, which can include kicanvas

Navigate doesn't include the kicad_pro file

Are you really talking about the navigate_results output?

Whatever produces this output: https://ci-includes.gitlab.io/kici/world/navigate/world-navigate.html where the kicad_pro is well hidden ;)

[set-soft]

I suppose 'just like kikit baketext does? for _file in $files; for _var in $vars; do sed -i 's|_var|$_var|g' $_file|g; done; done covers probably 99% of usecases?

KiKit doesn´t do. Doing it you risk to make the file invalid.

Whatever produces this output: https://ci-includes.gitlab.io/kici/world/navigate/world-navigate.html where the kicad_pro is well hidden ;)

Please read the docs to understand what this output does. In fact it can include the kicanvas and copy_files, so it can't be less than any of them.

[oliv3r]

I suppose 'just like kikit baketext does? for _file in $files; for _var in $vars; do sed -i 's|_var|$_var|g' $_file|g; done; done covers probably 99% of usecases?

KiKit doesn´t do. Doing it you risk to make the file invalid.

So then kibot does it? https://kibot.readthedocs.io/en/latest/configuration/outputs/PanelizeLayout.html baketext is what I'm referring to.

Whatever produces this output: https://ci-includes.gitlab.io/kici/world/navigate/world-navigate.html where the kicad_pro is well hidden ;)

Please read the docs to understand what this output does. In fact it can include the kicanvas and copy_files, so it can't be less than any of them.

I meant that kicanvas will include file.kicad_pro (and file.kicad_sch and file.kicad_pcb) whereas navigate only includes the later two. I didn't find file.kicad_pro in the GUI; I didn't see it in the output either https://gitlab.com/ci-includes/kici/-/jobs/8122456128/artifacts/browse/outputs/world/navigate/ so that's where the comment comes from. With the exportProject template/copy_files I do have it in the archive, so now navigate should be able to pick it up; but I did need exportProject first for that to work ;)

[set-soft]

So then kibot does it? https://kibot.readthedocs.io/en/latest/configuration/outputs/PanelizeLayout.html baketext is what I'm referring to.

Nope, KiKit, but it does something completely different to what you describe.

Whatever produces this output: https://ci-includes.gitlab.io/kici/world/navigate/world-navigate.html where the kicad_pro is well hidden ;)

Please read the docs to understand what this output does. In fact it can include the kicanvas and copy_files, so it can't be less than any of them.

I meant that kicanvas will include file.kicad_pro (and file.kicad_sch and file.kicad_pcb) whereas navigate only includes the later two.

This is not true.

1. navigate_results can include kicanvas, so anything included by kicanvas will be included in navigate_results.
2. navigate_results can include a project export, in fact it does it when using --quick-start, which includes the ExportProject template.
3. navigate_results is used to show what other outputs generated.

I didn't find file.kicad_pro in the GUI; I didn't see it in the output either https://gitlab.com/ci-includes/kici/-/jobs/8122456128/artifacts/browse/outputs/world/navigate/ so that's where the comment comes from.

Because you didn't read the docs to discover what navigate_results does.

With the exportProject template/copy_files I do have it in the archive, so now navigate should be able to pick it up; but I did need exportProject first for that to work ;)

Again something you should get from the docs.

[set-soft]

I still don't get the main use for such a feature.

First time you mentioned it I though it was related to your manual panelization process

[oliv3r]

So then kibot does it? https://kibot.readthedocs.io/en/latest/configuration/outputs/PanelizeLayout.html baketext is what I'm referring to.

Nope, KiKit, but it does something completely different to what you describe.

Hmm, what kitkit baketext seems to do, is just 'sed' all the variables into the value fields. E.g. if I have a silk-screen text, that has as value ${COMMENT1} kikit will replace that text with whatever the value was, e.g. it resolves the variable. But I've only used/seen this for panels.

This is important to me, when sharing just the kicad_pcb file, because it contains only resolved variables. Without this feature, the PCB now reads ${COMMENT1} on the silkscreen, as there is no kicad_pro to resolve the variable from.

Even more so important, because I actually put git describe in a variable, which by design, always changes. However during 'release' of a PCB, we want to 'bake the text' into the actual value.

Whatever produces this output: https://ci-includes.gitlab.io/kici/world/navigate/world-navigate.html where the kicad_pro is well hidden ;)

Please read the docs to understand what this output does. In fact it can include the kicanvas and copy_files, so it can't be less than any of them.

I meant that kicanvas will include file.kicad_pro (and file.kicad_sch and file.kicad_pcb) whereas navigate only includes the later two.

This is not true.

1. _navigate_results_ can include _kicanvas_, so anything included by _kicanvas_ will be included in _navigate_results_.

Yes, it CAN, but does not by default. E.g. I must include kicanvas, to get the kicanvas files. Without this include, I will not get the file.kicad_pro.

2. _navigate_results_ can include a project export, in fact it does it when using _--quick-start_, which includes the _ExportProject_ template.

Indeed, IF I include the ExportProject template (or the appropiate copy_files) output, then navigate will include those outputs. This was what I was missing and what made me chase this partial rabbit whole. I needed to learn about ExportProject/copy_files first :) So thank you for that!

3. _navigate_results_ is used to show what other outputs generated.

I didn't find file.kicad_pro in the GUI; I didn't see it in the output either https://gitlab.com/ci-includes/kici/-/jobs/8122456128/artifacts/browse/outputs/world/navigate/ so that's where the comment comes from.

Because you didn't read the docs to discover what navigate_results does.

No, I did, but was hoping/assuming as a basis, it would include the schematic and the PCB files by default. Where my confusion came from, is that I include a bunch of things, one being the pdf print, which has been so ingrained into my memory, I treated navigate as always having 'at least the schematic and pcb' files, which is not even true, it's the PDF's of these, and only because I have them in my list for navigate. I first needed to learn about ExportProject to also find the project included.

With the exportProject template/copy_files I do have it in the archive, so now navigate should be able to pick it up; but I did need exportProject first for that to work ;)

Again something you should get from the docs.

But you already told me before, it wasn't in the docs :p Or rather, exportProject is listed very briefly and not exactly clearly (unless you know what to expect) and copy_files was missing the documentation as you explained.

So this part is fully resolved now with your help, this is all not a problem any longer, except for

I still don't get the main use for such a feature.

First time you mentioned it I though it was related to your manual panelization process

When doing panelization, I found out about kikit's baketext feature, which is imo quite important when doing a 'release' of a PCB.

E.g. I have a preflight that does git describe to dynamically generate a variable for the revision. I tag my repository with the files to mark a release, the pipeline runs with kibot and runs exportProject. However because variables are not resolved, the file.kicad_pcb can not be shared without the file.kicad_pro as the project file holds the variables. Obviously, when doing a copy_files step, I want an option to baketext so that all variables are resolved.