Building under Windows

[marshalc]

[Windows] 'cp' is not recognised - npm install issue

Hi folks,

We've just tripped over a small detail whilst introducing a new team member to our hugo+docsy build process. Whilst our CI processes are linux based, and so are are some devs (or Mac in my case), more of our team coming on board are Windows based, and they're having this problem.

We've traced the issue to a modification at 1966f9604c89875c7bfd83b66fbfee4128297ccf which was introduced a couple of months ago. Specifically package.json, line 11:

"_cp:bs-rfs": "cp -R node_modules/bootstrap/scss/vendor/* assets/_vendor/bootstrap/scss/",

Running an npm install on a Windows machine (from either PowerShell or CMD) results in an error of:

Now, I appreciate this is use using pre-release versions of the platform - but I wasn't sure if anyone else had found this yet.

The common answer to solving this is to monkey patch the code for Windows users to use copy or similar (and change the slashes in the command string too it seems); however this then breaks the code for the *nix users, so we don't like this option.

The next answer has been to get our Windows users to either install Windows Subsystem for Linux (WSL) and use bash there, or the git-bash that comes from with Git-for-windows as part of the mingw32 package.

What I'm wondering is if anyone else has tripped on this, and if so, have they come up with a multi-platform solution yet?

[deining]

What I'm wondering is if anyone else has tripped on this, and if so, have they come up with a multi-platform solution yet?

Yes a somehow similiar issue was raised some time ago. In this context the cross-env package was identified as a potential multi-platform solution. In your case, I guess the shx package can be used to author a multi-platform solution. Can you try this package out, and potentially provide a pull request that solves your issue?

[marshalc]

They look like potential pathways to a cross platform answer, thanks - we can take a look at that and let you know what we find. If we can get it to work then we'll propose a PR fix.

[chalin]

Hi @marshalc. Just so I can better understand your use case: are you just trying to use Docsy as a dependency, or are you also attempting to built it yourselves?

[chalin]

@marshalc - Also, can you tell me how your are bringing in Docsy: NPM, Git submodule, or Hugo modules? Thx

[marshalc]

Thanks @deining - the shx solution appears to be working on the linux/Mac setups. I'm awaiting one of my windows-based colleagues testing it next week to confirm we're happy with it. The edit being tested is tiny (install shx, modify the one line in package.json to prepend shx to the cp line). You can see the edit at ouhft@f044c34

Hi @chalin - we're using Docsy in what I think is a typical theme dependancy use case in our various project's for documentation needs. Essentially it's project based docs, built with Hugo, themed by a lightly modified Docsy theme (see https://github.com/ouhft/docsy/tree/ouhft) added as a git sub-module (linked to that repo branch), and published via an internal Gitlab Pages build process to an internal only webserver (which unfortunately means showing build examples a tad more challenging though you can see the CI process in that link).

HTH

[chalin]

Thanks for the details @marshalc and the links to your project. Could you move shx into devDependencies instead?

My main problem is that I don't have access to a Windows environment so I can't validate the solutions I'd like to experiment with 🤷🏼‍♂️. (I'm looking forward to the likes of GitPod support for Windows :))

I finally started work on adding smoke tests (#1776), and I was able to validate (as I suspected) that Docsy can be used just fine under Windows. You're building Docsy, so that's another story.

I'd rather not add another pkg dependency if I can avoid it. I'm beginning to think that moving that prep step (involving the cp) into a bash script instead would be a simpler workaround. Would that work for you?

[chalin]

Ok, we now have smoke and built tests successfully passing under *nix and Windows. In particular you'll notice from this run https://github.com/google/docsy/actions/runs/7512943762/job/20454181915, that the cp command doesn't seem to cause a problem:

Can you give it a try @marshalc? If it doesn't work for you, I'm unsure what tooling GH might be using that is different from yours. Maybe WSL?

[marshalc]

Thanks for the updates - this response is a bit of a placeholder as in trying to investigate this compared to the path we were on has opened up a can of worms, and we're now in several rabbit holes (he says, badly mixing metaphors) trying to work out a clean approach that allows us to test your approach and then work out how it might apply in our context.

We did a quick test of our shx patch using one of our Windows desktops and that worked - but as you note, that adds quite a range of new package dependancies, so we'd like to be able to stay closer to the core approach and introduce as few divergencies for our branch as possible.

Our first challenge is we're using the git submodule rather than Hugo module approach and this seems to change the overall build and dependancies structures. We were unable to drop in the plain Docsy theme into any of our projects in place of our customised version (and are still trying to determine why), so testing the latest docsy:main hasn't worked - on either our Mac or Windows desktops.

To test some basic setup, we've been winding back to whether we can run the docsy:main user guide build in a clean download.. and we've had that work (but only if the local git repo folder is explicitly called docsy and not our usual docs-repo - so we're trying to find out why that specific path name appears to be a limit.

Our next step when we can return to this will be to try this first step on Windows desktop machines as well as my macOS one... then we work forward again, though I think that means we'll need to work out how to repackage our Docsy branch as a Hugo module first, before we can get to parity with you. (pointers welcome!)

One thing I noted from your output is that GH appears to be forcing the use of PowerShell in that setup, whereas npm on our local Windows desktop was running those sub-processes in cmd (even though it was called from a PowerShell prompt). The ins and outs of this are presently obscure to me.

[chalin]

Hi @marshalc, thanks for the updates.

Our first challenge is we're using the git submodule

All of my Hugo+Docsy-based projects in production are using Docsy via a git submodule. So, if you have specific questions about using Docsy in this way, let me know and I'll do my best to answer them.

To test some basic setup, we've been winding back to whether we can run the docsy:main user guide build in a clean download.. and we've had that work

Good.
Does that include Windows builds of the user guide?

(but only if the local git repo folder is explicitly called docsy and not our usual docs-repo - so we're trying to find out why that specific path name appears to be a limit.

You're hitting a Hugo requirement: if a Hugo project is configured to use a theme named abc, then Hugo will look for a folder with the same name under the themes directory. For details, see Theme components. So, if you put your Docsy theme in a folder named docs-repo, then it won't find it. You have two options: change the theme name to match the folder name or vice versa.

repackage our Docsy branch as a Hugo module first

That shouldn't be necessary -- we officially support use of Docsy as a git submodule.

forcing the use of PowerShell

Yes, that appears to be key.

---

Btw, since Docsy and the User Guide are successfully building under GH actions, I'm going to move this into the discussions area was we continue to explore your issues of building under Windows. Also note that I've tweaked the title.

[deining]

I added some mount points to hugo.yaml. After that change, I could drop _cp:bs-rfs from package.json. See #1797 for details.

@marshalc: once you ported these changes to your repo, it should work both on Linux and Windows machines. Can you give this a try, please?

[marshalc]

Thank you both - we're back to looking at this again today. First step, establish a working baseline for comparison with yourselves.

Test case: install the docsy main branch (ouhft@d99b473), and build the user guide.

Two desktop machines being used:

• macOS (14.2.1)
• windows 10 enterprise (21H2)¹

We noted the PowerShell difference, so we updated the default (v5?) PowerShell app from the Win10 build to the latest 7.4 release to see if that was a differentiator - spoiler alert, it was not.

Hugo versions installed:

• mac: hugo v0.121.2-6d5b44305eaa9d0a157946492a6f319da38de154+extended darwin/amd64 BuildDate=2024-01-05T12:21:15Z VendorInfo=brew
• pc: hugo v0.121.1-00b46fed8e47f7bb0a85d7cfc2d9f1356379b740+extended windows/amd64 BuildDate=2023-12-08T08:47:45Z VendorInfo=gohugoio

Assuming local path now: ~/Projects/github/ouhft/docsy/²

Download the main docsy project git clone --depth 1 https://github.com/ouhft/docsy.git ./docsy-repo

Cloning into './docsy-repo'...
remote: Enumerating objects: 333, done.
remote: Counting objects: 100% (333/333), done.
remote: Compressing objects: 100% (299/299), done.
remote: Total 333 (delta 25), reused 191 (delta 11), pack-reused 0
Receiving objects: 100% (333/333), 1.89 MiB | 15.21 MiB/s, done.
Resolving deltas: 100% (25/25), done.

Move into user guide: cd docsy-repo/userguide/

Now try the npm install...

Mac: works as expected, i.e.:

npm install

> prepare
> cd .. && npm install


> docsy@0.8.1-dev.0-unreleased postinstall
> npm run _mkdir:hugo-mod


> docsy@0.8.1-dev.0-unreleased _mkdir:hugo-mod
> npx mkdirp ../github.com/FortAwesome/Font-Awesome ../github.com/twbs/bootstrap


> docsy@0.8.1-dev.0-unreleased prepare
> npm run _cp:bs-rfs


> docsy@0.8.1-dev.0-unreleased _cp:bs-rfs
> cp -R node_modules/bootstrap/scss/vendor/* assets/_vendor/bootstrap/scss/


added 168 packages, and audited 169 packages in 27s

53 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

added 75 packages, and audited 76 packages in 32s

19 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

PC: the same issue we had originally when logging the Issue...

npm install

> prepare
> cd .. && npm install


> docsy@0.8.1-dev.0-unreleased postinstall
> npm run _mkdir:hugo-mod


> docsy@0.8.1-dev.0-unreleased _mkdir:hugo-mod
> npx mkdirp ../github.com/FortAwesome/Font-Awesome ../github.com/twbs/bootstrap


> docsy@0.8.1-dev.0-unreleased prepare
> npm run _cp:bs-rfs


> docsy@0.8.1-dev.0-unreleased _cp:bs-rfs
> cp -R node_modules/bootstrap/scss/vendor/* assets/_vendor/bootstrap/scss/

'cp' is not recognized as an internal or external command,
operable program or batch file.
npm ERR! code 1
npm ERR! path C:\Users\Joseph.Baboo\Projects\github\ouhft\docsy\docsy-repo
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c npm run _cp:bs-rfs

npm ERR! A complete log of this run can be found in: C:\Users\Joseph.Baboo\AppData\Local\npm-cache\_logs\2024-01-24T11_32_02_719Z-debug-0.log
npm ERR! code 1
npm ERR! path C:\Users\Joseph.Baboo\Projects\github\ouhft\docsy\docsy-repo\userguide
npm ERR! command failed
npm ERR! command C:\Windows\system32\cmd.exe /d /s /c cd .. && npm install

npm ERR! A complete log of this run can be found in: C:\Users\Joseph.Baboo\AppData\Local\npm-cache\_logs\2024-01-24T11_31_52_610Z-debug-0.log

@chalin - I think this suggests that the GH Windows build definitely has some other implicit/internal configurations setup to ensure the processes stay in PowerShell and don't move to cmd as appears to be happening in this npm process. We'll look at @deining's suggestion next from this setup and see if that works for us and report back...

Re: issue vs discussion - quite understand, this could very well be an "us" problem and you've got to work with the tools at your disposal, so this may be of limited to use to a wider cross-platform compatibility. Happy to engage either way.

Re: git submodule vs Hugo module - since that's likely an unrelated change to this issue, we've parked that for another time, thanks for the pointer.

Re: the folder paths

• For this test you can see our local setup from the above; the git repo itself is distinguished from the parent project name folder so that we avoid conversations about "which docsy folder?" that occur when the path looks like /google/docsy/docsy/content/ etc
• For when we include the theme as a sub module, you're quite right, we do that so that it ends up in project-folder/docs/themes/docsy/ (where docs is also the Hugo root location) as we recognise that the folder name in the themes folder needs to match the yaml config name of the theme

Footnotes

1. Corporate maintained build, so we recognise this isn't the latest Win10 (which ought to be the final version of 22H2), but this is what most of our developers are working on (I'm a rare case of using my own hardware) ↩

2. or ~\Projects\github\ouhft\docsy\ for those playing on windows ;-) ↩

[marshalc]

@chalin - I also have the sense that I'm missing some understanding between your distinction of "using" docsy opposed to "building" docsy for each project that includes our modified theme... Is there something you could explain/expand-on/point-me-at that might help me there please?

[marshalc]

Good news for @deining - we can confirm your approach works on our https://github.com/ouhft/docsy/tree/npm-no-cp-test (manual copy of your branch)

Output from our PC test:

npm install     

> prepare
> cd .. && npm install


> docsy@0.8.1-dev.0-unreleased postinstall
> npm run _mkdir:hugo-mod


> docsy@0.8.1-dev.0-unreleased _mkdir:hugo-mod
> npx mkdirp ../github.com/FortAwesome/Font-Awesome ../github.com/twbs/bootstrap


added 168 packages, and audited 169 packages in 26s

53 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

added 74 packages, and audited 75 packages in 36s

19 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

and then a successful serve:

npm run serve  

> serve
> npm run _serve


> _serve
> npm run _hugo-dev -- --minify serve


> _hugo-dev
> npm run _hugo -- -e dev -DFE --minify serve


> _hugo
> hugo --cleanDestinationDir --themesDir ../.. -e dev -DFE --minify serve

Watching for changes in C:\Users\Joseph.Baboo\Projects\github\ouhft\docsy\docsy-repo\{assets,i18n,layouts,node_modules,package.json,postcss.config.js,static,userguide}
Watching for config changes in C:\Users\Joseph.Baboo\Projects\github\ouhft\docsy\docsy-repo\userguide\hugo.yaml, C:\Users\Joseph.Baboo\Projects\github\ouhft\docsy\docsy-repo\hugo.yaml
Start building sites … 
hugo v0.120.4-f11bca5fec2ebb3a02727fb2a5cfb08da96fd9df+extended windows/amd64 BuildDate=2023-11-08T11:18:07Z VendorInfo=gohugoio


                   | EN  
-------------------+-----
  Pages            | 82
  Paginator pages  |  0
  Non-page files   |  8
  Static files     | 35
  Processed images |  5
  Aliases          |  3
  Sitemaps         |  1
  Cleaned          |  0

Built in 1991 ms
Environment: "dev"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)

So this (like the shx option) provides another viable path for us - but does it more closely align with your wishes @chalin?

[chalin]

Hi @marshalc - thanks for all the details. Yes I had high hopes for @deining's proposal but, in the end, it falls short of what we need.

I'll get back to you as soon as I can. In the meantime, could you let me know if #1806 works in your environment? Thanks.

[chalin]

Riding on the idea of shx, I opted for another NPM package instead, hopefully this will allow both build and use of Docsy to work under Windows:

• Use cpy-cli for OS-independent file copy #1810

@marshalc can you confirm? (If this works, then I won't need to further explain build vs use :)).

[marshalc]

Thanks @chalin - I'm just trying to pick up on this this week and hopefully will get some time with my colleagues to test