Requested: info on CI test runners

[abelbraaksma]

I got stuck for a few days trying to understand what goes on in the CI runners. The thing is, some language features and library functions behave differently between x86, x64, RyuJIT, LegacyJIT and x86 JIT (based on Legacy JIT), Mono, dotnet and .NET Framework, Linux vs Mac vs Windows, debug vs release, JIT optimizations on or off, AOT through ngen or otherwise, .NET 5 vs the rest.

Probably all these can be answered if we knew what the CI jobs run in:

• Do we use most recent RyuJIT 64, or the RTM version?
• Is any result NGEN'ed before tests are run?
• Are VS tests run against latest VS Preview or other?
• Are any tests run in x86 mode?
• For MacOS and Linux CI jobs, I'd assume Mono 64 bit, is this correct?
• Is Mono run on any Windows machine?
• Is anything run as Debug build or with the checked runtime?
• What are the commandlines used for the ultimate run (i.e., can we mimic this locally)?

Code generation, emitted JIT and emitted IL can be different between versions of JIT and frameworks, this influences many tests.

The Test Guide has just this line:

Note that for compatibility reasons, the IDE unit tests should be run in a 32-bit process, using the --x86 flag to nunit3-console.exe

But that's only the IDE unit tests...

This all boils down to a simple question: when looking at the CI, and/or looking at the logs, it would be very helpful if one could determine what system it is running on.

I'll gladly update the test guide to help future contributors understanding what's what w.r.t. the CI.

PS: here's the mess this can cause when running into the "runs perfectly fine locally but not on CI" situation: #9516.

[abelbraaksma]

I've again hit this while working on #9622. The TLDR of this request is: what .NET Core and .NET Framework versions, and what JIT versions are used for compiling and running code and tests on CI?

There appear to be subtle differences that lead to "runs locally, but not on CI" problems, and it would be nice to catch them earlier. I'd be happy to update the CONTRIBUTOR.md guide with this info.

[abelbraaksma]

Hmm, going through a recent build log showed a link of what's installed on the VM: https://github.com/actions/virtual-environments/blob/win19/20200621.1/images/win/Windows2019-Readme.md.

Of note is: Visual Studio Enterprise 2019, version 16.6.30204.135.

And literally every .NET Core SDK is available, since 2.1. Surprisingly, only one .NET Framework, so that answers that part of the questions:

And this line in the log suggests Release build, at least for one of my builds:

Posting job info: { "QueueId": "Windows_NT", "Source": "pr/dotnet/fsharp/refs/pull/9622/merge", "Type": "build/product/", "Build": "20200704.11", "Attempt": "1", "Properties": { "operatingSystem": "Windows_NT", "configuration": "Release" } }

And lines like this suggest that it is indeed not the same as using build.cmd or Build from VS:

[...]\dotnet.EXE" restore FSharp.Compiler.Service.sln -v n (In: false, Out: false, Err: false)

[cartermp]

@KevinRansom and @brettfo know the most about this.

[KevinRansom]

fcs build came from the community, one of these days we will subsume into our build proper.

[cartermp]

Now the FCS build is no longer, which is nice