Unable to load service index when running the bootstrapper from SSH shell

[glopesdev]

There seems to be some permission or NuGet client configuration when bootstrapping under SSH shell at least on some clusters. Full error stack trace below:

NuGet.Protocol.Core.Types.FatalProtocolException: Unable to load the service index for source https://www.myget.org/F/bonsai/api/v3/index.json. ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /nfs/nhome/live/goncalolopes/mono-6.12.0.205/external/boringssl/ssl/handshake_client.c:1132
  at Mono.Btls.MonoBtlsContext.ProcessHandshake () [0x00048] in <c315ecb2688f4c48b4b9d7e5dad29eb0>:0
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) [0x000da] in <c315ecb2688f4c48b4b9d7e5dad29eb0>:0
  at (wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
  at Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) [0x00006] in <c315ecb2688f4c48b4b9d7e5dad29eb0>:0
  at Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) [0x000fc] in <c315ecb2688f4c48b4b9d7e5dad29eb0>:0
   --- End of inner exception stack trace ---
  at Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) [0x00262] in <c315ecb2688f4c48b4b9d7e5dad29eb0>:0
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x000c3] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
   --- End of inner exception stack trace ---
  at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore (System.IO.Stream stream, System.Net.Security.SslClientAuthenticationOptions sslOptions, System.Threading.CancellationToken cancellationToken) [0x00102] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.HttpConnectionPool.CreateConnectionAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00322] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync (System.Threading.Tasks.ValueTask`1[TResult] creationTask) [0x000a2] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.HttpConnectionPool.SendWithRetryAsync (System.Net.Http.HttpRequestMessage request, System.Boolean doRequestAuth, System.Threading.CancellationToken cancellationToken) [0x00089] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.AuthenticationHelper.SendWithAuthAsync (System.Net.Http.HttpRequestMessage request, System.Uri authUri, System.Net.ICredentials credentials, System.Boolean preAuthenticate, System.Boolean isProxyAuth, System.Boolean doRequestAuth, System.Net.Http.HttpConnectionPool pool, System.Threading.CancellationToken cancellationToken) [0x00112] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.RedirectHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ba] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at System.Net.Http.DecompressionHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ca] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at NuGet.Protocol.ServerWarningLogHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0007f] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.StsAuthenticationHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000ce] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.HttpSourceAuthenticationHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x000e2] in <86d667fdac92404981c495ad3b503d7a>:0
  at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered (System.Threading.Tasks.Task`1[TResult] sendTask, System.Net.Http.HttpRequestMessage request, System.Threading.CancellationTokenSource cts, System.Boolean disposeCts) [0x000b3] in <5301e18e18904c4c9eb0e9e862fbdf4d>:0
  at NuGet.Protocol.HttpRetryHandler+<>c__DisplayClass5_1.<SendAsync>b__0 (System.Threading.CancellationToken timeoutToken) [0x000a9] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.TimeoutUtility.StartWithTimeout[T] (System.Func`2[T,TResult] getTask, System.TimeSpan timeout, System.String timeoutMessage, System.Threading.CancellationToken token) [0x001bd] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.HttpRetryHandler.SendAsync (NuGet.Protocol.HttpRetryHandlerRequest request, System.String source, NuGet.Common.ILogger log, System.Threading.CancellationToken cancellationToken) [0x00a8a] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.HttpSource.GetThrottledResponse (System.Func`1[TResult] requestFactory, System.TimeSpan requestTimeout, System.TimeSpan downloadTimeout, System.Int32 maxTries, System.Boolean isRetry, System.Boolean isLastAttempt, System.Guid sessionId, NuGet.Common.ILogger log, System.Threading.CancellationToken cancellationToken) [0x001fe] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.HttpSource+<>c__DisplayClass15_0`1[T].<GetAsync>b__0 (System.Threading.CancellationToken lockedToken) [0x00287] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T] (System.String filePath, System.Func`2[T,TResult] action, System.Threading.CancellationToken token) [0x002b8] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T] (System.String filePath, System.Func`2[T,TResult] action, System.Threading.CancellationToken token) [0x0037b] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.HttpSource.GetAsync[T] (NuGet.Protocol.HttpSourceCachedRequest request, System.Func`2[T,TResult] processAsync, NuGet.Common.ILogger log, System.Threading.CancellationToken token) [0x000e7] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.ServiceIndexResourceV3Provider.GetServiceIndexResourceV3 (NuGet.Protocol.Core.Types.SourceRepository source, System.DateTime utcNow, NuGet.Common.ILogger log, System.Threading.CancellationToken token) [0x0026a] in <86d667fdac92404981c495ad3b503d7a>:0
   --- End of inner exception stack trace ---
  at NuGet.Protocol.ServiceIndexResourceV3Provider.GetServiceIndexResourceV3 (NuGet.Protocol.Core.Types.SourceRepository source, System.DateTime utcNow, NuGet.Common.ILogger log, System.Threading.CancellationToken token) [0x00300] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.ServiceIndexResourceV3Provider.TryCreate (NuGet.Protocol.Core.Types.SourceRepository source, System.Threading.CancellationToken token) [0x00204] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T] (System.Threading.CancellationToken token) [0x000b3] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.DependencyInfoResourceV3Provider.TryCreate (NuGet.Protocol.Core.Types.SourceRepository source, System.Threading.CancellationToken token) [0x0007f] in <86d667fdac92404981c495ad3b503d7a>:0
  at NuGet.Protocol.Core.Types.SourceRepository.GetResourceAsync[T] (System.Threading.CancellationToken token) [0x000b3] in <86d667fdac92404981c495ad3b503d7a>:0
  at Bonsai.NuGet.PackageManager.GetPackageDependencies (System.String packageId, NuGet.Versioning.VersionRange versionRange, NuGet.Frameworks.NuGetFramework projectFramework, NuGet.Protocol.Core.Types.SourceCacheContext cacheContext, System.Collections.Generic.IEnumerable`1[T] repositories, System.Collections.Generic.IDictionary`2[TKey,TValue] availablePackages, NuGet.Common.ILogger logger, System.Boolean ignoreDependencies, System.Threading.CancellationToken token) [0x0009f] in <86d667fdac92404981c495ad3b503d7a>:0
  at Bonsai.NuGet.PackageManager.InstallPackageAsync (NuGet.Packaging.Core.PackageIdentity package, NuGet.Frameworks.NuGetFramework projectFramework, System.Boolean ignoreDependencies, System.Threading.CancellationToken token) [0x0031c] in <86d667fdac92404981c495ad3b503d7a>:0
  at Bonsai.NuGet.PackageHelper.StartInstallPackage (Bonsai.NuGet.IPackageManager packageManager, System.String packageId, NuGet.Versioning.NuGetVersion version, NuGet.Frameworks.NuGetFramework projectFramework) [0x000c2] in <86d667fdac92404981c495ad3b503d7a>:0
  at Bonsai.Configuration.Bootstrapper+<>c__DisplayClass7_0.<RunAsync>g__RestoreEditorPackage|1 () [0x000bd] in <86d667fdac92404981c495ad3b503d7a>:0
  at Bonsai.Configuration.ConsoleBootstrapper.RunPackageOperationAsync (System.Func`1[TResult] operationFactory) [0x00078] in <86d667fdac92404981c495ad3b503d7a>:0

We did confirm that wget https://www.myget.org/F/bonsai/api/v3/index.json works on that shell, and also dotnet restore but it could be that the particular default NuGet client configuration on .NET framework is not compatible with security settings of the environment.

[glopesdev]

It turns out this is related to the way Mono handles certificates by having its own separate certificate store.

In the documentation there is mention of a built-in tool to sync certificates with the system, but the actual examples are tucked away in the Release Notes for Mono 3.12.0. The command for Debian systems is reproduced below for reference:

cert-sync /etc/ssl/certs/ca-certificates.crt

This might still fail if you don't have root access to your system, in which case it might help to run the command with the --user flag:

cert-sync --user /etc/ssl/certs/ca-certificates.crt