Apply secrets to function result properties marked as such

[mikhailshilkov]

Function output properties can be annotated as secrets. That may have two effects:

1. The property is marked as a secret in Pulumi schema (already works in pulumi-go-provider). As of today, this will have no effect on Pulumi SDK and the result will still be returned to the engine as plain text.
2. The provider marks the output property value as a secret and that's what gets sent over gRPC to the client SDK.

This PR implements (2) by applying secrets to inferred function results.

Added an invoke test that verifies this behavior. Also, extended the credentials sample program with an implementation of a secretty invokes and verified that it works manually.

Note that this requires 3.131.0 or later of Node.js/Go/YAML SDK or a later version for Python SDK support. Earlier versions fail to deserialize a secretty invoke output.

Resolves #259

[mikhailshilkov]

This will need a pulumi-yaml update before the tests go green

[iwahbe]

For this to be safe, I think it will need an engine level feature flag: The engine tells the provider that it's ok to send secrets on invokes.

Otherwise this will break existing providers on old SDKs without any explanation.

[mikhailshilkov]

For this to be safe, I think it will need an engine level feature flag: The engine tells the provider that it's ok to send secrets on invokes.
Otherwise this will break existing providers on old SDKs without any explanation.

How so? In my mind, the flow goes like this:

1. We upgrade this PR to pu/pu that has all codegen fixed up.
2. We merge it and ship it in a new release.
3. A provider upgrades to this new release + pu/pu version it requires, re-runs the codegen to take advantage of the new SDKs.
4. Output-based invokes work with secrets.

The CLI version makes no difference here - so where should the flag be and why?

[iwahbe]

3. A provider upgrades to this new release + pu/pu version it requires, re-runs the codegen to take advantage of the new SDKs.

Will the newly generated SDKs require version's of their languages core SDKs that support this?

[mikhailshilkov]

Will the newly generated SDKs require version's of their languages core SDKs that support this?

I think so... E.g. Node implementation relies on new functions in the core SDK: https://github.com/pulumi/pulumi/pull/17237/files#diff-f786ce189e2b1bb74d5f9425d102b713287faea09bc04b692ea0d42e1b035a10

Besides, taking advantage of output secretness requires a code change from the provider's author, hopefully they will do so only if they know what they do.

[iwahbe]

Will the newly generated SDKs require version's of their languages core SDKs that support this?

I think so... E.g. Node implementation relies on new functions in the core SDK: https://github.com/pulumi/pulumi/pull/17237/files#diff-f786ce189e2b1bb74d5f9425d102b713287faea09bc04b692ea0d42e1b035a10

I don't see any required version bump in that diff. On master it's still on 3.42.0:

https://github.com/pulumi/pulumi/blob/189559fa8e43bb88b3f4e0ae9ca3093c74103a2b/pkg/codegen/nodejs/gen.go#L49-L50

If they did bump the required SDK version, then I think this would be ok.

Besides, taking advantage of output secretness requires a code change from the provider's author, hopefully they will do so only if they know what they do.

I don't think that's true. They could have marked field as secret (perhaps in a shared type) and forgot about it. Imagine if there was a remote.Run function in command. It would be affected by this change.

[iwahbe]

Thinking about it some more, the only case where this introduces a break is:

1. Someone will upgrade their pulumi-go-provider version to v${next}, which includes outputty functions.
2. Someone will generate an SDK with an old version of pulumi, which doesn’t require the new output-supporting SDKs.
3. The provider will start sending outputs in invokes, which will break existing usage.

This requires a pulumi package gen-sdk version that's behind what pulumi-go-provider uses, which is very hard to guard against.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 13 lines in your changes missing coverage. Please review.

Project coverage is 38.32%. Comparing base (c63d6b6) to head (2ee2a12).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
examples/credentials/main.go	0.00%	12 Missing ⚠️
infer/function.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #272      +/-   ##
==========================================
- Coverage   38.42%   38.32%   -0.10%     
==========================================
  Files          45       45              
  Lines        4705     4717      +12     
==========================================
  Hits         1808     1808              
- Misses       2733     2745      +12     
  Partials      164      164              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pulumi-bot]

This PR has been shipped in release v0.23.0.