Hook up github.com/cloudy-sky-software/pulschema

[thomas11]

This PR adds a package openapipulschema that uses pulschema to load an API spec and create a Pulumi schema from it. A new provider
is created based on the schema plus provided metadata.

Theoretically, this would allow generating the provider and schema parts
for an OpenAPI-based provider, leaving only the CRUD implementations.
I wasn't able to successfully load an OpenAPI spec with pulschema so far due to issues in the OpenAPI specs that are discussed in this PR.

See also Handle some edge cases in various OpenAPI schemas #129 that I submitted.

[EronWright]

For reference, here's where the pulumi-kubernetes provider generates Pulumi schema from the Kubernetes swagger doc:
https://github.com/pulumi/pulumi-kubernetes/blob/78ab38748b9101d98890d86af44f5addb6c852dd/provider/cmd/pulumi-gen-kubernetes/main.go#L134

[thomas11]

For reference, here's where the pulumi-kubernetes provider generates Pulumi schema from the Kubernetes swagger doc: https://github.com/pulumi/pulumi-kubernetes/blob/78ab38748b9101d98890d86af44f5addb6c852dd/provider/cmd/pulumi-gen-kubernetes/main.go#L134

Thanks! We have similar code in the Azure native provider as well. For the hackathon, I wanted to try out a generic solution.

[praneetloke]

However, I wasn't able to successfully load an OpenAPI spec with pulschema so far.

I came here through the linked issue in pulschema. Let me know how I can help. If you want examples of native providers built using pulschema, have a look at https://github.com/search?q=topic%3Apulumi-provider+org%3Acloudy-sky-software+fork%3Atrue&type=repositories.

[thomas11]

However, I wasn't able to successfully load an OpenAPI spec with pulschema so far.

I came here through the linked issue in pulschema. Let me know how I can help. If you want examples of native providers built using pulschema, have a look at https://github.com/search?q=topic%3Apulumi-provider+org%3Acloudy-sky-software+fork%3Atrue&type=repositories.

Thank you @praneetloke! Here are the OpenAPI schemas I attempted with notes on issues:

	// Comes out empty after loading by kin-openapi/openapi3
	apigatewayV2Spec = "https://raw.githubusercontent.com/aws/aws-sdk-js/master/apis/apigatewayv2-2018-11-29.normal.json"

	// Can be loaded by kin-openapi/openapi3 but pulschema fails. In genGetFunc, funcPkgCtx.propertyTypeSpec returns error
	// "failed to generate property types for {Extensions:map[] OneOf:[] AnyOf:[] AllOf:[] Not:<nil> Type:<nil> Title: Format: Description:*Conditional.* This property is only present when the **field's** `action` is `added` _and_ the `added_value` is an Asana resource. This will be only the `gid` and `resource_type` of the resource when the events come from webhooks; this will be the compact representation (and can have fields expanded with [opt_fields](/docs/input-output-options)) when using the [Events](/docs/asana-events) endpoint. Enum:[] Default:<nil> Example:map[gid:12345 resource_type:user] ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:false ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	asanaSpec = "https://raw.githubusercontent.com/Asana/developer-docs/master/defs/asana_oas.yaml"

	// Can be loaded by kin-openapi/openapi3 but pulschema fails. In genGetFunc, funcPkgCtx.propertyTypeSpec returns error
	// "failed to generate property types for {Extensions:map[] OneOf:[] AnyOf:[] AllOf:[] Not:<nil> Type: Title: Format: Description:The value of the property. Required on create and update. Enum:[] Default:<nil> Example:<nil> ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:false ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	jiraSpec = "https://developer.atlassian.com/cloud/jira/platform/swagger-v3.v3.json"

	// Without the x-oaiType special case,  pulschema fails. In genPropertySpec, ctx.propertyTypeSpec(propName, p) returns error
	// "failed to generate property types for {Extensions:map[x-oaiTypeLabel:string] OneOf:[] AnyOf:[0x14000b68e70 0x14000b68f60] AllOf:[] Not:<nil> Type:<nil> Title: Format: Description:The ID of the [Model](/docs/api-reference/models) to be used to execute this run. If a value is provided here, it will override the model associated with the assistant. If not, the model associated with the assistant will be used. Enum:[] Default:<nil> Example:gpt-4-turbo ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:true ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	// Without the x-oaiType special case, can be loaded by pulschema but fails jsonschema validation of `pulumi package get-schema`
	openaiSpec = "https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml"

	// Cannot be loaded by pulschema due to the use of $ref. In GatherResourcesFromAPI, the expression
	// `pathItem.Get.Responses.Status(200).Value.Content.Get(jsonMimeType)` is a nil pointer dereference. `Content` has a nil value,
	// instead it has `.Extensions["schema"]["$ref"]` to specify the type.
	azureResourcesSpec = "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/resources/resource-manager/Microsoft.Resources/stable/2019-08-01/resources.json"

With the pulumi-go-provider integration from this PR, the provider code is simply

func main() {
	specUri, err := url.Parse(azureResourcesSpec)
	exitIfErr(err)

	m := schema.Metadata{
		DisplayName: "test 1 2 3",
	}

	provider, err := Provider(*specUri, m)
	exitIfErr(err)

	err = p.RunProvider("myname", "0.1.0", provider)
	exitIfErr(err)
}

// Create a new inferred provider from the given OpenAPI spec and `metadata`.
func Provider(openApiUrl url.URL, metadata schema.Metadata) (p.Provider, error) {
	return openapipulschema.Wrap(p.Provider{}, openApiUrl, metadata)
}

[iwahbe]

I don't know how far you want to push this hackathon project, but I think it's very cool. If you want to keep going, I have a couple of suggestions.

I would move this out of infer into its own top level package, since it doesn't interact with infer in any way. Conceptually, it's a companion to infer: infer is a Pulumi provider from Go code. openapipulschema is a Pulumi provider from an API spec.

[thomas11]

However, I wasn't able to successfully load an OpenAPI spec with pulschema so far.

I came here through the linked issue in pulschema. Let me know how I can help. If you want examples of native providers built using pulschema, have a look at https://github.com/search?q=topic%3Apulumi-provider+org%3Acloudy-sky-software+fork%3Atrue&type=repositories.

Thank you @praneetloke! Here are the OpenAPI schemas I attempted with notes on issues:

	// Comes out empty after loading by kin-openapi/openapi3
	apigatewayV2Spec = "https://raw.githubusercontent.com/aws/aws-sdk-js/master/apis/apigatewayv2-2018-11-29.normal.json"

	// Can be loaded by kin-openapi/openapi3 but pulschema fails. In genGetFunc, funcPkgCtx.propertyTypeSpec returns error
	// "failed to generate property types for {Extensions:map[] OneOf:[] AnyOf:[] AllOf:[] Not:<nil> Type:<nil> Title: Format: Description:*Conditional.* This property is only present when the **field's** `action` is `added` _and_ the `added_value` is an Asana resource. This will be only the `gid` and `resource_type` of the resource when the events come from webhooks; this will be the compact representation (and can have fields expanded with [opt_fields](/docs/input-output-options)) when using the [Events](/docs/asana-events) endpoint. Enum:[] Default:<nil> Example:map[gid:12345 resource_type:user] ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:false ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	asanaSpec = "https://raw.githubusercontent.com/Asana/developer-docs/master/defs/asana_oas.yaml"

	// Can be loaded by kin-openapi/openapi3 but pulschema fails. In genGetFunc, funcPkgCtx.propertyTypeSpec returns error
	// "failed to generate property types for {Extensions:map[] OneOf:[] AnyOf:[] AllOf:[] Not:<nil> Type: Title: Format: Description:The value of the property. Required on create and update. Enum:[] Default:<nil> Example:<nil> ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:false ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	jiraSpec = "https://developer.atlassian.com/cloud/jira/platform/swagger-v3.v3.json"

	// Without the x-oaiType special case,  pulschema fails. In genPropertySpec, ctx.propertyTypeSpec(propName, p) returns error
	// "failed to generate property types for {Extensions:map[x-oaiTypeLabel:string] OneOf:[] AnyOf:[0x14000b68e70 0x14000b68f60] AllOf:[] Not:<nil> Type:<nil> Title: Format: Description:The ID of the [Model](/docs/api-reference/models) to be used to execute this run. If a value is provided here, it will override the model associated with the assistant. If not, the model associated with the assistant will be used. Enum:[] Default:<nil> Example:gpt-4-turbo ExternalDocs:<nil> UniqueItems:false ExclusiveMin:false ExclusiveMax:false Nullable:true ReadOnly:false WriteOnly:false AllowEmptyValue:false Deprecated:false XML:<nil> Min:<nil> Max:<nil> MultipleOf:<nil> MinLength:0 MaxLength:<nil> Pattern: MinItems:0 MaxItems:<nil> Items:<nil> Required:[] Properties:map[] MinProps:0 MaxProps:<nil> AdditionalProperties:{Has:<nil> Schema:<nil>} Discriminator:<nil>}"
	// Without the x-oaiType special case, can be loaded by pulschema but fails jsonschema validation of `pulumi package get-schema`
	openaiSpec = "https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml"

	// Cannot be loaded by pulschema due to the use of $ref. In GatherResourcesFromAPI, the expression
	// `pathItem.Get.Responses.Status(200).Value.Content.Get(jsonMimeType)` is a nil pointer dereference. `Content` has a nil value,
	// instead it has `.Extensions["schema"]["$ref"]` to specify the type.
	azureResourcesSpec = "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/master/specification/resources/resource-manager/Microsoft.Resources/stable/2019-08-01/resources.json"

Hi @praneetloke, I'm curious if you were able to reproduce my issues with loading the above schemas, or if I did something wrong here.

[praneetloke]

@thomas11

AWS API Gateway

How are you loading this file? As for pulschema, it doesn't deal with loading the raw spec and needs the loaded spec from kinopenapi, so you'll want to check what's going on with loading the spec.

Asana

It seems that these are legit errors in their OpenAPI schema. Have a look at the change type schema. It has properties that don't have a type property. pulschema just cannot make assumptions about the type of such properties. You'll need to patch the OpenAPI schema prior to handing it off to pulschema. See my inline comment below on one of the properties that has this problem.

Expand to see details

change:
          type: object
          description: >-
            Information about the type of change that has occurred. This field
            is only present when the value of the property `action`, describing
            the action taken on the **resource**, is `changed`.
          readOnly: true
          properties:
            field:
              description: The name of the field that has changed in the resource.
              type: string
              readOnly: true
              example: assignee
            action:
              description: >-
                The type of action taken on the **field** which has been
                changed.  This can be one of `changed`, `added`, or `removed`
                depending on the nature of the change.
              type: string
              readOnly: true
              example: changed
            new_value:

              # This is missing a `type` property and looking at its description (and the example),
              # it seems that it likely needs to have a `ref` to another type that
              # has `gid` and `resource_type` as its properties. It's the same with
              # the `added_value` and `removed_value` properties below. All three
              # of these seem to need a ref to the same schema type.

              description: >-
                *Conditional.* This property is only present when the
                **field's** `action` is `changed` _and_ the `new_value` is an
                Asana resource. This will be only the `gid` and `resource_type`
                of the resource when the events come from webhooks; this will
                be the compact representation (and can have fields expanded
                with [opt_fields](/docs/input-output-options)) when using the
                [Events](/docs/asana-events) endpoint.
              example:
                gid: '12345'
                resource_type: user
            added_value:
              description: >-
                *Conditional.* This property is only present when the
                **field's** `action` is `added` _and_ the `added_value` is an
                Asana resource. This will be only the `gid` and `resource_type`
                of the resource when the events come from webhooks; this will
                be the compact representation (and can have fields expanded
                with [opt_fields](/docs/input-output-options)) when using the
                [Events](/docs/asana-events) endpoint.
              example:
                gid: '12345'
                resource_type: user
            removed_value:
              description: >-
                *Conditional.* This property is only present when the
                **field's** `action` is `removed` _and_ the `removed_value` is an
                Asana resource. This will be only the `gid` and `resource_type`
                of the resource when the events come from webhooks; this will
                be the compact representation (and can have fields expanded
                with [opt_fields](/docs/input-output-options)) when using the
                [Events](/docs/asana-events) endpoint.
              example:
                gid: '12345'
                resource_type: user

Atlassian Jira

This one is similar to the Asana issue. Looking at the type that pulschema had trouble generating a spec for, it seems that it is a property that is missing a type. And once again, you'll need to patch this API spec yourself before you can hand it off to pulschema.

Expand to see details

"EntityProperty": {
        "additionalProperties": false,
        "description": "An entity property, for more information see [Entity properties](https://developer.atlassian.com/cloud/jira/platform/jira-entity-properties/).",
        "properties": {
          "key": {
            "description": "The key of the property. Required on create and update.",
            "type": "string"
          },
          "value": {
             <-- There should be a type property here just like `key` does.
            "description": "The value of the property. Required on create and update."
          }
        },
        "type": "object"
      },

For issues where you have to patch the API spec yourself before passing it off to pulschema I highly encourage that you do it using kinopenapi library. It very easy to use that lib to do that. This is also the reason why pulschema itself returns the OpenAPI spec doc bytes back to you along with the Pulumi schema package because it's likely that whatever provider you create might need to have access to the exact same OpenAPI doc that was used to generate the Pulumi schema.

Lastly, when building a native provider from an OpenAPI doc, you'll want to review the conformance doc I wrote for cloud providers: https://github.com/cloudy-sky-software/cloud-provider-api-conformance. The conformance repo ensures that you have a successful conversion to Pulumi schema from an OpenAPI doc.

[praneetloke]

As for the situation with OpenAI and its use of x-oaiTypeLabel where anyOf definition occurs. I think pulschema needs to handle anyOf better. Right now, it has a very purpose-driven handling of anyOf for PATCH endpoints alone. It needs to handle anyOf for property types just like it handles oneOf and allOf today.

[praneetloke]

@thomas11 (or anyone else following this PR) I've spent some more time thinking about the situation with OpenAI's use of x-oaiTypeLabel and I don't think there is a generic fix to make in pulschema. I think you should patch the OpenAI spec to rewrite the anyOf as oneOf definitions before handing it off to pulschema. Please have a look at cloudy-sky-software/pulschema#130.

[thomas11]

I would move this out of infer into its own top level package, since it doesn't interact with infer in any way. Conceptually, it's a companion to infer: infer is a Pulumi provider from Go code. openapipulschema is a Pulumi provider from an API spec.

I think it might make sense to group them somehow, the "things that create providers from some source", since there will hopefully be others. Haven't found a great name yet, though. We can postpone this until the third one comes along, if ever.

[thomas11]

For issues where you have to patch the API spec yourself before passing it off to pulschema I highly encourage that you do it using kinopenapi library. It very easy to use that lib to do that. This is also the reason why pulschema itself returns the OpenAPI spec doc bytes back to you along with the Pulumi schema package because it's likely that whatever provider you create might need to have access to the exact same OpenAPI doc that was used to generate the Pulumi schema.

Lastly, when building a native provider from an OpenAPI doc, you'll want to review the conformance doc I wrote for cloud providers: https://github.com/cloudy-sky-software/cloud-provider-api-conformance. The conformance repo ensures that you have a successful conversion to Pulumi schema from an OpenAPI doc.

Thank you for your detailed analysis here, @praneetloke!

I'm in two minds about this. On the one hand, yes, services should follow the spec and we should keep pulschema clean of service-specific hacks. On the other hand, in terms of ease of use, many schemas don't work out of the box and it's a high barrier to expect users to patch the schema first. And everyone who wants to use, say, OpenAPI, needs to make the same patches.

Maybe a well-separated compatibility layer in pulschema, that only patches schemas to make them compliant, would be a good compromise?

[thomas11]

I don't know how far you want to push this hackathon project, but I think it's very cool. If you want to keep going, I have a couple of suggestions.

I would move this out of infer into its own top level package, since it doesn't interact with infer in any way. Conceptually, it's a companion to infer: infer is a Pulumi provider from Go code. openapipulschema is a Pulumi provider from an API spec.

@iwahbe I believe all comments are addressed.

[iwahbe]

This change feels unnecessary. I'll leave a comment explaining how we can get around it in provider.go.

[iwahbe]

If you want to go in the direction of merging this, it needs to have documentation on [openapipulschema.Provider] and an end to end example.

I'm having trouble thinking of a practical use case here. As far as I see, it gives a schema but no backing implementation, which means the underlying provider would need to have a full knowledge of the schema it is serving anyway.

I'm also concerned that if the schema behind the link changes, the provider will become invalid. Have you considered a design where we embed the schema instead of a link to the schema?

[iwahbe]

What does consuming this provider look like. I don't see where the provider actually implements any methods beyond GetSchema. If this provider does nothing but provide an (incorrect) schema, I don't think its a valid example.

[iwahbe]

Instead of injecting like this, we should wrap the returned provider with schema.Wrap.

[iwahbe]

Providers are not allowed to know their name until they are run, so this must be a temporary value. Can we just leave it blank?

[iwahbe]

This needs documentation to merge. What are the constraints on oaURL? What does the resulting provider have?

[praneetloke]

@thomas11 apologies for extremely late reply. You have an interesting idea. I'll try to keep my response brief.

I'm in two minds about this. On the one hand, yes, services should follow the spec and we should keep pulschema clean of service-specific hacks. On the other hand, in terms of ease of use, many schemas don't work out of the box and it's a high barrier to expect users to patch the schema first. And everyone who wants to use, say, OpenAPI, needs to make the same patches.

Maybe a well-separated compatibility layer in pulschema, that only patches schemas to make them compliant, would be a good compromise?

You spoke my mind with that. pulschema originally started as a way for me to tackle writing a Render provider. In fact, it was originally inside the provider code after which I split out to be its own thing recognizing that it needed to be separate and generic. Having said that, the compatibility layer you speak of sort of already exists today by way of error handling from bad OpenAPI specs. Unfortunately, my current thinking is that it can't go beyond that by a whole lot due to how specific the API spec issues are sometimes. That doesn't mean that it can't gain more "generic" fault tolerance though. So yeah it's a case-by-case basis.

If you would like to discuss further, I would like to invite you to start a discussion in the pulschema repo when you get a chance.

P.S: Also I want to let you know that, as of last night, I have open-sourced pulumi-provider-framework. It's a comprehensive library designed to handle the output from pulschema and does all the heavy-lifting of a resource provider server. It's used in all of the native providers I have built thus far. In fact, I am currently tackling build a native provider for DigitalOcean, which is nearly complete.

[thomas11]

If you want to go in the direction of merging this, it needs to have documentation on [openapipulschema.Provider] and an end to end example.

I'm having trouble thinking of a practical use case here. As far as I see, it gives a schema but no backing implementation, which means the underlying provider would need to have a full knowledge of the schema it is serving anyway.

I'm also concerned that if the schema behind the link changes, the provider will become invalid. Have you considered a design where we embed the schema instead of a link to the schema?

I think @praneetloke's release of pulumi-provider-framework answers that question - OpenAPI provider authors can use that one. I hope pulumi-provider-framework can be based on pulumi-go-provider one day.

[thomas11]

P.S: Also I want to let you know that, as of last night, I have open-sourced pulumi-provider-framework. It's a comprehensive library designed to handle the output from pulschema and does all the heavy-lifting of a resource provider server. It's used in all of the native providers I have built thus far. In fact, I am currently tackling build a native provider for DigitalOcean, which is nearly complete.

Nice work @praneetloke! I wish the name of the project indicated that it's specific to OpenAPI to avoid confusion with, e.g., pulumi-go-provider and pulumi-provider-boilerplate. Nevertheless, great stuff.