Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent from accepting schema that are not uniquely identifiable from the current parser #717

Conversation

eliax1996
Copy link
Contributor

with the pr !716 it's possible to insert a schema like the following one:

syntax = "proto3";

package fancy.company.in.party.v1;
message AnotherMessage {
  enum BamFancyEnum {
    // Hei! This is a comment!
    MY_AWESOME_FIELD = 0;
  }
  message WowANestedMessage {
    message DeeplyNestedMsg {
      enum BamFancyEnum {
      // Hei! This is a comment!
         MY_AWESOME_FIELD = 0;
      }
      message AnotherLevelOfNesting {
         BamFancyEnum im_tricky_im_referring_to_the_previous_enum = 1;
      }
    }
  }
}

The problem is that currently the parser is a double pass parser that tries to expand all the types both providing the "local" name, the fully scoped name and a name composed by the first type anchestror and the type name.
We can allow types to come in but we cannot uniquely associate the type with which (resulting in a schema evolution potentially broken).
This pr is meant to avoid this situation by blocking the protobuf schemas that are not uniquely identifiable.

@eliax1996 eliax1996 requested review from a team as code owners September 18, 2023 11:39
@eliax1996 eliax1996 force-pushed the eliax1996/re-define-messages-in-the-same-scope-is-illegal branch 2 times, most recently from b602004 to fa20068 Compare September 18, 2023 14:03
@eliax1996 eliax1996 force-pushed the eliax1996/re-define-messages-in-the-same-scope-is-illegal branch from fa20068 to fb59e41 Compare September 19, 2023 07:14
Copy link
Contributor

@tvainika tvainika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs some clarification. In my testing the schema you are claiming to be "non-uniquely identifirable" is, according to protoc, just valid ProtoBuf schema. It should not be prevented. Is there some misunderstanding here?

@eliax1996
Copy link
Contributor Author

This needs some clarification. In my testing the schema you are claiming to be "non-uniquely identifirable" is, according to protoc, just valid ProtoBuf schema. It should not be prevented. Is there some misunderstanding here?

Yeah, I mean that isn't uniquely identifiable in our current parsing strategy.
Our current parsing mechanism work in that way:

  1. we scan each file and we collect every immport
  2. once done that we start from the leaf objects and we add to a set that contains all the valid types (types that are defined by basic types or types composed by other known types), basically we collect a set of string that are allowed to be used in the sources
  3. for each source we scan the content and we check that the file uses only files declared previously

The problem of that strategy it's that we lose the reference from "where" the type it's used with respect to the definitions. Because I imagine that the other parsers use the "distance" between the scopes to assign implicitly the nearest one.

In our case if you have the same message defined twice it would be tricky to verify for e.g. the schema evolution, Not 100% sure until we write a test trying to break the evolution but I feel that could be problematic

@eliax1996
Copy link
Contributor Author

Also, as we discussed with @jjaakola-aiven protoc has probably some way to deal with identify who is referring to who.
Probably we should test also how the schema evolution is handled by protoc and we should try to replicate as much as possible the behaviour, moreover I think that if we leave a schema like the previous one be ingested by the registry we might get into bugs

@eliax1996
Copy link
Contributor Author

This problem it's handled by the new parser implementation, it's not identical anymore, we are expanding the types so in that case, since it's the firstly defined type the reference will refer the AnotherMessage.BamFancyEnum field.

@eliax1996 eliax1996 closed this Oct 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants