Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save extensions from field options in message props #85

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

artemeff
Copy link

This is first step to implement plugins. Protobuf allows to declare and use custom options for fields too (currently only file options is supported), but this library omits that info, so I think neat place to put that info is __message_props__/0:

message Foo {
  optional string a = 1;
  optional string b = 2 [(mypkg.myopt_bool)=true];
  optional string c = 3 [(mypkg.myopt_string)="test"];
}
iex(1)> Protobuf.Protoc.ExtTest.Foo.__message_props__()
%Protobuf.MessageProps{
  ...
  field_props: %{
    1 => %Protobuf.FieldProps{
      default: nil,
      deprecated?: false,
      embedded?: false,
      encoded_fnum: "\n",
      enum?: false,
      extensions: %{},
      fnum: 1,
      map?: false,
      name: "a",
      name_atom: :a,
      oneof: nil,
      optional?: true,
      packed?: false,
      repeated?: false,
      required?: false,
      type: :string,
      wire_type: 2
    },
    2 => %Protobuf.FieldProps{
      default: nil,
      deprecated?: false,
      embedded?: false,
      encoded_fnum: <<18>>,
      enum?: false,
      extensions: %{{Mypkg.PbExtension, :myopt_bool} => true},
      fnum: 2,
      map?: false,
      name: "b",
      name_atom: :b,
      oneof: nil,
      optional?: true,
      packed?: false,
      repeated?: false,
      required?: false,
      type: :string,
      wire_type: 2
    },
    3 => %Protobuf.FieldProps{
      ...
      extensions: %{{Mypkg.PbExtension, :myopt_string} => "test"},
      ...
    }
  },
  ...
}

I thought a lot about how Plugins interface should looks like to support wide area of features, but I get stuck :(

In my project I have a few different types, that requires pre- and post-processing, like UUID that should be transformed from string representation to binary, or enum constants that should be converted from human readable to protobuf-acceptable (accordingly to protobuf style guide).

Keep that meta information in options looks like a solution (but it looks like workaround without plugins system), with that meta I can write Elixir's Protocol that process protobuf structs and automatically generate encoders/decoders for each struct/field.

The most important thing — this is not a duct tape for this library, my PR just add able to hold custom options :)

@artemeff artemeff force-pushed the field-extensions branch 3 times, most recently from dbba09c to 124b0fc Compare February 18, 2020 17:44
@artemeff
Copy link
Author

I'm also added handy mix task to compile proto schema: mix protobuf, no longer need to install escript globally in system, it's per-project now. Create protoc-gen-elixir bash script with contents:

#!/bin/sh
mix protobuf

and run protoc like: protoc --elixir_out=./lib --plugin=./protoc-gen-elixir *.proto.


Added config :protobuf, extension_paths: [path1, glob2] to support custom extensions placed in project (not in this library), protobuf-elixir will look in this paths and compile everything before generating protobuf schema, eg:

config :protobuf,
  extensions: :enabled,
  extension_paths: ["lib/my_app/myext.pb.ex"]

@artemeff artemeff requested a review from tony612 February 20, 2020 06:28
@tony612
Copy link
Collaborator

tony612 commented Mar 21, 2020

I feel the correct direction for plugins should be writing a separated protoc plugin and generate separated files. For example, these plugins https://github.com/envoyproxy/protoc-gen-validate & https://github.com/grpc-ecosystem/grpc-gateway work like this. And we can add some functions
to simplify writing a protoc plugin in Elixir, for example this project https://github.com/lyft/protoc-gen-star, but we may not need to be as powerful as protoc-gen-star at first.

In this way, we have to do things in separated module, like

person = Person.new() # a protobuf message
Validator.Person.validate(person) # Validator.Person is generated by a plugin

instead of

person = Person.new()
Person.validate(person) # inject into protobuf modules, which is hard in Elixir

If we have to inject something into original protobuf modules, we can discuss case by case. But I prefer avoiding doing this if we can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants