Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add method to get the schema (version) used for serialization of a message #362

Open
acromarco opened this issue Jan 28, 2024 · 0 comments

Comments

@acromarco
Copy link

acromarco commented Jan 28, 2024

There are client libraries for other languages (e.g. java) that support automatic (de)serialization of message data based on a schema (https://pulsar.apache.org/docs/3.1.x/schema-overview/).
The node client does not have this feature yet (#242).

In theory it should be possible to deserialize the message data manually, BUT...
Some serialization formats like Avro require to know the exact schema that was used for serialization (mtth/avsc#447). In order to deal with schema evolution the client needs to know it's own compatible schema AND the schema used for serialization.
As far I understand, the automatic (de)serialization feature of Pulsar solves this problem by keeping a schema registry and tagging the messages with the used schema version.
If I understand right, the node client does not provide a method to get the schema used for serialization and not even a method to get the schema version of a message.
Assuming the need for schema evolution, this makes it impossible to deserialize reliably messages written by a java client library using an Avro schema,

I'm new to Pulsar and Avro, so please forgive (and correct) me if my understanding is wrong.

If my understanding is right, I wonder how difficult it would be to add a method to lookup the serialization schema on an message.

@acromarco acromarco changed the title Add method to get the schema (version) used for serialization of an message Add method to get the schema (version) used for serialization of a message Jan 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant