Json serialization thing

[jnorthrup]

commenting on @AaronGoldman 's comment of JSON interop @ https://youtu.be/nUC0KZlQyHw?list=PL05j31Knswhn7RLk-VKHZ6RI4e9D4d-6e&t=5312

Having worked with GWT and Autobeans vs. Gson and Jackson there were some distinctions between the two relative featuresets: (Security is some other discussion than a json spec) , I would go with GWT's lead in this area vs. the gson/jackson/xstream models which are very biased towards non-interface ser/de products

Autobeans java -> js transpiler facade	java json serialization API's
interface based transpiler based code-gen	class mapped and reflection based
The "Splittable" composition	Map<String,?> , List<?>, and so on
generated shims from Interfaces with pre-and-post reification access	generally only whitelisted pre-mapped Interfaces, all else being Class

the GWT approach being compile-centric limits the Autobeans logic to an AutobeansCodex factory type java-side class for compiled js output, but was actually a bit more convenient for writing interface based contracts as well

not that AA needs the GWT libs specifically, but the builk of the json interop for the GWT serverside -> client js is in https://github.com/gwtproject/gwt/tree/main/user/src/com/google/web/bindery/autobean/shared

[cliffclick]

Reflection by default, followed by user-can-override with an interface spec?

[jnorthrup]

Reflection by default, followed by user-can-override with an interface spec?

honestly I would not use json as a first reflection-based automatic ser-de that really leads to death of a thousand papercuts due to the impedance mismatches and the high likelihood of rigorously getting them all tested very quickly by a user-base. Oracle-style backwards compatibility to band-aid fixes live here.

pascal-bounded or zigzag bounded binary fields with NIO buffers or similar would be a better discussion topic for automatic wire protocol IMHO. maybe metaprogramming based IDL from AA to slightly fluffier AA which achieves both IDL and DRY is feasable.

accomplishing pickle and beating it to death before json would be my priorities, while my personal experience says there have been non-serialization JSON interop that is prefereable to builtins.

[jnorthrup]

https://github.com/0xCopy/json-lazy-autobean answered some questions we had about json serialization upper bounds. i would not presume that a serialization model is going to be as awesome as a query-by-example visitor parser or the off-heap mmap options you get for it.

I have personally managed my own seekable columnar disk layouts in kotlin with NIO which has given me no surprises, and removes pandas from the critical path of hammering down imports.

Kotlin jumped feet first into serialization with a clean slate and present an interesting and unexpectedly complex supra-stdlib requirement for using it (new libs, gradle/maven, compioler switches) . however CBOR does look pretty cool. these don't all work to the same degree of capability, leading me to think i'd rather have a good reliable library than a wobbly language with permanent retardation in this area.

kotlinx.serialization includes libraries for various serialization formats:

JSON: kotlinx-serialization-json

Protocol buffers: kotlinx-serialization-protobuf

CBOR: kotlinx-serialization-cbor

Properties: kotlinx-serialization-properties

HOCON: kotlinx-serialization-hocon (only on JVM)

[AaronGoldman]

Is the distinction between the two stiles primarily a Run Time vs Ahead of Time one?
Sounds like by "GWT approach" you mean defining the interface in an interface description language and compile time generating the parser code.

Some systems like protobufs can be hard to work with if you don't have the protobufs schema. In a JiT environment I wander if it is better to have reification but never save out the generated code. The Reflection can be used to build the functions you need and the JiT can compile them with the "generated code" not being an artifact that you need to deal with.
Does that mitigate some of the distinctions between these methods or am I misunderstanding the tradeoff you are pointing out?

[jnorthrup]

a look at GWT RPCfactory would in fact go a logn ways toward addressing the hour long field-level security discussion beyond just serialization, the munged symbols in release mode were just md5's and offsets revealing nothing about the wire traffic other than the contents. json was a last resort in GWT until one day the json wire protocol made some new features available with less boilerplate (better kolmogorov score to define RPC)

[AaronGoldman]

I wander if we should look at the trio of edn, transit, and fressian from Clojure.

https://github.com/edn-format/edn
https://github.com/cognitect/transit-format
https://github.com/Datomic/fressian/wiki

edn is the languages literal syntax having an object notation can make a lot of things simpler. just put some literal data that both programer and jit can read.

fressian a dense column store. It is convenient to have a binary format that knows the common tricks to get data dense data. Big data and disk archival.

• dictionaries
• variable-length encodings
• delta encoding
• dubble delta encoding
• Huffman code
• Lempel–Ziv–Welch

e.g. If the logs are a struct with a format string and a struct of data we can dict encode the small number of format strings and use all the column store tricks on the data. Scanning those logs would be much faster than string parsing. We mostly just log strings because the log library in the standard library dose not have good tools for logging and viewing in a dense format.

transit is there live wire protocol. If I have a duplex channel there are tricks to have agreed on a format for the one conversation and get more dense representations than the disk format. Only the disk format needs to be stable across time and versions.

Obviously there are lots of resons to serialize to lots of formats but if there is a human readable format and a dense format that have low impedance mismatch with the language in the standard library devs will use then as a default and more code with will "just work" with other code in the ecosystem.

maybe AAlang should have:

• .aal AA literal
  • a AA notation encoding of objects that just decode but don't run arbatrary code as they do.
  • basically just AA but with call disallowed. You can still define a function just not call one.
• .aad AA dense
  • a dense binary columnar with standard encoding tricks that maps 1:1 with .aal

[jnorthrup]

looking back at technical debt once "reflection based ser/des" was added to standard contracts, a great idea in the 90's when this was uncommon-- age was not kind to any of the long-lived examples I can come up with; there are no mainstay serialization based OODBMS success stories of any importance that survived from the c++ era and even to give one recent example, hazelcast offloading all such distributed object matters to jvm serialization became a showstopper for migrating from javalang to jvm-language-x. boost serialization is "there", and adequate as an option to serialize by convention, but again I would personally discount this feature from a language's minimum core features. pickle is present on a lot of python notebooks and such, but generally some other IO library becomes more useful very quickly. object serialization seems synonymous with toy-project state-dump.

transit looks interesting while at the same time brings the question to my mind why not add those encoding capabilities as meta-programming and operator overloading capabilities up front in the preprocessor to enable AA self-maintained minification for those who would dare and give carte blanche to the Unicode symbol characters as a similar aliasing stepping stone; strongly indicated with #pragma/magic preamble to manage reader/complexity-budget expectations.

not to stray too far from the topic of [json]serialization but language construction freedoms can conquer impedance mappings while preserving whatever semantics that can be expressed in both the host and the source grammars.

supporting .aal (someone is going to say aason, why not here now?) and .aad as runtime has the effect of adding to the existing or new parser in the form of a wire protocol. seems like an eval-centric language can avoid this kink in the hose, but if pickle is absolutely a minimum requirement, please no not json as the straw-man.

[jnorthrup]

it occurred to me to mention what I find more useful than serialization, even though it may involve setting up the code to do similar things -

I use destructuring syntax val (a,b)=pairAB along with delegation and mixins and occasional import aliases to approximate the readable statements in the interesting points of transforms.

destructuring and infix modifiers on unary/binary functions make for readable generative expressions when the method size is filed down to a handful of lines. despite for instance having access to kotlin serialization library, for comparable serialization usecases it often seems more expedient/expressive to read a line, split/regex it, and destructure the result array; having FP available tends to make my codebases more script-like than OO which at least succeeds in keeping most of a program skeleton well described in a line and allowing mixins shoved off to the side rather than explicit class abstractions for top level code.

I still do use gson in recent stuff to capture object graphs to json as well, but given the presence of top-level (global) state and functions there is no one-class that holds this necessary graph until i write a shim expressly for serialization which serves no other purpose. prior to FP exposure, reflection based serialization was in fact a major asset to a lot of things I did in java though NIO ByteBuffes for anything important to maintain.

[cliffclick]

Looks like a cool discussion to restart when AA gets further along.