v0.9.0

v0.9.0 is a pretty significant release, including several neat new convenience features, but most noticeably, significantly reworking how linking works.

Almost any code that deals with storing and linking data will need some adaptation to handle this release.
We're sorry about the effort this may require, but it should be worth it.
The new LinkSystem API should let us introduce a lot more convenience features in the future, and do so without pushing additional breakage out to downstream users; this is an investment in the future.

The bullet points below contain all the fun details.

Note that a v0.8.0 release version has been skipped.
We use odd numbers to indicate the existence of significant changes;
and while usually we try to tag an even-number release between each odd number release so that migrations can be smoothed out,
in this case there simply weren't enough interesting points in between to be worth doing so.

• Change: linking has been significantly reworked, and now primarily works through the ipld.LinkSystem type.
  • This is cool, because it makes a lot of things less circuitous. Previously, working with links was a complicated combination of Loader and Storer functions, the Link interface contained the Load method, it was just... complicated to figure out where to start. Now, the answer is simple and constant: "Start with LinkSystem". Clearer to use; clearer to document; and also coincidentally a lot clearer to develop for, internally.
  • The PR can be found here: #143
  • Link.Load -> LinkSystem.Load (or, new: LinkSystem.Fill, which lets you control memory allocation more explicitly).
  • LinkBuilder.Build -> LinkSystem.Store.
  • LinkSystem.ComputeLink is a new feature that produces a Link without needing to store the data anywhere.
  • The ipld.Loader function is now most analogous to ipld.BlockReadOpener. You now put it into use by assigning it to a LinkLoader's StorageReadOpener field.
  • The ipld.Storer function is now most analogous to ipld.BlockWriteOpener. You now put it into use by assigning it to a LinkLoader's StorageWriteOpener field.
  • 99% of the time, you'll probably start with linking/cid.DefaultLinkSystem(). You can assign to fields of this to customize it further, but it'll get you started with multihashes and multicodecs and all the behavior you expect when working with CIDs.
    • (So, no -- the cidlink package hasn't gone anywhere. Hopefully it's a bit less obtrusive now, but it's still here.)
  • The traversal package's Config struct now uses a LinkSystem instead of a Loader and Storer pair, as you would now probably expect.
    • If you had code that was also previously passing around Loader and Storer, it's likely a similar pattern of change will be the right direction for that code.
  • In the future, further improvements will come from this: we're now much, much closer to making a bunch of transitive dependencies become optional (especially, various hashers, which currently, whenever you pull in the linking/cid package, come due to go-cid, and are quite large). When these improvements land (again, they're not in this release), you'll need to update your applications to import hashers you need if they're not in the golang standard library. For now: there's no change.
• Change: multicodec registration is now in the go-ipld-prime/multicodec package.
  • Previously, this registry was in the linking/cid package. These things are now better decoupled.
  • This will require packages which register codecs to make some very small updates: e.g. s/cidlink.RegisterMulticodecDecoder/multicodec.RegisterDecoder/, and correspondingly, update the package imports at the top of the file.
• New: some pre-made storage options (e.g. satisfying the ipld.StorageReadOpener and ipld.StorageWriteOpener function interfaces) have appeared! Find these in the go-ipld-prime/storage package.
  • Currently this only includes a simple in-memory storage option. This may be useful for testing and examples, but probably not much else :)
  • These are mostly intended to be illustrative. You should still expect to find better storage mechanisms in other repos.
• Change: some function names in codec packages are ever-so-slightly updated. (They're verbs now, instead of nouns, which makes sense because they're functions. I have no idea what I was thinking with the previous naming. Sorry.)
  • s/dagjson.Decoder/dagjson.Decode/g
  • s/dagjson.Decoder/dagjson.Encode/g
  • s/dagcbor.Decoder/dagcbor.Decode/g
  • s/dagcbor.Encoder/dagcbor.Encode/g
  • If you've only been using these indirectly, via their multicodec indicators, you won't have to update anything at all to account for this change.
• New: several new forms of helpers to make it syntactically easy to create new IPLD data trees with golang code!
  • Check out the go-ipld-prime/fluent/quip package! See #134, where it was introduced, for more details.
  • Check out the go-ipld-prime/fluent/qp package! See #138, where it was introduced, for more details.
  • Both of these offer variations on fluent which have much lower costs to use. (fluent incurs allocations during operation, which has a noticable impact on performance if used in a "hot" code path. Neither of these two new solutions do!)
  • For now, both quip and qp will be maintained. They have similar goals, but different syntaxes. If one is shown drastically more popular over time, we might begin to consider deprecating one in favor of the other, but we'll need lots of data before considering that.
  • We won't be removing the fluent package anytime soon, but we probably wouldn't recommend building new stuff on it. qp and quip are both drastically preferable for performance reasons.
• New: there is now an interface called ipld.ADL which can be used for a certain kind of feature detection.
  • This is an experimental new concept and likely subject to change.
  • The one key trait we've found all ADLs tend to share right now is that they have a "synthesized" view and "substrate" view of their data. So: the ipld.ADL interface states that a thing is an ipld.Node (for the synthesized view), and from it you should be able to access a Substrate() ipld.Node, and that's about it.

v0.7.0

v0.7.0 is a small release that makes a couple of breaking changes since v0.6.0.
However, the good news is: they're all very small changes, and we've kept them in a tiny group,
so if you're already on v0.6.0, this update should be easy.
And we've got scripts to help you.

There's also one cool new feature: traversal.FocusedTransform is now available to help you make mutations to large documents conveniently.

• Change: all interfaces and APIs now use golang int64 rather than golang int. #125
  • This is necessary because the IPLD Data Model specifies that integers must be "at least 2^53" in range, and so since go-ipld-prime may also be used on 32-bit architectures, it is necessary that we not use the int type, which would fail to be Data Model-compliant on those architectures.
  • The following GNU sed lines should assist this transition in your code, although some other changes that are more difficult automate may also be necessary:

     sed -ri 's/(func.* AsInt.*)\<int\>/\1int64/g' **/*.go
     sed -ri 's/(func.* AssignInt.*)\<int\>/\1int64/g' **/*.go
     sed -ri 's/(func.* Length.*)\<int\>/\1int64/g' **/*.go
     sed -ri 's/(func.* LookupByIndex.*)\<int\>/\1int64/g' **/*.go
     sed -ri 's/(func.* Next.*)\<int\>/\1int64/g' **/*.go
     sed -ri 's/(func.* ValuePrototype.*)\<int\>/\1int64/g' **/*.go
    

• Change: we've renamed the types talking about "kinds" for greater clarity. ipld.ReprKind is now just ipld.Kind; schema.Kind is now schema.TypeKind. We expect to use "kind" and "typekind" consistently in prose and documentation from now on, as well. #127
  • Pretty much everyone who's used this library has said "ReprKind" didn't really make sense as a type name, so, uh, yeah. You were all correct. It's fixed now.
  • "kind" now always means "IPLD Data Model kind", and "typekind" now always means "the kinds which an IPLD Schema type can have".
  • You can find more examples of how we expect to use this in a sentence from now on in the discussion that lead to the rename: #94 (comment)
  • The following GNU sed lines should assist this transition in your code:

     sed -ri 's/\<Kind\(\)/TypeKind()/g' **/*.go
     sed -ri 's/\<Kind_/TypeKind_/g' **/*.go
     sed -i 's/\<Kind\>/TypeKind/g' **/*.go
     sed -i 's/ReprKind/Kind/g' **/*.go
    

• Feature: traversal.FocusedTransform works now! 🎉 You can use this to take a node, say what path inside it you want to update, and then give it an updated value. Super handy. #130

v0.6.0

v0.6.0 is a feature-packed release and has a few bugfixes, and no significant breaking changes. Update at your earliest convenience.

Most of the features have to do with codegen, which we now consider to be in alpha -- go ahead and use it! (We're starting to self-host some things in it, so any changes will definitely be managed from here on out.)
A few other small handy helper APIs have appeared as well; see the detailed notes for those.

Like with the last couple of releases, our intent is to follow this smooth-sailing change with another release shortly which will include some minor but noticable API changes, and that release may require you to make some code changes.
Therefore, we suggest upgrading to this one first, beacuse it's an easy waypoint before the next change.

• Feature: codegen is a reasonably usable alpha! We now encourage trying it out (but still only for those willing to experience an "alpha" level of friction -- UX still rough, and we know it).
  • Consult the feature table in the codegen package readme: many major features of IPLD Schemas are now supported.
    • Structs with tuple representations? Yes.
    • Keyed unions? Yes.
    • Structs with stringjoin representations? Yes. Including nested? Yes.
    • Lots of powerful stuff is now available to use.
    • See the feature table in the codegen readme for details.
  • Many generated types now have more methods for accessing them in typed ways (in addition to the usual ipld.Node interfaces, which can access the same data, but lose explicit typing). #106
    • Maps and lists now have both lookup methods and iterators which know the type of the child keys and values explicitly.
  • Cool: when generating unions, you can choose between different implementation strategies (favoring either interfaces, or embedded values) by using Adjunct Config. This lets you tune for either speed (reduced allocation count) or memory footprint (less allocation size, but more granular allocations).
    • See notes in #60 for more detail on this. We'll be aiming to make configurability of this more approachable and better documented in future releases, as we move towards codegen tools usable as CLI tools.
  • Cyclic references in types are now supported.
    • ... mostly. Some manual configuration may sometimes be required to make sure the generated structure wouldn't have an infinite memory size. We'll keep working on making this smoother in the future.
  • Field symbol overrides now work properly. (E.g., if you have a schema with a field called "type", you can make that work now. Just needs a field symbol override in the Adjunct Config when doing codegen!)
  • Codegen'd link types now implemented the schema.TypedLinkNode interface where applicable.
  • Structs now actually validate all required fields are present before allowing themselves to finish building. Ditto for their map representations.
  • Much more testing. And we've got a nice new declarative testcase system that makes it easier to write descriptions of how data should behave (at both the typed and representation view levels), and then just call one function to run exhaustive tests to make sure it looks the same from every inspectable API.
  • Change: codegen now outputs a fixed set of files. (Previously, it output one file per type in your schema.) This makes codegen much more managable; if you remove a type from your schema, you don't have to chase down the orphaned file. It's also just plain less clutter to look at on the filesystem.
• Demo: as proof of the kind of work that can be done now with codegen, we've implemented the IPLD Schema schema -- the schema that describes IPLD Schema declarations -- using codegen. It's pretty neat.
  • Future: we'll be replacing most of the current current schema package with code based on this generated stuff. Not there yet, though. Taking this slow.
    • You can see the drafts of this, along with new features based on it, in #107.
• Feature: the schema typesystem info packages are improved.
  • Cyclic references in types are now supported.
    • (Mind that there are still some caveats about this when fed to codegen, though.)
  • Graph completeness is now validated (e.g. missing type references emit useful errors)!
• Feature: there's a traversal.Get function. It's like traversal.Focus, but just returns the reached data instead of dragging you through a callback. Handy.
• Feature/bugfix: the DAG-CBOR codec now includes resource budgeting limits. This means it's a lot harder for a badly-formed (or maliciously formed!) message to cause you to run out of memory while processing it. #85
• Bugfix: several other panics from the DAG-CBOR codec on malformed data are now nice politely-returned errors, as they should be.
• Bugfix: in codegen, there was a parity break between the AssembleEntry method and AssembleKey+AssembleValue in generated struct NodeAssemblers. This has been fixed.
• Minor: ErrNoSuchField now uses PathSegment instead of a string. You probably won't notice (but this was important interally: we need it so we're able to describe structs with tuple representations).
• Bugfix: an error path during CID creation is no longer incorrectly dropped. (I don't think anyone ever ran into this; it only handled situations where the CID parameters were in some way invalid. But anyway, it's fixed now.)
• Performance: when cidlink.Link.Load is used, it will do feature detection on its io.Reader, and if it looks like an already-in-memory buffer, take shortcuts that do bulk operations. I've heard this can reduce memory pressure and allocation counts nicely in applications where that's a common scenario.
• Feature: there's now a fluent.Reflect convenience method. Its job is to take some common golang structs like maps and slices of primitives, and flip them into an IPLD Node tree. #81
  • This isn't very high-performance, so we don't really recommend using it in production code (certainly not in any hot paths where performance matters)... but it's dang convenient sometimes.
• Feature: there's now a traversal.SelectLinks convenience method. Its job is to walk a node tree and return a list of all the link nodes. #110
  • This is both convenient, and faster than doing the same thing using general-purpose Selectors (we implemented it as a special case).
• Demo: you can now find a "rot13" ADL in the adl/rot13adl package. This might be useful reference material if you're interested in writing an ADL and wondering what that entails. #98
• In progress: we've started working on some new library features for working with data as streams of "tokens". You can find some of this in the new codec/codectools package.
  • Functions are available for taking a stream of tokens and feeding them into a NodeAssembler; and for taking a Node and reading it out as a stream of tokens.
  • The main goal in mind for this is to provide reusable components to make it easier to implement new codecs. But maybe there will be other uses for this feature too!
  • These APIs are brand new and are extremely subject to change, much more so than any other packages in this repo. If you work with them at this stage, do expect to need to update your code when things shift.