v1.0.0-rc.3

Summary

More fixes for Postgres database migrations.

What's Changed

• Fix postgres migrations and add to PR checks by @nguyer in #690

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

v1.0.0-rc.2

Summary

Minor internal rearrangement and fixes for Postgres database migrations.

What's Changed

• Configuration docs, and packaging config/i18n for re-use by microservices by @peterbroadhurst in #678
• Fix make docker command in release build GitHub Action by @nguyer in #682
• Enforce config descriptions by @nguyer in #681
• Remove extra postgres migrations by @nguyer in #683

Full Changelog: v1.0.0-rc.1...v1.0.0-rc.2

v1.0.0-rc.1

Summary

• Significant performance improvements
  • More types of background workers
  • Better caching and indexing
  • Batching of database updates
• Greatly improved OpenAPI spec, including more accurate input/output params and text descriptions of all fields
• UI updates and general bug fixes

Breaking Changes and Migrations

• firefly-dataexchange-https must be upgraded to the version listed below, as the ack event has been renamed from "commit" to "ack". Running mismatched versions of FireFly and dataexchange in this case will cause event processing to stall completely.

Updated Dependencies

• firefly-ethconnect v3.1.6
• firefly-dataexchange-https v0.11.1
• firefly-tokens-erc1155 v0.10.8
• firefly-tokens-erc20-erc721 v0.2.3
• firefly-ui v0.6.10

What's Changed

• Cache blockchain events for event enrichment by @peterbroadhurst in #638
• Add index on message_id for token transfers by @peterbroadhurst in #639
• Update txType in private message schema by @gabriel-indik in #637
• Fix transactions_id index by @peterbroadhurst in #643
• Update our cache for event enrichment on message confirm by @peterbroadhurst in #642
• Add extra convenience functions for identities by @peterbroadhurst in #641
• Fix IF EXISTS check on transactions_id INDEX migration by @peterbroadhurst in #645
• Call out memory needed in docs by @keeganhrobinson in #649
• Add background workers to flush updates to operations by @peterbroadhurst in #646
• Use a single commit to generate all batch ops, before initiating the HTTP calls by @peterbroadhurst in #647
• Assign nonces more efficiently, with minimal DB ops by @peterbroadhurst in #650
• Change messages_data join table indexes for data_id lookup by @peterbroadhurst in #653
• Cannot assume zero as reason to insert vs. update by @peterbroadhurst in #652
• [charts-config] chart buckets are now capped at a configurable max by @eberger727 in #651
• [chart-test-fix] chart testing fix by @eberger727 in #655
• Move DX events interface to async with callback by @peterbroadhurst in #656
• Avoid blocking event processing threads when queuing rewinds by @peterbroadhurst in #657
• Proper handling of token approval events by @shorsher in #662
• Latest UI v0.6.9 by @hfuss in #667
• [charts-isCapped] adding isCapped key to charts response by @eberger727 in #669
• Remove misleading payload_ref DB field and fix Group JSON tag by @peterbroadhurst in #666
• Add rewinder to handle enrichment off event poller critical path by @peterbroadhurst in #665
• Refactor OpenAPI Spec generation by @nguyer in #658
• Remove references to old firefly-e2e stack name by @awrichar in #671
• Clean up blockchain parameters returned by token connectors by @awrichar in #673
• Check up-front for duplicate token pool name by @awrichar in #674
• Do not update cache until messages/data assured to be in DB by @peterbroadhurst in #670
• [ui-v0.6.10] manifest by @eberger727 in #676
• Update manifest for v1.0.0-rc.1 by @nguyer in #677

New Contributors

• @keeganhrobinson made their first contribution in #649

Full Changelog: v0.14.1...v1.0.0-rc.1

v0.14.1

What's Changed

• Switch from https to ffdx by @gabriel-indik in #590
• [fetchreferences2] Adding event enrichment for all event types by @eberger727 in #623
• Re-read configuration on restart of orchestrator by @peterbroadhurst in #624
• Add pool config to activate call by @peterbroadhurst in #626
• [ui-v0.6.3] v0.6.3 release of ui in manifest by @eberger727 in #629
• Do not return an error from status if node lookup fails by @peterbroadhurst in #630
• Update batch manager dispatch to track inflight and fix private blobs by @peterbroadhurst in #633
• Created timestamp index on events shouldn't be unique by @peterbroadhurst in #634
• [ui-v0.6.6] manifest release by @eberger727 in #635
• [ui-v0.6.7] ui manifest by @eberger727 in #636

Full Changelog: v0.14.0...v0.14.1

v0.14.0

Summary

• Major UI Update - Includes Activity, Blockchain, Off-Chain, Tokens, Network Map, and My Node sections (https://github.com/hyperledger/firefly-ui)
• Custom contract APIs complete
• Enhanced subscription filters
• Event API enrichment
• Performance updates
• Bug fixes

Breaking Changes and Migrations

• FIR-9 Enhanced Identity API - custom identities and DID support
  • New namespaced API endpoints to manage identities within a namespace:
    • POST /api/v1/namespaces/identities - async broadcast, with confirm=true option for sync
    • GET /api/v1/namespaces/identities - collection of identities within namespace
    • GET /api/v1/namespaces/identities/{iid} - individual identity
    • GET /api/v1/namespaces/identities/{iid}/did - generated DID document
    • GET /api/v1/namespaces/identities/{iid}/verifiers - collection of keys
  • New non-namespaced endpoint is provided to look up identities by DID:
    • GET /api/v1/network/identities/{did} - efficient cached DID to identity resolution
  • Minor changes to existing APIs
    • GET /api/v1/network/organizations - now returns identity object
    • GET /api/v1/network/nodes - now returns identity object
    • GET /api/v1/status - now returns the did of the local organization
    • The identity field in nodes/orgs has been removed (used to contain the signing key)
    • The did field from identity is newly available on nodes and orgs
  • Existing node and organization identities are migrated to the new identity API
    • No migration support for non-root organizations, as the FIR-9 security model requires a verification transaction from the parent organization that is not available.
    • Existing root organization and node objects are migrated
    • New nodes joining a network can process previously published organization and node identities
    • All organizations now represented as an identity in the ff_system namespace with special types org and node
• Operation type strings have been updated for consistency:
  • blockchain_pin_batch - previously blockchain_batch_pin
  • dataexchange_send_batch - previously dataexchange_batch_send
  • dataexchange_send_blob - previously dataexchange_blob_send
  • sharedstorage_upload_batch - previously sharedstorage_batch_broadcast
  • sharedstorage_upload_blob - new
  • sharedstorage_download_batch - new
  • sharedstorage_download_blob - new
• Blockchain events
  • The /api/v1/contracts/subscriptions endpoint has been renamed to /api/v1/contracts/listeners
  • The /messages/{msgid}/operations endpoint has been removed
  • Events of type blockchain_event are now type blockchain_event_received
• The Ethereum plugin now expects a different input format to generate a FireFly Contract Interface from an Ethereum ABI. Please see the docs for full details.
• The Public Storage plugin interface has been renamed to Shared Storage. The FireFly core config should be updated to replace publicstorage with sharedstorage as a root key. This release of FireFly is still backwards compatible with the old config key.

Updated Dependencies

• firefly-ethconnect v3.1.5
• firefly-fabconnect v0.9.13
• firefly-tokens-erc1155 v0.10.6
• firefly-tokens-erc20-erc721 v0.2.0
• firefly-ui v0.6.0

What's Changed

• FIR-9: Identity enhancements by @peterbroadhurst in #549
• Rename contract subscriptions to contract listeners by @nguyer in #548
• Rename Public Storage to Shared storage by @nguyer in #538
• Add missing itype column in SQLite migration by @peterbroadhurst in #577
• Add Operations Manager, wrap all operations, and add retry functionality by @awrichar in #517
• Handle operations with retries in transaction status by @awrichar in #578
• [charts-by-type] histogram broken down by type by @eberger727 in #547
• Enhance subscription filters & event enrichment by @shorsher in #579
• add fetchreferences param to events api by @shorsher in #587
• Use CLI to deploy test contract by @nguyer in #589
• Update batch logic to only store+hash the manifest by @peterbroadhurst in #582
• Batching message writer routines, to use multi-value insert across API calls on message/data insert by @peterbroadhurst in #592
• Event lookup cache by @peterbroadhurst in #593
• Add DB tuning to keep conns option, and correct cache bypass on group lookup by @peterbroadhurst in #594
• Scope subscription check to the appropriate stream ID by @jebonfig in #591
• Move blob publish to broadcast batch dispatch, ensure public refs on inbound by @peterbroadhurst in #596
• Performance enhancements from flame graph analysis by @peterbroadhurst in #599
• Remove route /messages/{msgid}/operations by @awrichar in #602
• Upgrade dependencies by @nguyer in #605
• Add "info" field to token pools, consume "symbol" from connectors by @awrichar in #606
• [blockchainevents-charts] support for blockchain event charting by @eberger727 in #612
• Add E2E test support for ERC20/ERC721 by @awrichar in #611
• Add firstEvent field to contract listeners by @nguyer in #603
• Dependency updates for v0.14 testing by @peterbroadhurst in #613
• Batch confirm messages, and update events to contain topic+tag by @peterbroadhurst in #600
• Change node/org lookups to support name or ID by @awrichar in #616
• Update microservices versions for v0.14.0 by @nguyer in #615
• Restart stack and re-run E2E tests in daily integration job by @nguyer in #608
• Add Shared Storage Download Manager by @peterbroadhurst in #607
• Add route to lookup identity by DID by @awrichar in #617
• Blockchain metrics for Ethereum by @nguyer in #584
• [ui-v0.6.0] PR for new UI release by @eberger727 in #619
• Custom contracts docs by @nguyer in #604
• [ui-v0.6.0-release] new ui release by @eberger727 in #620

Full Changelog: v0.13.1...v0.14.0

v0.13.1

Summary

• Support for token approvals (only supported by erc1155 connector as of this release)
• Bug fixes

Updated Dependencies

• firefly-ethconnect v3.1.4
• firefly-fabconnect v0.9.11
• firefly-tokens-erc1155 v0.10.5
• firefly-tokens-erc20-erc721 v0.1.6

What's Changed

• Add additional backwards compatibility check on ethereum instance path by @nguyer in #531
• Build a predictable response body for async "InvokeContract" calls by @awrichar in #532
• Add endpoint to automatically generate an FFI by @nguyer in #511
• Updates to DX by @gabriel-indik in #534
• Add Transaction ID to Events by @awrichar in #522
• Add query support for Fabric by @nguyer in #533
• [transfer-type-query] allow filtering by type on token transfers by @eberger727 in #544
• Change query method to POST for custom contracts by @nguyer in #542
• Fix empty swagger URLs on contract API creation by @nguyer in #543
• Support for token approvals by @shorsher in #529
• Docs "bundle update" to pick up new nokogiri build by @peterbroadhurst in #552
• Documentation updates from @TheSwarnim by @peterbroadhurst in #553
• Updated documentation to make it more useful by @TheSwarnim in #451
• Roll requestConflictEmptyResult from Group to Data and Message too by @peterbroadhurst in #551
• Explicitly name all subscriptions by @peterbroadhurst in #555
• Update manifest versions by @awrichar in #556

New Contributors

• @gabriel-indik made their first contribution in #534
• @TheSwarnim made their first contribution in #451

Full Changelog: v0.13.0...v0.13.1

v0.13.0

Summary

This is an important hardening release - while very little has changed on the surface, it includes a lot of significant rework to the core of FireFly, mostly to fix issues exposed by the performance testing in #316.

Breaking changes:

• The token_pool_rejected event was removed, along with the recently-added contract_interface_rejected and contract_api_rejected (only message_rejected will now be emitted for rejected definitions)
• The expected format of the FireFly contract address in the "instance" config entry for Ethereum blockchains has changed (see "Migrations" below)
• The default data exchange plugin has been renamed from https to ffdx (old name will still be accepted with a warning, but this will eventually become a fatal config error)

Other major items included in this release:

• Support for running on ARM-based M1 processors
• Rewrite of the message batching and event aggregation logic inside FireFly, to fix numerous edge cases with lost or hung messages
• Hardening of operations and transactions to behave more consistently across all types
• Metrics reporting to Prometheus
• Continued development to support custom on-chain logic (still in preview)

Migrations

• The Ethereum plugin now expects just a contract address in the instance field of the FireFly core config file. The value should be a hex string (with the 0x prefix) which is the Ethereum address of the FireFly smart contract. It is recommended to update any existing config file for this release, though this version will attempt to read values from existing configs and determine the contract address from the old value. Note: FireFly will not automatically update the config file as it treats it as read-only.

Here is an example of what your blockchain section should look like (your actual instance value will be a different hex string):

blockchain:
  type: ethereum
  ethereum:
    ethconnect:
      instance: 0x8707d1f1151220430fc8f89836b20bcf05d6eb41

Updated Dependencies

• firefly-ethconnect v3.1.3
• firefly-fabconnect v0.9.9
• firefly-dataexchange-https v0.10.4
• firefly-tokens-erc20-erc721 v0.1.5

What's Changed

• Store all token transfer inputs on operation by @awrichar in #478
• Make the factories act like factories by @awrichar in #474
• Add missing test coverage in txcommon by @awrichar in #476
• Use PreFinalize/Finalize in all definition handlers by @awrichar in #475
• wsReader cannot rely on testing.T as it might extend past test scope by @peterbroadhurst in #472
• Set default keepalive timeout to 5s on clients for Node.js connectors by @peterbroadhurst in #470
• Address some inconsistencies in Operation behavior by @awrichar in #480
• Remove extra "rejected" events by @awrichar in #482
• Update aggregator batch processing to maintain in-memory pin state until OnFinalize by @peterbroadhurst in #483
• Fix fabric event timestamp parsing by @jimthematrix in #491
• Fix Docker build on ARM by @nguyer in #473
• Add extra logging for message/event ids by @peterbroadhurst in #488
• Pass arguments from Makefile to docker build script by @nguyer in #495
• Fix possibility for nil access in sendloop by @peterbroadhurst in #486
• Fix group to be constant throughout test by @peterbroadhurst in #490
• Compile directly with solc to avoid full Node.js npm install of Truffle by @peterbroadhurst in #494
• Rewind when messages appear behind offset by @peterbroadhurst in #496
• Pass requestId to DX and remove backend_id from Operation by @awrichar in #497
• Rename data exchange plugin to "ffdx" by @awrichar in #505
• Misc fixes for operations by @awrichar in #498
• Use inline ABI for all Ethconnect contract interactions by @nguyer in #502
• Prometheus metrics by @eberger727 in #503
• Re-work batch logic for simplicity, efficiency, and restart recovery by @peterbroadhurst in #501
• Add "initEnabled" config to DX by @awrichar in #507
• Move insert of events to a special pre-commit phase of the DB transaction by @peterbroadhurst in #510
• Flatten the DXInfo back to the orginal payload on the ffdx API to connectors by @peterbroadhurst in #514
• Implement and use UpsertOptimization for groups by @peterbroadhurst in #513
• Update components ready for v0.13.0 by @peterbroadhurst in #516
• Update EthConnect to 3.1.3 by @peterbroadhurst in #518
• [metrics-types] using fftypes in metrics manager by @eberger727 in #520
• Provide Migration / Registration / Debugging Utilities in Docker Image by @hfuss in #521
• go-migrate in Dockerfile by @hfuss in #524
• Fix pages build by @peterbroadhurst in #526
• Use bash for variable expansion prior to make by @peterbroadhurst in #528
• Allow building in docker by removing .git from .dockerignore by @peterbroadhurst in #530

Full Changelog: v0.12.0...v0.13.0

v0.12.0

Summary

This is a large release as we come closer to the 1.0.0 milestone. All users are recommended to upgrade to this release when possible, taking into account all breaking API changes noted here.

Breaking API changes:

• All APIs deprecated in v0.11.0 or earlier are removed - see detailed migration notes
• Richer query parameters for filtering collections - with breaking changes to the ^ operator

Other major items included in this release:

• Preview of custom on-chain logic
  • Provides flexible methods for interacting with your own contracts, deployed to the blockchain of your choice
  • New APIs are available under /namespaces/{ns}/contracts and/namespaces/{ns}/apis
  • Not yet guaranteed to be stable, and API is subject to change
• Support for new ERC20 / ERC721 connector
• Overhaul of Transaction type, and new BlockchainEvent type
• Support for delivery confirmations via DX plugin
• Support for more flexible signing via new config pointing at an external address resolver
• First-class support for blob filenames on Data objects
• UI enhancements
  • Improved support for navigating and downloading file blobs
  • Filtering for token screens (pools, transfers, accounts)
• Bug fixes and improvements

Updated Dependencies

• firefly-ethconnect v3.1.2
• firefly-fabconnect v0.9.7
• firefly-dataexchange-https v0.10.3
• firefly-tokens-erc1155 v0.10.4
• firefly-tokens-erc20-erc721 v0.1.4
• firefly-ui v0.5.0

What's Changed

• Fixing Websocket Connections when Prometheus Metrics Enabled by @hfuss in #371
• Set fetch-depth to get the full history in GitHub Actions by @nguyer in #377
• Fetch full history on all GitHub Actions by @nguyer in #378
• [helm] Publishing Helm Chart to GHCR via OCI by @hfuss in #359
• [psql-maxconns] default connection limit for postgresql by @eberger727 in #385
• Allow any company name in license header by @nguyer in #379
• Prevent send/receive deadlock in wsclient by @awrichar in #387
• API changes for Blob Friendly Names by @peterbroadhurst in #381
• Fix unit test for wsclient send failure by @awrichar in #388
• Fixing Duplicate Metrics Registration when Unsetting preInit by @hfuss in #389
• Have DB creation script account for username by @drewmarshburn in #375
• Add Performance Testing Details to README by @eberger727 in #373
• Add postgres migrations for onchain-logic by @nguyer in #390
• FIR-4 (rich query updates) and FIR-7 (DX manifests) integration to onchain-logic branch by @peterbroadhurst in #403
• Rectify Transactions and BlockchainEvents by @awrichar in #408
• Build the URL without the firefly port if it was omitted from stack.json by @jebonfig in #411
• Collapse onchain-logic into main by @awrichar in #410
• [erc20-payload] pass name and symbol to CreateTokenPool() by @eberger727 in #413
• Removing Helm Chart and Related CI Workflow by @hfuss in #412
• Update containerd by @peterbroadhurst in #414
• PostgreSQL updates following merging of onchain-logic branch by @peterbroadhurst in #417
• Fix token account listing on Postgres by @awrichar in #422
• Fix intermittent Fabric E2E test failure by @nguyer in #424
• Add ping/pong heartbeating to WSClient, and fix concurrent map on config by @peterbroadhurst in #420
• Fail transaction when token transfer operation fails by @awrichar in #435
• Ready state changes require a bump to the message to re-sequence it by @peterbroadhurst in #438
• JSON Schema validation for FFIs by @nguyer in #419
• Restore Transaction reference on TokenTransfer by @awrichar in #444
• Create a Transaction and Operation for contract invoke requests by @awrichar in #441
• Rename /contracts/events to /blockchainevents by @awrichar in #443
• Add convenience URL /transactions/{txnid}/blockchainevents by @awrichar in #446
• Allow calling JSONObject() on nil JSONAny by @peterbroadhurst in #456
• Add Blockchain Transaction IDs to FireFly Transaction objects and add blockchain IDs to events by @peterbroadhurst in #448
• Fixing E2E Tests for Multi-Member Networks by @hfuss in #459
• Avoid potential double-close of channels due to reconnects after tests complete by @peterbroadhurst in #458
• Add route for /transactions/{txnid}/status by @awrichar in #457
• Add AddressResolver to Ethereum for key-to-address mapping by @peterbroadhurst in #436
• Allow synchronous 200 response for token pool creation by @awrichar in #460
• Remove deprecated APIs by @peterbroadhurst in #452
• ui version 0.5.0 by @shorsher in #464
• Add DefinitionBatchActions to execute after a batch of system definitions by @awrichar in #462
• Use /firefly prefix on local docs dev by @peterbroadhurst in #461
• Allow passing a tokenIndex to "mint" by @awrichar in #465
• [v0.12.0-manifests] manifests for v0.12.0 by @eberger727 in #466

New Contributors

• @drewmarshburn made their first contribution in #375

Full Changelog: v0.11.4...v0.12.0

v0.11.5

Summary

• Backported bug fixes

What's Changed

• v0.11.x backport: Fix querying transfers by tx or type, batches by tx by @awrichar in #429
• v0.11.x backport: Fixing Websocket Connections when Prometheus Metrics Enabled + containerd update by @peterbroadhurst in #432
• v0.11.x backport: Fix token account listing on Postgres by @peterbroadhurst in #431
• v0.11.x backport: Add ping/pong heartbeating to WSClient, and fix concurrent map on config by @peterbroadhurst in #430
• v0.11.x backport: Ready state changes require a bump to the message to re-sequence it by @peterbroadhurst in #439

Full Changelog: v0.11.4...v0.11.5

v0.11.4

Summary

• Dependency updates and bug fixes

Updated Dependencies

• firefly-fabconnect v0.9.4

What's Changed

• Config map length fix, fix bug with missing events by closing the websocket by @jebonfig in #367
• Update dependencies for next release by @peterbroadhurst in #369
• Fix misspelled filename (bytetypes.go) by @awrichar in #368
• Validate message topics/tag before sending by @awrichar in #365

Full Changelog: v0.11.3...v0.11.4