v0.20.4

This patch release brings an urgent update to the DD scheduler interface, as well as a fix for a rare crash caused by an invalid workflow template structure.

• [core] Bump DD scheduler Protobuf interface
• [core] Prevent crash on empty defaults/vars maps

v0.20.3

This patch release fixes an unusual build issue with certain dependencies of the template system.

• [build] Update dependencies

v0.20.2

This patch release adds a DataDistribution scheduler plugin, as well as improvements to integration plugin loading and better coconut role query output.

• [apricot] Better init output
• [core] DD scheduler integration plugin
• [core][coconut] Display consolidated variable stack in role query
• [core] Fix integration plugin loading

v0.20.1

This is the first patch release in the v0.20 series, it brings a single fix to restore access to Consul-provided defaults and vars.

• [apricot] Fix defaults/vars path

v0.20.0

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new microservice Apricot ("a processor and repository for configuration templates"). AliECS now supports integration plugins, a new kind of plugin that exposes callable functions to workflow templates, and a prototype DCS integration plugin is provided based on this new mechanism. Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• Integration plugins and DCS:

  • [core] Add new INVARIANT machine state for callRoles
  • [core] Support for integration plugins
  • [core] Support for callRoles as triggers for integration plugins
  • [core] Ensure environments with calls still update status correctly

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [core] Fix NewID template function signature
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.90

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments. An in-depth overhaul of the O²/FLP configuration system is also included, represented by the new component Apricot (a processor and repository for configuration templates). Other changes include a rework of gRPC/Protobuf code generation, a reimplementation of the coconut conf subsystem, and miscellaneous bug fixes.

• Automatic state machine progression (environment auto driver) and event system:

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Apricot and coconut conf overhaul:

  • [apricot] A Processor and Repository for Configuration Templates
  • [apricot] Initial commit for apricot microservice
  • [apricot] Better component detection via viper in apricot.Instance()
  • [apricot] README.md
  • [apricot] Rebase coconut component commands on apricot
  • [apricot] Rebase and regenerate proto
  • [coconut] Initial support for RUNTYPE/rolename in config paths
  • [coconut] Improve coconut conf output
  • [coconut] Fix coconut conf import behavior
  • [coconut] Update coconut conf show for new Consul structure
  • [coconut] New --simulate flag to coconut conf show
  • [coconut] Update inline documentation for coconut conf
  • [common] Suppress bothersome log output
  • [configuration] Improve Get, GetKeysByPrefix; add IsDir
  • [configuration] Move component cfg template processing into confSvc
  • [configuration] Add IsDir to configuration.YamlSource
  • [configuration] Fix GetConfig logic for unversioned configuration
  • [configuration] Move repos package and "the" singleton to core
  • [core] Add GetConfigLiquid/GetConfigPongo to evaluate these engines
  • [core] Add support for config template inclusion
  • [core] Add support for RUNTYPE and rolename in config paths
  • [core][configuration] Move high level conf mgmt to configuration package
  • [core][apricot] Rearrange how the core is initialized from configuration

• gRPC/Protobuf generator:

  • [core] Migrate from gogo/protobuf to protobuf generator v2
  • [misc] Overhaul of Protobuf/gRPC tool handling in AliECS

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

• Miscellaneous:

  • [core] Fix StringWrapMap unmarshal occasionally breaking the var system
  • [core] Add utils.TimeTrack to be used with defer to measure exec time
  • [core] Add NewID function to workflow template environment
  • [misc] Update documentation
  • [occ] Dump incoming configuration payloads to file
  • [occ] Print out OCC version from dummyprocess

v0.19.80

This release brings major new features to AliECS, including post-DESTROY hooks, automatic state machine progression and an event system allowing real time state updates from tasks and environments.

• Automatic state machine progression (environment auto driver) and event system

  • [coconut] create auto subcommand for environment
  • [coconut] use --auto flag instead of subcommand
  • [coconut] Subscribe using xid
  • [coconut] Handle auto environment error
  • [common] Introduce Role, Task, Environment Event
  • [common] Add more info to the events structs
  • [common] Add message field to Environment Events
  • [core] environment autorun feature
  • [core] taskman utilize safeAcks
  • [core] NewAutoEnvironment should be a stream
  • [core] use EventFeed for subscriptions
  • [core] Get run number from args
  • [core] Connections struct to keep track of streams
  • [core] Remove obsolete events and update protos
  • [core] stream events per environment
  • [core] Wait for release tasks when cleanup a failing new Environment
  • [core] Send Environment teardown message
  • [core] create channel and add stream in NewAutoEnvironment
  • [core] Environment unsubscribe once
  • [core] Environment fix push to closed channel crash

• Logging:

  • [core] Output sandbox log URI in each ShortTaskInfo
  • [core] update protos

• DESTROY hooks:

  • [core][executor] Add support for DESTROY hooks
  • [executor] Ensure DESTROY hooks can be triggered even after task KILL

v0.19.2

This is a patch release that fixes several issues of varying severity that were discovered during Milestone Week 1.

Logging fixes:

• [common] Strip newlines from outbound InfoLogger messages
• [core] Reduce log verbosity
• [executor] stderr output of controlled processes now forwarded at DebugLevel
• [executor] Pipe controlled task stdout/stderr at TRACE level

Miscellaneous fixes:

• [common] Fix nil pointer crash when event is nil
• [core] Fix deadlock when unsubscribing from Wf state
• [core] Fix data race when gathering responses

v0.19.1

This is a patch release that fixes a cleanup issue on environment startup failure.

• [core] Fix clean up after failed creation of a new environment
• [misc] New script to create FLP-compatible LXD containers on demand
• [misc] Better wording for new root password prompt

v0.19.0

This release brings some performance and reliability improvements when terminating tasks. It also delivers some bug fixes, as well as an update to ODC integration.

Task termination:

• [executor] Tweak timeout system for the task kill procedure
• [executor] Always use the shell PGID as backup PID for killing
• [executor] Further tweak kill timeouts

ODC integration:

• [odcshim] Correctly handle early GetState calls

Miscellaneous:

• [core] Allow underscore in configuration path
• [misc] Gather more metrics in the test script
• [occ] Avoid using private namespace in gRPC
• [build] Toolchain version bump
• [build] gRPC, Protobuf version bump for OCC
• [build] Makefile output improvements on Mac