v0.13.1

This release brings critical fixes for the AliECS core and coconut.

• Miscellaneous:
  • [coconut] Accept CSV as well as JSON-formatted extra-vars
  • [core] Fix getWorkflowTemplates bug (OCTRL-219)
  • [core] Fix issue with constraint values not being template-processed

v0.13.0

This release adds an all-new template and variables system built on top of the workflow configuration mechanism. It also brings numerous improvements throughout AliECS, including reorganized path handling, task cleanup facilities, FLP Suite workflow versioning, an environment fast destroy operation, support for configuring shmem transport and ipc (named pipe) addressing, as well as API extensions for interfacing with the variables system.

• Variables system:
  • [coconut] Print variables in coconut env show
  • [coconut] Variables output in role query
  • [coconut] Acquire --extra-vars when creating environment
  • [coconut] Correctly handle env create with empty extra-vars
  • [coconut] Extract a new componentcfg package from configurationutil.go
  • [coconut] Add root role column in coconut env list
  • [coconut] Sort printed maps by key
  • [common] Make gera.Map implement yaml.Unmarshaler
  • [core] New hierarchical property map
  • [core] Add WrappedAndFlattened to hierarchical map
  • [core] Extend o2control.proto to accept environment properties
  • [core] Update proto and regenerate
  • [core] Add StringMap to gera package
  • [core] Add support for defaults and vars in confsys
  • [core] Implement variables KV in workflow
  • [core] Add properties to TaskInfo in protofile
  • [core] Task properties
  • [core] Use the full roleBase unmarshal path in aggregatorRole
  • [core] Output vars in GetEnvironment
  • [core][OCTRL-73] Add properties to running task
  • [core][OCTRL-73] Add properties to buildcommand properties
  • [core] Correctly unmarshal properties map into task.Class
  • [core] Add Copy method to gera.StringWrapMap
  • [core] Add defaults map to task template
  • [core] gera.FlattenStack convenience function
• Reorganized path handling:
  • [config] Add trailing slash to viper defaultRepo field
  • [core] Manage a single writable directory path
  • [misc] Update default executor path to a generic one
  • [repos] Consistent trailing slash use for repos
• Mesos lost task cleanup on startup:
  • [core] Persistent FID with reconciliation for previous running tasks
  • [core] Repos and runcounter.txt are stored inside coreWorkingDir
  • [core] Sanitize viper paths to remove trailing slash
  • [core] Use filepath.Join to concatenate paths
  • [core] Use path/filepath to remove trailing '/'
  • [core] Remove repositoriesPath from configuration
  • [core] Cleaner reposPath approach in service
  • [core] Use GetReposPath in repos manager
  • [core] Use service to get reposPath
• Mesos task cleanup on incoming UNIX signals:
  • [core] Handle UNIX signals
  • [core] Use taskman to cleanup and to get tasks
  • [core] Use proper exit status
• Support for per-repo default revision (FLP Suite workflow versioning):
  • [coconut] Fix repo default-revision bug & improve documentation
  • [coconut] Drop superfluous fmt.Sprintf from repo default-revision
  • [core+coconut] Introduce global default branch for repos
  • [core+coconut] Add default bracnh option for new repos and improve API and coconut
  • [core] Support perisistent default revisions per repo
  • [core] Update backend when a default-revision is explicitly set
  • [docs] Improve coconut repo add help dialogue
• Template engine:
  • [core] Implement new template engine
  • [core] Correctly handle precedence of template resolution
  • [core] Template support in constraints and objStack
  • [core] Add support for wrapping slices in template
  • [core] Parametrize basic task arguments and env vars
  • [core] Run template system on Task.properties
  • [core] Implement GetKeysByPrefix for all configuration.Sources
  • [core] Implement confsys.GetComponentConfiguration
  • [core] Implement To/FromJson and GetConfig functions for template system
  • [core] Template Task.Value and User
  • [core] Allow variables to control the expansion of an iteratorRole
  • [core] Fix issue with user provided vars not accessible
  • [core] Ensure correct template output when the result is a slice
  • [core] Fix bug with nil TaskCommandInfo preventing deployment
  • [core] Facilities for pushing runtime variables back up the stack
  • [core] Propagate task result to parent taskRole
  • [core] Include task class defaults in varStack for TaskCommandInfo
• Environment fast destroy feature:
  • [coconut][OCTRL-206] Add option in coconut to allow destroy env in running state
  • [core][OCTRL-206] Update core to destroy running env if specified
• Shared memory transport and named pipe addressing support:
  • [core] Overhaul the channel package to support TCP and IPC endpoints
  • [core] InboundChannels instead of BindPorts to support IPC files
  • [core] Use IPC-enabled endpoints instead of plain ports in task.Manager
  • [core] Update BindMap builder in scheduler to support named pipes
  • [core] Fix issue with misconfigured FairMQ transport
  • [core] Correctly acquire TCP/IPC addressing information
• Miscellaneous:
  • [build] Make "make generate" independent from $PATH and $GOPATH
  • [build] Improve Makefile output
  • [build] Bump and clean up dependencies
  • [coconut][OCTRL-209] Support --no-versioning flag in coconut conf import
  • [core] New proposed protofile for DCS integration
  • [core] Move NewUnixTimestamp to common/utils
  • [core] Suppress periodic output unless veryVerbose=true
  • [core] Add dumpWorkflows flag to core to control generation of workflow dumps
  • [core] Fix segfault for unsatisfied constraint or resource
  • [executor] Lower logrus severity of task stout/stderr
  • [executor] Make incoming MESSAGE handling concurrent
  • [executor] Ensure correct log level for task stdout/stderr
  • [executor] Prevent endless loop in ERROR state
  • [occ] Reduce log verbosity

v0.12.91

This is a pre-release in preparation for v0.13.

This release adds an all-new template and variables system built on top of the workflow configuration mechanism. It also brings numerous improvements throughout AliECS, including reorganized path handling, task cleanup facilities, FLP Suite workflow versioning, an environment fast destroy operation, as well as API extensions for interfacing with the variables system.

• Variables system:
  • [coconut] Print variables in coconut env show
  • [coconut] Variables output in role query
  • [coconut] Acquire --extra-vars when creating environment
  • [coconut] Correctly handle env create with empty extra-vars
  • [coconut] Extract a new componentcfg package from configurationutil.go
  • [coconut] Add root role column in coconut env list
  • [coconut] Sort printed maps by key
  • [common] Make gera.Map implement yaml.Unmarshaler
  • [core] New hierarchical property map
  • [core] Add WrappedAndFlattened to hierarchical map
  • [core] Extend o2control.proto to accept environment properties
  • [core] Update proto and regenerate
  • [core] Add StringMap to gera package
  • [core] Add support for defaults and vars in confsys
  • [core] Implement variables KV in workflow
  • [core] Add properties to TaskInfo in protofile
  • [core] Task properties
  • [core] Use the full roleBase unmarshal path in aggregatorRole
  • [core] Output vars in GetEnvironment
  • [core][OCTRL-73] Add properties to running task
  • [core][OCTRL-73] Add properties to buildcommand properties
  • [core] Correctly unmarshal properties map into task.Class
  • [core] Add Copy method to gera.StringWrapMap
  • [core] Add defaults map to task template
  • [core] gera.FlattenStack convenience function
• Reorganized path handling:
  • [config] Add trailing slash to viper defaultRepo field
  • [core] Manage a single writable directory path
  • [misc] Update default executor path to a generic one
  • [repos] Consistent trailing slash use for repos
• Mesos lost task cleanup on startup:
  • [core] Persistent FID with reconciliation for previous running tasks
  • [core] Repos and runcounter.txt are stored inside coreWorkingDir
  • [core] Sanitize viper paths to remove trailing slash
  • [core] Use filepath.Join to concatenate paths
  • [core] Use path/filepath to remove trailing '/'
  • [core] Remove repositoriesPath from configuration
  • [core] Cleaner reposPath approach in service
  • [core] Use GetReposPath in repos manager
  • [core] Use service to get reposPath
• Mesos task cleanup on incoming UNIX signals:
  • [core] Handle UNIX signals
  • [core] Use taskman to cleanup and to get tasks
  • [core] Use proper exit status
• Support for per-repo default revision (FLP Suite workflow versioning):
  • [core+coconut] Introduce global default branch for repos
  • [core+coconut] Add default bracnh option for new repos and improve API and coconut
  • [core] Support perisistent default revisions per repo
  • [core] Update backend when a default-revision is explicitly set
  • [docs] Improve coconut repo add help dialogue
• Template engine:
  • [core] Implement new template engine
  • [core] Correctly handle precedence of template resolution
  • [core] Template support in constraints and objStack
  • [core] Add support for wrapping slices in template
  • [core] Parametrize basic task arguments and env vars
  • [core] Run template system on Task.properties
  • [core] Implement GetKeysByPrefix for all configuration.Sources
  • [core] Implement confsys.GetComponentConfiguration
  • [core] Implement To/FromJson and GetConfig functions for template system
  • [core] Template Task.Value and User
  • [core] Allow variables to control the expansion of an iteratorRole
  • [core] Fix issue with user provided vars not accessible
  • [core] Ensure correct template output when the result is a slice
  • [core] Fix bug with nil TaskCommandInfo preventing deployment
  • [core] Facilities for pushing runtime variables back up the stack
  • [core] Propagate task result to parent taskRole
  • [core] Include task class defaults in varStack for TaskCommandInfo
• Environment fast destroy feature:
  • [coconut][OCTRL-206] Add option in coconut to allow destroy env in running state
  • [core][OCTRL-206] Update core to destroy running env if specified
• Miscellaneous:
  • [build] Make "make generate" independent from $PATH and $GOPATH
  • [build] Improve Makefile output
  • [coconut][OCTRL-209] Support --no-versioning flag in coconut conf import
  • [core] New proposed protofile for DCS integration
  • [core] Move NewUnixTimestamp to common/utils
  • [core] Suppress periodic output unless veryVerbose=true
  • [core] Add dumpWorkflows flag to core to control generation of workflow dumps
  • [executor] Lower logrus severity of task stout/stderr
  • [executor] Make incoming MESSAGE handling concurrent
  • [executor] Ensure correct log level for task stdout/stderr
  • [occ] Reduce log verbosity

v0.12.90

This is a pre-release in preparation for v0.13.

This release adds an all-new template and variables system built on top of the workflow configuration mechanism. It also brings numerous improvements throughout AliECS, including reorganized path handling, task cleanup facilities, FLP Suite workflow versioning, an environment fast destroy operation, as well as API extensions for interfacing with the variables system.

• Variables system:
  • [coconut] Print variables in coconut env show
  • [coconut] Variables output in role query
  • [coconut] Acquire --extra-vars when creating environment
  • [coconut] Correctly handle env create with empty extra-vars
  • [coconut] Extract a new componentcfg package from configurationutil.go
  • [coconut] Add root role column in coconut env list
  • [coconut] Sort printed maps by key
  • [common] Make gera.Map implement yaml.Unmarshaler
  • [core] New hierarchical property map
  • [core] Add WrappedAndFlattened to hierarchical map
  • [core] Extend o2control.proto to accept environment properties
  • [core] Update proto and regenerate
  • [core] Add StringMap to gera package
  • [core] Add support for defaults and vars in confsys
  • [core] Implement variables KV in workflow
  • [core] Add properties to TaskInfo in protofile
  • [core] Task properties
  • [core] Use the full roleBase unmarshal path in aggregatorRole
  • [core] Output vars in GetEnvironment
  • [core][OCTRL-73] Add properties to running task
  • [core][OCTRL-73] Add properties to buildcommand properties
  • [core] Correctly unmarshal properties map into task.Class
  • [core] Add Copy method to gera.StringWrapMap
  • [core] Add defaults map to task template
  • [core] gera.FlattenStack convenience function
• Reorganized path handling:
  • [config] Add trailing slash to viper defaultRepo field
  • [core] Manage a single writable directory path
  • [misc] Update default executor path to a generic one
  • [repos] Consistent trailing slash use for repos
• Mesos lost task cleanup on startup:
  • [core] Persistent FID with reconciliation for previous running tasks
  • [core] Repos and runcounter.txt are stored inside coreWorkingDir
  • [core] Sanitize viper paths to remove trailing slash
  • [core] Use filepath.Join to concatenate paths
  • [core] Use path/filepath to remove trailing '/'
  • [core] Remove repositoriesPath from configuration
  • [core] Cleaner reposPath approach in service
  • [core] Use GetReposPath in repos manager
  • [core] Use service to get reposPath
• Mesos task cleanup on incoming UNIX signals:
  • [core] Handle UNIX signals
  • [core] Use taskman to cleanup and to get tasks
  • [core] Use proper exit status
• Support for per-repo default revision (FLP Suite workflow versioning):
  • [core+coconut] Introduce global default branch for repos
  • [core+coconut] Add default bracnh option for new repos and improve API and coconut
  • [core] Support perisistent default revisions per repo
  • [core] Update backend when a default-revision is explicitly set
  • [docs] Improve coconut repo add help dialogue
• Template engine:
  • [core] Implement new template engine
  • [core] Correctly handle precedence of template resolution
  • [core] Template support in constraints and objStack
  • [core] Add support for wrapping slices in template
  • [core] Parametrize basic task arguments and env vars
  • [core] Run template system on Task.properties
  • [core] Implement GetKeysByPrefix for all configuration.Sources
  • [core] Implement confsys.GetComponentConfiguration
  • [core] Implement To/FromJson and GetConfig functions for template system
  • [core] Template Task.Value and User
  • [core] Allow variables to control the expansion of an iteratorRole
  • [core] Fix issue with user provided vars not accessible
  • [core] Ensure correct template output when the result is a slice
  • [core] Fix bug with nil TaskCommandInfo preventing deployment
  • [core] Facilities for pushing runtime variables back up the stack
  • [core] Propagate task result to parent taskRole
  • [core] Include task class defaults in varStack for TaskCommandInfo
• Environment fast destroy feature:
  • [coconut][OCTRL-206] Add option in coconut to allow destroy env in running state
  • [core][OCTRL-206] Update core to destroy running env if specified
• Miscellaneous:
  • [build] Make "make generate" independent from $PATH and $GOPATH
  • [build] Improve Makefile output
  • [core] New proposed protofile for DCS integration
  • [core] Move NewUnixTimestamp to common/utils
  • [core] Suppress periodic output unless veryVerbose=true
  • [executor] Lower logrus severity of task stout/stderr
  • [executor] Make incoming MESSAGE handling concurrent
  • [executor] Ensure correct log level for task stdout/stderr

v0.12.2

This release fixes a bug which resulted in missing log messages.

• Miscellaneous:
  • [core] Fill out logrus fields

v0.12.1

This release fixes a build system issue which resulted in a broken core binary.

• Miscellaneous:
  • [build] Bump dependencies
  • [build] Fix viper dependency and go mod tidy

v0.12.0

This release brings support for basic tasks and other improvements.

• Basic tasks:
  • [core] Add BASIC control mode
  • [core] Normalize interface of task.Task
  • [core] Fix CommandInfo for basic tasks and handle BASIC_TASK_TERMINATED
  • [core] Make transition reacting to END_OF_STREAM asynchronous
  • [core] SafeToStop mechanism in task.Task
  • [executor] Implement BasicTask transitioner
  • [executor] Fix crash on rerunning basic task within existing env
  • [executor] Implement basic task handlers and transitioner
  • [occ][executor] Add BASIC_TASK_TERMINATED event and update END_OF_STREAM
• Executor refactor:
  • [executor] Refactor for lockless concurrency and ControllableTask
  • [executor] Further refactor Transitioner-RPC relationship
  • [executor] Remove taskCmd from executor task base
• Miscellaneous:
  • [build] Support building on Mac
  • [coconut + core] Improvements in template list feature
  • [common] Improve log processing
  • [core] Tone down debug output
  • [core] Cleanup on failed deployment
  • [core] Correctly report deployment error
  • [misc] Add fx3-copy-executors utility script for quick lab deployment
  • [misc] Add protoString utility function to utils
  • [misc] Migrate from dep to Go Modules
  • [misc] Refresh coconut documentation

v0.11.1

This release brings a gRPC API fix to facilitate the implementation of the repo refresh feature in the AliECS GUI.

• Miscellaneous:
  • [core] Update repo rpcs to return errors
  • [core] Regenerate proto

v0.11.0

This release brings API improvements to the Git-based workflow configuration mechanism in order to support git-based workflow configuration in the AliECS GUI. It also adds a task configuration import and query feature to the coconut configuration command which can be used to manage task configuration in Consul, in preparation for the upcoming task configuration push mechanism.

• Workflow template API:
  • [coconut][core] Implement coconut template queries + API
  • [core] Task class always identified by hash
  • [core] Regenerate proto
  • [core] Update defaults for templ list API
• Task configuration Consul import:
  • [coconut] Update documentation
  • [coconut] Generate coconut user documentation
  • [coconut] coconut conf list command
  • [coconut] coconut conf show command
  • [coconut] coconut conf history command
  • [coconut] coconut conf import command
• Miscellaneous:
  • [executor] Fix bug which prevented configuration payload to be printed
  • [executor] Improve configuration kv output

v0.10.1

Merge pull request #61 from ktf/fix-warnings

Fix warnings on macOS