Omniconf

I I I 16.4k downloads I

Command-line arguments. Environment variables. Configuration files. Java properties. Almost every program requires some configuration which is usually spread around multiple sources. Keeping track of all of them, mapping ones to others, making sure they are present and correct, passing them around is a laborious and thankless task.

Configuration libraries, which there are plenty of, promise to solve the configuration problem, and they do. However, they usually provide only the mapping of configuration sources to Clojure data, leaving out the verification part. Omniconf’s value proposition, among other features, is the requirement to declare the expected configuration upfront, and the ability to validate the configuration early and display helpful messages if the application is misconfigured.

In terms of configuration sources, Omniconf supports:

• Environment variables
• CLI arguments
• Java properties
• EDN files
• AWS SSM (+dynamic reconfiguration)

Rationale

Omniconf is developed with the following principles in mind:

1. Explicit over implicit. Most configuration libraries allow to grab a configuration value (e.g. from ENV) at any point of time from any place in the code. This makes it impossible to list all configuration options that the program uses without reading the entire source. Omniconf requires you to declare all possible configuration options upfront, at the beginning of the program execution.
2. All configuration sources must be unified. It shouldn’t matter where the option is set from — environment variables, CLI arguments, or config files. It is uniformly initialized, verified, and accessed as regular Clojure data.
3. Maximum verification. We don’t want to see NumberFormatException stacktraces in the middle of your program run because of a typo in the environment variable. The whole configuration should be checked early and automatically before the rest of the code is executed. If there are any problems with it, a helpful message should be presented to the user.

Installation

Add this line to your list of dependencies:

Usage

1. You start by defining a set of supported options. cfg/define takes a map of options to their different parameters. The following small example shows the syntax:

   (require '[omniconf.core :as cfg])
   (cfg/define
     {:hostname {:description "where service is deployed"
                 :type :string
                 :required true}
      :port     {:description "HTTP port"
                 :type :number
                 :default 8080}})
       

   The full list of supported parameters is described here.

2. Populate the configuration from available sources:

   (cfg/populate-from-cmd args) ;; args is a command-line arguments list
   (when-let [conf (cfg/get :conf)]
     (cfg/populate-from-file conf))
   (cfg/populate-from-properties)
   (cfg/populate-from-env)
       

   The order in which to tap the sources is up to you. Perhaps you want to make environment variables overwrite command-line args, or give the highest priority to the config file. In the above example we get the path to the configuration file as --conf CMD argument. For more information, see this.

3. Call verify. It marks the boundary in your system, after which the whole configuration is guaranteed to be complete and correct.

   (cfg/verify)
       

   If there is something wrong with the configuration, verify will throw a proper exception. If called not in the REPL environment, the exception will be stripped of its stacktrace, so that you only see the exact error.

   If everything is alright, verify will pretty-print the whole configuration map into the standard output. It is convenient because it gives you one final chance to look at your config values and make sure they are good. :silent true can be passed to verify to prevent it from printing the map.

4. Use get to extract arbitrary value from the configuration.

   (cfg/get :hostname)
       

   For nested values you can pass an address of the value, either as a vector, or like varargs:

   (cfg/get :database :ip)
   (cfg/get [:database :ip])
       

   set allows you to change a value. It is definitely not recommended to be used in production code, but may be convenient during development:

   (cfg/set :database :port 3306)
   (cfg/set [:database :port] 3306)
       

Examples

Sample programs that use Omniconf: example-lein and example-boot. There is not much difference in using Omniconf with these build tools, but Boot requires a little hack to achieve parity with Leiningen.

Configuration scheme syntax

Configuration scheme is a map of option names to maps of their parameters. Option name is a keyword that denotes how the option is retrieved inside the program, and how it maps to configuration sources. Naming rules are the following:

For command-line arguments:

:some-option    =>    --some-option

For environment variables:

:some-option    =>    SOME_OPTION

For Java properties:

:some-option    =>    some-option   (java -Dsome-option=... if set from command line)

Each option can have the following parameters:

• :description — string that describes this option. This description will be used to generate a help message for the program.
• :type — currently the following types are supported: :string, :keyword, :number, :boolean, :edn, :file, :directory. Setting a type automatically defines how to parse a value for this option from a string, and also verifies that the resulting value has the correct Clojure type.

  Boolean types have special treatment. When setting them from the command line, one can omit the value completely.

  (cfg/define {:foo {:type :boolean}, :bar {:type :boolean}})
  ...
  $ my-app --foo --bar    # Confmap is {:foo true, :baz true}
      

  A string parser for booleans treats strings “0” and “false” as false, anything else as true.

• :parser — a single-arg function that converts a string value (given in command-line option or environment variable) into a Clojure value. This option can be used instead of :type if you need a custom option type.
• :default — the option will be initialized with this value. The default value must be specified as a Clojure datatype, not as a string yet to be parsed.

  The value for :default can be a nullary function used to generate the actual default value. This function will be invoked during the verification phase or on first direct access to the value, whichever happens first; thus, default functions will have access to other config values provided by the user. Note that you must invoke (cfg/enable-functions-as-defaults) first for this feature to work. Example:

  (cfg/enable-functions-as-defaults)
  (cfg/define {:host {:type :string}
               :port {:type :number}
               :connstring {:type :string
                            :default #(str (cfg/get :host) ":" (cfg/get :port))}})
  (cfg/populate-from-map {:host "localhost", :port 8888})
  (cfg/get :connstring) ;; => "localhost:8888"
      

  Even if a config option has a functional default, its value can be explicitly set from any configuration source to a normal value, and in that case the default function won’t be invoked.

  Make sure that you don’t try to cfg/get an option with a function default value before the values that function depends on are populated.

• :required — if true, the value for this option must be provided, otherwise verify will fail. The value of this parameter can also be a nullary function: if the function returns true then the option value must be provided. It is convenient if the necessity of an option depends on the values of some other options. Example:

  (cfg/define {:storage   {:one-of [:file :s3]}
               :s3-bucket {:required #(= (cfg/get :storage) :s3)}})
      

• :one-of — a sequence of values that an option is allowed to take. If the value isn’t present in the :one-of list, verify will fail. :one-of automatically implies :required true unless you add nil as a permitted value.
• :verifier — a function of [option-name value] that should throw an exception if the value is not correct. Verifier is only executed if the value is not nil, so it doesn’t imply :required true. Predefined verifiers:
  • cfg/verify-file-exists
  • cfg/verify-directory-non-empty — checks if the value is a directory, and if it is non-empty.
• :delayed-transform — a function of option value that will be called not immediately, but the first time when the option is accessed in the code. Transform will be applied only once, and after that the option will store the transformed value. Usefulness of this feature is yet in question. You can mimic it by using a custom parser that wraps the value in a delay, the only difference that you will also have to dereference it manually every time.
• :nested — a map that has the same structure as the top-level configuration scheme. Nested options have the same rights as top-level ones: they can have parsers, verifiers, defaults, etc. Example:

  (cfg/define
    {:statsd {:nested {:host {:type :string
                              :required true
                              :description "IP address of the StatsD server"}
                       :port {:type :number
                              :default 8125}}}})
      

  CLI and ENV arguments have special transformation rules for nested options — dot as a separator for CLI arguments and Java properties, and double underscore for ENV.

  [:statsd :host]    =>    --statsd.host   (cmdline args)
  [:statsd :host]    =>    -Dstatsd.host   (properties)
  [:statsd :host]    =>    STATSD__HOST    (env variables)
      

  In the program you can use cfg/get to fetch a concrete value, or a whole map at any level:

  (cfg/get :statsd :port) ;=> 8125
  (cfg/get :statsd) ;=> {:host "127.0.0.1", :port 8125}
      

• :secret — if true, the value of this option won’t be printed out by cfg/verify. You will see <SECRET> instead. Useful for passwords, API keys, and such.

Providing configuration as files

Omniconf can use EDN files as a configuration source. A file must contain a map of options to their values, which will be merged into the config when populate-from-file is called. The values should already have the format the option requires (number, keyword); but you can also use strings so that parser will be called on them.

You can hardcode the name of the file where to look for configuration (e.g. config.edn in the current directory). It is somewhat trickier to tell the name of the file dynamically. One of the solutions is to expect the configuration file to be provided in one of the command-line arguments. So you have to populate-from-cmd first, and then to populate from config file if it has been provided. However, this way the configuration file will have the priority over CLI arguments which is not always desirable. As a workaround, you can call populate-from-cmd again, but only if your CLI args are idempotent (i.e. they don’t contain ^:concat, see below).

Optionally, data-readers may be provided when reading from a file which contain library specific reader macros. For example:

(cfg/populate-from-cmd args) ;; args is a command-line arguments list
(when-let [conf (cfg/get :conf)]
  (binding [cfg/*data-readers* {'ig/ref ig/ref 'ig/refset ig/refset}] ; e.g. reader macros used by integrant
    (cfg/populate-from-file conf)))

Fetching configuration from AWS Systems Manager (SSM)

Since version 0.3, Omniconf supports Amazon SSM, particularly its Parameter Store, as a configuration source. SSM works well as a storage for secrets — passwords, tokens, and other sensitive things that you don’t want to check into the source control.

To use SSM backend, you’ll need to add an extra dependency:

The function omniconf.core/populate-from-ssm will be available now. It takes path as an argument which will be treated as root path to nested SSM parameters. For example:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true}}}})

(cfg/populate-from-ssm "/prod/myapp/")

This will fetch /prod/myapp/db/password parameter from SSM and save it as [:db :password] in Omniconf.

You can also specify explicit mapping between SSM and Omniconf like this:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true}}}
   :github-token {:type :string
                  :secret true
                  :ssm-name "/myteam/github/oauth-token"}})

(cfg/populate-from-ssm "/prod/myapp/")

Parameters with an absolute :ssm-name parameter will ignore the path argument and will fetch the value directly by name. In case you still want to use path for some keys but the layout in SSM differs from one in Omniconf, you can use ./ as a prefix to signify that it is relative to the path:

(cfg/define
  {:db {:nested {:password {:type :string
                            :secret true
                            :ssm-name "./db-pass"}}}})

(cfg/populate-from-ssm "/prod/myapp/")

This will set [:db :password] parameter from /prod/myapp/db-pass.

Dynamic reconfiguration from SSM

Unlike environment variables and command-line arguments, SSM Parameter Store values can change independently as your program is running. You might want to use this, so that you can change some configuration without restarting the program. There are plenty of usecases for this, like switching the upstream hostname on the fly, or gradually changing the rate of requests to an experimental server you are testing.

To tap into this functionality, use populate-from-ssm-continually instead of populate-from-ssm. It accepts the same path argument, and an extra one — interval in seconds between polling SSM. Polling is used because SSM doesn’t expose an event-based API for this; but it’s not too bad since you’d probably set the interval to 5-10 seconds, so the overhead of polling is not too big. Also, Omniconf would report setting only the values that actually have changed.

;; Poll values under /prod/myapp/ prefix (and all absolute :ssm-name values too) every 10 seconds.
(cfg/populate-from-ssm-continually "/prod/myapp/" 10)

Note that for now, the verification step is not re-run after fetching updated values from SSM, so it is possible to break :verifier invariants with this.

Tips, tricks, and FAQ

Are there any drawbacks? What’s the catch?

There are a few. First of all, Omniconf is much more complex and intertwined than, say, Environ. This might put off some developers, although we suspect they are re-implementing half of Omniconf functionality on top of Environ anyway (like we did before).

Omniconf configuration map is a global mutable singleton. It adds a bit of convenience that you don’t have to drag the config map around, or require it in every namespace. However, there might be usecases where this approach does not fit.

Omniconf is an application-level tool. You most likely don’t want to make your library depend on it, forcing the library users to configure through Omniconf too.

Why are there no convenient Leiningen plugins/Boot tasks for Omniconf?

In the end we distribute and deploy our applications as uberjars. As a standalone JAR our program doesn’t have access to Leiningen or Boot. Hence, it is better not to offload anything to plugins to avoid spawning differences between development and production time.

CLI help command

:help option gets a special treatment in Omniconf. It can have :help-name and :help-description parameters that will be used when printing the help message. If populate-from-cmd encounters --help on the arguments list, it prints the help message and quits.

Useful functions and macros

with-options works as let for configuration values, i.e. it takes a binding list of symbols that should have the same names as options’ keyword names. Only top-level options are supported, destructuring of nested values is not possible right now.

(cfg/with-options [username password]
  ;; Binds (cfg/get :username) to username, and (cfg/get :password) to password.
  ...)

Special operations for EDN options

Sometimes you don’t want to completely overwrite an EDN value, but append to it. For this case two special operations, — ^:concat and ^:merge — can be attached to a map or a list when setting them from any source. Example:

(cfg/define {:emails {:type :edn
                      :default ["admin1@corp.org" "admin2@corp.org"]}
             :roles  {:type :edn
                      :default {"admin1@corp.org" :admin
                                "admin2@corp.org" :admin}}})
...
$ my-app --emails '^:concat ["user1@corp.org"]' --roles '^:merge {"user1@corp.org" :user}'

Custom logging for Omniconf

By default, Omniconf prints errors and final configuration map to standard output. If you want it to use a special logging solution, call cfg/set-logging-fn and provide a vararg function for Omniconf to use it instead of println. For example:

(require '[taoensso.timbre :as log])
(cfg/set-logging-fn (fn [& args] (log/info (str/join " " args))))

License

© Copyright 2016-2020 Grammarly, Inc.

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.