Add support for environment variables

[Bramzor]

As best practices today is to use environment variables as config parameters, it would make sense to use it for environment based variables.
For example in my use case, I want to have rcpt_to.in_host_list plugin to check for environment variable to be set if there is no file. So kind of a fall back method.

[smfreegard]

That doesn't make sense - how can you possible replace the host_list file (which contains a list of domains) with an environment variable?

[Bramzor]

Easy... by setting HARAKA_HOST_LIST = "domain1.com domain2.com domain3.com"

[KingNoosh]

Linked to haraka/Haraka#2098

[ghost]

Good point @Bramzor you can have even an array environments in BASH nowadays.

[KingNoosh]

I think what would be easier is to allow environment variables to be used within the config files.

Less complicated to get working and easier to use.

[Bramzor]

+1 for environment variables inside config files

[ghost]

I was thinking the ability to override variables for example from smtp.ini.
Example (notice the __ separator)

HARAKA__SMTP_INI__LISTEN="[::0]:2525"

[KingNoosh]

https://gist.github.com/KingNoosh/4461e2cacff887d55604d149e06745f0

Is what we have atm, I can easily take this since I already have this.

[Bramzor]

Advantage of using environment variables inside config files is that you still keep one place where this is defined instead of having multiple places which might easily be overlooked.

[msimerson]

by setting HARAKA_HOST_LIST = "domain1.com domain2.com domain3.com"

another way is using URIs:

HARAKA_HOST_LIST="redis://redis.example.com:6179/0/host_list" or similar.
HARAKA_HOST_LIST="mysql://mysql.example.com/haraka/host_list" or similar.

[ghost]

@msimerson love the idea of fetching stuff directly from DB! How about passwords? inline, config or another env?

[smfreegard]

@msimerson love the idea of fetching stuff directly from DB!

Unfortunately being as config is a synchronous API, getting config from a database like this is impossible unless it's only ever read at start-up.

[ghost]

@smfreegard maybe cron (timer or something) like wrapper to refetch specific configs in master?

[smfreegard]

@smfreegard maybe cron like wrapper to refetch specific configs in master?

Yeah - that could work as I do something similar here - config gets read from the database in init_master and init_child and then an interval is set to re-read the database for only changes and deletions (I have a deleted flag and last_update timestamp in the database table that is set by triggers to do this).

However - you either need to use something that supports database triggers or pub/sub for it to work without requiring a complete re-read of the whole table on each interval, so PostgreSQL (like in my case) or Redis. I don't think MySQL would work as IIRC it doesn't have triggers like PostgreSQL.

It would need a fair amount of thought put into it for this to work generically for everyone that uses Haraka and not something that just works for a specific case.

Redis pub/sub seems like a sensible idea being as we already support Redis in one way or another. Redis keys would work fine for simple values, Hashes for INI/JSON type files (via dot notation) and Lists for List/Data types. However for list types, I don't think you can subscribe to see individual elements (but I might be wrong), so you might have to re-read the whole list - which wouldn't be ideal.

It's do-able, but I guess it would require a way to add plugins to config.

[ghost]

@smfreegard there actually are some triggers in MySQL:

CREATE     
[DEFINER = { user | CURRENT_USER }]     
TRIGGER trigger_name     
trigger_time trigger_event     
ON tbl_name FOR EACH ROW     
trigger_body
trigger_time: { BEFORE | AFTER } 
trigger_event: { INSERT | UPDATE | DELETE }

Maybe introduce some new event to trigger the update into the API? So plugins could also listen for it.
Redis would be neat because it is pretty fast from my experience.

[smfreegard]

@smfreegard there actually are some triggers in MySQL:

They're a bit too basic. e.g. you need to be able to intercept a DELETE and either UPDATE the deleted flag or allow the deletion to complete if it's already set.

Maybe introduce some new event to trigger the update into the API? So plugins could also listen for it.
Redis would be neat because it is pretty fast from my experience.

Start a page on the Wiki or something to brainstorm any ideas or start a new Issue etc.

[ghost]

@smfreegard

They're a bit too basic. e.g. you need to be able to intercept a DELETE and either UPDATE the deleted flag or allow the deletion to complete if it's already set.

Oh ok, I have only used some SET procedures on INSERT and UPDATE so far. So I have not faced this advanced issue yet.

Start a page on the Wiki or something to brainstorm any ideas or start a new Issue etc.

Yeah, I will probably leave that on someone more experienced. I was really throwing in my two cents here. 😄

[Bramzor]

We'll keep it simple for now and use the environments variables inside configs. Anosh already has this figured out anyway 👍

[jkroepke]

This patch wont work anymore

[celesteking]

Be extremely careful with these kind of changes. Using ENV vars is a security disaster.
Bringing expansion in will multiply the issue. (Unless there's someone very smart implementing those.)

[jkroepke]

Using ENV vars is a security disaster.

Explain why

[jchook]

Why not allow JavaScript configuration files, which would have access to process.env etc?

Edit: Looks like haraka-config does have support for JS.

[celesteking]

Using ENV vars is a security disaster.

Explain why

https://httpoxy.org/

I stay corrected, Using ENV vars *improperly* is a security disaster. And I'd like to repeat myself: Unless there's someone very smart implementing those..

Smart people are welcome with patches.

[jkroepke]

I got a full understanding of https://httpoxy.org/ (I know it before).

But this issue describe a problem in cgi applications, not a problem with environment variables itself.

Please get in detail what is your understanding of a very smart implementation.

[taobojlen]

I'm using .js config files to read configuration via environment variables, and it works pretty well.

The downside is that many/most of Haraka's configuration values have hard-coded filenames, e.g. config.get('host_list', 'list'). In order to use environment variables, you have to fork the plugins and change this to config.get('host_list.js', 'js').

This is fine for plugins, but it becomes more of a challenge when you want to use a .js file to configure e.g. the hostname that's usually stored in the me file. You can't configure this via environment variables without forking Haraka.

A potential solution: in the config get function, always check if the given file exists with a .js extension and use that if so.