…ification (linkedin#618)

* create endpoint to build and render messages

* update sender tests

* change cache plan delete log level (linkedin#619)

* skip error logging if iris bot fails to send messages to a channel it is not in (linkedin#621)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* rackspace webhook support (linkedin#405)

* Start rackspace webhook impl by copying alertmanager

* Plumb plan name through URL parameter for rackspace webhook

* add test for rackspace webhook (WIP)

* fix rackspace webhook test data

* rackspace webhook should parse plan inside its class

This detail is only relevant to the rackspace webhook class and doesn't belong
in api.py. This just reverts the changes in api.py and adds support for parsing
the plan in the rackspace webhook class.

* deduplicate webhook code with class inheritance

authored-by: Patrick Baxter <pb@coreos.com>

* Add support for custom Slack formatting via attachments/blocks (linkedin#624)

* Update iris_slack.py

* Update iris_slack.py

* bumnp version

* fix tests and incident id insert in slack message

* remove extra arguments (linkedin#626)

* added twillio number override mechanism (linkedin#627)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

* added twillio number override mechanism

* minor change in case application_override_mapping is not defined in the default config

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

Co-authored-by: ddurruty-li <85372760+ddurruty-li@users.noreply.github.com>
Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>
Co-authored-by: Patrick Baxter <patrickbx@gmail.com>
Co-authored-by: Luke Young <bored-engineer@users.noreply.github.com>

* change cache plan delete log level (linkedin#619)

* skip error logging if iris bot fails to send messages to a channel it is not in (linkedin#621)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* rackspace webhook support (linkedin#405)

* Start rackspace webhook impl by copying alertmanager

* Plumb plan name through URL parameter for rackspace webhook

* add test for rackspace webhook (WIP)

* fix rackspace webhook test data

* rackspace webhook should parse plan inside its class

This detail is only relevant to the rackspace webhook class and doesn't belong
in api.py. This just reverts the changes in api.py and adds support for parsing
the plan in the rackspace webhook class.

* deduplicate webhook code with class inheritance

authored-by: Patrick Baxter <pb@coreos.com>

* Add support for custom Slack formatting via attachments/blocks (linkedin#624)

* Update iris_slack.py

* Update iris_slack.py

* bumnp version

* fix tests and incident id insert in slack message

* remove extra arguments (linkedin#626)

* added twillio number override mechanism (linkedin#627)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

* added twillio number override mechanism

* minor change in case application_override_mapping is not defined in the default config

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* remove experimental-sender changes from master

* Update __init__.py

* check for active mailing_list target (linkedin#630)

* use app's category default if user's not defined

* Iris-message-processor cluster management

* incorporate suggestions

* remove redundant line

* remove internal app allowlist

* Update __init__.py

* remove non-inclusive language (linkedin#639)

* remove non-inclusive language (linkedin#635)

* master -> leader in config (linkedin#637)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc (linkedin#638)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc

Co-authored-by: ddurruty-li <85372760+ddurruty-li@users.noreply.github.com>
Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>
Co-authored-by: Patrick Baxter <patrickbx@gmail.com>
Co-authored-by: Luke Young <bored-engineer@users.noreply.github.com>

* include custom sender addresses for build_message

* add endpoint to fetch plan aggregation settings

…kedin#656)

* return device ids with target contact

* support multi-recipient msgs in external sender

* return device ids with target contact

* support multi-recipient msgs in external sender

* build messages for dynamic plans

* remove unused variables

* return device ids with target contact

* support multi-recipient msgs in external sender

* build messages for dynamic plans

* remove unused variables

* link ui to external sender if enabled

* fix wording of comment

* add external sender into incident target search

* return device ids with target contact

* support multi-recipient msgs in external sender

* build messages for dynamic plans

* remove unused variables

* link ui to external sender if enabled

* fix wording of comment

* add external sender into incident target search

* forward twilio deliver status to external sender

* handle external message responses

* set X-IRIS-INCIDENT header

* still display incidents if sender can't be reached

* add ecternal sender peer count endpoint

* flake8

linkedin#671)

* still display incidents if sender can't be reached

* add ecternal sender peer count endpoint

* flake8

* split external incident & notification hadling cfg

* change cache plan delete log level (linkedin#619)

* skip error logging if iris bot fails to send messages to a channel it is not in (linkedin#621)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* rackspace webhook support (linkedin#405)

* Start rackspace webhook impl by copying alertmanager

* Plumb plan name through URL parameter for rackspace webhook

* add test for rackspace webhook (WIP)

* fix rackspace webhook test data

* rackspace webhook should parse plan inside its class

This detail is only relevant to the rackspace webhook class and doesn't belong
in api.py. This just reverts the changes in api.py and adds support for parsing
the plan in the rackspace webhook class.

* deduplicate webhook code with class inheritance

authored-by: Patrick Baxter <pb@coreos.com>

* Add support for custom Slack formatting via attachments/blocks (linkedin#624)

* Update iris_slack.py

* Update iris_slack.py

* bumnp version

* fix tests and incident id insert in slack message

* remove extra arguments (linkedin#626)

* added twillio number override mechanism (linkedin#627)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

* added twillio number override mechanism

* minor change in case application_override_mapping is not defined in the default config

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* remove experimental-sender changes from master

* Update __init__.py

* check for active mailing_list target (linkedin#630)

* use app's category default if user's not defined

* remove internal app allowlist

* Update __init__.py

* remove non-inclusive language (linkedin#639)

* remove non-inclusive language (linkedin#635)

* master -> leader in config (linkedin#637)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc (linkedin#638)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc

* db setup instruction (linkedin#646)

the `schema_0.sql` contains `drop table` statement which could be not obvious.

I experienced this issue when using a docker image with `DOCKER_DB_BOOTSTRAP=1` (which does a schema and dummy data), and wondering why I lost all my custom plans and templates, after iris container restart.

Additionally provide alternative way to removing ONLY_FULL_GROUP_BY. This is useful when running mysql docker images. I.e. from oracla/mysql or bitnami mariadb images. This is easier than modifying the config file, or changing global server configuration when running in the container (i.e. docker or kubernetes).

* use DB port from the config in image entrypoint (linkedin#650)

* until now, 3306 was hardcoded
* port in config file stays optional for backward compatibility

Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>

* rebasing container image to ubuntu:20.04 (linkedin#649)

* rebasing container image to ubuntu:20.04

* switching to Python v3
  * which was driven by rsa>=3.1.4 -> oauth2client==1.4.12, it no longer supports Python v2
* adding ops/packer Makefile
  * to automate steps only found in README.md
  * done just for the docker image build
* instructing packer to clear ENTRYPOINT from original ubuntu image
  * CMD was not working with the /bin/sh in place of ENTRYPOINT

* fixes to actually make it work with Python 3.8

* correct plugin references for uwsgi
* fix of execv inputs to match validation, which changed in Python 3.6
* added missing mysql client package, which is used in the entrypoint
* not using 'gevent' parameter for uwsgi
  * it causes "DAMN ! worker N (pid: NNN) died :( trying respawn ..."
  * not sure why this happens for uwsgi+python3+gevent

Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>

* add max len to webhook context too long response (linkedin#653)

* Update __init__.py

* Create __init__.py

* Support multiple custom incident handler hooks (linkedin#664)

Refactor handling of custom incident handlers to support multiple handlers.

Retain compatibility with previous config syntax

* Calculate incident priority for incident creation hook

- Generate priority by looking for the highest severity priority across
  all of the plan notifications

- Set this as a new priority field within the incident details as passed
  to the process_create() hook

* Make application name in hook incident data support overwritten app name

When creating test incidents within the API, the application given to the
incident creation hooks is "iris" rather than the application being used.

Fix this such that the hooks receive the overwritten application name.

Also: move definition of `app` to the parent block in the function so it
is more obvious this variable is used later on.

* Don't prepend https:// to proxy hostnames (linkedin#668)

Historically, this approach has always worked because http forward proxies
generally only listen on http:// (not https://) and urllib3 has not supported
connecting to a http proxy via https:// so it has always ignored the scheme.

However, as of urllib3 >= 1.26 or so, urllib3 does support and attempt connecting
to proxies via https:// (if this schem is provided) and it raises an exception
if the proxy only listens on http://

Fix this by no longer enforcing a http:// prefix to proxy hostnames. If the user
desires connecting to a https:// proxy, this prefix can be provided within Iris's
configuration.

* remove old custom_incident_handler_dispatcher

Co-authored-by: ddurruty-li <85372760+ddurruty-li@users.noreply.github.com>
Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>
Co-authored-by: Patrick Baxter <patrickbx@gmail.com>
Co-authored-by: Luke Young <bored-engineer@users.noreply.github.com>
Co-authored-by: Witold Baryluk <witold.baryluk+github@gmail.com>
Co-authored-by: mighq <contact@mighq.net>
Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>
Co-authored-by: Joe Gillotti <joe@u13.net>
Co-authored-by: Joe Gillotti <jgillotti@linkedin.com>

* change cache plan delete log level (linkedin#619)

* skip error logging if iris bot fails to send messages to a channel it is not in (linkedin#621)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* rackspace webhook support (linkedin#405)

* Start rackspace webhook impl by copying alertmanager

* Plumb plan name through URL parameter for rackspace webhook

* add test for rackspace webhook (WIP)

* fix rackspace webhook test data

* rackspace webhook should parse plan inside its class

This detail is only relevant to the rackspace webhook class and doesn't belong
in api.py. This just reverts the changes in api.py and adds support for parsing
the plan in the rackspace webhook class.

* deduplicate webhook code with class inheritance

authored-by: Patrick Baxter <pb@coreos.com>

* Add support for custom Slack formatting via attachments/blocks (linkedin#624)

* Update iris_slack.py

* Update iris_slack.py

* bumnp version

* fix tests and incident id insert in slack message

* remove extra arguments (linkedin#626)

* added twillio number override mechanism (linkedin#627)

* skip error logging if iris bot fails to send messages to a channel it is not in

* removed redundant slack api call and added a warning

* added a return statement to keep the event from being counted as an error in the metrics

* added twillio number override mechanism

* minor change in case application_override_mapping is not defined in the default config

Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>

* remove experimental-sender changes from master

* Update __init__.py

* check for active mailing_list target (linkedin#630)

* use app's category default if user's not defined

* remove internal app allowlist

* Update __init__.py

* remove non-inclusive language (linkedin#639)

* remove non-inclusive language (linkedin#635)

* master -> leader in config (linkedin#637)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc (linkedin#638)

* remove non-inclusive language

* master -> leader in config

* restore sphinx required master_doc

* db setup instruction (linkedin#646)

the `schema_0.sql` contains `drop table` statement which could be not obvious.

I experienced this issue when using a docker image with `DOCKER_DB_BOOTSTRAP=1` (which does a schema and dummy data), and wondering why I lost all my custom plans and templates, after iris container restart.

Additionally provide alternative way to removing ONLY_FULL_GROUP_BY. This is useful when running mysql docker images. I.e. from oracla/mysql or bitnami mariadb images. This is easier than modifying the config file, or changing global server configuration when running in the container (i.e. docker or kubernetes).

* use DB port from the config in image entrypoint (linkedin#650)

* until now, 3306 was hardcoded
* port in config file stays optional for backward compatibility

Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>

* rebasing container image to ubuntu:20.04 (linkedin#649)

* rebasing container image to ubuntu:20.04

* switching to Python v3
  * which was driven by rsa>=3.1.4 -> oauth2client==1.4.12, it no longer supports Python v2
* adding ops/packer Makefile
  * to automate steps only found in README.md
  * done just for the docker image build
* instructing packer to clear ENTRYPOINT from original ubuntu image
  * CMD was not working with the /bin/sh in place of ENTRYPOINT

* fixes to actually make it work with Python 3.8

* correct plugin references for uwsgi
* fix of execv inputs to match validation, which changed in Python 3.6
* added missing mysql client package, which is used in the entrypoint
* not using 'gevent' parameter for uwsgi
  * it causes "DAMN ! worker N (pid: NNN) died :( trying respawn ..."
  * not sure why this happens for uwsgi+python3+gevent

Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>

* add max len to webhook context too long response (linkedin#653)

* Update __init__.py

* Create __init__.py

* Support multiple custom incident handler hooks (linkedin#664)

Refactor handling of custom incident handlers to support multiple handlers.

Retain compatibility with previous config syntax

* Calculate incident priority for incident creation hook

- Generate priority by looking for the highest severity priority across
  all of the plan notifications

- Set this as a new priority field within the incident details as passed
  to the process_create() hook

* Make application name in hook incident data support overwritten app name

When creating test incidents within the API, the application given to the
incident creation hooks is "iris" rather than the application being used.

Fix this such that the hooks receive the overwritten application name.

Also: move definition of `app` to the parent block in the function so it
is more obvious this variable is used later on.

* Don't prepend https:// to proxy hostnames (linkedin#668)

Historically, this approach has always worked because http forward proxies
generally only listen on http:// (not https://) and urllib3 has not supported
connecting to a http proxy via https:// so it has always ignored the scheme.

However, as of urllib3 >= 1.26 or so, urllib3 does support and attempt connecting
to proxies via https:// (if this schem is provided) and it raises an exception
if the proxy only listens on http://

Fix this by no longer enforcing a http:// prefix to proxy hostnames. If the user
desires connecting to a https:// proxy, this prefix can be provided within Iris's
configuration.

* remove old custom_incident_handler_dispatcher

* process ldap and oncall syncs concurrently (linkedin#677)

Co-authored-by: ddurruty-li <85372760+ddurruty-li@users.noreply.github.com>
Co-authored-by: Damian Durruty <ddurruty@ddurruty-mn2.linkedin.biz>
Co-authored-by: Patrick Baxter <patrickbx@gmail.com>
Co-authored-by: Luke Young <bored-engineer@users.noreply.github.com>
Co-authored-by: Witold Baryluk <witold.baryluk+github@gmail.com>
Co-authored-by: mighq <contact@mighq.net>
Co-authored-by: Michal Zubac <michal.zubac@inuits.eu>
Co-authored-by: Joe Gillotti <joe@u13.net>
Co-authored-by: Joe Gillotti <jgillotti@linkedin.com>

…rimental-sender

…rimental-sender

* do not load configs from util

* create external_sender_incident_dryrun category

* remove comment

* restore auth req

* fail build_message reqs when no messages made

* revert oncall metrics change

…rimental-sender

…inkedin#683)

* do not load configs from util

* create external_sender_incident_dryrun category

* remove comment

* restore auth req

* fail build_message reqs when no messages made

* revert oncall metrics change

* check oncall_error is defined in stats

* change key check syntax

…rimental-sender

* do not load configs from util

* create external_sender_incident_dryrun category

* remove comment

* restore auth req

* fail build_message reqs when no messages made

* revert oncall metrics change

* check oncall_error is defined in stats

* change key check syntax

* create ca_bundle_path config

…rimental-sender

* do not load configs from util

* create external_sender_incident_dryrun category

* remove comment

* restore auth req

* fail build_message reqs when no messages made

* revert oncall metrics change

* check oncall_error is defined in stats

* change key check syntax

* create ca_bundle_path config

* do not log every time lookup module is loaded

* make module log level debug

* change failed notification status code

* flake8

…rimental-sender

Working Docker setup with python3 (linkedin#684)

* do not load configs from util

* create external_sender_incident_dryrun category

* remove comment

* restore auth req

* fail build_message reqs when no messages made

* revert oncall metrics change

* check oncall_error is defined in stats

* change key check syntax

* create ca_bundle_path config

* do not log every time lookup module is loaded

* make module log level debug

* change failed notification status code

* flake8

* add ability to partially ramp notifications

* catch notification request exceptions

* flake8

* dont retry 400 notification requests

…rimental-sender

display message sent status correctly

…into linkedin-experimental-sender

Linkedin experimental sender

use locks with api cache

* use locks with api cache

* add ability to filter incidents by claimed

* use gevent lock directly

* use locks with api cache

* add ability to filter incidents by claimed

* use gevent lock directly

* import gevent.lock explicitly in cache

* use locks with api cache

* add ability to filter incidents by claimed

* use gevent lock directly

* import gevent.lock explicitly in cache

* use gevent BoundedSemaphore instead of Semaphore

* use locks with api cache

* add ability to filter incidents by claimed

* use gevent lock directly

* import gevent.lock explicitly in cache

* use gevent BoundedSemaphore instead of Semaphore

* init api cache immediately

* use locks with api cache

* add ability to filter incidents by claimed

* use gevent lock directly

* import gevent.lock explicitly in cache

* use gevent BoundedSemaphore instead of Semaphore

* init api cache immediately

* set iris client init log to debug

…rimental-sender

…into experimental-sender

Experimental sender