Skip to content

Home

Jump to bottom
disisto edited this page May 17, 2022 · 10 revisions

systemd status notifier

This bash script allows you to be notified via Email, Mattermost, PagerDuty, Pushover, Slack and/or SMS (sipgate) when an enabled systemd service is in a failed condition.

Content

Supported services

Service Method
Email MTA
Mattermost API
PagerDuty API
Pushover API
sipgate (SMS) API
Slack API

Requirements

  • Linux OS with systemd

  • For Email notifications: Mail Transfer Agent (MTA) like postfix

  • For PagerDuty API v2 notifications: pdagent

  • For all other notifications: cURL

Installation

This guide is based on the setup on Debian. Depending on the Linux distribution, certain deviations may occur.

Download

Download the script to the /bin folder...

cd /bin

Download via cURL...

sudo curl https://raw.githubusercontent.com/disisto/systemd-status-notifier/main/systemd-status-notifier.sh --output systemd-status-notifier.sh

...or via wget

sudo wget https://raw.githubusercontent.com/disisto/systemd-status-notifier/main/systemd-status-notifier.sh

Access and ownership permissions

...and set the correct access and ownership permissions

sudo chmod +x systemd-status-notifier.sh
sudo chown root:root systemd-status-notifier.sh

 

Create a systemd service

Set up a systemd service for the script.

sudo nano /etc/systemd/system/systemd-status-notifier@.service
[Unit]
Description=systemd status notifier
After=network.target
Documentation=https://github.com/disisto/systemd-status-notifier

[Service]
Type=simple
ExecStart=/bin/systemd-status-notifier.sh

Add the following string to each service you want to monitor. This will allow you to be notified as soon as the service changes to the faulty state.

Add the string in the Unit section right after the service description.

[Unit]
Description=...
OnFailure=systemd-status-notifier@%n.service

...
sudo systemctl daemon-reload

Optional: Cron job

Set up a cron job to check the status at regular intervals. In this example the interval is set to 15min. Adjust the interval according to your needs.

crontab -e
*/15 * * * * /bin/systemd-status-notifier.sh

Script adaptation

Select the systemd services to monitor

In line 40 you have the option to select which systemd services you want to be informed about.

You can select "all", then you will be informed about all enabled systemd services.

40   SYSTEMD_SERVICES=(all)

If the full list contains some services you want to exclude, add the service in question after the "all" tag.

40   SYSTEMD_SERVICES=(all postgresql nodeexporter)

^^ (Add "all", exclude postgresql nodeexporter)

Alternatively, you can select one or more services you want to be notified about by simply listing them by name.

40   SYSTEMD_SERVICES=(apache2 postfix mariadb)

^^ (Include only apache2 postfix mariadb)

ℹ️ Separate multiple entries with a space between them and without the .service extension.

Select the channel for notifications

In line 44 you can define which channel you want to be informed about in case of a stopped or failed status.

You can choose between email, mattermost, paderduty, pushover, slack and sipgate and combine them as you like.

ℹ️ paderduty is intended for communication via the PagerDuty API v2. If you prefer to trigger PagerDuty via email, omit paderduty from the listing.

44   MESSAGING_SERVICES=(paderduty)
44   MESSAGING_SERVICES=(email slack)
44   MESSAGING_SERVICES=(email mattermost sipgate)

ℹ️ Separate multiple entries with a space between them.

Channel configuration

Email notifications

In order to receive Email notifications a Mail Transfer Agent (MTA) is necessary. After setup, a sender and one or more recipient addresses must be specified in the script.


Enter a sender address in line 47.

47   EMAIL_SENDER=(noreply@troubleshooting.tools)

Enter one or more recipient addresses in line 50

50   EMAIL_RECIPIENTS=(admin_1@troubleshooting.tools admin_2@troubleshooting.tools)

ℹ️ Separate multiple entries with a space between them.


Depending on which MTA is used, the path /usr/sbin/sendmail in line 381 may have to be adjusted. The current script has been tested with postfix.

381   ) | /usr/sbin/sendmail -t $emailRecipient

Example how to "Configuring a Postfix Server to Send Email through Gmail or Google Workspace" by linode.com:

https://www.linode.com/docs/guides/configure-postfix-to-send-mail-using-gmail-and-google-workspace-on-debian-or-ubuntu/

Mattermost notifications

To receive Mattermost notifications, you need to set up an incoming webhook in Mattermost. Installation instructions can be found at the following link: https://mattermost.com/blog/mattermost-integrations-incoming-webhooks/


The script expects the variable MATTERMOST_WEBHOOK_URL followed by the token in the following notation:

MATTERMOST_WEBHOOK_URL=https://mattermost.troubleshooting.tools/hooks/xx0xx0xxxxxxxxxxxxxx0xx0xx

⚠️ During the setup phase, you can temporarily store the service key in the script itself. The corresponding entry is located starting at line 60. For productive use, you should remove the string from the script and store the token in local secret storage.


The script uses cURL to send webhooks. Make sure that your system has this module installed.


Quick Installation Guide for Debian/Ubuntu/Raspberry Pi OS:

sudo apt update && sudo apt upgrade -y && sudo apt install curl -y

PagerDuty notifications

To receive PagerDuty notifications, you need to set up the 'Events API V2' integration to an existing PagerDuty service to get a service key.


The script expects the variable PD_SERVICE_KEY followed by the token in the following notation:

PD_SERVICE_KEY=0x00x000xx00000xx000x00000xx0x00

⚠️ During the setup phase, you can temporarily store the service key in the script itself. The corresponding entry is located starting at line 60. For productive use, you should remove the string from the script and store the token in local secret storage.


In order to use the PagerDuty Events API, the PagerDuty Agent need to be installed. Installation instructions can be found at the following link. https://www.pagerduty.com/docs/guides/agent-install-guide/


Quick Installation Guide for Debian/Ubuntu/Raspberry Pi OS:

sudo apt update && sudo apt upgrade -y && sudo apt install wget gpg apt-transport-https -y 
wget -O - https://packages.pagerduty.com/GPG-KEY-pagerduty | sudo apt-key add -
sudo sh -c 'echo "deb https://packages.pagerduty.com/pdagent deb/" >/etc/apt/sources.list.d/pdagent.list'
sudo apt update && sudo apt install pdagent pdagent-integrations -y

Pushover notifications

To receive Pushover notifications, you need to set up an Application in Pushover. Installation instructions can be found at the following link: https://pushover.net/apps/build


The script expects the variable PUSHOVER_TOKEN followed by the token in the following notation:

PUSHOVER_TOKEN=xxx00xxxx0xxxxx0xxxx0xxxxxxx0x

and the variable PUSHOVER_USER followed by the token in the following notation:

PUSHOVER_USER=xxxxxxxx0xx0xxxxx0xx0xx00xx0

The user key is displayed in the upper right corner directly after logging in.


The configuration options are not exhausted and can be customized as desired. For example, in line 540 the notification sound type siren has been enabled. Replace the sound with one of the other available sounds, with an uploaded sound file of your choice, or replace it with none if you do not want to hear any sound.

540   --form-string "sound=siren" \

The script uses cURL to send webhooks. Make sure that your system has this module installed.


Quick Installation Guide for Debian/Ubuntu/Raspberry Pi OS:

sudo apt update && sudo apt upgrade -y && sudo apt install curl -y

sipgate SMS notifications

To receive sipgate SMS notifications, you need to set up

  • Personal Access Token which consits of a token and tokenid

  • Required Scope: sessions:sms:write

  • smsId, which is the ID to your mobile number


How to retrieve the Personal Access Token and set the scope can be found at the following link: https://developer.sipgate.io/rest-api/authentication/


To get the smsId an intermediate step is necessary:

  1. Visit https://app.sipgate.com/personal-access-token

  2. Select in the upper left corner the user from whom the SMS should be sent later (which will used as the sender phone number)

  3. After you select the user, the URL lists the UserID. Make a note of the number that begins with the lower case W.

https://app.sipgate.com/w999/...

  1. Visit https://app.sipgate.com/personal-access-token again

  2. Set up another token with the scope sms:read

  3. Now type the following command in your terminal and replace the placeholders {token}, {tokenid}, and {userid}.

curl --user {token}:{tokenid} https://api.sipgate.com/v2/{userid}/sms

(If cURL is not installed yet, you can find instructions for Debian/Ubuntu/Raspberry Pi OS at the end of this description.)

  1. Output will be like the following. Make a note of the number that begins with the lower case S.
{"items":[{"id":"s999","alias":"SMS from User X","callerId":"sipgate"}]}
  1. Delete the temporarily created token with the scope sms:read.

Enter the smsId address in line 53.

53   SIPGATE_SMSID=(s0)

Enter the recipients phone number(s) in line 56.

56   SIPGATE_RECIPIENT=(+4900000000000 +31999999999))

ℹ️ International writing:

  • beginning with a + sign
  • International prefix without leading zeros
  • SMS capable mobile number
  • Separate multiple entries with a space between them

The script expects the variable SIPGATE_TOKEN followed by the Personal Access Token in the following notation:

SIPGATE_TOKEN=token-XXX00X:0000xx0x-xxx0-0x00-x0xx-00000x0xxx0x

Logical structure:   token:tokenid (token + colon + tokenid)

⚠️ During the setup phase, you can temporarily store the service key in the script itself. The corresponding entry is located starting at line 60. For productive use, you should remove the string from the script and store the token in local secret storage.


The script uses cURL to send webhooks. Make sure that your system has this module installed.


Quick Installation Guide for Debian/Ubuntu/Raspberry Pi OS:

sudo apt update && sudo apt upgrade -y && sudo apt install curl -y

Slack notifications

To receive Slack notifications, you need to set up an incoming webhook in Slack. Installation instructions can be found at the following link: https://api.slack.com/messaging/webhooks


The script expects the variable SLACK_WEBHOOK_URL followed by the token in the following notation:

SLACK_WEBHOOK_URL=https://hooks.slack.com/services/X00XX0X0XXX/X00XXX00XXX/XXxxxxxxXxx0xxXxXX0X0xXX

⚠️ During the setup phase, you can temporarily store the service key in the script itself. The corresponding entry is located starting at line 60. For productive use, you should remove the string from the script and store the token in local secret storage.


The script uses cURL to send webhooks. Make sure that your system has this module installed.


Quick Installation Guide for Debian/Ubuntu/Raspberry Pi OS:

sudo apt update && sudo apt upgrade -y && sudo apt install curl -y

This project is not affiliated with Mattermost, PagerDuty, Pushover, sipgate and/or Slack
All mentioned trademarks are the property of their respective owners.

Clone this wiki locally