Heroku Drain for Datadog

A micro Ruby app that drains logs from Heroku, extracts the metrics, and forwards them to Datadog. Supported services are:

• Heroku Router
• Heroku Postgres
• Heroku Redis
• Dyno

By forwarding metrics to Datadog you could:

• Set an alert when dynos are low on free memory.
• Set an alert when Redis is approaching the maximum memory limit allowed by the current plan.
• Graph the dyno's memory usage and load for each type of dyno (e.g. Worker, Web)
• Graph the request queue

See config/default.yml for a full list of metrics and how they map between Heroku and Datadog.

Example Dashboards

System Dependencies

• Ruby 3.1.4
• Heroku Buildpack for DataDog Agent

Setup

Deploying the Drain

To deploy the drain:

$ git clone https://github.com/fivegoodfriends/heroku-drain-datadog.git
$ heroku apps create
$ heroku buildpacks:add https://github.com/DataDog/heroku-buildpack-datadog.git
$ heroku buildpacks:add heroku/ruby
$ heroku config:set DD_API_KEY=<YOUR_DATADOG_API_KEY>
$ heroku config:set DRAIN_PASSWORD=<YOUR_DRAIN_PASSWORD>
$ heroku config:set RACK_ENV=production
$ git push heroku main

The drain itself will not use the dyno, dynotype, and appname tags to avoid conflicting with forwarded metrics.

Instrumenting an App

To instrument an app:

$ heroku labs:enable log-runtime-metrics --app <MY-APP>
$ heroku drains:add https://user:<YOUR_DRAIN_PASSWORD>@<YOUR_APP>.herokuapp.com/logs --app <MY-APP>

All forwarded metrics will be tagged with their dyno and dynotype.

It's recommended to set additional tags such as service, env, and appname if you forward multiple metrics from multiple sources to Datadog.

$ heroku config:set DRAIN_TAGS_FOR_<LOGPLEX_DRAIN_TOKEN>="env:production,service:app" -a <DRAIN-APP>

The drain token can be found by running $ heroku drains -a <MY-APP>

Development

Setup the project:

$ bin/setup

Run the specs:

$ bin/rspec

Run the server and send a log entry:

$ DRAIN_PASSWORD=secret LOG_LEVEL=debug ./bin/puma -C config/puma.rb
$ curl -u :secret -X POST -d "338 <158>1 2016-08-20T02:15:10.862264+00:00 host heroku router - at=info method=GET path="/assets/admin-62f13e9f7cb78a2b3e436feaedd07fd67b74cce818f3bb7cfdab1e1c05dc2f89.css" host=app.fivegoodfriends.com.au request_id=bef7f609-eceb-4684-90ce-c249e6843112 fwd="58.6.203.42,54.239.202.42" dyno=web.1 connect=0ms service=2ms status=304 bytes=112" http://localhost:3000/logs

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/fivegoodfriends/heroku_drain_datadog.

License

The gem is available as open source under the terms of the MIT License.