events-counter
is a simple utility that can be used for accounting messages
in a Kafka topic. It supports accounting by number of bytes (regardless the
numer of messages).
Messages are expected to be on JSON format when counting messages.
events-counter
has two independent modules:
- UUIDCounter
- Reads messages from a set of input topics.
- Count the number of bytes of these messages.
- Sends a message periodically to an output topic with the account of bytes on the inputs topic.
- CountersMonitor
- The input topic of the counter monitor is the output topic of the UUIDCounter.
- Reads messages with the account of bytes. Discards old messages.
- If the number of total bytes exceeds the limit sends an alert.
events-counter
ignores Teldat sensors traffic, so these kind of sensors can
send an unlimited amount of bytes.
events-counter
limits can't be configured. These limits are loaded from a
LICENSE file. These licenses must be signed to be considered a valid license. Also, a
license has an expiration time and it won't be considered valid after
the expiration time.
The public key used to verify the license should be specified at compile time and can't be modified once the binary is compiled. To specify the public key, the application should be compiled like this:
export PUBLIC_KEY="-----BEGIN PUBLIC KEY-----\nMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCqGKukO1De7zhZj6+H0qtjTkVxwTCpvKe4eCZ0\nFPqri0cb2JZfXJ/DgYSF6vUpwmJG8wVQZKjeGcjDOL5UlsuusFncCzWBQ7RKNUSesmQRMSGkVb1/\n3j+skZ6UtW+5u09lHNsj6tQ51s1SPrCBkedbNf0Tp0GbMJDyR4e9T04ZZwIDAQAB\n-----END PUBLIC KEY-----"
make
Note that every line break should be replaced by "\n"
To install this application ensure you have the GOPATH environment variable set and glide installed.
curl https://glide.sh/get | sh
And then:
-
Clone this repo and cd to the project
git clone https://github.com/redBorder/events-counter && cd events-counter
-
Install dependencies and compile
make
-
Install on desired directory
prefix=/opt/events-counter/ make install
Usage of events-counter
:
--version
Show version info
--config string
Config file
--debug
Print debug info
counters:
batch_timeout_s: 5 # Max time to wait before send a count message
batch_max_messages: 1000 # Max number of messages to hold before send a count message
uuid_key: "sensor_uuid" # JSON key for the UUID
kafka: # Kafka configuration
write_topic: "rb_monitor" # Topic to send the count
read_topics: # Topics to read messages for accounting
- "rb_flow"
attributes: # Custom internal rdkafka attributes
bootstrap.servers: "kafka:9092"
group.id: "counters"
monitor:
timer:
period: 86400 # Width in seconds of the interval between counters reset (86400 -> 24h)
offset: 0 # Offset in seconds to change the start of the interval (0 -> 00:00h)
kafka: # Kafka configuration
write_topic: "rb_limits" # Topic to send the alerts
read_topics: # Topics to read messages with accounting info
- "rb_monitor"
attributes: # Custom internal rdkafka attributes
bootstrap.servers: "kafka:9092"
enable.auto.commit: "false" # IMPORTANT: Should be set to false
group.id: "monitor"
licenses_directory: /etc/licenses # Path to the directory where licenses are stored
organization_mode: false # Ignore organization_uuid. Count every message as belonging to org "*"