-
Ouput to clickhouse
-
Processor to process geo (Using maxmind database)
-
Processor to parse user agent
go build -o cmd/benthos .
1. Benthos output plugin to clickhouse
- Using clickhouse go client
- Create clickhouse table
CREATE TABLE IF NOT EXISTS
sample(hitmiss String, client_ip String, status Int32, timestamp DateTime)
engine=Memory;
- Using plugin
...
output:
type: clickhouse
plugin:
connection_string: "tcp://localhost:9000"
query: "insert into sample(hitmiss, client_ip, status, timestamp) values(?, ?, ?, ?)"
columns:
- hitmiss
- client_ip
- status$floatToInt32
- timestamp$stringToDateOrNow$2006-01-02T15:04:05.000Z
-
Conncetion string : The connection string
-
Query : insert query
-
Columns: Columns in jsonfields tobe inserted
- Supported converter: stringToInt32, floatToInt32, floatToUInt32, floatToUInt8, stringToDateOrNow, unixToDateOrNow
...
- type: geo
plugin:
file: GeoLite2-City.mmdb
field: client_ip
...
- file: maxmind mmdb file
- field: field to process
...
- type: useragent
plugin:
file: regexes.yaml
field: user_agent
...
- file: regex of useragent
- field: field to process