Data pipeline configuration (user journey and audit trail)

[luizaraujobliver]

Hey guys, how are you?

We are configuring HS locally and the use of analytics functions are extremely important for the company. We need to use the user journey function, system metrics, audit trail and report generation.

It seems that for this we need to configure an entire data pipeline (kafka, zookeeper, ckh etc) but there is no documentation for this. Could you help us with this complete configuration of the data pipeline so that at least the user journey and audit trail work?

[ShivanshMathurJuspay]

Hey @luizaraujobliver
The documentation for setup of Kafka, ClickHouse, Zookeeper is already available

Link for the same : https://github.com/juspay/hyperswitch/tree/main/crates/analytics/docs/clickhouse

[luizaraujobliver]

@ShivanshMathurJuspay We followed this tutorial, however, we had some problems along the way with some of the scripts not working. But I will try again and return with details of the errors.

Regarding the audit trail, does it depend on this configuration to work? Because for me it returns a SQLX configuration error, is it a different configuration of this pipeline?

[ShivanshMathurJuspay]

Hey @luizaraujobliver
Yes
With SQLX as the source for analytics
Audit trail won't work
Probably you would want to run dashboard without the feature flag audit_trail as false
It would perfectly with Kafka and Clickhouse
Lets get that working for better experience

[luizaraujobliver]

@ShivanshMathurJuspay We don't want to disable the flag because we need to use the audit trail. However, some scripts are giving errors, is it possible to provide a little more detail in the configuration doc? This audit trail feature is extremely important for us

[ShivanshMathurJuspay]

@luizaraujobliver Can we then work on setting up the clickhouse and kafka pipeline
with that audit trail will work fine

[luizaraujobliver]

Yes, we can. What is the guidance? @ShivanshMathurJuspay

[ShivanshMathurJuspay]

Can you please re share the output of the https://github.com/juspay/hyperswitch/tree/main/crates/analytics/docs/clickhouse

[ShivanshMathurJuspay]

@luizaraujobliver any luck with setting up the pipeline