Database password is included in execution plans with JDBC data sources

[97nitt]

I was doing some local testing (using v0.7.1 of the agent) to see what execution plans look like for different data sources, standing up a backend that simply logs the requests from the Spline agent. When running a Spark job that reads from a JDBC source, all of the connection parameters were included in the params of the read operation, including the password, which should probably be redacted. I didn't test writes, but I'm guessing we'd see the same.

[wajda]

There is a DataSourcePasswordReplacingFilter that removes sensitive information from the agent output. The filter isn't enabled by default, so you need to enable it explicitly via agent configuration. For example:

spline.postProcessingFilter.composite.filters=dsPasswordReplace

See https://github.com/AbsaOSS/spline-spark-agent/blob/release/0.7.1/core/src/main/resources/spline.default.properties#L104 for details.

I think, we need to enabled it by default in the next major release.

[97nitt]

@wajda I'm having trouble getting the composite filter working with two user-defined filters. I'm doing the Spline configuration in my PySpark application, so I have code that now looks like this:

        spark = SparkSession \
            .builder \
            .appName(app_name) \
            .config('spark.sql.queryExecutionListeners', 'za.co.absa.spline.harvester.listener.SplineQueryExecutionListener') \
            .config('spark.spline.lineageDispatcher', 'http') \
            .config('spark.spline.lineageDispatcher.http.producer.url', spline_config.url) \
            .config('spark.spline.mode', spline_config.mode.value) \
            .config('spark.spline.postProcessingFilter', 'composite') \
            .config('spark.spline.postProcessingFilter.composite.className', 'za.co.absa.spline.harvester.postprocessing.CompositePostProcessingFilter') \
            .config('spark.spline.postProcessingFilter.composite.filters', 'mcd,password') \
            ... (truncated for brevity)

where password is the DataSourcePasswordReplacingFilter, and mcd is another custom filter I've built.

When I run the Spark application the Spline agent initialization fails with:

21/12/21 17:32:15 ERROR QueryExecutionEventHandlerFactory: Spline initialization failed! Spark Lineage tracking is DISABLED.
java.util.NoSuchElementException: Missing configuration property spline.postProcessingFilter.mcd,password.className

Seems the comma in there isn't serving as a delimiter of filter names. I've tried a few different iterations of spacing around the comma with no effect. Any suggestions on how I can get this working in code instead of a config file?

[wajda]

Ok, that's definitely a bug, I'll create a separate issue for that.

[wajda]

Should be fixed in 0.7.2 - 837e76a
Please check.

[97nitt]

Confirmed fix in latest release. Thanks for turning that around so quickly.

…

On Dec 22, 2021, at 10:22 AM, Alex Vayda ***@***.***> wrote:  Should be fixed in 0.7.2 - 837e76a Please check. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.