How to config tokenurl in AWS glue

[KeyonQu]

Hi team,

I'm working on integrate the spline agent to AWS glue jobs and send the lineage data to API gateway. For security I've configured the API gateway as IAM authentication. And I'm using the IAM assume role to generate the token string. But I'm not sure how to configure the tokenurl (spline.lineageDispatcher.http.authentication.tokenUrl) Is it the combination of API endpoint and the token?

Eg.

spline.lineageDispatcher.http.authentication.tokenUrl = (API endpoint)?token=(token string)

Btw, I tried to set the tokenUrl as a parameter in glue as below code. But the glue job failed with message java.lang.IllegalArgumentException: Invalid input to --conf
Does any one know why the token string is invalid?

--conf spark.spline.lineageDispatcher.http.authentication.tokenUrl="<MyAPIEndpoint>?token=FwoGZXIvYXdzEFIaDGIuf+PuveDrcBTLwiK2AWHbz7oAmkgrj4iV85g+7xJeOMAP0V91/GBRMQbb/VCBEijedMfWaU3dJkebCLnNAouLyWEhJzFkMkAyfhZXSdfwFyanSoPFjbDambDSqKzAJ3HSqMoABpBAfrjVfrqSPnlcC9STbTxwxokcgZtaOjFsXu/tFFDZufmCiZm0tazaEi6TJkTfuBzOXkGzgoBrpXtsRKfGTwL++CptDafTPEUy5XYyJQqL+IOELFetkK9UehMxhhtCKNiotKUGMi3bq+hM+/XngDCZfblhqlJKrVzVIH4e69SjPeURuJTtjQaJqxIwJHYOrQkdd4w="

Thanks for any suggestion!

[wajda]

the authentication.tokenUrl property is used to fetch the token from a specified URL. It's not a combination of the API endpoint and the token. Instead, it should be a URL from which the Spline Agent can retrieve an OAuth authentication token. There is currently no way to pass an auth token directly to the Spline agent dispatcher. You can extend HTTP Dispatcher of write your own dispatcher that implements this kind of scenario.

Regarding the error you get, it's unrelated problem. The error message java.lang.IllegalArgumentException: Invalid input to --conf suggests that there might be an issue with the format of the command line argument you're passing to Spark. In Spark, configuration properties can be set directly on a SparkConf and through Spark's conf/spark-defaults.conf file. It's also possible to pass it as a command-line argument to spark-submit or spark-shell using --conf option. However, if the value of the configuration property contains special characters, it might cause issues. In your case, the token string might contain characters that need to be escaped. But anyway, in this case the token string you provided does not belong the 'tokenUrl' parameter.

[KeyonQu]

Thanks for the detailed explanation!