Spline Bundled Fat jar

[vishalag001]

Hi @wajda,
The Spline agent bundled jar is not ideally a fat jar. As per the pom.xml under the bundled directory, it only includes the agent-core dependency and all other dependencies are marked as provided. So ideally this is not a fat jar.
There when we use the --jars option on pyspark with the bundled agent, it doesnot resolve the transitive dependencies and it fails with ClassNotFoundException.
Any reason why are we using provided in the bundled pom.xml ? Can we have a true fat jar for the bundled version ?

[cerveada]

The provided dependencies are the ones provided by Spark itself.

Please let us know the following info to help you solve the issue:

• Which version of Spark are you using?
• Which version of Agent are you using?
• Please provide the whole exception.

[vishalag001]

Spark version: 2.4

Agent Version: 0.7.12

Mode of usage:
Using jar directly by passing it to "jar_file_uris" when submitting job via DataprocSubmitJobOperator.

Error:

ERROR org.apache.spark.deploy.yarn.Client: Application diagnostics message: User class threw exception: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1118)
	at org.apache.spark.sql.SparkSession.$anonfun$sessionState$2(SparkSession.scala:146)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144)
	at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141)
	at org.apache.spark.sql.SparkSession.conf$lzycompute(SparkSession.scala:170)
	at org.apache.spark.sql.SparkSession.conf(SparkSession.scala:170)
	at org.apache.spark.sql.SQLContext.setConf(SQLContext.scala:125)
	at com.walmart.cbb.common.Utils.createSparkSession(Utils.java:705)
	at com.walmart.cbb.common.Utils.createSparkSession(Utils.java:685)
	at com.walmart.cbb.unifiedschema.market.jobs.TestJobJar$.main(TestJobJar.scala:32)
	at com.walmart.cbb.unifiedschema.market.jobs.TestJobJar.main(TestJobJar.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
Caused by: java.lang.ClassNotFoundException: za.co.absa.spline.harvester.listener.SplineQueryExecutionListener
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
	at org.apache.spark.util.Utils$.$anonfun$loadExtensions$1(Utils.scala:2682)
	at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
	at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
	at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
	at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2680)
	at org.apache.spark.sql.util.ExecutionListenerManager.$anonfun$new$1(QueryExecutionListener.scala:85)
	at org.apache.spark.sql.util.ExecutionListenerManager.$anonfun$new$1$adapted(QueryExecutionListener.scala:84)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.sql.util.ExecutionListenerManager.<init>(QueryExecutionListener.scala:84)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$listenerManager$2(BaseSessionStateBuilder.scala:270)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.listenerManager(BaseSessionStateBuilder.scala:270)
	at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:297)
	at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1115)
	... 16 more

[cerveada]

SplineQueryExecutionListener is the main class in Spline Agent, so it looks like the jar is not even there.

Are you using the correct bundle for your Spark and Scala version? For Spark 2.4 it should be this one:
https://mvnrepository.com/artifact/za.co.absa.spline.agent.spark/spark-2.4-spline-agent-bundle

[wajda]

+1 to Adam's answer. Just to expand it a little bit more:

it only includes the agent-core dependency and all other dependencies are marked as provided

It is a true fat jar. But remember, the agent isn't executable - it's an add-on to Spark. Therefore it doesn't have to (and it even could not) include the whole world. There's no point in bundling dependencies that are a priori available in the target runtime. It could even lead to classpath conflict in certain circumstances. Those jars aren't just missing - thay are excluded in purpose. So no, do not include those "provided" jars additionally unless you know exactly what you arw doing.

[kuhnen]

Hello, I am trying to use a postProcessing filter:

but I am getting this error:
Make sure that type NamedEntity is in your classpath and check for conflicting dependencies with -Ylog-classpath.
A full rebuild may help if 'LineageDispatcher.class' was compiled against an incompatible version of za.co.absa.commons.

I am using this version: spark-3.3-spline-agent-bundle_2.12. and spark 3.3.1
It was working before version 1.0. thanks a lot

[wajda]

moved to a separate discussion - #722