[Bug] Exception in thread "spark-listener-group-shared" java.lang.NoClassDefFoundError: org/apache/kyuubi/util/reflect/ReflectUtils$

[lukaizj]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

beeline -u "jdbc:kyuubi://x.x.x.x:10009/;principal=hive/x.x.x.x@TEST-HADOOP.COM?spark.sql.queryExecutionListeners=org.apache.kyuubi.plugin.lineage.SparkOperationLineageQueryExecutionListener;spark.kyuubi.plugin.lineage.dispatchers=KYUUBI_EVENT"

Affects Version(s)

1.8.0~1.9.2

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

24/09/18 15:57:16 ERROR Utils: uncaught error in thread spark-listener-group-shared, stopping SparkContext
java.lang.NoClassDefFoundError: org/apache/kyuubi/util/reflect/ReflectUtils$
        at org.apache.kyuubi.plugin.lineage.helper.LineageParser.extractColumnsLineage(SparkSQLLineageParseHelper.scala:193)
        at org.apache.kyuubi.plugin.lineage.helper.LineageParser.$anonfun$extractColumnsLineage$22(SparkSQLLineageParseHelper.scala:341)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.IterableLike.foreach(IterableLike.scala:74)
        at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at org.apache.kyuubi.plugin.lineage.helper.LineageParser.extractColumnsLineage(SparkSQLLineageParseHelper.scala:341)
        at org.apache.kyuubi.plugin.lineage.helper.LineageParser.parse(SparkSQLLineageParseHelper.scala:54)
        at org.apache.kyuubi.plugin.lineage.helper.LineageParser.parse$(SparkSQLLineageParseHelper.scala:52)
        at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParseHelper.parse(SparkSQLLineageParseHelper.scala:510)
        at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParseHelper.$anonfun$transformToLineage$1(SparkSQLLineageParseHelper.scala:516)
        at scala.util.Try$.apply(Try.scala:213)
        at org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParseHelper.transformToLineage(SparkSQLLineageParseHelper.scala:516)
        at org.apache.kyuubi.plugin.lineage.SparkOpera

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @lukaizj,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[wForget]

@lukaizj Thank you for reporting this issue. This is because the spark engine jar and lineage plugin jar are loaded by different classloaders. We can use spark.jars to specify lineage plugin jar path (outside $SPARK_HOME/jars) so that it uses the same classloader as the spark engine jar to avoid this issue.

[pan3793]

@wForget maybe we should consider providing a shaded jar for lineage plugin, as we did for authz

[wForget]

@wForget maybe we should consider providing a shaded jar for lineage plugin, as we did for authz

Good idea, I will try to shade it