Travis build fails due to Kryo serialisation issue

[soxofaan]

Latest travis-ci builds are failing with (from https://travis-ci.org/github/Open-EO/openeo-geopyspark-driver/builds/675374224):

INTERNALERROR>   File "/home/travis/build/Open-EO/openeo-geopyspark-driver/tests/conftest.py", line 14, in pytest_configure
INTERNALERROR>     _setup_local_spark(terminal_reporter, verbosity=config.getoption("verbose"))
INTERNALERROR>   File "/home/travis/build/Open-EO/openeo-geopyspark-driver/tests/conftest.py", line 58, in _setup_local_spark
INTERNALERROR>     answer = context.parallelize([9, 10, 11, 12]).sum()
INTERNALERROR>   File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/pyspark/rdd.py", line 1044, in sum
INTERNALERROR>     return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
INTERNALERROR>   File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/pyspark/rdd.py", line 915, in fold
INTERNALERROR>     vals = self.mapPartitions(func).collect()
INTERNALERROR>   File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/pyspark/rdd.py", line 814, in collect
INTERNALERROR>     sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
INTERNALERROR>   File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in __call__
INTERNALERROR>     answer, self.gateway_client, self.target_id, self.name)
INTERNALERROR>   File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/py4j/protocol.py", line 328, in get_return_value
INTERNALERROR>     format(target_id, ".", name), value)
INTERNALERROR> py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
INTERNALERROR> : org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: org.apache.spark.SparkException: Failed to register classes with Kryo
INTERNALERROR> org.apache.spark.SparkException: Failed to register classes with Kryo
INTERNALERROR> 	at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
INTERNALERROR> 	at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:324)
INTERNALERROR> 	at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:309)
INTERNALERROR> 	at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:218)
INTERNALERROR> 	at org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:288)
INTERNALERROR> 	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:127)
INTERNALERROR> 	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
INTERNALERROR> 	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
INTERNALERROR> 	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
INTERNALERROR> 	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1482)
INTERNALERROR> 	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1039)

[soxofaan]

first build with this issue: 399
https://travis-ci.org/github/Open-EO/openeo-geopyspark-driver/builds/627130535
from 136360a

[soxofaan]

that commit is yours @bossie , any quick ideas?

[bossie]

The stack trace also contains this: java.lang.ClassNotFoundException: geopyspark.geotools.kryo.ExpandedKryoRegistrator, which sounds vaguely familiar.

This will download the "official" GeoPySpark back end assembly:

openeo-geopyspark-driver/.travis.yml

Line 13 in 157ddfc

- SPARK_HOME=$(find_spark_home.py) geopyspark install-jar

We want our Geotrellis 3.1 back end assembly at https://artifactory.vgt.vito.be/auxdata-public/openeo/geotrellis-backend-assembly-0.4.5-openeo.jar instead.

[soxofaan]

aha, and that install-jar will fail because it tries to download version 0.4.5-openeo from https://github.com/locationtech-labs/....

this also sound familiar to me: locationtech-labs/geopyspark#710 :)

[soxofaan]

travis build works again