-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Spark Version 2.2 #43
Comments
What’s the issue you’re seeing trying to use this with Spark 2.2? The Spark
dependency is “provided” meaning you should be able to use this jar with
Spark 2.2.
…On Fri, Mar 30, 2018 at 11:07 AM DataWanderer ***@***.***> wrote:
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#43>, or mute the
thread
<https://github.com/notifications/unsubscribe-auth/AAG2gqSH_5GoMwD5lzRfsOY-45WduYh1ks5tjnRugaJpZM4TB-Ug>
.
|
com.google.common.util.concurrent.RateLimiter.acquire(I)D at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1$$anonfun$apply$2$$anonfun$apply$1.apply$mcDI$sp(DynamoDBRelation.scala:137)
at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1$$anonfun$apply$2$$anonfun$apply$1.apply(DynamoDBRelation.scala:136)
at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1$$anonfun$apply$2$$anonfun$apply$1.apply(DynamoDBRelation.scala:136)
at scala.Option.foreach(Option.scala:257) at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1$$anonfun$apply$2.apply(DynamoDBRelation.scala:136)
at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1$$anonfun$apply$2.apply(DynamoDBRelation.scala:130)
at scala.Option.foreach(Option.scala:257) at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1.apply(DynamoDBRelation.scala:130)
at
com.github.traviscrawford.spark.dynamodb.DynamoDBRelation$$anonfun$scan$1.apply(DynamoDBRelation.scala:114)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at
On Fri, Mar 30, 2018 at 6:59 PM, Travis Crawford <notifications@github.com>
wrote:
… What’s the issue you’re seeing trying to use this with Spark 2.2? The Spark
dependency is “provided” meaning you should be able to use this jar with
Spark 2.2.
On Fri, Mar 30, 2018 at 11:07 AM DataWanderer ***@***.***>
wrote:
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <#43>, or mute
the
> thread
> <https://github.com/notifications/unsubscribe-
auth/AAG2gqSH_5GoMwD5lzRfsOY-45WduYh1ks5tjnRugaJpZM4TB-Ug>
> .
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#43 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AG94jobiTzZJiFB_wAl3fCqcc8eaUrvSks5tjsbhgaJpZM4TB-Ug>
.
|
Looking at https://github.com/traviscrawford/spark-dynamodb/blob/master/pom.xml we see there's no explicit Guava dependency, so whatever version Spark brings is what's used. Does your application override the Guava version, either explicitly or transitively from some other dependency? What version of Guava is on your application's classpath, and what's pulling it in? |
The application doesnot override Guava Version. But some how EMR spark versino 2.2.1 and 2.2.0 gives this issue |
It looks like it's not limited to EMR. The same issue appears on DataBricks. Here is my code: import com.github.traviscrawford.spark.dynamodb.DynamoScanner
val rdd = DynamoScanner(sc, "table", 8, 1000, None, Option(50), Option("us-east-1"))
rdd.collect() And the output:
I'm running it on a databricks cluster using their 3.5 LTS (includes Apache Spark 2.2.1, Scala 2.11). Can I ask on what kind of cluster are you running those spark jobs @traviscrawford ? EMR, Databricks or self-maintained cluster? EDIT: It's worth noting that this only occurs when you put a rate limit that isn't None. |
Here is how I fixed it. I added I think there is something funky with the Scala compilation. Indeed, this stack means that the library code is trying to call an acquire method that takes an Int as an argument and return a Double. Which is not possible if you're using a stock version of Spark (Guava is stuck at 15 because of Hadoop if I'm not mistaken). Hence, shadowing a Guava 16 solves the issue |
No description provided.
The text was updated successfully, but these errors were encountered: