You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When hitting an unsupported type it will immediately throw an Exception. By your comments there I assume this is expected :)
17/08/01 12:41:06 ERROR DynamoDBRelation: Failed converting item to row: {"id":"xxxx","sections":["AAA","BBB","CCC"]}
scala.MatchError: ArrayType(StringType,true) (of class org.apache.spark.sql.types.ArrayType)
at com.github.traviscrawford.spark.dynamodb.ItemConverter$$anonfun$1.apply(ItemConverter.scala:30)
I'm quite new to Scala but the following works for my current needs:
case _ =>
field.dataType match {
case ArrayType(StringType,true) => jsonFieldValue.extract[List[String]]
case IntegerType => jsonFieldValue.extract[Int]
case LongType => jsonFieldValue.extract[Long]
case DoubleType => jsonFieldValue.extract[Double]
case StringType => jsonFieldValue.extract[String]
}
I wonder if you are planning to update the logic with something more robust
The text was updated successfully, but these errors were encountered:
Agreed that ItemConverter is not fully featured. It was enough for my needs at the time and this has never been revisited. I just added support for simple arrays in branch https://github.com/traviscrawford/spark-dynamodb/compare/travis/improve-item-converter . Can you try that branch on your table and test that it works for you? If it does I'll merge to master and publish a new release.
https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/ItemConverter.scala#L17
When hitting an unsupported type it will immediately throw an Exception. By your comments there I assume this is expected :)
I'm quite new to Scala but the following works for my current needs:
I wonder if you are planning to update the logic with something more robust
The text was updated successfully, but these errors were encountered: