Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ItemConverter to support DynamoDB List #31

Open
pmminov opened this issue Aug 1, 2017 · 1 comment
Open

ItemConverter to support DynamoDB List #31

pmminov opened this issue Aug 1, 2017 · 1 comment

Comments

@pmminov
Copy link

pmminov commented Aug 1, 2017

https://github.com/traviscrawford/spark-dynamodb/blob/master/src/main/scala/com/github/traviscrawford/spark/dynamodb/ItemConverter.scala#L17

When hitting an unsupported type it will immediately throw an Exception. By your comments there I assume this is expected :)

17/08/01 12:41:06 ERROR DynamoDBRelation: Failed converting item to row: {"id":"xxxx","sections":["AAA","BBB","CCC"]}
scala.MatchError: ArrayType(StringType,true) (of class org.apache.spark.sql.types.ArrayType)
	at com.github.traviscrawford.spark.dynamodb.ItemConverter$$anonfun$1.apply(ItemConverter.scala:30)

I'm quite new to Scala but the following works for my current needs:

case _ =>
          field.dataType match {
            case ArrayType(StringType,true) => jsonFieldValue.extract[List[String]]
            case IntegerType => jsonFieldValue.extract[Int]
            case LongType => jsonFieldValue.extract[Long]
            case DoubleType => jsonFieldValue.extract[Double]
            case StringType => jsonFieldValue.extract[String]
          }

I wonder if you are planning to update the logic with something more robust

@traviscrawford
Copy link
Owner

Agreed that ItemConverter is not fully featured. It was enough for my needs at the time and this has never been revisited. I just added support for simple arrays in branch https://github.com/traviscrawford/spark-dynamodb/compare/travis/improve-item-converter . Can you try that branch on your table and test that it works for you? If it does I'll merge to master and publish a new release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants