#38 http client and config

[cerveada]

fixes #38

• Since no API is defined yet, just simple HTTP client implementation.
• Using dispatcher as an abstraction for publishing data, currently to http nad console, later kafka and other destination can be added
• added type safe config
• dispatcher type can be set via config

other possible improvements

• use async http
• better error handling

We need to have some discussion about how we want to handle error in the context of Spark.

[github-actions]

JaCoCo agent module code coverage report - spark:2 - scala 2.12.12

File	Coverage [43.87%]	❌
AtumAgent.scala	72.81%	❌
ConsoleDispatcher.scala	63.64%	❌
HttpDispatcher.scala	0%	❌

Total Project Coverage	55.54%	❌

[github-actions]

JaCoCo server module code coverage report - scala 2.12.12

There is no coverage information present for the Files changed

Total Project Coverage	12.45%	❌

[TebaleloS]

This looks good to me.

[lsulak]

How would retries work? I never used sttp library but I quickly checked it and it looks awesome!

[cerveada]

There is no direct retry support, but it could be implemented using: https://github.com/softwaremill/retry or https://github.com/hipjim/scala-retry

[cerveada]

Regarding the error handling, I think we have two options:

1. try our best to deliver the measure result, but when all fails, just log it and let the Spark job continue.
2. when sending measurement fails, throw an exception and let the Spark job fail. I am not sure if it will fail the whole Job, but since Atum runs as part of the job (not as listener llike Spline) it should, right?

To me it seems the number 1 makes more sense but at the end it depends on the requirements on Atum.

[TebaleloS]

Yah, in terms of error handling we need the requirements, but I think the first option would be great.

[benedeki]

I like it.
I could imagine:

• a dispatcher that would accumulate the results locally, for testing purposes
• dispatcher not built directly in AtumAgent but in a factory
• the HttpDispatcher doing a quick ping to the server on creation to fail fast (no endpoint for it now anyway)

And obviously the base trait's methods will change. PR is 😎
*

[benedeki]

Regarding the error handling, I think we have two options:

1. try our best to deliver the measure result, but when all fails, just log it and let the Spark job continue.
2. when sending measurement fails, throw an exception and let the Spark job fail. I am not sure if it will fail the whole Job, but since Atum runs as part of the job (not as listener llike Spline) it should, right?

To me it seems the number 1 makes more sense but at the end it depends on the requirements on Atum.

Eventually probably ErrorHandling from SparkCommons can be used. For now, let's keep it as it is, and throw and exception failing the whole job.
If any of us are bored can used the ErrorHandling right away with the default of failing with exception implementation.

[benedeki]

This I would removed though.

[cerveada]

done

[benedeki]

• code reviewed
• pulled
• built
• run