#280: agent and server compatibility for Additional Data PATCH operation #283

lsulak · 2024-10-01T13:19:43Z

Note: Additional Data = AD

What was done:

a lot more than I wanted :( (but most files changed are a result of moving files from server/ to model/ - REST envelope classes)
AD update functionality compatible between Agent <-> Server
Changing HTTP Backend on Agent from HttpURLConnectionBackend -> OkHttpSyncBackend - inspired by java.net.ProtocolException: Invalid HTTP method: PATCH softwaremill/sttp#316 and Get scopes integration tests working raster-foundry/raster-foundry#5306 because HttpURLConnectionBackend didn't support PATCH HTTP Method and we need it for AD update endpoint
Creating a general DTO Base 64 Encode function + tests
Moved Envelope related code from Server to Model, because Agent also needs it
Added E2E test using Balta to check the results - this is a new 'type' of test and will not be executed in CI yet. But if your local PG is ready & empty and the service is running locally, it will work with the proper service configuration
Removed a bunch of obsolete AD DTOs
Server supports these 2 endpoints: patchPartitioningAdditionalDataEndpointV2 and getPartitioningEndpointV2

Note: If you want to run AgentWithServerIntegrationTests then:

Prepare PG locally (make sure it has newest schemas / tables, and that it's empty)
Prepare Service - either as Dockerized one or a typical Localhost - and adjust src/test/resources/reference.conf accordingly - this is needed: atum.dispatcher.http.url="http://localhost:8080" and atum.dispatcher.type="http"
I personally run PG in Docker & Service in Docker and it all works very nicely

Closes #280

Release notes:

Support of Additional Data addition or update functionality

github-actions · 2024-10-01T16:30:36Z

JaCoCo model module code coverage report - scala 2.13.11

Overall Project	59.69% `-12.4%`	❌
Files changed	46.67%	❌

File	Coverage
JsonSyntaxExtensions.scala	59.29% `-14.16%`	❌

github-actions · 2024-10-01T16:30:37Z

JaCoCo agent module code coverage report - scala 2.13.11

Overall Project	78.49% `-18.64%`	🍏
Files changed	39.06%	❌

File	Coverage
AtumAgent.scala	96.99%	🍏
AtumContext.scala	92.56% `-0.97%`	🍏
CapturingDispatcher.scala	87.43% `-68%`	❌
HttpDispatcher.scala	34.12% `-51.76%`	❌

github-actions · 2024-10-01T16:30:39Z

JaCoCo reader module code coverage report - scala 2.13.11

Overall Project	100%	🍏

There is no coverage information present for the Files changed

github-actions · 2024-10-01T16:30:40Z

JaCoCo server module code coverage report - scala 2.13.11

Overall Project	68.87% `-1.42%`	🍏
Files changed	79.83%	❌

File	Coverage
PartitioningRepositoryImpl.scala	100%	🍏
PartitioningServiceImpl.scala	100%	🍏
FlowControllerImpl.scala	100%	🍏
CheckpointControllerImpl.scala	100%	🍏
PartitioningControllerImpl.scala	91.67%	🍏
BaseController.scala	85.29%	🍏
GetPartitioningAdditionalData.scala	56.41% `-49.74%`	❌

…ort PATCH; also updating config in our agent tests, also moving envelope related DTOs to model/ package

lsulak · 2024-10-02T10:21:30Z

Release Notes:

Support of Additional Data addition or update functionality

… service

…e and thus HTTP Dispatcher

…rs are not needed to be tested

… not needed to be tested

benedeki

Reviewed on a pair session

…nused endpoint and DTOs/aliases

…gent-to-server-in-v0-3-0 # Conflicts: # server/src/main/scala/za/co/absa/atum/server/api/controller/BaseController.scala # server/src/main/scala/za/co/absa/atum/server/api/controller/FlowController.scala # server/src/main/scala/za/co/absa/atum/server/api/controller/FlowControllerImpl.scala # server/src/main/scala/za/co/absa/atum/server/api/repository/PartitioningRepository.scala # server/src/test/scala/za/co/absa/atum/server/api/controller/FlowControllerUnitTests.scala

benedeki

LGTM

benedeki · 2024-10-03T13:22:08Z

...test/scala/za/co/absa/atum/database/runs/GetPartitioningAdditionalDataIntegrationTests.scala

@@ -88,19 +88,21 @@ class GetPartitioningAdditionalDataIntegrationTests extends DBTestSuite{
    )

    function(fncGetPartitioningAdditionalData)
-      .setParam("i_partitioning", partitioning1)
+      .setParam("i_partitioning_id", fkPartitioning1)


You know, that you don't have to name the params, you can put them there positionally? Especially in one param functions, it's easier

Yep, I know, but I find it nicer - more readable if it's named, especially if there are more functions; but for 1-2 params I think you are right, can be ommited

benedeki · 2024-10-03T13:22:52Z

...test/scala/za/co/absa/atum/database/runs/GetPartitioningAdditionalDataIntegrationTests.scala

      .execute { queryResult =>
        val results = queryResult.next()
        assert(results.getInt("status").contains(11))
        assert(results.getString("status_text").contains("OK"))
        assert(results.getString("ad_name").contains("ad_1"))
        assert(results.getString("ad_value").contains("This is the additional data for Joseph"))
+        assert(results.getString("ad_author").contains("Joseph"))


Really need to switch to Balta 0.3, so much tidier 😉

#280: agent and server compatibility in AD

a0e7045

lsulak self-assigned this Oct 1, 2024

lsulak added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Oct 1, 2024

lsulak added 3 commits October 1, 2024 18:09

#280: agent and server compatibility in AD

6f10138

Merge branch 'master' into feature/280-adopt-agent-to-server-in-v0-3-0

550d557

#280: removing unnecessary DTOs and refactoring

978ea21

lsulak added 4 commits October 1, 2024 18:33

#280: unit test of DTO Base 64 encoder

74f3837

#280: replacing HTTP backend, 'HttpURLConnectionBackend' doesn't supp…

ae9875d

…ort PATCH; also updating config in our agent tests, also moving envelope related DTOs to model/ package

#280: adding E2E integration test using Balta

242452d

#280: Adding e2e test - not part of CI yet!

88ec23d

lsulak marked this pull request as ready for review October 2, 2024 10:13

lsulak requested review from benedeki, TebaleloS, Zejnilovic, dk1844 and salamonpavel as code owners October 2, 2024 10:13

lsulak removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label Oct 2, 2024

#280: adding missing licence headers

4bdeb99

lsulak added 7 commits October 2, 2024 12:25

#280: changing back - unit tests would be otherwise working with real…

d01aaea

… service

#280: changing back

2534a6b

#280: refactoring

a3519c9

#280: ignoring e2e test for now

b479c62

#280: test file must be with this suffix

204f1b1

#280: AgentWithServerIntegrationTests should run against local servic…

fd2c52f

…e and thus HTTP Dispatcher

#280: ignoring e2e test for now

10ff455

lsulak added 3 commits October 2, 2024 13:09

#280: ignoring e2e test for now - proper way

caae842

#280: adding exclusions to jacoco coverage ignore - these 2 dispatche…

068dc66

…rs are not needed to be tested

#280: adding exclusions to jacoco coverage ignore - envelope DTOs are…

c785a49

… not needed to be tested

lsulak changed the title ~~#280: agent and server compatibility in AD~~ #280: agent and server compatibility for Additional Data PATCH operation Oct 2, 2024

lsulak added 3 commits October 2, 2024 18:48

#280: post-review improvements

95cbe0a

#280: syndrome under control

4ead4dc

#280: actually using Atum HTTP Dispatcher config

0cdc465

benedeki previously approved these changes Oct 3, 2024

View reviewed changes

#280: post-review changes, refactoring simplification, removing old u…

9b344b5

…nused endpoint and DTOs/aliases

lsulak dismissed benedeki’s stale review via 9b344b5 October 3, 2024 11:15

lsulak added 3 commits October 3, 2024 15:09

#280: post-merge conflict resolutions

522ad56

#280: post-merge conflict resolutions

2b20f17

benedeki approved these changes Oct 3, 2024

View reviewed changes

lsulak merged commit ca7d8f4 into master Oct 3, 2024
9 of 10 checks passed

lsulak deleted the feature/280-adopt-agent-to-server-in-v0-3-0 branch October 3, 2024 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#280: agent and server compatibility for Additional Data PATCH operation #283

#280: agent and server compatibility for Additional Data PATCH operation #283

lsulak commented Oct 1, 2024 •

edited by miroslavpojer

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

lsulak commented Oct 2, 2024 •

edited

Loading

benedeki left a comment

benedeki left a comment

benedeki Oct 3, 2024

lsulak Oct 3, 2024

benedeki Oct 3, 2024

#280: agent and server compatibility for Additional Data PATCH operation #283

#280: agent and server compatibility for Additional Data PATCH operation #283

Conversation

lsulak commented Oct 1, 2024 • edited by miroslavpojer Loading

github-actions bot commented Oct 1, 2024 • edited Loading

JaCoCo model module code coverage report - scala 2.13.11

github-actions bot commented Oct 1, 2024 • edited Loading

JaCoCo agent module code coverage report - scala 2.13.11

github-actions bot commented Oct 1, 2024 • edited Loading

JaCoCo reader module code coverage report - scala 2.13.11

github-actions bot commented Oct 1, 2024 • edited Loading

JaCoCo server module code coverage report - scala 2.13.11

lsulak commented Oct 2, 2024 • edited Loading

benedeki left a comment

Choose a reason for hiding this comment

benedeki left a comment

Choose a reason for hiding this comment

benedeki Oct 3, 2024

Choose a reason for hiding this comment

lsulak Oct 3, 2024

Choose a reason for hiding this comment

benedeki Oct 3, 2024

Choose a reason for hiding this comment

lsulak commented Oct 1, 2024 •

edited by miroslavpojer

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 1, 2024 •

edited

Loading

lsulak commented Oct 2, 2024 •

edited

Loading