Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/235 get flow partitionings #267

Merged
merged 22 commits into from
Sep 26, 2024

Conversation

salamonpavel
Copy link
Collaborator

@salamonpavel salamonpavel commented Sep 10, 2024

Introduces GET endpoint for retrieval of partitionings of a given flow
Closes #235

Release notes:

  • Introduces GET endpoint for retrieval of partitionings of a given flow

@salamonpavel salamonpavel self-assigned this Sep 10, 2024
@salamonpavel salamonpavel added the work in progress Work on this item is not yet finished (mainly intended for PRs) label Sep 10, 2024
@salamonpavel
Copy link
Collaborator Author

Release notes:

  • Introduces GET endpoint for retrieval of partitionings of a given flow

Copy link

github-actions bot commented Sep 10, 2024

JaCoCo model module code coverage report - scala 2.13.11

Overall Project 63.64% 🍏

There is no coverage information present for the Files changed

Copy link

github-actions bot commented Sep 10, 2024

JaCoCo agent module code coverage report - scala 2.13.11

Overall Project 84.63% 🍏

There is no coverage information present for the Files changed

Copy link

github-actions bot commented Sep 10, 2024

JaCoCo reader module code coverage report - scala 2.13.11

Overall Project 100% 🍏

There is no coverage information present for the Files changed

Copy link

github-actions bot commented Sep 10, 2024

JaCoCo server module code coverage report - scala 2.13.11

Overall Project 75.29% -7.88% 🍏
Files changed 62.57%

File Coverage
PartitioningRepositoryImpl.scala 100% 🍏
PartitioningServiceImpl.scala 100% 🍏
PartitioningControllerImpl.scala 89.62% 🍏
GetFlowPartitionings.scala 70.48%

@salamonpavel salamonpavel marked this pull request as ready for review September 16, 2024 13:04
@salamonpavel salamonpavel removed the work in progress Work on this item is not yet finished (mainly intended for PRs) label Sep 16, 2024
# Conflicts:
#	server/src/main/scala/za/co/absa/atum/server/api/repository/PartitioningRepositoryImpl.scala
#	server/src/test/scala/za/co/absa/atum/server/api/controller/PartitioningControllerUnitTests.scala
#	server/src/test/scala/za/co/absa/atum/server/api/repository/PartitioningRepositoryUnitTests.scala
Comment on lines 78 to 99
WITH limited_partitionings AS (
SELECT P.id_partitioning
FROM flows.partitioning_to_flow PF
JOIN runs.partitionings P ON PF.fk_partitioning = P.id_partitioning
WHERE PF.fk_flow = i_flow_id
ORDER BY P.created_at DESC, P.id_partitioning
LIMIT i_limit OFFSET i_offset
)
SELECT
11 AS status,
'OK' AS status_text,
P.id_partitioning AS id,
P.partitioning,
P.created_by AS author,
_has_more AS has_more
FROM
runs.partitionings P
WHERE
P.id_partitioning IN (SELECT LP.id_partitioning FROM limited_partitionings LP)
ORDER BY
P.created_at DESC,
P.id_partitioning;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
WITH limited_partitionings AS (
SELECT P.id_partitioning
FROM flows.partitioning_to_flow PF
JOIN runs.partitionings P ON PF.fk_partitioning = P.id_partitioning
WHERE PF.fk_flow = i_flow_id
ORDER BY P.created_at DESC, P.id_partitioning
LIMIT i_limit OFFSET i_offset
)
SELECT
11 AS status,
'OK' AS status_text,
P.id_partitioning AS id,
P.partitioning,
P.created_by AS author,
_has_more AS has_more
FROM
runs.partitionings P
WHERE
P.id_partitioning IN (SELECT LP.id_partitioning FROM limited_partitionings LP)
ORDER BY
P.created_at DESC,
P.id_partitioning;
SELECT
11 AS status,
'OK' AS status_text,
P.id_partitioning,
P.partitioning,
P.created_by,
has_more
FROM
runs.partitionings P INNER JOIN
flows.partitioning_to_flow PF ON PF.fk_partitioning = P.id_partitioning
WHERE
PF.fk_flow = i_flow
ORDER BY
P.id_partitioning,
P.created_at DESC
LIMIT i_limit OFFSET i_offset;

Several things:

  • You don't need the WITH here. Generally the order of approaches to use:
    • use JOIN
    • use subquery - there are cases where this can be better then JOIN, but we don't have that much data yet to run a meaningful comparison
    • use WITH
    • use temporary tables
  • in the RETURN QUERY statement it's not required the output column names to be the same as the OUT ones. Only the position matters.
  • I switched the order of ORDER BY columns, reasons described here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok changed

P.created_at DESC,
P.id_partitioning;

IF NOT FOUND THEN
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as here. Do we return an error?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok changed

OUT id BIGINT,
OUT partitioning JSONB,
OUT author TEXT,
OUT has_more BOOLEAN
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we also want to care whether a given partitioning was primary for the given flow?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Little value now, and adds a table to the JOIN.
IMHO not worth it.
Btw, we could add the main flow id to partitioning table (for easier search and one less index).

Copy link
Collaborator

@lsulak lsulak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review finished, apart from 1 q from me and 1 note from David it looks good

Comment on lines 95 to 99
IF NOT FOUND THEN
status := 12;
status_text := 'OK with no partitionings found';
RETURN NEXT;
END IF;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
IF NOT FOUND THEN
status := 12;
status_text := 'OK with no partitionings found';
RETURN NEXT;
END IF;

salamonpavel and others added 3 commits September 26, 2024 16:31
…s.sql

Co-authored-by: David Benedeki <14905969+benedeki@users.noreply.github.com>
# Conflicts:
#	server/src/main/scala/za/co/absa/atum/server/api/repository/PartitioningRepository.scala
#	server/src/main/scala/za/co/absa/atum/server/api/service/PartitioningServiceImpl.scala
@salamonpavel salamonpavel merged commit db4a867 into master Sep 26, 2024
10 checks passed
@salamonpavel salamonpavel deleted the feature/235-get-flow-partitionings branch September 26, 2024 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GET /flows/{flowId}/partitonings
3 participants