sync upstream 0.50.0 (#5)

* Fixing data quality display. (MarquezProject#2937) Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Dataset Version call simplification (MarquezProject#2938) * Fixing data quality display. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Fixing dataset version calls. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> --------- Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * feat: allow db-migrate without version (MarquezProject#2936) Signed-off-by: David Goss <david@davidgoss.co> * Display full `runID` and check icon when copied (MarquezProject#2940) Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Deferred copy revert. (MarquezProject#2941) Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Long text handling (MarquezProject#2942) * Deferred copy revert. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Long text handling. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Adding search back in. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> --------- Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Use project root for docker volume prefix (MarquezProject#2943) Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * fix: Correct SQL query pagination for DatasetVersion findAll method (MarquezProject#2945) Signed-off-by: Alper İnan <kurumsal.alper@gmail.com> Signed-off-by: Alper <kurumsal.alper@gmail.com> * Update changelog for `0.50.0` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Replace `redoc-cli` with `redocly` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Prepare for release 0.50.0 Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Prepare next development version 0.51.0-SNAPSHOT Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Templatize event time in `metadata.json` (MarquezProject#2946) * Templatize event time in `metadata.json` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Use `metadata.template.json` Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> --------- Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Update CHANGELOG.md * Update `web/docs/demo.gif` (MarquezProject#2948) Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * fix(deps): update dependency io.openlineage:openlineage-java to v1.23.0 (MarquezProject#2907) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update dependency org.assertj:assertj-core to v3.26.3 (MarquezProject#2909) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Willy Lulciuc <willy@datakin.com> * fix(deps): update dependency org.postgresql:postgresql to v42.7.4 (MarquezProject#2912) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * fix(deps): update dependency org.opensearch.client:opensearch-rest-client to v2.17.1 (MarquezProject#2911) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Willy Lulciuc <willy@datakin.com> * fix(deps): update dependency org.apache.commons:commons-lang3 to v3.17.0 (MarquezProject#2908) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * Ignore `**/stats/**` (MarquezProject#2952) Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Update compatibility for `0.50.0` * fix(deps): update dependency org.opensearch.client:opensearch-java to v2.16.0 (MarquezProject#2910) Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> * `Dataset.currentVersionUuid` `->` `DatasetVersion.uuid` (MarquezProject#2954) Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> * Update Events Page (MarquezProject#2955) * Tuning the events page for longer events. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Adding events file. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Refetch jobs button. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Refetch jobs button. Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> * Lint Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> --------- Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> Co-authored-by: Willy Lulciuc <willy@datakin.com> * Lineage run attachment issue. (MarquezProject#2953) Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> Co-authored-by: Willy Lulciuc <willy@datakin.com> * feature: Better handling of missing environment variables in setupProxy.js file. (MarquezProject#2956) Signed-off-by: Artur Owczarek <owczarek.artur@gmail.com> --------- Signed-off-by: phixMe <peter.hicks@pdtechsolutions.com> Signed-off-by: David Goss <david@davidgoss.co> Signed-off-by: Willy Lulciuc <willy.lulciuc@gmail.com> Signed-off-by: Alper İnan <kurumsal.alper@gmail.com> Signed-off-by: Alper <kurumsal.alper@gmail.com> Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Signed-off-by: Artur Owczarek <owczarek.artur@gmail.com> Co-authored-by: Peter Hicks <phixMe@users.noreply.github.com> Co-authored-by: davidjgoss <david@davidgoss.co> Co-authored-by: Willy Lulciuc <willy@datakin.com> Co-authored-by: Alper İnan <65857758+inanalper@users.noreply.github.com> Co-authored-by: Willy Lulciuc <willy.lulciuc@gmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Artur Owczarek <owczarek.artur@gmail.com>
nubank · Oct 28, 2024 · 5df8472 · 5df8472
1 parent 6dc1d99
commit 5df8472
Show file tree

Hide file tree

Showing 50 changed files with 1,098 additions and 2,730 deletions.
diff --git a/.circleci/api-load-test.sh b/.circleci/api-load-test.sh
@@ -14,7 +14,7 @@
 set -e
 
 # Build version of Marquez
-readonly MARQUEZ_VERSION=0.50.0-SNAPSHOT
+readonly MARQUEZ_VERSION=0.51.0-SNAPSHOT
 # Fully qualified path to marquez.jar
 readonly MARQUEZ_JAR="api/build/libs/marquez-api-${MARQUEZ_VERSION}.jar"
 

diff --git a/.circleci/db-migration.sh b/.circleci/db-migration.sh
@@ -13,7 +13,7 @@
 # Version of PostgreSQL
 readonly POSTGRES_VERSION="14"
 # Version of Marquez
-readonly MARQUEZ_VERSION=0.49.0
+readonly MARQUEZ_VERSION=0.50.0
 # Build version of Marquez
 readonly MARQUEZ_BUILD_VERSION="$(git log --pretty=format:'%h' -n 1)" # SHA1
 readonly POSTGRES_PORT=5432

diff --git a/.env.example b/.env.example
@@ -3,4 +3,4 @@ API_ADMIN_PORT=5001
 WEB_PORT=3000
 POSTGRES_PORT=5432
 SEARCH_PORT=9200
-TAG=0.49.0
+TAG=0.50.0
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,45 @@
 # Changelog
 
-## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.48.0...HEAD)
+## [Unreleased](https://github.com/MarquezProject/marquez/compare/0.50.0...HEAD)
+
+## [0.50.0](https://github.com/MarquezProject/marquez/compare/0.49.0...0.50.0) - 2024-10-23
+
+### Added
+
+* Web: **New** _Data Observability_ dashboard for _stats_ on OpenLineage events (`24hrs`, past `7.days`); views are also available for _sources_, _datasets_, and _jobs_; **new** job list view has also been introduced displaying the latest `N` runs (and duration) for a given job [`#2913`](https://github.com/MarquezProject/marquez/pull/2913) [@phixMe](https://github.com/phixMe)
+* Web: `404` page [`#2890`](https://github.com/MarquezProject/marquez/pull/2890) [@phixMe](https://github.com/phixMe)
+* Web: Display _parent_ job (if present) in _job_ panel [`#2868`](https://github.com/MarquezProject/marquez/pull/2868) [@phixMe](https://github.com/phixMe)
+* Web: Allow override of `web.port` via `WEB_PORT` environment variable [`#2838`](https://github.com/MarquezProject/marquez/pull/2838) [@bidlako](https://github.com/bidlako)
+* Web: Allow _nullable_ columns for schema in _dataset_ panel (use `N/A`) [`#2896`](https://github.com/MarquezProject/marquez/pull/2896) [@phixMe](https://github.com/phixMe)
+* Web: Better feedback when lineage events are loading [`#2916`](https://github.com/MarquezProject/marquez/pull/2916) [@NisargChokshi45](https://github.com/NisargChokshi45)
+* API: `Job` object will now return `Job.latestRuns` (latest `N` runs) and `Job.latestRun` (last run to execute) [`#2901`](https://github.com/MarquezProject/marquez/pull/2901) [@phixMe](https://github.com/phixMe)
+* API: Use `io.openlineage.server.*` pkg and **class** [`Metadata`](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/api/models/Metadata.java) (utility class for `OpenLineage.RunEvent`) [`#2853`](https://github.com/MarquezProject/marquez/pull/2853) [@wslulciuc](https://github.com/wslulciuc)
+* API: Use `TIMESTAMPTZ` for _timestamps_ in database; supports _Data Observability_ dashboard with timezone of user [`#2924`](https://github.com/MarquezProject/marquez/pull/2924) [@wslulciuc](https://github.com/wslulciuc)
+* API: Set `current_run_uuid` in **table** `jobs` optimizing query for `JobDao.findAll()` [`#2929`](https://github.com/MarquezProject/marquez/pull/2929) [@wslulciuc](https://github.com/wslulciuc)
+* API: **New** `GET` `/api/v1/jobs` [`#2930`](https://github.com/MarquezProject/marquez/pull/2930) [@wslulciuc](https://github.com/wslulciuc)
+* CLI: **New** cmd args for [`cli.MetadataCommand`](https://github.com/MarquezProject/marquez/blob/main/api/src/main/java/marquez/cli/MetadataCommand.java) [`#2923`](https://github.com/MarquezProject/marquez/pull/2923) [@wslulciuc](https://github.com/wslulciuc)
+  * `--jobs`: _limits OL jobs up to N (default: 5)_
+  * `--runs-per-job`: _limits OL run executions per job up to N (default: 10)_
+  * `--runs-active`: _limits OL run executions marked as active (='RUNNING') up to N_
+  * `--max-run-fails-per-job`: _maximum OL run fails per job (default: 2)_
+  * `--min-run-duration`: _minimum OL run duration (in seconds) per execution (default: 300)_
+  * `--run-start-time`: _specifies the OL run start time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs starting relative to the initial start time. (default: 2024-10-15T01:00:11.080828Z)_
+  * `--run-end-time`: _specifies the OL run end time in UTC ISO ('YYYY-MM-DDTHH:MM:SSZ'); used for the initial OL run, with subsequent runs ending relative to the initial end time. (default: 2024-10-15T01:07:25.080828Z)_
+
+### Fixed
+
+* Web: Better rendering of long text [`#2942`](https://github.com/MarquezProject/marquez/pull/2942) [@phixMe](https://github.com/phixMe)
+* Web: Display full `runID` and check icon when copied [`#2940`](https://github.com/MarquezProject/marquez/pull/2940) [`#2941`](https://github.com/MarquezProject/marquez/pull/2941) [@wslulciuc](https://github.com/wslulciuc) [@phixMe](https://github.com/phixMe)
+* Web: Use **DatasetVersionAPI** to display latest schema and remove extra job facets API call in _dataset_ panel [`#2938`](https://github.com/MarquezProject/marquez/pull/2938) [@phixMe](https://github.com/phixMe)
+* Web: Use **DatasetAPI** for data quality assertions in _dataset_ panel [`#2937`](https://github.com/MarquezProject/marquez/pull/2937) [@phixMe](https://github.com/phixMe)
+* Web: Fill-in _job_ node in lineage graph with correct color for `JobEvent`s [`#2934`](https://github.com/MarquezProject/marquez/pull/2934) [@phixMe](https://github.com/phixMe)
+* Web: Fill-in _job_ node in lineage graph with correct color for run states `RUNNING`, `COMPLETED`, etc [`#2897`](https://github.com/MarquezProject/marquez/pull/2897) [@phixMe](https://github.com/phixMe)
+* API: Pagination for `DatasetVersion.findAll()`; not all dataset versions were returned for `GET` `/api/v1/namespaces/{namespace}/datasets/{dataset}/versions` [`#2944`](https://github.com/MarquezProject/marquez/pull/2945) [@inanalper](https://github.com/inanalper)
+* API: `null` namespace and dataset name in **view** `dataset_view` for old versions; use **table** `dataset_versions` instead in column lineage query [#2881](https://github.com/MarquezProject/marquez/pull/2881) [@sophiely](https://github.com/sophiely)
+* API: Missing `DELETE CASCADE` on **table** `job_facets` [`#2878`](https://github.com/MarquezProject/marquez/pull/2878) [@mattwparas](https://github.com/mattwparas)
+* API: Ensure `Job.latestRun` in `Job` object is set for runs in a `RUNNING` state; before `Job.latestRun` was set only for a run in a _done_ state (`COMPLETED` / `FAILED`) [`#2933`](https://github.com/MarquezProject/marquez/pull/2933) [@phixMe](https://github.com/phixMe)
+* CLI: Repurpose cmd `db-migrate` to run all pending database migrations, no longer coupling migrations with HTTP server startup [`#2936`](https://github.com/MarquezProject/marquez/pull/2936) [@davidjgoss](https://github.com/davidjgoss)
+* Chart: Missing common `labels` for `deployment.replicas` [`#2877`](https://github.com/MarquezProject/marquez/pull/2877) [@alaturqua](https://github.com/alaturqua)
 
 ## [0.49.0](https://github.com/MarquezProject/marquez/compare/0.48.0...0.49.0) - 2024-08-07
 

diff --git a/README.md b/README.md
@@ -98,8 +98,8 @@ Versions of Marquez are compatible with OpenLineage unless noted otherwise. We e
 | **Marquez**                                                                                      | **OpenLineage**                                               | **Status**    |
 |--------------------------------------------------------------------------------------------------|---------------------------------------------------------------|---------------|
 | [`UNRELEASED`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#unreleased)      | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `CURRENT`     |
-| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |
-| [`0.48.0`](https://github.com/MarquezProject/marquez/blob/0.45.0/CHANGELOG.md#0480---2024-08-05) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |
+| [`0.50.0`](https://github.com/MarquezProject/marquez/blob/main/CHANGELOG.md#0500---2024-10-23)   | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `RECOMMENDED` |
+| [`0.49.0`](https://github.com/MarquezProject/marquez/blob/0.49.0/CHANGELOG.md#0490---2024-08-07) | [`2-0-2`](https://openlineage.io/spec/2-0-2/OpenLineage.json) | `MAINTENANCE` |
 
 > **Note:** The [`openlineage-python`](https://pypi.org/project/openlineage-python) and [`openlineage-java`](https://central.sonatype.com/artifact/io.openlineage/openlineage-java) libraries will a higher version than the OpenLineage [specification](https://github.com/OpenLineage/OpenLineage/tree/main/spec) as they have different version requirements.
 

diff --git a/api/build.gradle b/api/build.gradle
@@ -51,8 +51,8 @@ dependencies {
     implementation 'com.graphql-java:graphql-java:20.9'
     implementation 'com.graphql-java-kickstart:graphql-java-servlet:12.0.0'
 
-    implementation 'org.opensearch.client:opensearch-rest-client:2.15.0'
-    implementation 'org.opensearch.client:opensearch-java:2.6.0'
+    implementation 'org.opensearch.client:opensearch-rest-client:2.17.1'
+    implementation 'org.opensearch.client:opensearch-java:2.16.0'
 
     testImplementation "io.dropwizard:dropwizard-testing:${dropwizardVersion}"
     testImplementation "org.jdbi:jdbi3-testing:${jdbi3Version}"

diff --git a/api/src/main/java/marquez/MarquezApp.java b/api/src/main/java/marquez/MarquezApp.java
@@ -28,7 +28,7 @@
 import marquez.api.filter.JobRedirectFilter;
 import marquez.api.filter.exclusions.Exclusions;
 import marquez.api.filter.exclusions.ExclusionsConfig;
-import marquez.cli.DbMigrationCommand;
+import marquez.cli.DbMigrateCommand;
 import marquez.cli.DbRetentionCommand;
 import marquez.cli.MetadataCommand;
 import marquez.cli.SeedCommand;
@@ -90,6 +90,7 @@ public void initialize(@NonNull Bootstrap<MarquezConfig> bootstrap) {
             new EnvironmentVariableSubstitutor(ERROR_ON_UNDEFINED)));
 
     // Add CLI commands
+    bootstrap.addCommand(new DbMigrateCommand());
     bootstrap.addCommand(new DbRetentionCommand());
     bootstrap.addCommand(new MetadataCommand());
     bootstrap.addCommand(new SeedCommand());
@@ -202,12 +203,6 @@ public void registerResources(
     }
   }
 
-  @Override
-  protected void addDefaultCommands(Bootstrap<MarquezConfig> bootstrap) {
-    bootstrap.addCommand(new DbMigrationCommand<>(this));
-    super.addDefaultCommands(bootstrap);
-  }
-
   private void registerServlets(@NonNull Environment env) {
     log.debug("Registering servlets...");
 

diff --git a/api/src/main/java/marquez/MarquezContext.java b/api/src/main/java/marquez/MarquezContext.java
@@ -155,7 +155,7 @@ private MarquezContext(
     this.tagService = new TagService(baseDao);
     this.tagService.init(tags);
     this.openLineageService = new OpenLineageService(baseDao, runService);
-    this.lineageService = new LineageService(lineageDao, jobDao);
+    this.lineageService = new LineageService(lineageDao, jobDao, runDao);
     this.columnLineageService = new ColumnLineageService(columnLineageDao, datasetFieldDao);
     this.searchService = new SearchService(searchConfig);
     this.statsService = new StatsService(statsDao);

diff --git a/api/src/main/java/marquez/cli/DbMigrateCommand.java b/api/src/main/java/marquez/cli/DbMigrateCommand.java
@@ -0,0 +1,42 @@
+/*
+ * Copyright 2018-2022 contributors to the Marquez project
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package marquez.cli;
+
+import io.dropwizard.cli.ConfiguredCommand;
+import io.dropwizard.db.DataSourceFactory;
+import io.dropwizard.db.ManagedDataSource;
+import io.dropwizard.setup.Bootstrap;
+import lombok.NonNull;
+import lombok.extern.slf4j.Slf4j;
+import marquez.MarquezConfig;
+import marquez.db.DbMigration;
+import net.sourceforge.argparse4j.inf.Namespace;
+
+/**
+ * A command to manually run database migrations. This command to be used to run migrations
+ * decoupled from application deployment.
+ */
+@Slf4j
+public class DbMigrateCommand extends ConfiguredCommand<MarquezConfig> {
+
+  public DbMigrateCommand() {
+    super("db-migrate", "A command to manually run database migrations.");
+  }
+
+  @Override
+  protected void run(
+      @NonNull Bootstrap<MarquezConfig> bootstrap,
+      @NonNull Namespace namespace,
+      @NonNull MarquezConfig configuration)
+      throws Exception {
+
+    final DataSourceFactory sourceFactory = configuration.getDataSourceFactory();
+    final ManagedDataSource source =
+        sourceFactory.build(bootstrap.getMetricRegistry(), "MarquezApp-source");
+
+    DbMigration.migrateDbOrError(configuration.getFlywayFactory(), source, true);
+  }
+}
diff --git a/api/src/main/java/marquez/cli/DbMigrationCommand.java b/api/src/main/java/marquez/cli/DbMigrationCommand.java
diff --git a/api/src/main/java/marquez/db/DatasetVersionDao.java b/api/src/main/java/marquez/db/DatasetVersionDao.java
@@ -168,14 +168,14 @@ default void updateDatasetVersionMetric(
       WITH selected_dataset_versions AS (
           SELECT dv.*
           FROM dataset_versions dv
-          WHERE dv.version = :version
+          WHERE dv.uuid = :version
       ), selected_dataset_version_facets AS (
           SELECT dv.uuid, dv.dataset_name, dv.namespace_name, df.run_uuid, df.lineage_event_time, df.facet
           FROM selected_dataset_versions dv
           LEFT JOIN dataset_facets_view df ON df.dataset_version_uuid = dv.uuid
       )
       SELECT d.type, d.name, d.physical_name, d.namespace_name, d.source_name, d.description, dv.lifecycle_state,\s
-          dv.created_at, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
+          dv.created_at, dv.uuid AS current_version_uuid, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
           sv.schema_location, t.tags, f.facets
       FROM selected_dataset_versions dv
       LEFT JOIN datasets_view d ON d.uuid = dv.dataset_uuid
@@ -206,7 +206,7 @@ WITH selected_dataset_versions AS (
           LEFT JOIN dataset_facets_view df ON df.dataset_version_uuid = dv.uuid  AND (df.type ILIKE 'dataset' OR df.type ILIKE 'unknown' OR df.type ILIKE 'input')
       )
       SELECT d.type, d.name, d.physical_name, d.namespace_name, d.source_name, d.description, dv.lifecycle_state,\s
-          dv.created_at, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
+          dv.created_at, dv.uuid AS current_version_uuid, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
           sv.schema_location, t.tags, f.facets
       FROM selected_dataset_versions dv
       LEFT JOIN datasets_view d ON d.uuid = dv.dataset_uuid
@@ -255,7 +255,7 @@ default Optional<DatasetVersion> findByWithRun(UUID version) {
       """
       WITH dataset_info AS (
 	    SELECT d.type, d.name, d.physical_name, d.namespace_name, d.source_name, d.description, dv.lifecycle_state,
-		dv.created_at, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
+		dv.created_at, dv.uuid AS current_version_uuid, dv.version, dv.dataset_schema_version_uuid, dv.fields, dv.run_uuid AS createdByRunUuid,
 		sv.schema_location, t.tags, f.facets, f.lineage_event_time, f.dataset_version_uuid, facet_name
 		FROM dataset_versions dv
 		LEFT JOIN datasets_view d ON d.uuid = dv.dataset_uuid
@@ -277,7 +277,6 @@ LEFT JOIN (
       	) f ON f.dataset_version_uuid = dv.uuid
       	WHERE dv.namespace_name = :namespaceName
             AND dv.dataset_name = :datasetName
-      	LIMIT :limit OFFSET :offset
         ),
         dataset_symlinks_names as (
           SELECT DISTINCT dataset_uuid, name
@@ -286,15 +285,16 @@ dataset_symlinks_names as (
         )
         SELECT
 	        type, name, physical_name, namespace_name, source_name, description, lifecycle_state,
-            created_at, version, dataset_schema_version_uuid, fields, createdByRunUuid, schema_location,
+            created_at, current_version_uuid, version, dataset_schema_version_uuid, fields, createdByRunUuid, schema_location,
             tags, dataset_version_uuid,
 	        JSONB_AGG(facets ORDER BY lineage_event_time ASC) AS facets
         FROM dataset_info
         WHERE name NOT IN (SELECT name FROM dataset_symlinks_names)
         GROUP BY type, name, physical_name, namespace_name, source_name, description, lifecycle_state,
-            created_at, version, dataset_schema_version_uuid, fields, createdByRunUuid, schema_location,
+            created_at, current_version_uuid, version, dataset_schema_version_uuid, fields, createdByRunUuid, schema_location,
             tags, dataset_version_uuid
         ORDER BY created_at DESC
+        LIMIT :limit OFFSET :offset
   """)
   List<DatasetVersion> findAll(String namespaceName, String datasetName, int limit, int offset);