Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synchronization logic for exported API #8203

Merged
merged 8 commits into from
Nov 7, 2024

Conversation

jonasfj
Copy link
Member

@jonasfj jonasfj commented Oct 29, 2024

We identified 3 cases where we want to sync:

  • (A) Full daily background sync.
  • (B) Immediate sync after a package version has been published.
  • (C) When Package.updated is changed.

Leveraging the fact that Bucket.list from package:gcloud now returns ObjectInfo, we can do synchronizePackage such that it:

  • Updates advisories,
  • Updates version listing response,
  • List all versions from TarballStorage and takes the versions visible in version listing and uses synchronizeTarballs.

In the case where there is nothing to update, this should do:

  • datastore lookup of Package,
  • datastore query for PackageVersion entities,
  • Cache lookup for advisories.
  • 2 HEAD requests to GCS for advisories and version listing (to confirm content haven't changed using md5Hash).
  • A bucket.list request to canonical bucket using TarballStorage.
  • Two bucket.list requests to exported API bucket to confirm that archives exported match those from TarballStorage.

All in all, I think that's actually fairly efficient.

@jonasfj jonasfj requested a review from isoos October 29, 2024 21:32
@jonasfj jonasfj force-pushed the exported-api-draft-full-sync branch from 96be3e8 to e069030 Compare October 31, 2024 11:54
@jonasfj jonasfj force-pushed the exported-api-draft-full-sync branch 3 times, most recently from fb3a1f7 to b7f7a99 Compare November 6, 2024 14:18
// Only delete the item if it's older than _minGarbageAge
// This avoids any races where we delete files we've just created
// TODO: Conditionally deletion API from package:gcloud would be better!
await _bucket.tryDelete(item.name);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: should we log the fact of deletion (here and elsewhere)? Maybe for a cursory later verification that we are not doing something stupid...

app/test/shared/test_services.dart Outdated Show resolved Hide resolved
@jonasfj jonasfj force-pushed the exported-api-draft-full-sync branch from b7f7a99 to 55e624a Compare November 7, 2024 11:31
@jonasfj jonasfj force-pushed the exported-api-draft-full-sync branch from 55e624a to e722e19 Compare November 7, 2024 13:45
@jonasfj jonasfj marked this pull request as ready for review November 7, 2024 13:54
@jonasfj jonasfj requested a review from isoos November 7, 2024 13:54
@jonasfj jonasfj changed the title Draft of what a full sync could look like Synchronization logic for exported API Nov 7, 2024
@jonasfj jonasfj force-pushed the exported-api-draft-full-sync branch from 44a6479 to 51bb50d Compare November 7, 2024 15:28
@jonasfj jonasfj merged commit 1cff64b into dart-lang:master Nov 7, 2024
32 checks passed
@jonasfj jonasfj deleted the exported-api-draft-full-sync branch November 7, 2024 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants