Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bucket config + initialization for API JSON exporter. #7238

Merged
merged 2 commits into from
Dec 6, 2023
Merged

Bucket config + initialization for API JSON exporter. #7238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d4f71d5
Bucket config + initialization for API JSON exporter.
isoos Dec 1, 2023
c176636
Updated config
isoos Dec 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions app/config/dartlang-pub-dev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ defaultServiceBaseUrl: https://{{GAE_VERSION}}-dot-{{GOOGLE_CLOUD_PROJECT}}.apps
dartdocStorageBucketName: dartlang-pub-dev--dartdoc-storage
popularityDumpBucketName: dartlang-pub-dev--popularity
searchSnapshotBucketName: dartlang-pub-dev--search-snapshot
exportedApiBucketName: null
maxTaskInstances: 50
maxTaskRunHours: 2
taskResultBucketName: dartlang-pub-dev-task-output
Expand Down
1 change: 1 addition & 0 deletions app/config/dartlang-pub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ defaultServiceBaseUrl: https://{{GAE_VERSION}}-dot-{{GOOGLE_CLOUD_PROJECT}}.apps
dartdocStorageBucketName: dartlang-pub--dartdoc-storage
popularityDumpBucketName: dartlang-pub--popularity
searchSnapshotBucketName: dartlang-pub--search-snapshot
exportedApiBucketName: null
maxTaskInstances: 700
maxTaskRunHours: 2
taskResultBucketName: dartlang-pub-task-output
Expand Down
8 changes: 8 additions & 0 deletions app/lib/package/export_api_to_bucket.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import 'dart:io';
import 'package:basics/basics.dart';
import 'package:clock/clock.dart';
import 'package:crypto/crypto.dart';
import 'package:gcloud/service_scope.dart' as ss;
import 'package:gcloud/storage.dart';
import 'package:logging/logging.dart';
import 'package:meta/meta.dart';
Expand Down Expand Up @@ -41,6 +42,13 @@ List<String> _apiPkgNameCompletitionDataNames() => [
'current/api/package-name-completion-data',
];

/// Sets the API Exporter service.
void registerApiExporter(ApiExporter value) =>
ss.register(#_apiExporter, value);

/// The active API Exporter service or null if it hasn't been initialized.
ApiExporter? get apiExporter => ss.lookup(#_apiExporter) as ApiExporter?;

class ApiExporter {
final Bucket _bucket;
final int _concurrency;
Expand Down
19 changes: 18 additions & 1 deletion app/lib/service/entrypoint/analyzer.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ import 'dart:async';
import 'package:args/command_runner.dart';
import 'package:gcloud/service_scope.dart';
import 'package:logging/logging.dart';
import 'package:pub_dev/package/export_api_to_bucket.dart';
import 'package:pub_dev/search/backend.dart';
import 'package:pub_dev/shared/configuration.dart';

import '../../analyzer/handlers.dart';
import '../../service/services.dart';
Expand Down Expand Up @@ -43,6 +45,15 @@ class AnalyzerCommand extends Command {
);
registerScopeExitCallback(indexBuilder.close);

if (activeConfiguration.exportedApiBucketName != null) {
final apiExporterIsolate = await startWorkerIsolate(
logger: logger,
entryPoint: _apiExporterMain,
kind: 'api-exporter',
);
registerScopeExitCallback(apiExporterIsolate.close);
}

await runHandler(logger, analyzerServiceHandler);
});
}
Expand All @@ -51,11 +62,11 @@ class AnalyzerCommand extends Command {
Future _workerMain(EntryMessage message) async {
message.protocolSendPort.send(ReadyMessage());

await popularityStorage.start();
await taskBackend.start();

setupAnalyzerPeriodicTasks();
setupSearchPeriodicTasks();
await popularityStorage.start();

// wait indefinitely
await Completer().future;
Expand All @@ -66,3 +77,9 @@ Future _indexBuilderMain(EntryMessage message) async {
await popularityStorage.start();
await searchBackend.updateSnapshotInForeverLoop();
}

Future _apiExporterMain(EntryMessage message) async {
message.protocolSendPort.send(ReadyMessage());
await popularityStorage.start();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why popularityStorage.start();`?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need it yet, but we may if/when we export endpoints with popularity scores. It felt better to include it so we won't forget about it later.

await apiExporter!.uploadInForeverLoop();
}
6 changes: 6 additions & 0 deletions app/lib/service/services.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import 'package:gcloud/service_scope.dart';
import 'package:gcloud/storage.dart';
import 'package:googleapis_auth/auth_io.dart' as auth;
import 'package:logging/logging.dart';
import 'package:pub_dev/package/export_api_to_bucket.dart';
import 'package:pub_dev/service/async_queue/async_queue.dart';
import 'package:pub_dev/service/security_advisories/backend.dart';
import 'package:shelf/shelf.dart' as shelf;
Expand Down Expand Up @@ -228,6 +229,11 @@ Future<R> _withPubServices<R>(FutureOr<R> Function() fn) async {
registerAccountBackend(AccountBackend(dbService));
registerAdminBackend(AdminBackend(dbService));
registerAnnouncementBackend(AnnouncementBackend());
if (activeConfiguration.exportedApiBucketName != null) {
registerApiExporter(ApiExporter(
bucket: storageService
.bucket(activeConfiguration.exportedApiBucketName!)));
}
registerAsyncQueue(AsyncQueue());
registerAuditBackend(AuditBackend(dbService));
registerConsentBackend(ConsentBackend(dbService));
Expand Down
7 changes: 7 additions & 0 deletions app/lib/shared/configuration.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,9 @@ class Configuration {
/// The name of the Cloud Storage bucket to use for generated reports.
final String? reportsBucketName;

/// The name of the Cloud Storage bucket to use for exporting JSON API responses.
final String? exportedApiBucketName;

/// The Cloud project Id. This is only required when using Apiary to access
/// Datastore and/or Cloud Storage
final String projectId;
Expand Down Expand Up @@ -275,6 +278,7 @@ class Configuration {
required this.dartdocStorageBucketName,
required this.popularityDumpBucketName,
required this.searchSnapshotBucketName,
required this.exportedApiBucketName,
required this.maxTaskInstances,
required this.maxTaskRunHours,
required this.taskResultBucketName,
Expand Down Expand Up @@ -341,6 +345,7 @@ class Configuration {
dartdocStorageBucketName: 'fake-bucket-dartdoc',
popularityDumpBucketName: 'fake-bucket-popularity',
searchSnapshotBucketName: 'fake-bucket-search',
exportedApiBucketName: 'fake-exported-apis',
maxTaskInstances: 10,
maxTaskRunHours: 2,
taskResultBucketName: 'fake-bucket-task-result',
Expand Down Expand Up @@ -392,6 +397,7 @@ class Configuration {
dartdocStorageBucketName: 'fake-bucket-dartdoc',
popularityDumpBucketName: 'fake-bucket-popularity',
searchSnapshotBucketName: 'fake-bucket-search',
exportedApiBucketName: 'fake-exported-apis',
taskResultBucketName: 'fake-bucket-task-result',
maxTaskInstances: 10,
maxTaskRunHours: 2,
Expand Down Expand Up @@ -442,6 +448,7 @@ class Configuration {
publicPackagesBucketName!,
searchSnapshotBucketName!,
taskResultBucketName!,
if (exportedApiBucketName != null) exportedApiBucketName!,
]);

late final isProduction = projectId == 'dartlang-pub';
Expand Down
4 changes: 4 additions & 0 deletions app/lib/shared/configuration.g.dart
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 15 additions & 0 deletions app/lib/tool/neat_task/pub_dev_tasks.dart
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import 'dart:io';
import 'package:gcloud/service_scope.dart' as ss;
import 'package:logging/logging.dart';
import 'package:neat_periodic_task/neat_periodic_task.dart';
import 'package:pub_dev/package/export_api_to_bucket.dart';

import '../../account/backend.dart';
import '../../account/consent_backend.dart';
Expand Down Expand Up @@ -114,6 +115,13 @@ void _setupGenericPeriodicTasks() {
task: updatePublicArchiveBucket,
);

// Exports the package name completetion data to a bucket.
_daily(
name: 'export-package-name-completition-data-to-bucket',
isRuntimeVersioned: true,
task: () async => await apiExporter?.uploadPkgNameCompletionData(),
);

// Deletes task status entities where the status hasn't been updated
// for more than a month.
_weekly(
Expand Down Expand Up @@ -150,6 +158,13 @@ void _setupGenericPeriodicTasks() {
task: taskBackend.garbageCollect,
);

// Deletes exported API data for old runtime versions
_weekly(
name: 'garbage-collect-api-exports',
isRuntimeVersioned: true,
task: () async => apiExporter?.deleteObsoleteRuntimeContent(),
);

// Delete very old instances that have been abandoned
_daily(
name: 'garbage-collect-old-instances',
Expand Down
1 change: 1 addition & 0 deletions app/test/shared/test_data/foo_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ popularityDumpBucketName: foo
searchSnapshotBucketName: foo
canonicalPackagesBucketName: foo
reportsBucketName: foo
exportedApiBucketName: foo
maxTaskInstances: 0
maxTaskRunHours: 2
searchServicePrefix: 'https://search-dot-dartlang-pub-dev.appspot.com'
Expand Down