Whisper and GCP Compute Runners

CouncilDataProject cdp-backend v4.0.0

⚠️ ⚠️ This is a major breaking release. Instance maintainers should update the instance with just update-from-cookiecutter. ⚠️ ⚠️

You should re-read through the SETUP/README.md document as there is some new minor configuration required. Specifically the new PERSONAL_ACCESS_TOKEN and Quote Increase request should be the only things that need to be updated for existing instances.

You should also lower how often your CRON event gather runs prior to running just update-from-cookiecutter. All of the instances maintained by the CDP Core Team will be lowered to running only once per day.

---

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment ties together multiple projects to make a single deployable infrastructure.

v4.0.0

There are two main changes for this release.

1. We are swapping out Google Speech-to-Text for OpenAIs Whisper.

Specifically, we are using a forked version called faster-whisper. This new speech-to-text model performs much better (ranging from ~3.6% word-error-rate to ~9% word-error-rate on long audio files).

To use this new model efficiently, we need access to a GPU. Since GitHub Actions do not have GPUs available, we are using a system which spins up a Google Cloud Compute Engine instance, connects to it, runs our job, and then tears it down all in the course of a single GitHub Action workflow. From multiple tests, this should be a reduction in cost and processing time however with this release we will do more testing to get a better estimate.

2. We have switched from MIT to MPLv2 License.

Unless you are trying to fork our code and take it private, this won't affect you.

Parametrizable Event Gather CRON and Default Timedelta

Council Data Project v3.2.1

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment is a cookiecutter template to deploy a whole new CDP infrastructure.

There are no new library changes from cdp-backend or cdp-frontend this release. This is a release that simply adds new parameters to the cookiecutter to make it easier to randomize CRON event gather schedules (via deployment bot), or provide a specific CRON from cookiecutter CLI, as well as set the default event gather timedelta lookback (how many days are checked for events each time the pipeline runs).

Those changes mean that the template also needed to be updated to utilize the new infrastructure deployment strategy.

⚠️ ⚠️

This release is a "breaking" change for instances.

Instance maintainers will want to do the following:

• Add a value to the .cookiecutter.yaml file called: event_gather_timedelta_lookback_days with an integer value for the number of days to check during the event gather. The default (and current value prior to this update to make it parametrizable) is 2 for two days. To decide what this parameter should be, a general rule of thumb is: "how long does it take for the municipality to post events"
• Add a value to the .cookiecutter.yaml file called: event_gather_cron with a CRON string for how often to run the event gather pipeline. The default (and current value prior to this update to make it parametrizable) is "26 0,6,12,18 * * *". Which means: "run the pipeline at 12:26am, 6:26am, 12:26pm and 6:26pm UTC everyday". A general rule of thumb is: "if the event gather timedelta is large, the pipeline doesn't need to run as often."
• Run just update-from-cookiecutter or make update-from-cookiecutter

⚠️ ⚠️

Full Changelog: v3.2.0...v3.2.1

New Infrastructure Management and Admin Updates

Council Data Project v3.2.0

Council Data Project is a backend, frontend, and cookiecutter deployment for creating a whole database, storage system, and website, for archiving, exploring, and tracking municipal council action.

This library, cookiecutter-cdp-deployment is a cookiecutter template to deploy a whole new CDP infrastructure.

The recent changes to cdp-backend is meant to fix the currently broken infrastructure deployment system. We are ripping out all of Pulumi and using gcloud, gsutil, and firebase CLI tooling to achieve the same result (which are all first-party tools so should be better supported).

Those changes mean that the template also needed to be updated to utilize the new infrastructure deployment strategy.

⚠️ ⚠️
This release is a "breaking" change for instances. Instance maintainers will want to update the instance with make update-from-cookiecutter.
⚠️ ⚠️

What's Changed

• bugfix/infra-stack by @evamaxfield in #111

Full Changelog: v3.1.1...v3.2.0

New Indexing Workflow

This change utilizes the new indexing pipelines for events! Instance maintainers will need to run make update-from-cookiecutter to upgrade.

What's Changed

• feature/split-indexing-into-multi-step-workflow by @JacksonMaxfield in #108

Full Changelog: v3.1.0...v3.1.1

Legislation Tracking and Easier Deployment Bots!

The main reason for this version bump is due to https://github.com/CouncilDataProject/cdp-frontend/releases/tag/v3.1.0
Which introduces quite a lot but mainly the legislation tracking page!

There is a breaking change introduced in this release but it is minor and should be easily handlable. See the release notes in cdp-frontend linked above for details.

There are also many changes to this repo itself but they are largely administrative. Specifically the GitHub Action deployment bots now require timezone and many bugs have been fixed.

Full Changelog: v3.0.6...v3.1.0

More Docs and Frontend Optimizations

This release brings some better docs to the cookiecutter from Smai! Thanks Smai!

Additionally, this brings a lot of frontend optimizations but with those optimizations comes a requirement on instance maintainers to rerun the cookiecutter. The main culprit here is that we have upgraded the frontend from node v12 to node v16.

Instead of listening to dependabot, please run make update-from-cookiecutter to pull in these changes.

What's Changed

• docs/new-deployment-clarifications-around-event-gathering-and-cost by @smai-f in #81
• admin/upgrade-deps-and-build-eng by @JacksonMaxfield in #84

New Contributors

• @smai-f made their first contribution in #81

Full Changelog: v3.0.5...v3.0.6

Process YouTube Videos, Embed YouTube URLs for Video Source, and Paginate Voting Records

Contains minimum version bumps for fresh CDP instance to backend v3.0.9 and frontend v3.0.7 which contain features for handling YouTube video URLs and on the frontend pagination of voting records due to simply having more data.

Some additional bug fixes in this repo (related to GitHub Issue Bot) and in both libraries (Index Creation, Localization, etc.) are additionally added in these updates.

Person Page and Database Patches for Programmatic UX

See changelog for backend here: https://github.com/CouncilDataProject/cdp-backend/releases/tag/v3.0.4

See changelog for frontend here: https://github.com/CouncilDataProject/cdp-frontend/releases/tag/v3.0.4

There were also changes to the deployment instance cookiecutter that document how to rollout the backend changes. These are optional but recommended.

Full Cookiecutter Changelog: v3.0.3...v3.0.4

GSR Improvements and Frontend Bugfixes

v3.0.3

Update cdp lib version to 3.0.3

Event Gather Pipeline Fix and Web App Build Fix

Event Gather Pipeline bugfix addresses a bug found when trying to download a picture for person or seat and the download being rejected by the host. Web app build fix solves the whole web app build and deploy due to upstream deps / warnings from react-scripts.