Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADAP-1124: Establish shared workflows for the monorepo migration #362

Merged
merged 66 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
fbb1a88
create new batch of workflows
mikealfare Nov 22, 2024
378fe73
remove old workflows
mikealfare Nov 22, 2024
e71f0b3
update publish to work with dbt-adapters
mikealfare Nov 22, 2024
ff8fa55
clean up existing workflows
mikealfare Nov 22, 2024
607074e
update pull-request-checks to work with dbt-adapters
mikealfare Nov 22, 2024
f5dfafc
add pull_request target for checking workflows
mikealfare Nov 22, 2024
0ae769e
Merge branch 'main' into monorepo/shared-workflows
mikealfare Nov 22, 2024
07dd825
fix typo in _package-directory.yml
mikealfare Nov 22, 2024
997404f
remove env vars from inputs
mikealfare Nov 22, 2024
ea68191
add python version to unit test matrix
mikealfare Nov 22, 2024
e424dfc
register workflow with gha
mikealfare Nov 22, 2024
8ba75f7
revert defaults
mikealfare Nov 22, 2024
76a18a6
update workflow deps for publish.yml
mikealfare Nov 23, 2024
139bcf8
bump version just to test changelog generation
mikealfare Nov 23, 2024
d16ec97
fix temp branch name for changelog generation
mikealfare Nov 23, 2024
cdecc9d
revert version bump used for testing publish action
mikealfare Nov 23, 2024
7080796
revert pull_request trigger that was used for testing
mikealfare Nov 23, 2024
87f439a
add in changelog entry check
mikealfare Dec 5, 2024
6d1b194
Merge branch 'main' into monorepo/shared-workflows
mikealfare Dec 5, 2024
bcaaa09
rename files for git rebase
mikealfare Dec 6, 2024
c527fe9
create new batch of workflows
mikealfare Nov 22, 2024
f7586aa
remove old workflows
mikealfare Nov 22, 2024
ceadd71
update publish to work with dbt-adapters
mikealfare Nov 22, 2024
e57b1b2
clean up existing workflows
mikealfare Nov 22, 2024
44fa2e0
update pull-request-checks to work with dbt-adapters
mikealfare Nov 22, 2024
f997219
add pull_request target for checking workflows
mikealfare Nov 22, 2024
c3dc37f
fix typo in _package-directory.yml
mikealfare Nov 22, 2024
f7162c1
remove env vars from inputs
mikealfare Nov 22, 2024
4be4c46
add python version to unit test matrix
mikealfare Nov 22, 2024
2f5917e
register workflow with gha
mikealfare Nov 22, 2024
c8ec78a
revert defaults
mikealfare Nov 22, 2024
1aeafdf
update workflow deps for publish.yml
mikealfare Nov 23, 2024
5860f1d
bump version just to test changelog generation
mikealfare Nov 23, 2024
3771603
fix temp branch name for changelog generation
mikealfare Nov 23, 2024
3297a3c
revert version bump used for testing publish action
mikealfare Nov 23, 2024
740a8d6
revert pull_request trigger that was used for testing
mikealfare Nov 23, 2024
32eab3a
add in changelog entry check
mikealfare Dec 5, 2024
e044482
Merge remote-tracking branch 'origin/monorepo/shared-workflows' into …
mikealfare Dec 6, 2024
1142f3c
remove duplicate action
mikealfare Dec 6, 2024
6eb3a26
update changelog check to use the correct subdirectory
mikealfare Dec 6, 2024
e0fe7f7
add dispatch trigger to changelog check
mikealfare Dec 6, 2024
77083a2
update runner to reference the default runner env var
mikealfare Dec 6, 2024
b0dde3f
remove rebasing artifact
mikealfare Dec 6, 2024
96325a2
generalize changelog inputs
mikealfare Dec 7, 2024
d5981da
account for adapter subpackage
mikealfare Dec 7, 2024
67c803d
allow for running against a fork
mikealfare Dec 7, 2024
0bfa652
add run names and concurrency conditions to all workflows
mikealfare Dec 7, 2024
39b6ffc
pull stale workflow local to update labels
mikealfare Dec 7, 2024
849c6b8
update format to match other workflows
mikealfare Dec 7, 2024
0ecccb5
register atomic workflows by adding a pull_request trigger; to be rem…
mikealfare Dec 7, 2024
c468e54
register atomic workflows by adding a pull_request trigger; to be rem…
mikealfare Dec 7, 2024
febbb9c
fix version
mikealfare Dec 7, 2024
9f257a7
fix package directory outputs
mikealfare Dec 7, 2024
59c68b8
fix package directory outputs
mikealfare Dec 7, 2024
ba65a24
fix package directory outputs
mikealfare Dec 7, 2024
8217651
fix package directory input
mikealfare Dec 7, 2024
3dfa158
fix package directory
mikealfare Dec 7, 2024
a3bf5b8
remove workflow dispatch testing for changelog-entry-check since it i…
mikealfare Dec 7, 2024
68c8b57
add workflow test script for code quality
mikealfare Dec 7, 2024
c4249c5
add workflow test script for unit tests
mikealfare Dec 7, 2024
6a5062d
add workflow test script for verify build
mikealfare Dec 7, 2024
e5e2fed
bump version to test changelog generation
mikealfare Dec 7, 2024
8782e47
bump version to test changelog generation
mikealfare Dec 7, 2024
1d36766
fix version things for publishing
mikealfare Dec 7, 2024
325f534
remove internal release workaround for dbt-adapters since it is not c…
mikealfare Dec 7, 2024
33abd53
remove pull request trigger on pull request checks, this should only …
mikealfare Dec 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/workflows/_changelog-entry-check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: "Changelog entry check"

# this cannot be tested via workflow_dispatch
# dorny/paths-filter inspects the current trigger to determine how to compare branches
on:
workflow_call:
inputs:
package:
description: "Choose the package to test"
type: string
default: "dbt-adapters"
pull-request:
description: "The PR number"
type: string
required: true

permissions:
contents: read
pull-requests: write

jobs:
package:
uses: ./.github/workflows/_package-directory.yml
with:
package: ${{ inputs.package }}

changelog-check:
needs: package
if: ${{ !contains(github.event.pull_request.labels.*.name, 'Skip Changelog') }}
outputs:
exists: ${{ steps.changelog.outputs.exists }}
runs-on: ${{ vars.DEFAULT_RUNNER }}
steps:
- id: changelog
uses: dorny/paths-filter@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
filters: |
exists:
- added|modified: '${{ needs.package.outputs.directory }}.changes/unreleased/**.yaml'

comment:
needs: changelog-check
if: needs.changelog-check.outputs.exists == false
runs-on: ${{ vars.DEFAULT_RUNNER }}
env:
AUTHOR: "github-actions[bot]"
COMMENT: >-
Thank you for your pull request! We could not find a changelog entry for this change in the ${{ inputs.package }} package.
For details on how to document a change, see the [Contributing Guide](https://github.com/dbt-labs/dbt-adapters/blob/main/CONTRIBUTING.md).
steps:
- id: comment
uses: peter-evans/find-comment@v3
with:
issue-number: ${{ inputs.pull-request }}
comment-author: ${{ env.AUTHOR }}
body-includes: ${{ env.COMMENT }}
- if: steps.comment.outputs.comment-body == ''
run: gh issue comment ${{ inputs.pull-request }} --repo ${{ github.repository }} --body "${{ env.COMMENT }}"
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/github-script@v7
with:
script: core.setFailed('Changelog entry required to merge.')
39 changes: 39 additions & 0 deletions .github/workflows/_code-quality.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: "Code quality"

on:
workflow_call:
inputs:
branch:
description: "Choose the branch to check"
type: string
default: "main"
repository:
description: "Choose the repository to check, when using a fork"
type: string
default: "dbt-labs/dbt-adapters"
workflow_dispatch:
inputs:
branch:
description: "Choose the branch to check"
type: string
default: "main"
repository:
description: "Choose the repository to check, when using a fork"
type: string
default: "dbt-labs/dbt-adapters"

permissions:
contents: read

jobs:
code-quality:
runs-on: ${{ vars.DEFAULT_RUNNER }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.branch }}
repository: ${{ inputs.repository }}
- uses: actions/setup-python@v5
with:
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
- uses: pre-commit/action@v3.0.1
229 changes: 229 additions & 0 deletions .github/workflows/_generate-changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
name: "Changelog generation"

on:
workflow_call:
inputs:
package:
description: "Choose the package getting published"
type: string
default: "dbt-adapters"
merge:
description: "Choose whether to merge the changelog branch"
type: boolean
default: true
branch:
description: "Choose the branch to use"
type: string
default: "main"
outputs:
branch-name:
description: "The SHA to release"
value: ${{ jobs.branch.outputs.name }}
secrets:
FISHTOWN_BOT_PAT:
description: "Token to commit/merge changes into branches"
required: true
IT_TEAM_MEMBERSHIP:
description: "Token that can view org level teams"
required: true
workflow_dispatch:
inputs:
package:
description: "Choose the package getting published"
type: string
default: "dbt-adapters"
merge:
description: "Choose whether to merge the changelog branch"
type: boolean
default: false
branch:
description: "Choose the branch to use"
type: string
default: "main"
secrets:
FISHTOWN_BOT_PAT:
description: "Token to commit/merge changes into branches"
required: true
IT_TEAM_MEMBERSHIP:
description: "Token that can view org level teams"
required: true

permissions:
contents: write

defaults:
run:
shell: bash

jobs:
package:
uses: ./.github/workflows/_package-directory.yml
with:
package: ${{ inputs.package }}

version:
needs: package
runs-on: ${{ vars.DEFAULT_RUNNER }}
outputs:
raw: ${{ steps.version.outputs.raw }}
base: ${{ steps.semver.outputs.base-version }}
prerelease: ${{ steps.semver.outputs.pre-release }}
is-prerelease: ${{ steps.semver.outputs.is-pre-release }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.branch }}
- uses: actions/setup-python@v5
with:
python-version: ${{ vars.DEFAULT_PYTHON_VERSION }}
- uses: pypa/hatch@install
- id: version
run: echo "raw=$(hatch version)" >> $GITHUB_OUTPUT
working-directory: ./${{ needs.package.outputs.directory }}
- id: semver
uses: dbt-labs/actions/parse-semver@v1.1.1
with:
version: ${{ steps.version.outputs.raw }}

changelog:
needs: [package, version]
runs-on: ${{ vars.DEFAULT_RUNNER }}
outputs:
path: ${{ steps.changelog.outputs.path }}
exists: ${{ steps.changelog.outputs.exists }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.branch }}
- id: changelog
run: |
path=".changes/${{ needs.version.outputs.base }}"
if [[ ${{ needs.version.outputs.is-prerelease }} -eq 1 ]]
then
path+="-${{ needs.version.outputs.prerelease }}"
fi
path+=".md"

echo "path=$path" >> $GITHUB_OUTPUT

exists=false
if test -f $path
then
exists=true
fi
echo "exists=$exists">> $GITHUB_OUTPUT
working-directory: ./${{ needs.package.outputs.directory }}

temp-branch:
needs: [version, changelog]
if: needs.changelog.outputs.exists == 'false'
runs-on: ${{ vars.DEFAULT_RUNNER }}
outputs:
name: ${{ steps.branch.outputs.name }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.branch }}
- id: branch
run: |
name="prep-release/${{ inputs.package }}/$GITHUB_RUN_ID"
echo "name=$name" >> $GITHUB_OUTPUT
- run: |
git checkout -b ${{ steps.branch.outputs.name }}
git push -u origin ${{ steps.branch.outputs.name }}

dbt-membership:
needs: [version, changelog]
if: needs.changelog.outputs.exists == 'false'
runs-on: ${{ vars.DEFAULT_RUNNER }}
outputs:
team: ${{ steps.team.outputs.team }}
steps:
- id: temp-file
run: echo "name=output_$GITHUB_RUN_ID.json" >> $GITHUB_OUTPUT
- run: |
gh api -H "Accept: application/vnd.github+json" orgs/dbt-labs/teams/core-group/members > ${{ steps.temp-file.outputs.name }}
env:
GH_TOKEN: ${{ secrets.IT_TEAM_MEMBERSHIP }}
- id: team
run: |
team_list=$(jq -r '.[].login' ${{ steps.temp-file.outputs.name }})
team_list_single=$(echo $team_list | tr '\n' ' ')
echo "team=$team_list_single" >> $GITHUB_OUTPUT
- run: rm ${{ steps.temp-file.outputs.name }}

generate-changelog:
needs: [package, version, changelog, temp-branch, dbt-membership]
if: needs.changelog.outputs.exists == 'false'
runs-on: ${{ vars.DEFAULT_RUNNER }}
steps:
- uses: actions/checkout@v4
with:
ref: ${{ needs.temp-branch.outputs.name }}
- run: echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
- run: |
brew install pre-commit
brew tap miniscruff/changie https://github.com/miniscruff/changie
brew install changie
- run: |
if [[ ${{ needs.version.outputs.is-prerelease }} -eq 1 ]]
then
changie batch ${{ needs.version.outputs.base }} --move-dir '${{ needs.version.outputs.base }}' --prerelease ${{ needs.version.outputs.prerelease }}
elif [[ -d ".changes/${{ needs.version.outputs.base }}" ]]
then
changie batch ${{ needs.version.outputs.base }} --include '${{ needs.version.outputs.base }}' --remove-prereleases
else # releasing a final patch with no prereleases
changie batch ${{ needs.version.outputs.base }}
fi
changie merge
working-directory: ./${{ needs.package.outputs.directory }}
env:
CHANGIE_CORE_TEAM: ${{ needs.dbt-membership.outputs.team }}
- run: |
pre-commit run trailing-whitespace --files __version__.py CHANGELOG.md .changes/*
pre-commit run end-of-file-fixer --files __version__.py CHANGELOG.md .changes/*
working-directory: ./${{ needs.package.outputs.directory }}
continue-on-error: true
- run: |
git config user.name "Github Build Bot"
git config user.email "buildbot@fishtownanalytics.com"
git pull
git add .
git commit -m "generate changelog"
git push
working-directory: ./${{ needs.package.outputs.directory }}

merge-changes:
needs: [temp-branch, generate-changelog]
if: ${{ needs.temp-branch.outputs.name != '' && inputs.merge }}
runs-on: ${{ vars.DEFAULT_RUNNER }}
steps:
- uses: everlytic/branch-merge@1.1.5
with:
source_ref: ${{ needs.temp-branch.outputs.name }}
target_branch: ${{ inputs.branch }}
github_token: ${{ secrets.FISHTOWN_BOT_PAT }}
commit_message_template: "[Automated] Merged {source_ref} into target {target_branch} during release process"
- run: git push origin -d ${{ needs.temp-branch.outputs.name }}

branch:
needs: [temp-branch, merge-changes]
if: ${{ !failure() && !cancelled() }}
runs-on: ${{ vars.DEFAULT_RUNNER }}
# always run this job, regardless of whether changelog generation was skipped
# Get the sha that will be released. If the changelog already exists on the input sha and the version has already been bumped,
# then it is what we will release. Otherwise, we generated a changelog and did the version bump in this workflow and there is a
# new sha to use from the merge we just did. Grab that here instead.
outputs:
name: ${{ steps.branch.outputs.name }}
steps:
- id: branch
run: |
branch=""
if [ ${{ needs.temp-branch.outputs.name == '' || inputs.merge }}]
then
branch="${{ inputs.branch }}"
else
branch=${{ needs.temp-branch.outputs.name }}
fi
echo "name=$branch" >> $GITHUB_OUTPUT
33 changes: 33 additions & 0 deletions .github/workflows/_package-directory.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: "Package directory"

on:
workflow_call:
inputs:
package:
description: "Choose the package whose directory you need"
type: string
default: "dbt-adapters"
outputs:
directory:
description: "The root directory of the package"
value: ${{ jobs.package.outputs.directory }}

defaults:
run:
shell: bash

jobs:
package:
runs-on: ${{ vars.DEFAULT_RUNNER }}
outputs:
directory: ${{ steps.package.outputs.directory }}
steps:
- id: package
run: |
if [[ ${{ inputs.package }} == "dbt-adapters" ]]
then
directory=""
else
directory="${{ inputs.package }}/"
fi
echo "directory=$directory" >> $GITHUB_OUTPUT
Loading
Loading