Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: optimize setup-with-retry CI to run faster #32338

Closed
wants to merge 44 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
83e45f1
optimize docker
NripeshN May 3, 2024
262c343
cahcing
NripeshN May 3, 2024
1db7a58
revert action
NripeshN May 3, 2024
74dfd25
revert
NripeshN May 3, 2024
f838618
docker multistage
NripeshN May 3, 2024
4844f34
test
NripeshN May 3, 2024
5110720
revert
NripeshN May 3, 2024
e3438e4
ditch docker
NripeshN May 3, 2024
e7df497
add shell
NripeshN May 3, 2024
093e76d
3.11
NripeshN May 3, 2024
62ce2d7
use 3.11
NripeshN May 3, 2024
3a87926
install 3.11
NripeshN May 3, 2024
c57ed4d
Update Python version to 3.10 and numpy version to >=1.23.5 in poetry…
NripeshN May 3, 2024
4a90a63
remove python 3.11
NripeshN May 3, 2024
ee66a9a
change to python 3.10
NripeshN May 3, 2024
aa7cc22
update
NripeshN May 3, 2024
1d17ef2
try deleting pip installation
NripeshN May 3, 2024
af37ba3
Update runs-on to ubuntu-20.04 in .github/workflows/setup/action.yaml
NripeshN May 3, 2024
620d5ec
revert ubuntu
NripeshN May 3, 2024
549d891
Upgrade pip and install poetry in setup action.yaml
NripeshN May 3, 2024
06dd3fa
Update poetry installation in setup action.yaml
NripeshN May 3, 2024
434162e
Update actions/cache version to v4 in .github/workflows/setup/action.…
NripeshN May 3, 2024
fb7f769
Refactor setup workflow to include retry logic and sleep between retries
NripeshN May 3, 2024
6dda223
Refactor setup workflow to include retry logic and sleep between retries
NripeshN May 3, 2024
9b2778c
revert
NripeshN May 3, 2024
b023a34
format
NripeshN May 3, 2024
b926592
install portaudio
NripeshN May 3, 2024
25eca79
test cache
NripeshN May 3, 2024
fc11716
format
NripeshN May 3, 2024
1c4257e
update docs
NripeshN May 3, 2024
109e3e2
install qmake
NripeshN May 3, 2024
614620d
update
NripeshN May 3, 2024
7e4aaf4
update
NripeshN May 3, 2024
5635171
gcc install
NripeshN May 3, 2024
b81c103
capnproto
NripeshN May 3, 2024
48f2189
dev
NripeshN May 3, 2024
5cf7eb0
zeromq
NripeshN May 3, 2024
ba53b63
openCL
NripeshN May 3, 2024
0887839
Add git lfs install command to build openpilot and install git-lfs in…
NripeshN May 3, 2024
0611056
pull lfs
NripeshN May 3, 2024
93e8d5f
Update setup action to include libeigen3-dev in system dependencies
NripeshN May 3, 2024
7590a40
Update system dependencies in setup action
NripeshN May 3, 2024
621972b
Update system dependencies in setup action to include libusb-1.0-0-dev
NripeshN May 3, 2024
720199a
Add enum34 to Python dependencies in setup action
NripeshN May 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cahcing
  • Loading branch information
NripeshN committed May 3, 2024
commit 262c343ebf19784d2e68ff5c393f6588db06051f
106 changes: 55 additions & 51 deletions .github/workflows/setup/action.yaml
Original file line number Diff line number Diff line change
@@ -10,54 +10,58 @@ inputs:
required: false
default: 'false'

runs:
using: "composite"
steps:
# assert that this action is retried using the setup-with-retry
- shell: bash
if: ${{ inputs.is_retried == 'false' }}
run: |
echo "You should not run this action directly. Use setup-with-retry instead"
exit 1

# do this after checkout to ensure our custom LFS config is used to pull from GitLab
- shell: bash
run: git lfs pull

# on BuildJet runners, must be logged into DockerHub to avoid rate limiting
# https://buildjet.com/for-github-actions/docs/guides/docker
- shell: bash
if: ${{ contains(runner.name, 'buildjet') && inputs.docker_hub_pat == '' }}
run: |
echo "Need to set the Docker Hub PAT secret as an input to this action"
exit 1
- name: Login to Docker Hub
if: contains(runner.name, 'buildjet')
shell: bash
run: |
docker login -u adeebshihadeh -p ${{ inputs.docker_hub_pat }}

# build cache
- id: date
shell: bash
run: echo "CACHE_COMMIT_DATE=$(git log -1 --pretty='format:%cd' --date=format:'%Y-%m-%d-%H:%M')" >> $GITHUB_ENV
- shell: bash
run: echo "$CACHE_COMMIT_DATE"
- id: scons-cache
uses: ./.github/workflows/auto-cache
with:
path: .ci_cache/scons_cache
key: scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
scons-${{ runner.arch }}-${{ env.CACHE_COMMIT_DATE }}
scons-${{ runner.arch }}
# as suggested here: https://github.com/moby/moby/issues/32816#issuecomment-910030001
- id: normalize-file-permissions
shell: bash
name: Normalize file permissions to ensure a consistent docker build cache
run: |
find . -type f -executable -not -perm 755 -exec chmod 755 {} \;
find . -type f -not -executable -not -perm 644 -exec chmod 644 {} \;
# build our docker image
- shell: bash
run: eval ${{ env.BUILD }}
runs:
using: "composite"
steps:
# Verify if the setup is retried using setup-with-retry
- name: Verify Retry Setup
shell: bash
if: ${{ inputs.is_retried == 'false' }}
run: |
echo "This action should not run directly. Use setup-with-retry instead."
exit 1

# Perform git lfs pull if necessary, ensure condition checks for files existence
- name: Git LFS Pull
shell: bash
run: git lfs pull
if: ${{ github.event_name == 'push' || github.event_name == 'pull_request' }}

# Login to Docker Hub to avoid rate limiting on BuildJet
- name: Login to Docker Hub
shell: bash
if: ${{ contains(runner.name, 'buildjet') }}
run: |
if [ -z "${{ inputs.docker_hub_pat }}" ]; then
echo "Docker Hub PAT is required."
exit 1
fi
docker login -u adeebshihadeh -p ${{ inputs.docker_hub_pat }}

# Set up caching for faster builds, using more specific keys and restoration logic
- name: Set Cache Dates
shell: bash
run: echo "CACHE_COMMIT_DATE=$(date +%Y-%m-%d-%H:%M)" >> $GITHUB_ENV

- name: Restore Build Cache
uses: actions/cache@v2
with:
path: .ci_cache/scons_cache
key: ${{ runner.os }}-scons-${{ env.CACHE_COMMIT_DATE }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-scons-${{ env.CACHE_COMMIT_DATE }}
${{ runner.os }}-scons-

# Normalize file permissions to ensure a consistent Docker build cache
- name: Normalize File Permissions
shell: bash
run: |
find . -type f -executable -not -perm 755 -exec chmod 755 {} \;
find . -type f -not -executable -not -perm 644 -exec chmod 644 {} \;

# Build Docker image
- name: Build Docker Image
shell: bash
run: eval ${{ env.BUILD }}
env:
BUILD: docker build -t openpilot-env .
7 changes: 3 additions & 4 deletions Dockerfile.openpilot_base
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# Use a multi-stage build to keep the final image clean and minimal
# Base stage for installations
FROM ubuntu:20.04 as builder

@@ -24,12 +23,12 @@ ARG INTEL_DRIVER_URL=https://registrationcenter-download.intel.com/akdlm/irc_nas

RUN mkdir /tmp/opencl-driver-intel \
&& cd /tmp/opencl-driver-intel \
&& curl -O $INTEL_DRIVER_URL/$INTEL_DRIVER \
&& curl -fsSL $INTEL_DRIVER_URL/$INTEL_DRIVER -o $INTEL_DRIVER \
&& tar -xzf $INTEL_DRIVER \
&& for i in $(basename $INTEL_DRIVER .tgz)/rpm/*.rpm; do alien --to-deb $i; done \
&& find . -name '*.rpm' -exec alien --to-deb {} \; \
&& dpkg -i *.deb \
&& echo /opt/intel/opencl_compilers_and_libraries_18.1.0.015/linux/compiler/lib/intel64_lin/libintelocl.so > /etc/OpenCL/vendors/intel.icd \
&& rm -rf $INTEL_DRIVER $(basename $INTEL_DRIVER .tgz) *.deb /tmp/opencl-driver-intel
&& rm -rf /tmp/opencl-driver-intel $(basename $INTEL_DRIVER .tgz) *.deb

# Final stage to reduce size
FROM ubuntu:20.04
Loading
Oops, something went wrong.