Skip to content

e2e tests on Cirrus Labs runners #5485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alwx wants to merge 30 commits into
base: main
Choose a base branch
from
Open

e2e tests on Cirrus Labs runners #5485

Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e2397d0
e2e tests on Cirrus Labs runners
alwx Dec 18, 2025
a04f351
Checking another group
alwx Dec 18, 2025
83a3d6a
rolling back last change
alwx Dec 18, 2025
1cb4e72
Merge branch 'main' into alwx/experiment/cirrus-labs
alwx Dec 23, 2025
bd991eb
Merge branch 'main' into alwx/experiment/cirrus-labs
alwx Jan 26, 2026
a933ded
Corepack enable
alwx Jan 28, 2026
9899e3d
Corepack enable
alwx Jan 28, 2026
61211d5
Ninja
alwx Jan 28, 2026
b5c48c1
Ruby install
alwx Jan 28, 2026
9142e2f
Ruby install
alwx Jan 28, 2026
da9811a
Updated env params
alwx Jan 29, 2026
3034d58
Trying to fix metrics job
alwx Jan 29, 2026
8c3a337
Merge branch 'main' into alwx/experiment/cirrus-labs
alwx Jan 29, 2026
550e671
Trying it out with Macos 26.0.1
alwx Jan 29, 2026
064ae98
iOS version env param change
alwx Jan 29, 2026
78bf149
Debug outputs
alwx Jan 29, 2026
92b254f
iDB companion update to potentially fix things on iOS side
alwx Jan 29, 2026
bd23cc4
Another attempt
alwx Jan 29, 2026
7da4011
Typo
alwx Jan 29, 2026
4c7db57
GH runners for tests
alwx Jan 29, 2026
d9d34c0
e2e fix attempt (with a temp workaround of using GH runners for tests)
alwx Jan 29, 2026
0c1d899
Rolling back some of experiments
alwx Jan 29, 2026
3ca98e6
Solving concurrency issues
alwx Jan 29, 2026
a685372
More GH runner issues
alwx Jan 29, 2026
6a25aaa
Merge branch 'main' into alwx/experiment/cirrus-labs
alwx Jan 29, 2026
cbd93fc
Merge branch 'main' into alwx/experiment/cirrus-labs
alwx Jan 30, 2026
ebb8321
Small fix
alwx Jan 30, 2026
bf21537
Fix-2
alwx Jan 30, 2026
a0aadff
Fix-3
alwx Jan 30, 2026
94cbad2
Hopefully final fix
alwx Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 43 additions & 24 deletions .github/workflows/e2e-v2.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,17 +42,18 @@ jobs:
strategy:
# we want that the matrix keeps running, default is to cancel them if it fails.
fail-fast: false
# Limit parallel jobs to avoid hitting Sauce Labs concurrency limits (max 3 sessions)
max-parallel: 2
matrix:
rn-architecture: ['legacy', 'new']
platform: ["ios", "android"]
include:
- platform: ios
runs-on: macos-26
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2.0", "runner_group_id:12"]
name: iOS
appPlain: performance-tests/test-app-plain.ipa
- platform: android
# Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
runs-on: ubuntu-22.04
runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
name: Android
appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
steps:
Expand All @@ -61,17 +62,18 @@ jobs:
- uses: ./.github/actions/disk-cleanup
if: ${{ matrix.platform == 'android' }}

- run: sudo xcode-select -s /Applications/Xcode_26.1.1.app/Contents/Developer
if: ${{ matrix.platform == 'ios' }}

- run: npm i -g corepack
- run: corepack enable
- uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6
with:
package-manager-cache: false
node-version: 20
cache: 'yarn'
cache-dependency-path: yarn.lock

- name: Install Ninja
if: ${{ matrix.platform == 'android' }}
run: sudo apt-get update && sudo apt-get install -y ninja-build

- uses: actions/setup-java@v5
with:
java-version: '17'
Expand All @@ -80,6 +82,11 @@ jobs:
- name: Gradle cache
uses: gradle/gradle-build-action@v3

- uses: ruby/setup-ruby@v1
if: ${{ matrix.platform == 'ios' }}
with:
ruby-version: '3.3.0'

- name: Install Global Dependencies
run: npm i -g react-native-cli @sentry/cli

Expand Down Expand Up @@ -115,9 +122,11 @@ jobs:
if [[ ${{ matrix.rn-architecture }} == 'new' ]]; then
export RCT_NEW_ARCH_ENABLED=1
fi
pod install
bundle install
bundle exec pod install
Comment on lines +125 to +126
Copy link

@sentry sentry bot Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The bundle install command in the iOS build step will fail because it is run from the ios subdirectory, which does not contain a Gemfile.
Severity: HIGH

Suggested Fix

Move the bundle install command so it executes in the correct directory (./performance-tests/TestAppPlain) before the script changes directory into the ios subdirectory. This will ensure dependencies are installed before pod install is called.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: .github/workflows/e2e-v2.yml#L125-L126

Potential issue: In the `e2e-v2.yml` workflow, the `Build app plain` and `Build app with
Sentry` jobs for the iOS platform will fail. The script changes the directory to `ios`
before running `bundle install`. However, the `Gemfile` is located in the parent
directory (`./performance-tests/TestAppPlain`), not in the `ios` subdirectory. Since
`bundle install` does not search parent directories for a `Gemfile` by default, the
command will fail with a "Gemfile not found" error, which in turn causes the `pod
install` command to fail and breaks the entire iOS build process within the CI pipeline.

cd ../..
fastlane build_perf_test_app_plain
bundle install
bundle exec fastlane build_perf_test_app_plain
fi
env:
APP_STORE_CONNECT_KEY_ID: ${{ secrets.APP_STORE_CONNECT_KEY_ID }}
Expand All @@ -143,9 +152,11 @@ jobs:
if [[ ${{ matrix.rn-architecture }} == 'new' ]]; then
export RCT_NEW_ARCH_ENABLED=1
fi
pod install
bundle install
bundle exec pod install
cd ../..
fastlane build_perf_test_app_sentry
bundle install
bundle exec fastlane build_perf_test_app_sentry
cd TestAppSentry
fi
env:
Expand Down Expand Up @@ -189,15 +200,13 @@ jobs:
# Use Xcode 16 for older RN versions
- platform: ios
rn-version: '0.71.19'
xcode-version: '16.4'
runs-on: macos-15
runs-on: ["ghcr.io/cirruslabs/macos-sequoia-xcode:16.4", "runner_group_id:12"]
# Use Xcode 26 for newer RN versions (0.83.0)
- platform: ios
rn-version: '0.83.0'
xcode-version: '26.1.1'
runs-on: macos-26
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2.0", "runner_group_id:12"]
- platform: android
runs-on: ubuntu-latest
runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
exclude:
# exclude JSC for new RN versions (keeping the matrix manageable)
- rn-version: '0.83.0'
Expand Down Expand Up @@ -247,17 +256,18 @@ jobs:
echo "SENTRY_RELEASE=$SENTRY_RELEASE"
echo "SENTRY_DIST=$SENTRY_DIST"

- run: sudo xcode-select -s /Applications/Xcode_${{ matrix.xcode-version }}.app/Contents/Developer
if: ${{ matrix.platform == 'ios' }}

- run: npm i -g corepack
- run: corepack enable
- uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6
with:
package-manager-cache: false
node-version: 20
cache: 'yarn'
cache-dependency-path: yarn.lock

- name: Install Ninja
if: ${{ matrix.platform == 'android' }}
run: sudo apt-get update && sudo apt-get install -y ninja-build

- uses: actions/setup-java@v5
with:
java-version: '17'
Expand Down Expand Up @@ -323,9 +333,9 @@ jobs:
include:
- platform: ios
rn-version: '0.83.0'
runs-on: macos-26
runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2.0", "runner_group_id:12"]
- platform: android
runs-on: ubuntu-latest
runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]

steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
Expand Down Expand Up @@ -354,14 +364,19 @@ jobs:
path: dev-packages/e2e-tests

- name: Enable Corepack
run: npm i -g corepack
run: corepack enable

- uses: actions/setup-node@6044e13b5dc448c55e2357c09f80417699197238 # v6
with:
package-manager-cache: false
node-version: 20
cache: 'yarn'
cache-dependency-path: yarn.lock

- name: Install Ninja
if: ${{ matrix.platform == 'android' }}
run: sudo apt-get update && sudo apt-get install -y ninja-build

- uses: actions/setup-java@v5
with:
java-version: '17'
Expand Down Expand Up @@ -417,11 +432,15 @@ jobs:

- name: Run tests on iOS
if: ${{ matrix.platform == 'ios' }}
env:
# Increase timeout for Maestro iOS driver startup (default is 60s, some CI runners need more time)
MAESTRO_DRIVER_STARTUP_TIMEOUT: 120000
run: ./dev-packages/e2e-tests/cli.mjs ${{ matrix.platform }} --test

- name: Upload logs
if: ${{ always() }}
uses: actions/upload-artifact@v6
with:
name: ${{ matrix.rn-version }}-${{ matrix.rn-architecture }}-${{ matrix.engine }}-${{ matrix.platform }}-${{ matrix.build-type }}-${{ matrix.ios-use-frameworks }}-logs
path: ./dev-packages/e2e-tests/maestro-logs
path: |
./dev-packages/e2e-tests/maestro-logs
Loading