Skip to content

feat: integration test batching with timing-based auto-balancing #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
nick-y-snyk wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

feat: integration test batching with timing-based auto-balancing #1132

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
25c377d
feat: add integration test batching with timing-based auto-balancing
nick-y-snyk Jan 28, 2026
308ee36
temp
nick-y-snyk Jan 28, 2026
98587c8
Merge branch 'main' into feat/integration-test-batching
nick-y-snyk Jan 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 102 additions & 26 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
split-index: [0, 1, 2, 3]
steps:
- name: Prepare git
run: |
Expand All @@ -104,53 +105,128 @@ jobs:
with:
go-version-file: 'go.mod'

- name: Download JUnit timing summary from main
uses: dawidd6/action-download-artifact@v6
continue-on-error: true
with:
name: junit-test-summary-${{ matrix.os }}
path: .
workflow: build.yaml
# branch: main
workflow_conclusion: success
if_no_artifact_found: warn

- name: Setup tools (`make tools`)
if: matrix.os != 'windows-latest'
uses: ./.github/actions/setup-tools

- name: Run integration & smoke tests with Pact
- name: Install gotestsum
run: go install gotest.tools/gotestsum@latest

- name: Split tests
id: test_split
uses: hashicorp-forge/go-test-split-action@v1
with:
total: 4
index: ${{ matrix.split-index }}
junit-summary: ./junit-test-summary-${{ matrix.os }}.xml

- name: Setup Xvfb and clipboard tools (Ubuntu)
if: matrix.os == 'ubuntu-latest'
env:
DEEPROXY_API_URL: ${{secrets.DEEPROXY_API_URL}}
SNYK_TOKEN: ${{secrets.SNYK_TOKEN }}
INTEG_TESTS: "true"
SMOKE_TESTS: "true"
run: |
export PATH=$PWD/.bin/pact/bin:$PATH

# this is required to be able to test the clipboard
export DISPLAY=:99
echo "DISPLAY=:99" >> $GITHUB_ENV
sudo Xvfb -ac :99 -screen 0 1280x1024x24 > /dev/null 2>&1 &
sudo apt-get install -y xsel xclip wl-clipboard

make clean test

- name: Run integration tests with Pact
- name: Setup Xvfb and display (macOS)
if: matrix.os == 'macos-latest'
env:
DEEPROXY_API_URL: ${{secrets.DEEPROXY_API_URL}}
SNYK_TOKEN: ${{secrets.SNYK_TOKEN }}
INTEG_TESTS: "true"
SMOKE_TESTS: "true"
run: |
export PATH=$PWD/.bin/pact/bin:$PATH

# this is required to be able to test the clipboard
export DISPLAY=:99
echo "DISPLAY=:99" >> $GITHUB_ENV
sudo Xvfb -ac :99 -screen 0 1280x1024x24 > /dev/null 2>&1 &

make clean test
- name: Add Pact to PATH
if: matrix.os != 'windows-latest'
run: echo "$PWD/.bin/pact/bin" >> $GITHUB_PATH

- name: Clean build artifacts
run: make clean

- name: Verify test environment variables
run: |
echo "INTEG_TESTS=${{ env.INTEG_TESTS || 'not set' }}"
echo "SMOKE_TESTS=${{ env.SMOKE_TESTS || 'not set' }}"

- name: Run integration tests without Pact
if: matrix.os == 'windows-latest'
- name: Run integration & smoke tests
env:
DEEPROXY_API_URL: ${{secrets.DEEPROXY_API_URL}}
SNYK_TOKEN: ${{secrets.SNYK_TOKEN }}
INTEG_TESTS: "true"
SMOKE_TESTS: "true"
run: |
make clean test
gotestsum --junitfile node-summary.xml -- \
-timeout=45m -failfast \
-run "${{ steps.test_split.outputs.run }}" \
./...

- name: Verify tests ran
shell: bash
run: |
# Verify the junit file exists and contains test results
if [ ! -f node-summary.xml ]; then
echo "ERROR: JUnit XML file not found - no tests ran!"
exit 1
fi
# Check that at least one test executed (look for testcase elements)
if ! grep -q '<testcase' node-summary.xml; then
echo "WARNING: No test cases found in split ${{ matrix.split-index }} for ${{ matrix.os }}"
echo "This may be expected if tests are unevenly distributed, but verify manually."
fi
# Verify we're running integration/smoke tests, not just unit tests
# Check for test names containing "Integ" or "Smoke" patterns
if grep -q '<testcase.*name=".*\(Integ\|Smoke\)' node-summary.xml; then
echo "✓ Confirmed: Integration/smoke tests are executing (env vars working)"
else
echo "INFO: No integration/smoke test patterns found in this split"
echo "This may be normal depending on test distribution"
fi

- name: Upload test summary
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-test-summary-${{ matrix.os }}-${{ matrix.split-index }}
path: node-summary.xml
retention-days: 1


combine-test-summaries:
name: Combine Test Summaries
needs: [integration-tests]
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/setup-node@v4
with:
node-version: 20

- name: Download artifacts
uses: actions/download-artifact@v4

- name: Install junit-report-merger
run: npm install -g junit-report-merger

- name: Merge reports
run: jrm ./junit-test-summary-${{ matrix.os }}.xml "junit-test-summary-${{ matrix.os }}-0/*.xml" "junit-test-summary-${{ matrix.os }}-1/*.xml" "junit-test-summary-${{ matrix.os }}-2/*.xml" "junit-test-summary-${{ matrix.os }}-3/*.xml"

- name: Upload combined test summary
uses: actions/upload-artifact@v4
with:
name: junit-test-summary-${{ matrix.os }}
path: ./junit-test-summary-${{ matrix.os }}.xml
retention-days: 90

proxy-test:
name: proxy-test
Expand Down
7 changes: 6 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,14 @@ TOOLS_BIN := $(shell pwd)/.bin
OVERRIDE_GOCI_LINT_V := v2.6.1
GOLICENSES_V := v1.6.0
PACT_V := 2.4.2
GOTESTSUM_V := latest
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Jan 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

The GOTESTSUM_V variable is set to latest, which causes the go install command to pull the most recent version of the tool every time the tools target is run. This introduces non-determinism into the build process and poses a supply chain risk. If the upstream gotestsum package is compromised, the malicious version will be automatically incorporated into the build environment. To ensure build stability and reproducibility, it is recommended to pin the version to a specific, verified tag, consistent with the other tools defined in this Makefile (e.g., OVERRIDE_GOCI_LINT_V, GOLICENSES_V).

GOTESTSUM_V := v1.12.0


TIMEOUT := "-timeout=45m"


## tools: Install required tooling.
.PHONY: tools
tools: $(TOOLS_BIN)/go-licenses $(TOOLS_BIN)/golangci-lint $(TOOLS_BIN)/pact/bin/pact
tools: $(TOOLS_BIN)/go-licenses $(TOOLS_BIN)/golangci-lint $(TOOLS_BIN)/pact/bin/pact $(TOOLS_BIN)/gotestsum

.PHONY: hooks
hooks:
Expand All @@ -55,6 +56,10 @@ $(TOOLS_BIN)/golangci-lint:
$(TOOLS_BIN)/pact/bin/pact:
cd $(TOOLS_BIN); curl -fsSL https://raw.githubusercontent.com/pact-foundation/pact-ruby-standalone/v$(PACT_V)/install.sh | PACT_CLI_VERSION=v$(PACT_V) bash

$(TOOLS_BIN)/gotestsum:
@echo "==> Installing gotestsum"
@GOBIN=$(TOOLS_BIN) go install gotest.tools/gotestsum@$(GOTESTSUM_V)

## clean: Delete the build directory
.PHONY: clean
clean:
Expand Down
Loading