[ML] Add serverless MKI testing for ml-cpp PR builds#3027
Open
edsavage wants to merge 2 commits intoelastic:mainfrom
Open
[ML] Add serverless MKI testing for ml-cpp PR builds#3027edsavage wants to merge 2 commits intoelastic:mainfrom
edsavage wants to merge 2 commits intoelastic:mainfrom
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
edsavage
force-pushed
the
feature/serverless-pr-testing
branch
4 times, most recently
from
d089817 to
86a5bb1
Compare
Add pipeline and scripts to build a serverless ES Docker image with custom ml-cpp artifacts and run E2E tests against MKI QA. Triggered via PR comment (`buildkite run_serverless_tests`) or label (`ci:run-serverless-tests`). Builds Linux x86_64 ml-cpp, sets up a local Ivy repo, clones elasticsearch-serverless, builds the Docker image with -Dbuild.ml_cpp.repo override, pushes to the CI registry, and triggers the E2E QA pipeline. Relates elastic/ml-team#1243 Made-with: Cursor
edsavage
force-pushed
the
feature/serverless-pr-testing
branch
from
86a5bb1 to
0c5b4ea
Compare
|
Pinging @elastic/ml-core (Team:ML) |
edsavage
force-pushed
the
feature/serverless-pr-testing
branch
4 times, most recently
from
e493cb4 to
59cc65e
Compare
Instead of cloning elasticsearch-serverless from the ml-cpp pipeline (which requires cross-repo GitHub token access), upload the custom ml-cpp artifacts to an S3 staging path under the existing prelert-artifacts bucket, then trigger the serverless validation pipeline which runs in its own context with full permissions. The serverless pipeline picks up the custom artifacts via the ML_CPP_REPO_OVERRIDE env var, which is passed through as -Dbuild.ml_cpp.repo to the Gradle build. This requires a small change in elasticsearch-serverless (documented in docs/serverless-integration/). Made-with: Cursor
edsavage
force-pushed
the
feature/serverless-pr-testing
branch
from
59cc65e to
df41985
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the ability to build a serverless Elasticsearch Docker image with custom ml-cpp artifacts from a PR build and run E2E tests against MKI QA. This automates the manual process documented in elastic/ml-team#1243.
How to trigger
PR comment:
PR label:
ci:run-serverless-testsWhat it does
./gradlew buildDockerImage -Dbuild.ml_cpp.repo=<local-ivy>to inject the custom ml-cppdocker.elastic.co/elasticsearch-ci/elasticsearch-serverlesselasticsearch-serverless-e2e-tests-qafor both new and upgraded project scenariosFiles
.buildkite/pipelines/run_serverless_tests.yml.sh— pipeline definition chaining Docker build → E2E tests.buildkite/scripts/steps/build_serverless_docker.sh— core script: Ivy repo setup, serverless Docker build, push.buildkite/ml_pipeline/config.py—run_serverless_testsflag with label/comment parsing.buildkite/pipeline.json.py— wires serverless step into the PR pipeline.buildkite/pull-requests.json— addsrun_serverless_teststo trigger comment regexPrerequisites verified
secret/ci/elastic-elasticsearch-serverless/migrated/es-license(license key) — accessiblesecret/ci/elastic-elasticsearch-serverless/prod_docker_registry_credentials(Docker push) — accessibleelasticsearch-serverless-e2e-tests-qahastrigger_mode: nonewith no restrictions, same pattern as existingappex-qa-stateful-custom-ml-cpp-build-testingRelates elastic/ml-team#1243
Made with Cursor