DRIVERS-3564: bootstrap Python with uv in primary scripts - #808
Merged
Conversation
Adds ensure-uv.sh and a root .python-version pin, and switches setup.sh, install-cli.sh, download-mongodb.sh, and several test/helper scripts over to uv-managed Python instead of scanning the filesystem for a suitable interpreter. Lays the groundwork for the uv workspace consolidation that follows.
pip --user installs console scripts to a platform/version-specific directory (e.g. the Library Python bin dir on macOS), not the Linux default. Ask the interpreter via site --user-base instead of assuming.
pip install --user fails on some Evergreen hosts (old RHEL7/rhel8-zseries pip with no matching uv wheel, and PEP 668 externally-managed-environment blocks on newer Ubuntu/OIDC hosts). Fall back to the official install script, which fetches a prebuilt binary directly and sidesteps both issues.
Avoid fetching the standalone installer via curl on hosts. Instead, set PIP_BREAK_SYSTEM_PACKAGES=1 to bypass PEP 668's externally-managed-environment guard on Debian/Ubuntu, and fall back to upgrading pip itself (then retrying) on hosts like RHEL7 whose pip is too old to resolve a matching uv wheel.
RHEL7 hosts have no python3 on PATH at all (only an ancient Python 2 python), so pip could never resolve uv regardless of pip version; fall back to the MongoDB toolchain's python3 in that case. Separately, the deadsnakes PPA python3 used in the docker test images has no pip module at all; bootstrap it via the stdlib ensurepip (no network needed) before trying pip install.
RHEL8 zseries/power8 hosts have pyenv installed, whose shims intercept python/python3 and enforce the repo's .python-version file, failing outright since that exact version isn't pyenv-installed there. Export PYENV_VERSION=system to make pyenv defer to the real system interpreter. Also fix a latent set -e bug: the last command in an A || B chain is not exempt from errexit, so the ensurepip fallback needs an explicit || true.
Forcing PYENV_VERSION=system broke rhel8-power8, where uv was already installed under pyenv's own configured global version (3.10.14), not system. Defer to 'pyenv global' instead. Also switch the uv-availability checks from 'command -v uv' (which only checks that a pyenv shim file exists, not that it resolves under the active version) to 'uv --version' actually invoking it.
…d builds rhel8-zseries (s390x) hosts fail SSL certificate verification in mongodl.py and mongodb-runner: uv's managed Python build and Node don't discover this host's CA trust store by default. Detect the system CA bundle in ensure_uv and export SSL_CERT_FILE (Python) / NODE_EXTRA_CA_CERTS (Node), persisting to $DRIVERS_TOOLS/.env for later task steps. Also have test-cli.sh source ensure-uv.sh directly, since it invokes the installed mongodl/mongosh-dl binaries itself rather than through a child process that already calls ensure_uv.
SSL_CERT_FILE/NODE_EXTRA_CA_CERTS affect Node and uv broadly, not just the uv-bootstrap concern ensure-uv.sh is scoped to. setup.sh already runs once per task (via the global pre: hook) and its persistence to $DRIVERS_TOOLS/.env is picked up by every later script via handle-paths.sh, so detecting the cert bundle there once is simpler than repeating it in every ensure_uv call. This also makes the test-cli.sh ensure-uv sourcing added for this purpose unnecessary.
The setup.sh-based SSL_CERT_FILE fix still doesn't resolve the CERTIFICATE_VERIFY_FAILED failures on rhel8-zseries; add visibility to find out whether the block found a candidate path at all (echo to stderr and inputs.log) and whether the value actually reaches test-cli.sh's own shell (debug echo). Also add the ca-trust extracted bundle path as a fourth candidate.
blink1073
changed the title
blink1073
changed the title
…in self-test CI Root cause found: when this repo tests itself, prepare-env-and-resources.sh copies the checkout into a separate $DRIVERS_TOOLS directory before invoking setup.sh on that copy, but config.yml invokes test-cli.sh from the *original* checkout ($PROJECT_DIRECTORY), whose own handle-paths.sh recomputes DRIVERS_TOOLS by walking up directories and resolves the original location instead. Two physically different .env files. Append the CA bundle env vars to both $DRIVERS_TOOLS/.env and $PROJECT_DIRECTORY/.env so whichever one a given script reads has the value.
Root cause of the persistent zseries SSL failures: handle-paths.sh disabled auto-export (set +a) before sourcing $DRIVERS_TOOLS/.env and only re-enabled it (set -a) afterward, so values like SSL_CERT_FILE became plain shell variables rather than real environment variables. They were visible within the sourcing script itself (hence the propagation debug line showing the right value) but never inherited by a child process like mongodl.py. Swap the order, following the same save/restore pattern already used for errexit/nounset. Remove the now-unneeded debug echo from test-cli.sh.
The handle-paths.sh auto-export fix and the setup.sh CA-bundle-detection/.env-propagation work were addressing the wrong layer. The actual failures (test-cli-partial, test-mongodb-runner-partial on rhel8-zseries) trace back to mongodl.py's own SSL context missing certifi in its dependency environment, not to environment variable propagation. Reverting this out-of-scope detour; a targeted fix follows.
…find it mongodl.py already tries to load certifi's bundled CA bundle into its SSL context (falling back silently via except ImportError if unavailable), but .evergreen/pyproject.toml never declared certifi as a dependency, so it was never actually installed and the fallback silently no-opped. This is the real root cause of the persistent CERTIFICATE_VERIFY_FAILED failures on rhel8-zseries: uv's managed CPython build for that platform has default cert paths that don't match the host, and there was nothing overriding them. Adding certifi fixes this portably, without depending on any host-specific file layout. Verified: both uv tool install --editable . and uv run mongodl.py now have certifi available; uv tool install ignores PEP 723 inline script metadata (confirmed empirically), so a pyproject.toml dependency is the only mechanism that covers both invocation paths this repo uses. Regenerated .evergreen/orchestration/uv.lock to keep its dependency graph consistent (it also picked up an incidental mongo-orchestration version bump, an unrelated side effect of relocking an unpinned master-branch archive dependency).
Always upgrade pip before installing uv instead of installing, checking, and retrying with an upgrade only on failure. pip's own upgrade check is a fast no-op when already current, so this removes a duplicated install path with no behavior change.
Addresses a shellcheck info-level finding (SC2012) on the RHEL7 toolchain-python fallback glob.
There was a problem hiding this comment.
Pull request overview
This PR begins migrating Python dependency/interpreter management to a uv workspace by introducing a shared ensure-uv.sh bootstrap helper and updating key Evergreen scripts/tests to run Python via uv (instead of scanning the filesystem for an interpreter).
Changes:
- Add a root
.python-versionpin and a new.evergreen/ensure-uv.shhelper to ensureuvis present and usable across CI hosts. - Switch primary setup/install/download scripts and several Evergreen test helpers to use
ensure_uv+uv run/uv python find. - Update
.evergreenPython project metadata/locks (including addingcertifi).
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| CONTRIBUTING.md | Documents the new uv/.python-version approach and notes remaining legacy venv scripts. |
| .python-version | Pins the default Python version used by uv. |
| .evergreen/uv.lock | Updates the uv lock for the .evergreen project (adds certifi). |
| .evergreen/tests/test-mongodb-runner.sh | Switches runner test to bootstrap uv and execute download via uv. |
| .evergreen/tests/test-install-binaries.sh | Uses uv python find to select the interpreter for subsequent checks. |
| .evergreen/tests/test-happy-eyeballs.sh | Runs the happy-eyeballs client via uv run. |
| .evergreen/tests/test-csfle.sh | Resolves Python via uv python find for CSFLE test flow. |
| .evergreen/setup.sh | Ensures uv is available before orchestration/CLI installation steps. |
| .evergreen/serverless/create-instance.sh | Uses uv run python for JSON parsing related to serverless instance metadata. |
| .evergreen/pyproject.toml | Adds certifi dependency and documents why it’s required for mongodl.py. |
| .evergreen/orchestration/uv.lock | Updates orchestration lock content (adds certifi, updates mongo-orchestration version/hash). |
| .evergreen/install-cli.sh | Simplifies CLI install bootstrap to rely on ensure_uv rather than preselected Python. |
| .evergreen/happy_eyeballs/teardown.sh | Runs teardown server control via uv run python. |
| .evergreen/happy_eyeballs/setup.sh | Runs server startup/wait via uv run python (including nohup path). |
| .evergreen/ensure-uv.sh | New helper that installs/locates uv (with pyenv/toolchain considerations). |
| .evergreen/download-mongodb.sh | Uses uv to run the Python helpers that compute download URLs. |
| .evergreen/atlas/atlas-utils.sh | Uses uv run python for JSON parsing while polling Atlas deployment readiness. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Pass the curl response to json.loads via sys.argv instead of interpolating it into the Python source string, which breaks (or is exploitable) if the response contains quotes or newlines. Also call `uv run python script.py` explicitly instead of `uv run script.py` for clarity, closing out the remaining review comments.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 17 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
.evergreen/ensure-uv.sh:60
- Toolchain python discovery uses
compgen | sort -V | tailwhich (1) relies on GNUsort -V(not available on macOS/BSD sort) and (2) can cause an unexpected exit underset -euo pipefailwhen there are no matches or any pipeline component fails (e.g..evergreen/tests/test-install-binaries.shenablespipefail). Rewriting this without external pipelines makesensure_uvmore portable andpipefail-safe.
declare toolchain_py
toolchain_py="$(compgen -G '/opt/mongodbtoolchain/v*/bin/python3' | sort -V | tail -n1)"
if [ -n "$toolchain_py" ] && [ -x "$toolchain_py" ]; then
py="$toolchain_py"
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 15 out of 17 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
.evergreen/ensure-uv.sh:63
- The toolchain-python fallback uses
compgen | sort -V | tailwithout guarding failures. When there is no match (or on platforms withoutsort -V), this can fail underset -e/pipefailand abort the caller even though the later fallback topythonwould work. Prefer a pure-Bash glob loop that is safe when there are no matches and doesn’t depend on GNUsort -V.
declare toolchain_py
toolchain_py="$(compgen -G '/opt/mongodbtoolchain/v*/bin/python3' | sort -V | tail -n1)"
if [ -n "$toolchain_py" ] && [ -x "$toolchain_py" ]; then
py="$toolchain_py"
elif command -v python >/dev/null 2>&1; then
py=python
fi
Add ensure_uv_scoped_paths() to ensure-uv.sh so every uv-using script confines uv's cache, tool installs, and downloaded Python interpreters to $DRIVERS_TOOLS/.local instead of uv's default shared home-directory locations, avoiding cross-task/cross-host contention on reused hosts. install-cli.sh already did this ad hoc for cache/tool dirs; it now shares the same helper, which also covers python installs.
Move the UV_CACHE_DIR/UV_TOOL_DIR/UV_PYTHON_INSTALL_DIR scoping from a separate ensure_uv_scoped_paths() call into ensure_uv() itself, so every call site gets scoped paths automatically instead of needing a second call after ensure_uv.
Make _ensure_uv_scope_paths a no-op unless $CI is set, so local developer runs keep uv's normal shared cache instead of being forced into checkout-scoped paths (and requiring $DRIVERS_TOOLS) outside CI.
eramongodb
requested changes
Jul 28, 2026
Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
The `${DRIVERS_TOOLS:?}` guard exited the shell outright rather than
returning non-zero, which broke atlas/setup.sh and serverless/setup.sh:
they reach ensure_uv through the `bash -c` in check-connection.sh, and
handle-paths.sh assigns DRIVERS_TOOLS before its `set -a` so the child
never inherits it. Path scoping is best-effort hygiene, so treat a
missing DRIVERS_TOOLS as a no-op instead.
Also stop gating UV_TOOL_DIR on $CI. install-cli.sh scoped it
unconditionally before this branch, and it runs
`uv tool install --force "uv~=0.8.0"`, which would otherwise overwrite
a developer's globally installed uv on local runs. The cache and managed
interpreters stay CI-only so local runs keep their warm shared cache.
Exclude the scoped uv directories from the drivers-dir.tar.gz failure
artifact, since uv-managed interpreters are hundreds of MB and are not
useful for debugging.
Before this branch, setup.sh computed DRIVERS_TOOLS_PYTHON and appended it to $DRIVERS_TOOLS/.env, which handle-paths.sh sources. Deprecating find-python3.sh dropped both lines, so the per-folder virtualenv scripts still on that mechanism (auth_aws, auth_oidc, csfle, docker, ocsp) lost the hint and fall back to scanning the filesystem for an interpreter. Have ensure_uv export the uv-resolved interpreter, and restore the .env write in setup.sh -- the export alone is not enough, since those scripts source neither ensure-uv.sh nor handle-paths.sh and each Evergreen command runs in a fresh shell. An existing DRIVERS_TOOLS_PYTHON is left alone, since find-python3.sh documents it as an override.
Python 3.9 reached end of life in October 2025, so raise requires-python to >=3.10 in both pyproject.toml files (regenerating the lockfiles) and in the is_python3.py gate used by find-python3.sh. Also remove the .python-version pin added earlier on this branch. It forced a 3.13 download on any host without exactly that version, which made the requires-python floor largely meaningless. Without it uv reuses any host interpreter that satisfies requires-python. Use `uv python find --system` for DRIVERS_TOOLS_PYTHON now that no pin narrows resolution: a plain `uv python find` returns whichever virtualenv it discovers from the working directory, and consumers feed this to venvcreate, which needs a base interpreter.
Reverts the DRIVERS_TOOLS_PYTHON export added earlier in this branch, which broke the docker test. Seeding the variable makes ensure_python3 take its override branch and skip find_python3 entirely, replacing find_python3's ordered, vetted selection with uv's. In the Ubuntu test image those disagree: the image installs python3.11 and symlinks /usr/bin/python3 to it but never creates /usr/bin/python, so `uv python find --system` returns /usr/bin/python (the older system 3.10) while find_python3 correctly prefers python3. Because venvcreate passes --system-site-packages, the 3.10 dist-packages then leaked a stale pyOpenSSL into the venv, failing against the venv's newer cryptography: AttributeError: module 'lib' has no attribute X509_V_FLAG_NOTIFY_POLICY The original .env write this was restoring only cached find_python3's own result to avoid rescanning. The scripts still on that mechanism call ensure_python3 themselves, so dropping the cache costs a repeated scan but selects the same interpreter.
Restores the .python-version pin, at 3.13. Dropping it let uv resolve the newest interpreter, and on amazon-2023 mongo-orchestration then failed to reach a requireTLS mongod: ssl.SSLError: unknown error (0x0) (_ssl.c:3175) mongo_orchestration.errors.TimeoutError: Could not start Server Task logs show uv downloading cpython-3.14.0 and installing mongo-orchestration into lib/python3.14. Those three test-8.0-* tasks pass on the last patch that still had the pin and fail on both patches without it. The mechanism is not yet understood. Ruled out so far: the x509gen certs are 2048-bit/SHA-256 and valid until 2039; the bundled OpenSSL version is not the trigger (3.12 passes on the same OpenSSL 3.5.4 that 3.13 and 3.14 fail on); VERIFY_X509_STRICT is irrelevant because pymongo disables it; and pymongo's TLS handshake against these certs succeeds on 3.14 in isolation. Reproducing it needs a Linux mongod, so this commit confirms the attribution in CI instead. Removing the pin is still the goal, per review feedback on PR 808 -- it lets uv reuse an interpreter the host already has instead of forcing a download. That should follow once 3.14 works.
Three amazon-2023 test-8.0-* tasks started failing in `bootstrap mongo-orchestration`, with mongo-orchestration unable to reach a requireTLS mongod: ssl.SSLError: unknown error (0x0) (_ssl.c:3175) They pass on the last patch before this branch's recent commits and fail on every patch since, so the cause is in that window. This restores .python-version, both requires-python values, both lockfiles, and is_python3.py to their state in that last green patch, leaving only the uv path-scoping guard fix. If the tasks go green, the Python version work is responsible and can be redone separately; if not, the guard fix is implicated. Note that the earlier attempt to hold the interpreter at 3.13 was ineffective: `uv tool install` does not honor .python-version and resolves its own interpreter, so mongo-orchestration ran on 3.14 both before and after. The Python version is not the trigger. Also documents that .python-version does not apply to `uv tool install`, since that was not obvious and cost a round of debugging. Still ruled out as causes: the x509gen certs (2048-bit/SHA-256, valid to 2039), the OpenSSL version bundled with the interpreter, VERIFY_X509_STRICT (pymongo disables it), pymongo's TLS handshake under 3.14 in isolation, and the locked mongo-orchestration and pymongo versions, which this branch never changed.
setup.sh stopped seeding DRIVERS_TOOLS_PYTHON when it moved from find-python3.sh to ensure-uv.sh. The per-folder virtualenv scripts (auth_aws, auth_oidc, csfle, docker, ocsp) still read it through ensure_python3, and downstream repos read it out of .env, so restore it. Take the value from ensure_python3 rather than from uv. ensure_python3 selects in a specific order -- toolchain, then `python3`, then `python` -- and the result is handed to venvcreate, which passes --system-site-packages. An earlier attempt to resolve this with `uv python find` picked /usr/bin/python (3.10) in the Ubuntu test image where find_python3 correctly picks python3 (3.11); the 3.10 dist-packages then leaked a stale pyOpenSSL into the venv and broke test-docker. This means setup.sh keeps sourcing find-python3.sh until those per-folder scripts migrate to uv.
Addresses review feedback on PR 808: pinning to one exact version forced a download on any host that did not already have it, which made the requires-python floor largely moot. Keeping the floor at >=3.9 is what makes this safe. uv reuses the first interpreter it finds that satisfies the floor and only downloads when none does, so on hosts whose system Python is 3.9 -- amazon-2023 among them -- nothing is downloaded. Raising the floor to >=3.10 is what broke those hosts earlier in this branch: it pushed them off the system 3.9 onto a downloaded 3.14, where mongo-orchestration fails its TLS connection to mongod. Dropping the pin without touching the floor does not move any host's interpreter. The pin never applied to `uv tool install` in any case -- that resolves its own interpreter -- so removing it does not affect the orchestration tool environment.
The repo ships no such file, so describing its absence was noise. Keeps the part that is actionable: uv reuses a satisfying interpreter the host already has, so raising requires-python forces an interpreter upgrade rather than only tightening metadata.
The previous push updated the branch ref but GitHub did not fire the synchronize event, leaving PR 808 pointing at the prior commit and showing a stale CONTRIBUTING.md.
Evergreen sets TMPDIR to a per-task directory that sits outside both $DRIVERS_TOOLS and $PROJECT_DIRECTORY. The failure artifacts are packed from those two trees, so putting uv's cache, tools, and managed interpreters under TMPDIR keeps them out of the uploads without needing exclude_files entries, and lets Evergreen recycle them with the task. Drops the exclusions added for the previous layout. Outside CI only UV_TOOL_DIR is redirected, still under $DRIVERS_TOOLS/.local, so `uv tool install --force` cannot overwrite a developer's global tools while the cache stays shared. TMPDIR is not assumed: it falls back to TEMP then TMP, as clean.sh already does, and then to $DRIVERS_TOOLS/.local for the tool dir if none is set. The value is run through `cygpath -m` on Windows to match handle-paths.sh, since uv rejects /cygdrive/c/... paths.
eramongodb
reviewed
Jul 30, 2026
eramongodb
left a comment
eramongodb
left a comment
Contributor
There was a problem hiding this comment.
Minor feedback remaining; otherwise, LGTM.
eramongodb
approved these changes
Jul 30, 2026
Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
Co-authored-by: Ezra Chung <88335979+eramongodb@users.noreply.github.com>
Member
Author
|
Thanks @eramongodb! I'll merge once CI passes again |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRIVERS-3564
Summary
First step of migrating Python dependency management to a
uvworkspace: switch the primary setup/install/download scripts and test helpers from scanning the filesystem for a Python interpreter to lettinguvmanage it.Changes in this PR
Adds a new
ensure-uv.shhelper, used by the main scripts and a handful of test helpers. The per-folder virtual environment scripts (auth, docker, ocsp, csfle) are intentionally left on the existing mechanism for now and will move over in follow-up PRs.setup.shtherefore still seedsDRIVERS_TOOLS_PYTHONfromensure_python3, both for those scripts and for downstream repos that read it out of.env.Confines uv's shared state to the Evergreen checkout rather than uv's shared home-directory locations, avoiding contention when Evergreen hosts are reused or share a home directory.
install-cli.shpreviously did this ad hoc for two of the three directories; everyensure_uvcall site now gets it. Tool installs are always contained so a local run cannot overwrite a developer's globaluv, while the cache stays shared outside CI to avoid needless re-downloads.Also adds
certifias a dependency somongodl.pycan load a CA bundle explicitly, rather than relying on interpreter default cert paths that don't always match the host.Supported Python is unchanged at 3.9+.
Notes for reviewers
One thing surfaced while getting CI green that is worth knowing, since it isn't obvious: raising
requires-pythonis not merely a metadata change.uvreuses the first interpreter satisfying the floor and only downloads when none does, so raising the floor pushes hosts sitting at the old one onto a freshly downloaded interpreter. Trying>=3.10moved amazon-2023 off its system 3.9 onto a downloaded 3.14, where mongo-orchestration fails its TLS connection tomongod. Dropping EOL 3.9 is blocked on that until the 3.14 failure is understood, so the floor is left alone here.Test Plan
Ran the affected scripts directly and
pre-commitpasses.Evergreen patch 10730 was fully green, with zero failed tasks. The two most recent commits have a patch in flight and are not yet confirmed.
Checklist
Checklist for Author
Checklist for Reviewer