Skip to content

fix(render): two silent hangs in the turbo capture path - #132

Merged
andylizf merged 3 commits into
mainfrom
fix/shm-scratch-per-user
Jul 31, 2026
Merged

fix(render): two silent hangs in the turbo capture path#132
andylizf merged 3 commits into
mainfrom
fix/shm-scratch-per-user

Conversation

@andylizf

@andylizf andylizf commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Two failures in the turbo capture path that both end the same way: the run reports
success and nothing is wrong in the logs.

1. The scratch directory belongs to whoever ran first

The turbo path writes Chrome's raw BGRA tiles to a fixed
/dev/shm/pixelrag_render/raw. /dev/shm is shared by every user on the box, so
that directory belongs to whoever ran first — it lands at 0775 owned by them, and
every later user is denied write access to it.

Nothing about that surfaces:

  • mkdir(exist_ok=True) still succeeds against the existing directory
  • Chrome's write failure goes to its stderr, which this backend sends to DEVNULL
  • the resulting FileNotFoundError in every compression worker was swallowed by a
    bare except Exception: pass

A second user on the same host gets Done: 13 tiles, a manifest listing 13 tiles
with complete: true, and an output directory holding nothing but tiles.json.

Fix: scope the scratch path per uid, refuse to start when it isn't writable, and
count and log compression failures instead of dropping them.

Measured on a shared box, as the non-owning user:

tiles reported images on disk
before 13 0
after 13 13 (2,611,261 bytes)

2. The compression pool forked from a thread

fast_cdp builds its compression pool from inside a thread, while the capture side
is already running threads of its own. The pool used the platform default start
method, fork — and forking a multi-threaded process can leave the child holding a
lock nobody will ever release, so it blocks forever. CPython has warned about this
since 3.12.

A single-threaded caller forks cleanly, which is why it survived: pixelshot from a
shell works every time. Under a threaded host it hangs with no output and no error.
The test suite is one such host:

result
tests/test_render.py on its own 5.8s, passes
same file, after the async tests hangs indefinitely

Fix: start the pool with spawn. That means the pool initializer has to survive
pickling, so the core-affinity helper moves from a closure to a module-level function.

full suite
before exceeded a 400s timeout, no result
after 77 passed, 2 skipped in 12.17s

Based on #131fast_cdp.py here already carries that PR's change.

@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
web Ready Ready Preview Jul 31, 2026 8:50am

@andylizf andylizf changed the title fix(render): scope the /dev/shm scratch dir to the current user fix(render): two silent hangs in the turbo capture path Jul 31, 2026
@andylizf
andylizf changed the base branch from fix/page-height-viewport-bounded-body to main July 31, 2026 08:44
andylizf added 2 commits July 31, 2026 16:45
The turbo path writes Chrome's raw BGRA tiles to a fixed
`/dev/shm/pixelrag_render/raw`. `/dev/shm` is shared by every user on the box,
so that directory belongs to whoever ran first — it lands at 0775 owned by them,
and every later user is denied write access to it.

Nothing about that surfaced. `mkdir(exist_ok=True)` still succeeds against the
existing directory; Chrome's write failure goes to its stderr, which this backend
sends to DEVNULL; and the resulting FileNotFoundError in every compression worker
was swallowed by a bare `except Exception: pass`. A second user on the same host
therefore got "Done: 13 tiles", a manifest listing 13 tiles with
`complete: true`, and an output directory holding nothing but `tiles.json`.

Scope the scratch path per uid, refuse to start when it isn't writable, and count
and log compression failures instead of dropping them.

Measured on a shared box, as the non-owning user:

    before:  Done: 13 tiles  ->  0 images on disk
    after:   Done: 13 tiles  -> 13 images on disk (2,611,261 bytes)
The pool is created from a thread while the capture side already runs its own
threads. Forking a multi-threaded process can leave the child holding a lock
that was never released, and the child then blocks forever — CPython has warned
about this since 3.12.

A single-threaded caller forks cleanly, which is why this survived: `pixelshot`
from a shell works every time. Under a threaded host it hangs with no output and
no error. The full test suite is one such host: `tests/test_render.py` passes in
5.8s on its own and hangs indefinitely when the async tests run first.

Switching to spawn also means the pool initializer has to survive pickling, so
the core-affinity helper moves from a closure to a module-level function.

    before:  full suite exceeded a 400s timeout, no result
    after:   77 passed, 2 skipped in 12.17s
@andylizf
andylizf merged commit c1dae49 into main Jul 31, 2026
6 checks passed
@andylizf
andylizf deleted the fix/shm-scratch-per-user branch July 31, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant