feat: documentation build-test proof of concept (dasharo-issues#1153)#1276
Draft
ameyypawar wants to merge 3 commits into
Draft
feat: documentation build-test proof of concept (dasharo-issues#1153)#1276ameyypawar wants to merge 3 commits into
ameyypawar wants to merge 3 commits into
Conversation
Proof-of-concept for dasharo-issues#1153. mkdocs_build_extractor resolves a single build path through the tabbed MkDocs "building manual" decision tree instead of linearising every fenced block, plus a signature-aware reproducibility verdict and documentation diagnostics.
CLI exposes list/extract/script/verify/diagnose over the extractor. The self-tests inline markdown fixtures (mirroring regression_scope_selftests.py) and cover single-path resolution, linked-tab dedup, version substitution and the documentation diagnostics.
- verify(): drop the invented REPRODUCIBLE_MODULO_SIGNATURE verdict (romscope
does not emit those strings); return IDENTICAL/DIFFERS and surface romscope's
raw report for human interpretation
- execute only shell-labelled fences; parse {.bash} attribute-list fences so
sample-output blocks are not run and tagged commands are not dropped
- resolve(): raise on ambiguous selection and on unresolved placeholders
(e.g. --version without --revision)
- diagnose: dedupe identical build commands (removes a false positive)
- README: correct the root-cause framing and the reproducibility/artifact
over-claims; note fresh-OS dependency capture, macpijan's plan, the
universal-build alternative and docs#1240
- tests: cover unlabelled/attribute fences, missing-revision and dedupe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A proof of concept toward Dasharo/dasharo-issues#1153 (automatically test that
following the build documentation reproduces the published firmware). It's a
draft, and I'm treating it as concrete input to the discussion on that
issue rather than a finished solution — happy to move, reshape, or close it.
Where this fits (and what it deliberately doesn't do)
This follows the "parse the docs, then hash-compare" plan @macpijan sketched in
#545. There's an alternative in that thread — standardising the build (a
universal script like Dasharo/coreboot#579, or Jinja-generated docs) so parsing
becomes unnecessary — and this PoC doesn't preclude it; it tackles the
"read the existing docs" slice.
It intentionally does not attempt the harder half of the issue:
nothing missing (the
sudo/toolchain gap hit when trying doc-detective).That's the part that guarantees "no dependency is missing from the docs," and
it's out of scope here.
published release set + hashes.
Relationship to Dasharo/docs#1240
@WuXieSec's Dasharo/docs#1240 ("phase-1 build-docs command checks") is already
in this space, in the docs repo. This harness is complementary — it
resolves a single build path through the MkDocs tabbed decision tree, generates
a runnable build script, and adds a reproducibility comparison — but overlaps
on the "extract commands from building-manual.md" part. I'd like to reconcile
with it rather than duplicate; guidance on whether this belongs here (osfv) or
in docs alongside Dasharo/docs#1240 would help.
What the tool does
A
building-manual.mdpage is a MkDocs Material decision tree ofpymdownx.tabbedtabs (firmware x device x options). Extracting every fencedblock concatenates mutually exclusive branches; this resolves a single path
for a selection, following MkDocs'
content.tabs.linksemantics.list/extract/script— enumerate targets, resolve one, emit arunnable build script.
verify— sha256 compare; on a difference it surfacesromscope compareoutput to interpret (it does not guess a verdict — a correct reproducible
build legitimately differs from the signed release).
diagnose— flags doc patterns that block mechanical testing (versionchoices hidden in prose, inconsistent checkout/build tab labels, ambiguous
selections). Findings come from running it against the live manuals; the
committed tests use small inlined fixtures.
Standard library only;
pre-commit(black, isort, codespell, markdownlint,reuse) passes.
Asking first
@macpijan — you asked people to discuss on the issue before picking this up,
so: is this direction useful, and if so where should it live and how should it
relate to Dasharo/docs#1240? I'll follow your lead.