Skip to content

final_report_backport_scaffold.md

Latest commit

 

History

History
151 lines (123 loc) · 7.24 KB

final_report_backport_scaffold.md

File metadata and controls

151 lines (123 loc) · 7.24 KB
Raw
title Final Report Back-Port Scaffold
slug final-report-backport-scaffold
type reference
status live
created 2026-05-01
updated 2026-05-07
owner Team 13
scope team-repo
canonical true

Final Report Back-Port Scaffold

Last updated: 2026-05-03 Owner: Alex Xin Issues: #40, #78, with source draft tracked by #5 / #39

This is the control surface for turning the NeurIPS Evaluations & Datasets draft (formerly Datasets & Benchmarks) into the class IEEE-format final report without content drift. The source of truth for claims and prose remains docs/neurips_draft.md; this file defines the conversion path. The first content-bearing class report drafting surface is now reports/final_report_ieee_draft.md.

Overleaf transfer surface: docs/neurips_overleaf_transfer_plan.md.

May 3 readiness status

The back-port checklist and the first content-bearing IEEE Markdown report draft now exist. #78 is therefore no longer blocked on checklist creation; its remaining value is to make sure the checklist is used during the final IEEE LaTeX transfer. #40 remains open until the class-report template is actually populated, compiled/exported, and checked against the final NeurIPS source.

Current source posture:

  • NeurIPS Overleaf has a first real draft at commit 4a85633.
  • The NeurIPS source is not yet frozen because compile proof, checklist answers, final references, scenario-count wording, and mitigation-rerun disposition remain open.
  • The IEEE Markdown draft can keep improving now, but final IEEE export should wait until the NeurIPS source text is coherent enough to avoid a divergent second paper.

Source-of-truth order

Use this order when the report and paper disagree:

  1. docs/validation_log.md and committed benchmark artifacts for run IDs, artifact paths, and what a run proves.
  2. results/metrics/*.csv and results/figures/* for result tables and figure inputs.
  3. docs/neurips_draft.md for canonical paper framing and reusable prose.
  4. This file for class-report section placement and conversion checks.

Do not introduce class-report-only claims unless they are explicitly marked as class-format framing. If a factual claim changes, update docs/neurips_draft.md first and then back-port it here.

Section map

IEEE final report section Source in NeurIPS draft / repo Conversion note
Abstract docs/neurips_draft.md abstract Same core paragraph; trim venue-specific wording if needed.
Introduction Introduction + contribution paragraphs Keep the benchmark gap, MCP systems variable, and AaT vs PE framing.
Models and Data Description Benchmark Extension + docs/data_pipeline.tex Emphasize source datasets, shared transformer_id, scenarios, and four tool domains.
Training and Profiling Methodology System Design + runbook/profiling docs The class heading says training, but our work is inference/profiling; phrase the section as serving, orchestration, and profiling methodology.
Performance Tuning Methodology Experiment 1 Cell C / D / ZSD optimization discussion Separate clean transport optimization from exploratory optimized-serving ablations.
Experimental Results Results + failure analysis sections Use Notebook 02/03 exports, validation ledger references, and the merged PR #151 failure-taxonomy artifacts: results/metrics/failure_taxonomy_counts.csv, results/metrics/failure_stage_cell_counts.csv, results/figures/failure_taxonomy_counts.svg, and results/figures/failure_stage_cell_heatmap.svg.
Conclusion Discussion / limitations / future work Keep claims conservative; include AOB upstream path as future work, not a deadline blocker.

Required report figures and tables

Minimum report-ready set:

  • Experiment 1 latency comparison from Notebook 02.
  • Experiment 2 orchestration comparison from Notebook 03.
  • PE-family Self-Ask rows from results/metrics/notebook03_self_ask_ablation.csv and results/metrics/experiment_matrix_summary.csv; do not cite notebook03_pe_family_follow_on.csv as the source for YS / ZS unless it is regenerated.
  • Failure taxonomy count figure from results/figures/failure_taxonomy_counts.svg, backed by results/metrics/failure_taxonomy_counts.csv.
  • Failure stage-by-cell heatmap from results/figures/failure_stage_cell_heatmap.svg, backed by results/metrics/failure_stage_cell_counts.csv.
  • Experiment matrix / trial-policy table.
  • Artifact ledger table with Slurm run IDs and repo paths.

Optional if evidence lands in time:

  • Mitigation before/after figure from results/metrics/mitigation_before_after.csv, regenerated from results/metrics/gcp_post175_mitigation_4tier_summary.csv and the tracked raw run artifacts.
  • PE-family optimized-serving ablation table.
  • Scenario-realism / generated-scenario validation figure.

May 3 draft status

reports/final_report_ieee_draft.md now contains the first full IEEE-section draft with current evidence tables for:

  • Experiment 1 A/B/C transport latency.
  • Experiment 2 B/Y/Z and PE-family Self-Ask quality, with source separation between notebook03_orchestration_comparison.csv and notebook03_self_ask_ablation.csv.
  • Failure taxonomy class counts.
  • Figure and artifact checklist.

The draft intentionally distinguishes canonical team13/main facts from pending deadline work. In particular, it does not claim the 30-scenario floor is complete until PR #156 plus generator-accepted scenarios are merged and validated.

The next useful report step is to re-check this Markdown draft against the populated NeurIPS Overleaf source, then copy into the IEEE template only after the NeurIPS source compiles and its TODO markers are under control. This keeps #40 and #78 aligned with #5/#39 instead of creating a second independent paper. Until the NeurIPS source is compiled, the Markdown report draft should remain a working conversion surface rather than the class-submission artifact.

Back-port checklist

  • Freeze the result tables and figure files that the class report cites.
  • Copy the NeurIPS abstract into the report and trim only for class format.
  • Convert the benchmark-extension section into Models and Data Description.
  • Convert the system-design section into methodology, avoiding any claim that we trained a model when we only served and profiled inference.
  • Convert the optimization discussion into Performance Tuning Methodology.
  • Insert result figures with captions that include source CSV / run IDs in the final IEEE LaTeX surface.
  • Insert the artifact ledger and validation caveats in the final IEEE LaTeX surface.
  • Check every numeric claim against results/metrics/ or docs/validation_log.md.
  • Compile the IEEE Overleaf report and record the compile/export status in the issue or PR thread.
  • After the NeurIPS Overleaf source is populated, re-check this Markdown draft against main.tex before copying into the IEEE template.

Current status

The report back-port now has a content-bearing Markdown draft, but the IEEE-format LaTeX report has not yet been compiled/exported. #40 should remain open until the final IEEE Overleaf report is generated from the NeurIPS source without content drift and the numeric claims are checked against the final frozen artifacts.