| status | canonical-index |
|---|---|
| scope | team-repo |
| owner | Team 13 |
| canonical | true |
Course: COMS E6998: High Performance Machine Learning Institution: Columbia University Semester: Spring 2026 Instructor: Dr. Kaoutar El Maghraoui Mentor: Dr. Dhaval Patel, IBM Research
- Team Name: Team 13 / District 1101 (SmartGridBench)
- Members:
- Akshat Bhandari (ab6174) — LLM / eval harness
- Aaron Fan (af3623) — EE / systems / Slurm
- Tanisha Rathod (tr2828) — distributed systems / MCP
- Wei Alexander Xin (wax1) — PM / data / Agent-as-Tool orchestration
- Mentor: Dr. Dhaval Patel, IBM Research
- GitHub repository: https://github.com/HPML6998-S26-Team13/hpml-assetopsbench-smart-grid-mcp
- Final presentation:
deliverables/SmartGridBench-Final-Presentation.pptxanddeliverables/SmartGridBench-Final-Presentation.pdf - Final report:
deliverables/SmartGridBench_Final_Paper.pdf - Experiment-tracking dashboard: https://wandb.ai/assetopsbench-smartgrid/assetopsbench-smartgrid/reports/SmartGridBench-Final-Evidence-Dashboard--VmlldzoxNjgyODI4NA
This project extends IBM's AssetOpsBench industrial AI agent benchmark (467 scenarios across 6 HuggingFace subsets) by adding a 7th domain -- Smart Grid power transformers:
- Creating 36 paper-grade validated maintenance scenarios for Smart Grid transformers using public-safe synthetic telemetry data (the canonical corpus used for NeurIPS and CourseWorks claims; 25 additional scenarios added post-submission via PR #199 are present in
data/scenarios/but excluded from paper claims) - Wrapping four AssetOpsBench tool domains (IoT, TSFM, FMSR, WO) as MCP servers
- Profiling and optimizing the LLM agent inference pipeline when operating through MCP
- Comparing two orchestration paradigms (Agent-as-Tool vs Plan-Execute) on end-to-end multi-domain scenarios
We serve Llama-3.1-8B-Instruct via vLLM, profile end-to-end with PyTorch Profiler, apply 2-3 optimization techniques (INT8 quantization, KV-cache tuning, batched tool-call scheduling), and compare MCP-mediated tool calling against direct function calls to quantify protocol overhead. All experiments tracked with WandB.
| Component | Project implementation |
|---|---|
| Base model | Llama-3.1-8B-Instruct served through vLLM; selected WatsonX Llama-3.3-70B spot checks |
| Application | Industrial smart-grid maintenance benchmark extension for AssetOpsBench |
| Tool interface | Four Smart Grid MCP servers: IoT, TSFM, FMSR, and work orders |
| Orchestration variants | Agent-as-Tool, Plan-Execute, Verified PE, and Self-Ask follow-ons |
| Profiling stack | PyTorch Profiler, nvidia-smi telemetry, W&B run metadata, and committed result tables |
| Dataset | Rows | Primary Agent | License | Source |
|---|---|---|---|---|
| Power Transformers FDD & RUL | 3,000 files x 420 | TSFM, IoT | CC0 | Kaggle |
| DGA Fault Classification | 201 | FMSR | CC0 | Kaggle |
| Transformer Health Index | 470 | FMSR | ODbL | Kaggle |
| Current & Voltage Monitoring | 19,352 | IoT, TSFM | © Authors | Kaggle |
| Smart Grid Fault Records | 506 | WO | CC0 | Kaggle |
The repo's tracked data/processed/ artifacts are kept synthetic/public-safe. Restricted-source Kaggle joins remain a local benchmarking path, not a tracked publication path.
| Model | Params | FP16 VRAM | Tool-calling support | Fit for this project |
|---|---|---|---|---|
| Llama-3.1-8B-Instruct | 8B | ~16GB | Good, well-documented, strong vLLM support | Best -- fits A6000, enables rapid iteration, INT8 optimization is meaningful (16→8GB) |
| Phi-4-14B | 14B | ~28GB | Strong reasoning, less proven for tool calling | Reasonable but less community tooling |
| Mistral-Small-24B | 24B | ~48GB | Good | Needs A100, fewer experiment iterations per dollar |
| Gemma-3-27B | 27B | ~54GB | Good | Needs A100 80GB, overkill for benchmarking |
We select Llama-3.1-8B-Instruct for its favorable profiling characteristics: it fits comfortably on an A6000 with room for KV-cache experiments, and INT8 quantization produces a meaningful memory reduction. We optionally compare against Llama-3.3-70B (AssetOpsBench's default model) via WatsonX API to assess scaling effects.
The final package is anchored by the submitted report, slide deck, W&B dashboard, and committed
metrics under results/metrics/.
| Result area | Evidence |
|---|---|
| Scenario corpus | Current repo contains 61 SmartGridBench scenario files plus 5 negative validation fixtures; the final paper/result claims preserve the paper-grade frozen scope described in the result tables and report. |
| MCP transport benchmark | First-capture AaT runs show direct, MCP baseline, and optimized MCP paths in results/metrics/notebook02_latency_summary.csv. |
| Orchestration comparison | Verified PE + Self-Ask was the strongest first-capture PE-family row in the notebook export: 5/6 judge pass rate and 0.833 mean judge score in results/metrics/notebook03_self_ask_ablation.csv. |
| Full judged floor | The post-PR175 31-scenario aggregate is summarized in results/metrics/gcp_post175_core31_summary.csv. |
| Profiling evidence | W&B-linked and profiler-linked run inventory is in results/metrics/profiling_inventory.csv. |
.
├── README.md # This file - project overview, submission links, reproducibility
├── LICENSE # MIT license
├── requirements.txt # Python dependencies (ibm-watsonx-ai, pandas, etc)
├── .github/workflows/ # CI (Black formatting check)
├── deliverables/ # CourseWorks final report/deck files uploaded with the submission
│
├── data/ # Data pipeline + processed datasets - see data/README.md
│ ├── build_processed.py # Downloads + joins 5 Kaggle datasets via synthesized transformer_id
│ ├── generate_synthetic.py # Offline synthetic equivalent (no Kaggle needed)
│ ├── processed/ # 6 public-safe synthetic CSVs tracked in git (asset_metadata, dga_records, …)
│ ├── scenarios/ # AssetOpsBench-format scenario files - see data/scenarios/README.md
│ ├── knowledge/ # Structured standards artifacts for PS B generation - see data/knowledge/README.md
│ └── raw/ # GITIGNORED raw Kaggle downloads
│
├── mcp_servers/ # 4 MCP servers on shared base - see mcp_servers/README.md
│ ├── base.py # Shared data loader + utilities
│ ├── iot_server/ # Sensor reads, asset metadata
│ ├── fmsr_server/ # Failure search, IEC 60599 Rogers Ratio DGA analysis
│ ├── tsfm_server/ # RUL forecast, z-score anomaly, OLS trend
│ └── wo_server/ # Work order CRUD, downtime estimation
│
├── scripts/ # Utility scripts - see scripts/README.md
│ ├── verify_watsonx.py # WatsonX access verification + latency benchmarking
│ ├── run_experiment.sh # Canonical SmartGridBench runner (PE baseline + Self-Ask/Verified PE follow-ons)
│ ├── setup_insomnia.sh # Shared Insomnia environment setup
│ ├── vllm_serve.sh # Self-hosted vLLM serve + smoke path on Insomnia
│ ├── test_inference.sh # Sanity checks against a live vLLM endpoint
│ └── benchmark_prompts/ # Prompt templates for latency tests
│
├── benchmarks/ # Raw latency/throughput/judge runs - see benchmarks/README.md
│ ├── cell_A_direct/ # Direct-tool baseline artifacts
│ ├── cell_B_mcp_baseline/ # MCP baseline artifacts
│ ├── cell_C_mcp_optimized/ # Optimized MCP path artifacts
│ ├── cell_Y_plan_execute/ # Plan-Execute proof path (WatsonX smoke landed)
│ └── cell_Z_hybrid/ # Optional third-method slot (Verified PE preferred over generic Hybrid)
│
├── notebooks/ # Jupyter notebooks - see notebooks/README.md
├── profiling/ # PyTorch Profiler + Nsight - see profiling/README.md
├── results/ # Curated metrics + figures - see results/README.md
│
├── docs/ # Living authored documentation - see docs/README.md
│ └── knowledge-base/reference/ # Lower-churn class / mentor / setup reference docs
├── planning/ # Meeting agendas + working notes
└── reports/ # Build notes, historical submissions, and report work products - see reports/README.md
└── archive/ # Superseded drafts
- Python 3.12+
- Docker (for CouchDB)
- CUDA-capable GPU (16GB+ VRAM recommended)
- Access to Columbia Insomnia cluster or Google Cloud GPU instance
- WatsonX API key (via Codabench)
git clone https://github.com/HPML6998-S26-Team13/hpml-assetopsbench-smart-grid-mcp.git
cd hpml-assetopsbench-smart-grid-mcp
uv venv .venv
source .venv/bin/activate
uv pip install -r requirements.txtOptional overlays:
# interactive notebook authoring
uv pip install -r requirements-notebooks.txt
# Insomnia / cluster serving stack
uv pip install -r requirements-insomnia.txtrequirements.txt now also includes the portable AssetOpsBench PE-client
dependencies (litellm and mcp[cli]) needed by the repo-local Self-Ask PE /
Verified PE runners. The cluster overlay in requirements-insomnia.txt layers
vLLM and CUDA-specific pins on top of that base.
- W&B dashboard (final evidence): https://wandb.ai/assetopsbench-smartgrid/assetopsbench-smartgrid/reports/SmartGridBench-Final-Evidence-Dashboard--VmlldzoxNjgyODI4NA==
- W&B project (all runs): https://wandb.ai/assetopsbench-smartgrid/assetopsbench-smartgrid
The repo tracks public-safe synthetic processed data under data/processed/ and
AssetOpsBench-format Smart Grid scenarios under data/scenarios/. Raw Kaggle
downloads remain gitignored under data/raw/ and are not required to inspect
the submitted results.
python data/generate_synthetic.pyThis project benchmarks inference and tool orchestration; it does not fine-tune
the base Llama model. The model-serving path is configured through vLLM,
WatsonX-compatible LiteLLM endpoints, and the configs/*.env experiment files.
Judge outputs and summarized metrics are committed under results/. To score a
new completed run directory, use the judge helper with the run directory and
matching output path:
python scripts/judge_trajectory.py \
--run-dir benchmarks/cell_B_mcp_baseline/raw/<run-id> \
--scenario-dir data/scenarios \
--out results/metrics/scenario_scores.jsonl \
--log-dir results/judge_logsProfiling is enabled through the runner environment and summarized in
results/metrics/profiling_inventory.csv.
ENABLE_WANDB=1 TORCH_PROFILE=1 bash scripts/run_experiment.sh configs/example_baseline.env
python scripts/build_profiling_inventory.pyFor the currently proven benchmark-facing path:
bash scripts/run_experiment.sh configs/example_baseline.envSee:
- docs/orchestration_wiring.md for what is wired today versus only adapter-ready
- benchmarks/cell_Y_plan_execute/ for the first kept WatsonX smoke proof run and artifact layout
- docs/insomnia_runbook.md for the shared Insomnia self-hosted vLLM path
- The AaT direct, MCP baseline, and optimized MCP latency comparison is
exported in
results/metrics/notebook02_latency_summary.csv. - The first-capture orchestration and Self-Ask comparison is exported in
results/metrics/notebook03_self_ask_ablation.csv. - The final submitted claim set is backed by committed result tables, W&B run links, benchmark artifacts, and exported report/deck deliverables.
Last updated: Apr 18, 2026 - Apr 16 post-call audit resolved the WO architecture decision, closed the first-WandB-run milestone, and narrowed the active risk to overdue W2 closeout plus W3 profiling / PS B execution.
Week 1 (complete):
- Problem statement finalized (four contributions)
- Research proposal drafted and shared with mentor via Overleaf; mentor endorsed NeurIPS 2026 Datasets & Benchmarks track
- GitHub repo scaffolded, WandB team created, repo now public (Apr 7)
- 5 Kaggle datasets identified, AssetOpsBench forked and reviewed
- Compute confirmed (Insomnia cluster + GCP credits) and compute plan committed (
docs/archive/compute_plan.md) - WatsonX API access received from mentor (Apr 5) and verified end-to-end - 6 Llama models available; Llama-4-Maverick-17B and Llama-3.3-70B-instruct benchmarked (
docs/knowledge-base/reference/2026-04-06_watsonx_access.md) - Data pipeline + tracked public-safe processed datasets landed (
data/processed/with synthetic asset metadata, DGA records, failure modes, fault records, RUL labels, and sensor readings - development-ready and safe to publish) - MCP server skeletons landed for all four domains (IoT, FMSR, TSFM, WO) on a shared base class with substantive domain logic (IEC 60599 Rogers Ratio DGA analysis, RUL forecast, anomaly detection, work-order CRUD)
-
docs/data_pipeline.texpaper section drafted - Mid-point report submitted (
reports/2026-04-06_midpoint_submission.pdf) to Courseworks on Mon Apr 6
Week 2 (landed on canonical history):
- GitHub Projects reset as the canonical planning surface, with weekly iterations, workstream parent issues, and delivery-gate milestones
- Successful Insomnia A6000 vLLM serve smoke test for Llama-3.1-8B-Instruct, with kept smoke-test logs and fixed serve/test scripts
- First real SmartGridBench WandB run is live and back-linked to committed benchmark artifacts
- Plan-Execute is wired to the team’s Smart Grid MCP servers as a real experiment condition, with a successful WatsonX-hosted 70B / Mac 1-scenario smoke proof run under
benchmarks/cell_Y_plan_execute/ - Scenario realism validation note landed with IEEE / IEC grounded findings for Dhaval-facing review
Still open from the original W2 critical path / backlog:
-
#3Canonical benchmark scenario proof on the AssetOpsBench stack (Akshat) -
#58Benchmark-Llama-path validation closeout plus#9-#12MCP hardening/tests (Tanisha) - PR#115now contains a real Insomnia A6000 / self-hosted 8B / all-4-server proof run plus branch-specific long-context serve notes, but the merge/readme cleanup is still open (seedocs/insomnia_runbook.md) -
#7/#59profiling wrappers and the first profiling-linked experiment capture path (Aaron) -
#15/#17/#18/#20scenario-count, judge, and first trajectory artifacts (Akshat)
Paper-grade scenario corpus: 36 validated SmartGridBench scenarios — 31 hand-authored + 5 promoted generated. This is the canonical count used for the final NeurIPS and CourseWorks claims (frozen at NeurIPS submission, 2026-05-07).
Current repo corpus: 61 scenario files under data/scenarios/ plus 5 negative validation fixtures. The 25 scenarios beyond the 36 paper-grade canonical were added post-submission via PR #199 (#55 batch A and B) to grow the repo corpus and are deliberately not part of any evaluated/judged claims.
Resolved during the Apr 16 post-call audit:
-
#13WO architecture review closed as a documented keep-vs-pivot decision -
#28closed against the first real shared WandB run and its committed benchmark artifacts
W3 focus (Apr 14-20):
- Experiment 1 profiling captures plus profiling-to-WandB linkage (
#25,#27) - Problem Statement B kickoff: Knowledge Plugin, first generation prototype, and evaluation methodology (
#50,#2,#51) - NeurIPS abstract outline and title candidates (
#77) - Runbook consolidation for the infra / serve / profiling path (
#37)
Committed W3-W5 tracks:
- Experiment 1: MCP overhead and optimization (Direct vs MCP-baseline vs MCP-optimized)
- Experiment 2: orchestration comparison and failure analysis
- Problem Statement B extension: scenario generation pipeline, Knowledge Plugin, and validation methodology
- NeurIPS 2026 draft first, then back-port to the class IEEE report format
Current default scope decision: the working comparison is still vanilla Agent-as-Tool vs vanilla Plan-Execute. The repo now also has an active local mitigation stream for PE + Self-Ask and an optional Verified PE third-method prototype, but neither should muddy the honesty of the core AaT vs PE story. The primary local benchmark model remains self-hosted Llama-3.1-8B-Instruct with 70B reserved for selective WatsonX spot-checks.
- docs/README.md - living docs index and runbook navigation
- scripts/README.md - executable entrypoints and helper scripts
- configs/README.md - benchmark config schema and cell mapping
- data/README.md - data pipeline and processed dataset policy
- data/scenarios/README.md - scenario authoring rules and validation
- mcp_servers/README.md - Smart Grid MCP server layout
- benchmarks/README.md - raw benchmark artifact layout
- profiling/README.md - profiling capture workflow
| Date | Milestone |
|---|---|
| Mon Apr 6 | Mid-point report due (Courseworks, 11:59pm) |
| Tue May 5 | NeurIPS 2026 Datasets & Benchmarks abstract submitted |
| Thu May 7 | NeurIPS 2026 full submission submitted |
| Thu May 7 | Final class presentation delivered |
| Sun May 10 | CourseWorks final report/package due (extended deadline) |
- AssetOpsBench -- IBM's industrial AI agent benchmark
- AssetOpsBench on HuggingFace
- AssetOpsBench competition
- Model Context Protocol -- open protocol for LLM tool integration
- vLLM -- High-throughput LLM serving
- PyTorch Profiler
- Weights & Biases
See also: docs/knowledge-base/reference/2026-04-05_project_reference.md for class requirements, grading, and mentor guidance.
Per the HPML AI Use Policy, Team 13 used AI assistance during this project.
Tools used: Claude and Codex.
Specific purpose: We used AI tools for brainstorming, debugging support, code navigation, runbook/checklist assistance, and prose polishing. Experimental design decisions, reported measurements, profiling interpretations, quantitative analysis, and final claims were checked by team members against committed code, experiment logs, dashboard records, and exported result artifacts.
Sections affected: README/runbook wording, report and presentation polishing, checklist maintenance, and selected implementation/debugging workflows.
How we verified correctness: We ran the reported experiments ourselves, preserved result artifacts under benchmarks/, results/, and reports/, cross-checked headline claims against the committed evidence tables, and manually reviewed final public-facing wording before submission.
By submitting this project, the team confirms that the analysis, interpretations, and conclusions are our own, and that AI assistance is disclosed here.
MIT; see LICENSE.
If citing this course project, use:
SmartGridBench Team 13. "MCP-Based Industrial Agent Benchmarking for Smart Grid Operations." COMS E6998 High Performance Machine Learning Final Project, Columbia University, Spring 2026.
For project questions, use the GitHub issue tracker or contact the Team 13 members listed above.