Feature/sciot 042 proper evaluation output#43
Open
lbedogni wants to merge 3 commits into
Open
Conversation
added 2 commits
July 5, 2026 20:59
Implements a proper structured evaluation output for server inference cycles. Each inference cycle produces one JSONL row with all analysis-ready metrics. Key changes: - New src/sciot/evaluation.py with inference-cycle.v1 schema - Structured fields: timing, network, device_layers, edge_layers, offloading_decision, layer_sizes_bytes - Replaces the old flat evaluation CSV write with structured JSONL - Versioned schema (inference-cycle.v1) for future compatibility - File rotation with configurable max bytes and backups - Summary frame for flat-table analysis - 7 unit tests covering event building, append/read, rotation, and summary Data per inference cycle: - Timing: acquire, device compute, edge compute, network, total - Network: avg speed, payload size, NTP latency - Device/edge layers: per-layer times, EMA times, count, total - Offloading decision: selected layer, reason, candidates count, estimated cost - Layer sizes for model reference All unit tests pass (49 passed, 0 failed).
The structured evaluation JSONL file now includes the server start timestamp in its name (format: inference_cycles_YYYYMMDD_HHMMSS.jsonl) so different server runs are easily distinguishable. - EvaluationFiles.structured_evaluations_file_path() now returns path with timestamp - RequestHandler.__init__() sets the server start timestamp on first instantiation - All unit tests pass (49 tests total)
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…d toggles Replace the single-file + size-rotation scheme (which discarded old data) for both the structured-evaluation and offloading-decision JSONL outputs with never-deleted segment files. A new segment is opened at the first tripped criterion (evaluation.split.max_rows or evaluation.split.interval_minutes) via a shared segment writer reused by both modules. - Add evaluation.split (rows/time) and evaluation.outputs (per-output enable/disable) config, validated in config.py, with example in server.full.yaml. - Skip event build + enqueue entirely for disabled outputs, so there is no cost. - Normalize device_layers_inference_time (a numpy array) via _to_float_list: the previous `array or []` raised "truth value of an array is ambiguous" and aborted the inference-cycle write before it was enqueued. Also log the endpoint ValueError and background I/O failures with tracebacks so such write failures are no longer swallowed. - Read across all segments in the dashboard and via run_segment_paths / iter_run_offloading_decisions / iter_run_evaluations helpers. - Update CONFIGURATION.md and OFFLOADING_DECISION_EVENTS.md. - Add tests for splitting, per-output toggles, config validation, and the numpy-normalization fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.