Skip to content

AGILE_ACTION_PLAN.md

Latest commit

 

History

History
1567 lines (1417 loc) · 82.8 KB

AGILE_ACTION_PLAN.md

File metadata and controls

1567 lines (1417 loc) · 82.8 KB
Raw

SCIoT Agile Action Plan

This document is the working backlog for improving SCIoT. Keep items small, testable, and ordered by value and risk. Add new items to the Product Backlog using the template at the end.

Product Goal

Provide a reliable, maintainable split-computing platform in which supported clients can execute local inference, offload valid model segments to the edge, recover from network failures, and produce trustworthy performance measurements.

Working Agreement

  • Work in short iterations, preferably one or two weeks.
  • Prioritize working software, runtime reliability, and automated verification.
  • Keep stories small enough to complete within one iteration.
  • Define acceptance criteria before implementation.
  • Do not mix unrelated changes in the same pull request.
  • Update documentation and tests as part of the story, not afterward.
  • Review backlog priorities at the start of each iteration.
  • Demonstrate completed behavior at the end of each iteration.
  • Move unfinished work back to the backlog and re-estimate it.

Status Labels

  • BACKLOG: Not yet selected for an iteration.
  • READY: Clear requirements, acceptance criteria, and dependencies.
  • IN PROGRESS: Actively being implemented.
  • REVIEW: Implementation complete and awaiting review.
  • DONE: Meets the Definition of Done.
  • BLOCKED: Cannot progress until a named dependency is resolved.

Priority Labels

  • P0: Prevents the primary system from running.
  • P1: Major correctness, reliability, or maintainability issue.
  • P2: Useful improvement with moderate impact.
  • P3: Optional enhancement or research task.

Definition of Ready

An item is ready when:

  • Its value or problem is clearly stated.
  • Acceptance criteria are testable.
  • Dependencies and affected components are known.
  • The item is small enough for one iteration.
  • Open technical or product decisions have been resolved.

Definition of Done

An item is done when:

  • Acceptance criteria pass.
  • Relevant automated tests have been added or updated.
  • Existing tests pass.
  • Code is reviewed and merged.
  • Configuration and documentation are updated.
  • No credentials, generated logs, or local artifacts are committed.
  • Runtime behavior has been demonstrated on the relevant platform.

Product Backlog

Epic 0: Separate logics

SCIOT-000 - Separate logics

  • Status: IN PROGRESS
  • Priority: P0
  • Value: Allow the edge server and clients to be able to run separately.
  • Problem: The package is a a unique one. Server should be able to be run without client dependencies, and different clients have their own implementations and should not depend among each other.
  • Task breakdown:
    • Inventory imports and map dependencies to runtime profiles.
    • Define server, static client, laptop client, Raspberry Pi client, analysis, research, test, and development dependency profiles.
    • Add OS and CPU markers for TensorFlow and isolate Picamera2 as a Raspberry Pi OS dependency.
    • Regenerate and validate uv.lock.
    • Verify a clean server-only installation.
    • Verify clean static and laptop client installations.
    • Verify the Raspberry Pi dependency profile and document hardware-only validation.
    • Update installation and supported-platform documentation.
  • Actions:
    • Separate dependencies inside the pyproject.toml
    • Add specific dependencies based on the system (e.g. tensorflow on Macos vs Linux)
    • Verify that the server runs correctly only with its dependencies alone
    • Document the supported environment.
  • Acceptance criteria:
    • uv sync produces a usable environment.
    • uv can be run to install dependencies only for server or for specific client
    • sciot-server --config src/server/settings.yaml reaches server startup.
  • Verification evidence:
    • Base uv sync --no-dev installed only NumPy, Pillow, PyYAML, and SCIoT.
    • Clean server, client-static, and client-laptop environments were created successfully.
    • Package inspection confirmed that server, camera, analysis, and research dependencies remain isolated.
    • The server remained active for the agreed 15-second startup check.
    • The static client remained active for the agreed 15-second runtime check.
    • Laptop runtime imports, including OpenCV, completed successfully.
    • Raspberry Pi dependency resolution completed; Picamera2 remains a Raspberry Pi OS system dependency requiring hardware verification.
    • Dependency-profile regression assertions passed.

Epic 1: Restore a Reproducible Runtime

SCIOT-001 — Resolve TensorFlow/Keras compatibility

  • Status: DONE
  • Priority: P0
  • Value: Allow the edge server and tests to start in a clean environment.
  • Problem: The server forces TF_USE_LEGACY_KERAS=1, but the required legacy Keras package is not installed.
  • Task breakdown:
    • Remove the forced legacy-Keras environment override from server startup.
    • Confirm that standard TensorFlow 2.15/Keras can load the checked-in models on the current development platform.
    • Record the supported TensorFlow/Keras policy in one authoritative document.
    • Audit all model utilities for legacy tf_keras, private Keras APIs, and compatibility shims.
    • Decide which compatibility shims remain required and add focused model-loading tests for them.
    • Verify model loading on Linux x86-64.
    • Verify model loading on macOS Intel.
    • Verify model loading on macOS Apple Silicon.
    • Add a CI or documented manual verification matrix for supported platforms.
    • Add troubleshooting guidance for incompatible model exports and Keras versions.
  • Actions:
    • Decide whether the project will use standard TensorFlow 2.15 Keras or tf-keras.
    • Remove incompatible environment overrides or declare the required package.
    • Verify model loading on Linux, macOS Intel, and macOS Apple Silicon where supported.
    • Document the supported environment.
  • Acceptance criteria:
    • uv sync produces a usable environment.
    • sciot-server --config src/server/settings.yaml reaches server startup.
    • The test suite can be collected without a Keras import failure.
  • Verification evidence:
    • docs/TENSORFLOW_KERAS_POLICY.md now defines the standard TensorFlow 2.15 tf.keras policy, platform matrix, supported loader shim, manual checks, and troubleshooting.
    • Active code and comments were audited for TF_USE_LEGACY_KERAS, tf_keras, and tfk; no legacy Keras dependency remains outside policy/troubleshooting text.
    • Model loading now uses tf.keras.models.load_model(..., compile=False) through load_keras_model_for_inference.
    • The only retained compatibility shim is CompatDepthwiseConv2D, which strips legacy groups from older H5 exports.
    • 4 focused model-loading compatibility tests pass.
    • On macOS Intel / Darwin x86_64 with TensorFlow 2.15.0, all checked-in FOMO H5 models loaded successfully: FOMO_48_CUT, FOMO_96_CUT, FOMO_192_CUT, FOMO_224_CUT, and FOMO_300_CUT.
    • tests/test_edge/test_edge.py now uses the current FOMO_96_CUT artifact and passes.

SCIOT-003 — Correct the standard client lifecycle

  • Status: DONE
  • Priority: P0
  • Value: Prevent accidental execution and preserve local-only fallback.
  • Task breakdown:
    • Add an if __name__ == "__main__": import guard.
    • Preserve failed registration state and enter local-only mode.
    • Extract testable local-fallback and reconnection helpers.
    • Introduce an explicit client lifecycle object or cancellation signal.
    • Add a configurable maximum cycle count and optional run duration.
    • Stop accepting new work when shutdown begins.
    • Wait for or cancel the pending inference-result upload deterministically.
    • Shut down request and upload thread pools.
    • Close HTTP sessions and profiling resources.
    • Handle SIGINT/SIGTERM without os._exit.
    • Add lifecycle tests for normal completion, Ctrl+C, failed upload, and shutdown during reconnect.
  • Actions:
    • Add an if __name__ == "__main__": guard.
    • Stop resetting server_available to True after failed registration.
    • Add graceful shutdown for thread pools and pending uploads.
    • Add a configurable cycle limit for tests and demonstrations.
  • Acceptance criteria:
    • Importing the client does not start inference.
    • Failed registration starts local-only operation.
    • The client reconnects when the server returns.
    • Ctrl+C shuts down without forcing the process to exit abruptly.
  • Verification evidence:
    • ClientLifecycle owns cancellation, signal handling, bounded run limits, pending-upload cleanup, request/upload pools, sessions, and profiler shutdown.
    • main() accepts max_cycles, run_duration_seconds, and test-safe argv overrides while preserving direct script execution.
    • Lifecycle coverage verifies normal bounded completion, local-only startup after failed registration, failed upload cleanup, Ctrl+C-style cancellation, no new work during shutdown, and skipped reconnect work after shutdown begins.
    • 7 focused lifecycle tests pass.
    • The current fast CI lane reports 61 passed.
    • The current HTTP integration lane reports 8 passed.

Epic 2: Communication Correctness

SCIOT-004 — Repair WebSocket communication

  • Status: DONE
  • Priority: P1
  • Value: Restore the advertised WebSocket transport.
  • Problem: WebSocket code expects two return values from the request handler, which now returns three.
  • Task breakdown:
    • Define the WebSocket registration, inference, prediction, and split-decision message schema.
    • Align WebSocket binary inference payloads with the versioned binary protocol.
    • Update the handler call to consume split layer, device ID, and prediction.
    • Return both the prediction and next split decision to the correct connection.
    • Replace payload-length heuristics with an explicit message type.
    • Track the registered device and selected model per WebSocket connection.
    • Return structured client errors for malformed registration and binary messages.
    • Add connection-close and server-shutdown cleanup.
    • Add an isolated WebSocket integration test for registration and split inference.
    • Document whether WebSocket support is stable or experimental.
  • Acceptance criteria:
    • Registration works over WebSocket.
    • An inference request returns both the next split decision and prediction as defined by the protocol.
    • Automated integration coverage verifies the flow.
  • Implementation notes:
    • Added shared transport response helpers for JSON-safe predictions, registration acknowledgements, structured errors, and MQTT topic resolution.
    • WebSocket registration supports explicit type: registration messages and the legacy untyped JSON registration format.
    • WebSocket binary frames now support explicit JSON control messages for device_input and device_inference_result; the old payload-length fallback remains for backwards compatibility.
    • Added focused WebSocket integration tests for registration, inference response payloads, explicit binary frame typing, device input, and malformed JSON errors.
  • Verification:
    • UV_CACHE_DIR=.uv-cache uv run pytest tests/integration/test_websocket_transport.py -q

SCIOT-005 — Repair MQTT communication

  • Status: DONE
  • Priority: P1
  • Value: Restore MQTT support for IoT deployments.
  • Problem: MQTT code expects an obsolete request-handler return signature.
  • Task breakdown:
    • Define MQTT registration, acknowledgement, input, inference, prediction, and split topics.
    • Make topic routing include a device identifier rather than one fixed device.
    • Update inference handling to consume split layer, device ID, and prediction.
    • Publish the prediction and next split decision to device-specific topics.
    • Define registration rejection behavior for an unknown model hash.
    • Add payload validation and structured error reporting.
    • Add reconnect, subscription restoration, and clean disconnect behavior.
    • Verify the Docker Compose broker configuration.
    • Add isolated unit tests using a reliable MQTT test double.
    • Add an optional broker-backed integration smoke test.
  • Acceptance criteria:
    • The broker starts with Docker Compose.
    • A client can register and complete an inference exchange.
    • MQTT topics support multiple devices without collisions.
    • Automated tests use an isolated broker or a reliable test double.
  • Implementation notes:
    • MQTT registration now supports wildcard registration subscriptions for devices/ style topics.
    • Device-specific publish topics now resolve either {device_id} templates or legacy device_01/... topics.
    • Inference handling consumes the current three-value request-handler return signature and publishes both the next split decision and JSON-safe prediction.
    • Registration acknowledgements, prediction-only publishes, and structured error publishes are enabled when their optional topics are configured.
    • Disconnect now cancels the NTP timer and shuts down queued MQTT work.
    • Added an opt-in broker-backed smoke test guarded by SCIOT_RUN_MQTT_BROKER_SMOKE=1 so normal CI does not require Docker.
  • Verification:
    • UV_CACHE_DIR=.uv-cache uv run pytest tests/test_mqtt_client/test_mqtt_client.py -q
    • UV_CACHE_DIR=.uv-cache uv run pytest tests/test_mqtt_client tests/integration/test_websocket_transport.py tests/integration/test_http_end_to_end.py tests/integration/test_http_protocol_validation.py -q
    • docker compose config
    • UV_CACHE_DIR=.uv-cache uv run pytest tests/integration/test_mqtt_broker_smoke.py -q verifies the optional test skips by default.
    • Live broker smoke command: SCIOT_RUN_MQTT_BROKER_SMOKE=1 uv run pytest tests/integration/test_mqtt_broker_smoke.py -q.
    • Local live-broker execution was not possible in this session because the Docker daemon was unavailable.

SCIOT-006 — Run enabled communication transports concurrently

  • Status: DONE
  • Priority: P1
  • Depends on: SCIOT-004, SCIOT-005
  • Value: Allow HTTP, WebSocket, and MQTT to operate together.
  • Problem: Each transport uses a blocking run() call, preventing later transports from starting.
  • Task breakdown:
    • Define a common transport lifecycle interface: start, ready, stop, and join.
    • Add a coordinator that creates only the transports enabled in configuration.
    • Start blocking transports in managed threads or asynchronous tasks.
    • Expose per-transport readiness and startup failure state.
    • Keep healthy transports running when another transport fails after startup.
    • Propagate SIGINT/SIGTERM to every enabled transport.
    • Stop NTP timers, executors, sockets, and broker connections cleanly.
    • Add tests for HTTP-only, multiple enabled transports, partial startup failure, and shutdown.
    • Document resource ownership and process topology.
  • Acceptance criteria:
    • Every transport enabled in settings.yaml starts.
    • A failure in one transport is reported without silently disabling the others.
    • Shutdown closes all transport resources cleanly.
  • Implementation notes:
    • Added ManagedTransport and TransportCoordinator to run blocking transports in managed daemon threads.
    • Updated run_edge.py so communication.mode creates only the enabled transports and starts them together.
    • HTTP and WebSocket transports now expose stop() hooks that cancel NTP timers and request Uvicorn shutdown.
    • MQTT stop() already disconnects the broker and now participates in the common lifecycle.
    • The coordinator captures per-transport failure state, logs startup failures, keeps other transports alive, and propagates SIGINT/SIGTERM from the main process.
    • Documented resource ownership and process topology in docs/TRANSPORTS.md.
  • Verification:
    • UV_CACHE_DIR=.uv-cache uv run pytest tests/unit/test_transport_lifecycle.py tests/unit/test_run_edge_transports.py -q

SCIOT-007 — Version and validate the binary protocol

  • Status: DONE
  • Priority: P1
  • Value: Prevent silent incompatibilities across machines and client versions.
  • Task breakdown:
    • Use fixed little-endian encoding for the inference-result payload.
    • Add bounded decoding for device IDs, tensors, timing arrays, and total payload size.
    • Return structured errors for malformed inference payloads.
    • Choose a protocol magic value and version-field layout.
    • Define compatibility rules for current unversioned clients.
    • Add version negotiation or explicit unsupported-version responses.
    • Document registration, device-input, inference-result, prediction, and split-decision formats.
    • Define field types, units, limits, optional fields, and error codes.
    • Add golden payload fixtures for each supported protocol version.
    • Verify golden fixtures on supported CPU and operating-system architectures.
    • Add tests for unknown versions, trailing fields, and backward compatibility.
  • Actions:
    • Define message fields, byte order, data types, and length limits.
    • Add a protocol version.
    • Validate payload sizes before allocating or reshaping arrays.
    • Return structured errors for malformed messages.
  • Acceptance criteria:
    • Protocol documentation includes registration, image, inference result, and response formats.
    • Golden payload fixtures decode consistently across supported platforms.
    • Invalid and truncated payload tests pass.
  • Implementation notes:
    • Added an optional versioned inference-result header: magic SCIP, version uint16, flags uint16.
    • Version 1 uses the existing bounded little-endian body layout.
    • Legacy payloads without SCIP continue to decode as protocol version 0.
    • Unknown versions and non-zero flags are rejected before body decoding.
    • Added a deterministic v1 golden fixture in tests/fixtures/protocol/inference_v1.hex.
    • Documented exact field types, units, limits, compatibility rules, and transport error behavior in docs/TRANSPORTS.md.
  • Verification:
    • UV_CACHE_DIR=.uv-cache uv run --extra server --extra test pytest tests/unit/test_inference_protocol.py tests/integration/test_http_protocol_validation.py -q

SCIOT-008 — Consolidate inference endpoints

  • Status: BACKLOG
  • Priority: P2
  • Value: Remove duplicated inference logic and hard-coded model mappings.
  • Problem: /api/split_inference implements a second inference path with model-specific assumptions.
  • Task breakdown:
    • Inventory the behavior and consumers of every inference endpoint.
    • Define one transport-independent suffix-inference service interface.
    • Move model lookup from endpoint code into the shared service.
    • Generate or load skip-connection metadata per model.
    • Remove hard-coded resolution-to-directory and skip-layer mappings.
    • Make the primary inference-result endpoint delegate to the shared service.
    • Make /api/split_inference delegate to the shared service or deprecate it.
    • Define a migration period and compatibility response for deprecated callers.
    • Add parity tests showing both accepted paths produce identical predictions.
    • Remove obsolete endpoint code after migration is complete.
  • Acceptance criteria:
    • One shared service performs model selection and suffix inference.
    • HTTP endpoints delegate to that service.
    • Skip-connection validation comes from model metadata rather than hard-coded layer numbers.

Epic 3: Multi-Device State and Concurrency

SCIOT-009 — Introduce explicit per-device runtime state

  • Status: BACKLOG
  • Priority: P1
  • Value: Prevent one client from influencing another client's decisions.
  • Task breakdown:
    • Inventory all device-specific values currently stored globally or on shared handlers.
    • Define a typed DeviceRuntimeState model.
    • Include model identity, latest split, network estimator, device timings, edge timings, and variance state.
    • Introduce a device-state registry with explicit create, get, update, and remove operations.
    • Move registration initialization into the registry.
    • Move inference-result updates and offloading decisions to device state.
    • Define duplicate registration and model-change behavior.
    • Define expiration, disconnection, and reconnection retention policies.
    • Add a read-only inspection representation for diagnostics.
    • Add tests proving two devices maintain independent measurements and decisions.
  • Actions:
    • Create a typed device-state object.
    • Store network speed, model selection, timings, variance history, and latest split per device.
    • Define registration, expiration, and reconnection behavior.
  • Acceptance criteria:
    • Two clients with different network speeds receive independent split decisions.
    • Reconnecting a device restores or deliberately resets its state.
    • Device state can be inspected without reading global dictionaries directly.

SCIOT-010 — Make shared state thread-safe

  • Status: BACKLOG
  • Priority: P1
  • Depends on: SCIOT-009
  • Value: Avoid races under concurrent HTTP or MQTT requests.
  • Task breakdown:
    • Inventory every mutable registry, cache, counter, writer, and profiler shared across threads.
    • Assign an owner and synchronization strategy to each shared object.
    • Protect device-state creation and mutation with per-device or registry-level synchronization.
    • Make model-registry and metadata-cache reads safe during initialization and reload.
    • Serialize or queue CSV and evaluation writes through one owned writer.
    • Make counters and periodic exports atomic.
    • Define lock ordering to prevent deadlocks.
    • Add concurrent registration and inference tests for multiple devices.
    • Add a stress test that detects lost updates and corrupted output.
    • Measure lock contention and reduce unnecessarily broad critical sections.
  • Acceptance criteria:
    • Shared registries, counters, CSV writers, and profiles have defined ownership or synchronization.
    • A concurrent multi-client test runs without corrupting state or output files.
    • Lock scope does not serialize all inference work unnecessarily.

SCIOT-011 — Define model and interpreter concurrency behavior

  • Status: BACKLOG
  • Priority: P1
  • Value: Ensure cached interpreters are safe and performant under load.
  • Task breakdown:
    • Document TensorFlow Lite interpreter thread-safety constraints.
    • Choose between per-layer locks, per-worker interpreters, or an interpreter pool.
    • Key caches by model identity, model version, layer, and execution backend.
    • Define interpreter creation, warm-up, reuse, and failure recovery.
    • Add cache size limits and an eviction or shutdown policy.
    • Ensure model reloads cannot reuse stale interpreters.
    • Add concurrent prediction tests that compare results with serial execution.
    • Add tests for simultaneous use of different models and layers.
    • Benchmark throughput and memory for the selected strategy.
    • Document the concurrency guarantees for transport handlers.
  • Acceptance criteria:
    • Interpreter access is isolated or synchronized per model layer.
    • Load tests verify correct predictions with concurrent clients.
    • Cache limits and cleanup behavior are documented.

Epic 4: Offloading Algorithm and Adaptation

SCIOT-012 — Include edge-only execution in optimization

  • Status: BACKLOG
  • Priority: P1
  • Value: Evaluate every meaningful execution strategy.
  • Problem: Edge-only cost is calculated but does not participate in selecting the minimum.
  • Task breakdown:
    • Define the protocol representation for edge-only execution.
    • Define one candidate-cost structure shared by edge-only, mixed, and device-only strategies.
    • Include edge-only in the minimum-cost comparison.
    • Define deterministic tie-breaking between strategies.
    • Validate that network input cost is included correctly for edge-only execution.
    • Handle unavailable or incomplete timing estimates explicitly.
    • Add unit tests for slow device, slow edge, slow network, and tied costs.
    • Add integration coverage proving the client follows an edge-only decision.
    • Expose the selected strategy and candidate costs in diagnostics.
    • Document the new decision and protocol semantics.
  • Acceptance criteria:
    • Device-only, mixed, and edge-only strategies use the same cost comparison.
    • Unit tests cover slow-device, slow-edge, and slow-network cases.
    • The representation of edge-only execution is documented in the protocol.

SCIOT-013 — Integrate variance detection with adaptation

  • Status: BACKLOG
  • Priority: P2
  • Value: Make collected variance information affect runtime behavior.
  • Problem: Variance is recorded, but production code does not call the retest decision.
  • Task breakdown:
    • Define when variance is considered actionable rather than merely observable.
    • Move variance histories and flags into per-device state.
    • Define which layer measurements are invalidated after a variance event.
    • Define the local or partial retest request sent to the client.
    • Add cooldown, hysteresis, and minimum-sample rules to prevent repeated retests.
    • Clear or age variance state after successful stable measurements.
    • Connect variance events to the next offloading decision.
    • Add deterministic tests for stable, noisy, step-change, and recovery sequences.
    • Add a simulation showing the split adapts after a performance change.
    • Document the adaptation policy and tunable parameters.
  • Acceptance criteria:
    • A documented policy explains how unstable layer measurements trigger refreshes.
    • Variance state is maintained per device.
    • Stable measurements do not cause unnecessary full local inference.
    • Tests demonstrate adaptation after a simulated performance change.

SCIOT-014 — Improve network-cost estimation

  • Status: BACKLOG
  • Priority: P2
  • Value: Produce more stable split decisions from real network measurements.
  • Task breakdown:
    • Define separate metrics for round-trip latency, upload throughput, and server processing time.
    • Create a per-device network-estimator component.
    • Define warm-up defaults before enough samples are available.
    • Smooth measurements with a documented window or exponential average.
    • Reject negative, zero-duration, impossible, and non-finite samples.
    • Expire stale network measurements.
    • Add outlier handling and confidence information.
    • Feed the estimator output into the offloading cost model.
    • Add tests for zero, negative, extreme, stale, and noisy measurements.
    • Add a simulation that measures decision oscillation under small variations.
  • Actions:
    • Separate upload throughput, round-trip latency, and payload processing time.
    • Smooth measurements and reject impossible or stale values.
    • Define behavior before enough samples exist.
  • Acceptance criteria:
    • Unit tests cover zero, negative, extreme, and noisy measurements.
    • The chosen split does not oscillate excessively under small variations.

SCIOT-015 — Persist and inspect offloading decisions

  • Status: DONE
  • Priority: P2
  • Value: Make algorithm behavior explainable.
  • Task breakdown:
    • Define a versioned offloading-decision event schema.
    • Include device, model, timestamp, measurement confidence, and current strategy.
    • Record every candidate strategy and its device, network, and edge cost components.
    • Record the selected strategy, layer, tie-break, and fallback reason.
    • Add one persistence sink with bounded asynchronous writes.
    • Define file rotation, retention, and failure behavior.
    • Add correlation IDs linking request, prediction, and decision.
    • Update analysis tooling to compare estimated and observed latency.
    • Add tests for event completeness and persistence failures.
    • Document the schema and privacy implications.
  • Acceptance criteria:
    • Each decision records candidate costs and the reason for the selected layer.
    • Logs include device ID, model, network estimate, and selected strategy.
    • Analysis tools can compare predicted and observed latency.
  • Implementation notes:
    • Added server.telemetry.offloading_decisions with offloading-decision.v1 JSONL events.
    • OffloadingAlgo now exposes candidate cost components without changing selected-layer behavior.
    • RequestHandler persists decision events through the existing background I/O queue to data/results/offloading_decisions.jsonl.
    • Documented the schema and privacy notes in docs/OFFLOADING_DECISION_EVENTS.md.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_offloading_decision_events.py tests/unit/test_background_io_queue.py tests/unit/test_offloading_decision_summary.py tests/unit/test_policy_docs.py tests/test_offloading_algo -q

Epic 5: Model Management

SCIOT-016 — Make model initialization lazy

  • Status: BACKLOG
  • Priority: P1
  • Value: Reduce server startup time and memory usage.
  • Problem: Startup loads and executes every configured model, and models may later be loaded again.
  • Task breakdown:
    • Measure current startup time, peak memory, and per-model initialization cost.
    • Separate static model metadata generation from runtime server startup.
    • Define a model manifest containing hashes, layer counts, sizes, and valid split points.
    • Load manifests and registration hashes without loading Keras models.
    • Load a model and its interpreters only on first use.
    • Coordinate concurrent first-use requests so a model is loaded once.
    • Define cache retention, preload options, and unload behavior.
    • Handle missing or invalid manifests with actionable errors.
    • Add tests for startup without model execution and first-use loading.
    • Compare startup and first-inference measurements with the baseline.
  • Acceptance criteria:
    • Model metadata can be prepared offline or loaded without executing every layer at startup.
    • A model is fully loaded only when first requested.
    • Startup and first-inference times are measured.

SCIOT-017 — Create a supported model preparation command

  • Status: BACKLOG
  • Priority: P1
  • Value: Replace undocumented and obsolete model-generation instructions.
  • Task breakdown:
    • Define supported source model formats and TensorFlow/Keras versions.
    • Design the CLI command, arguments, exit codes, and output layout.
    • Validate source model input/output shapes and supported layer types.
    • Split the model into executable layer or subgraph artifacts.
    • Detect skip connections and generate valid split-point metadata.
    • Generate activation sizes, hashes, model identity, and version metadata.
    • Verify adjacent generated fragments have compatible tensors.
    • Write outputs atomically and clean partial output on failure.
    • Add fixture-model success tests and invalid-model failure tests.
    • Add --help, examples, and migration documentation.
  • Actions:
    • Provide one CLI for validating and splitting a model.
    • Generate TFLite fragments, activation sizes, hashes, and valid split points.
    • Validate skip connections and tensor compatibility.
  • Acceptance criteria:
    • The command has --help and documented inputs and outputs.
    • Generated artifacts can be registered and used by server and client.
    • Invalid models fail with actionable errors.

SCIOT-018 — Define model artifact storage

  • Status: IN PROGRESS
  • Priority: P1
  • Value: Reduce repository size and simplify model versioning.
  • Task breakdown:
    • Inventory model artifacts, duplicates, file sizes, and current consumers.
    • Compare Git LFS, release assets, and external artifact storage against project needs.
    • Record the selected storage approach in the decision log.
    • Define a versioned model manifest with URLs, checksums, sizes, and compatibility metadata.
    • Add a reproducible model-fetch and verification command.
    • Update server and client setup to locate fetched artifacts.
    • Migrate required models and preserve any minimal test fixtures in Git.
    • Remove duplicate generated artifacts from the normal source tree.
    • Add CI verification for manifest integrity and missing artifacts.
    • Document offline use, cache location, updates, and rollback.
  • Actions:
    • Choose Git LFS, release assets, or an external artifact store.
    • Add checksums and model-version metadata.
    • Remove generated duplicates from normal source history where practical.
  • Acceptance criteria:
    • A clean source checkout is reasonably sized.
    • Required models can be fetched reproducibly.
    • Source code and model artifacts have independently traceable versions.
  • Implementation notes:
    • Added docs/MODEL_ARTIFACTS.md with artifact classes, target storage approach, manifest fields, local cache/offline expectations, and a rule against adding new large binaries to Git.
    • Remaining work: inventory actual artifact sizes/consumers, implement fetch/verify command, migrate artifacts, and add CI verification.

Epic 6: Testing and Continuous Integration

SCIOT-019 — Repair the CI workflow

  • Status: DONE
  • Priority: P0
  • Depends on: SCIOT-001
  • Value: Restore trustworthy automated feedback.
  • Problem: CI references outdated test paths and an invalid coverage package.
  • Task breakdown:
    • Inventory current test locations, dependency profiles, and required services.
    • Replace obsolete test paths in the GitHub Actions workflow.
    • Install the exact server, client, and test extras required by each job.
    • Split fast unit/resilience tests from slower integration and model tests.
    • Configure model- or hardware-dependent tests to use fixtures or explicit skips.
    • Correct coverage package targets and remove nonexistent modules.
    • Publish readable test and coverage artifacts.
    • Add dependency and uv.lock consistency checks.
    • Add caching without allowing stale environments to hide lockfile changes.
    • Make required jobs fail on collection, test, or coverage-command errors.
    • Document the local commands that reproduce every CI job.
  • Acceptance criteria:
    • CI runs the current unit and integration test locations.
    • Test collection succeeds on Python 3.11.
    • Coverage targets importable project packages.
    • CI fails on test failures.
  • Verification evidence:
    • GitHub Actions now has separate dependency-check, fast test, and HTTP integration jobs.
    • Workflow installs from uv.lock with uv sync --frozen --extra server --extra client-static --extra test.
    • Obsolete test paths and the nonexistent server_client_light coverage target were removed.
    • Coverage now targets importable client and server packages and uploads per-lane XML plus JUnit artifacts.
    • uv.lock consistency is checked with uv lock --check; cache keys include the uv.lock hash.
    • Model-artifact and external-server accuracy tests are explicit opt-in suites using SCIOT_RUN_MODEL_TESTS=1 and SCIOT_RUN_ACCURACY_TESTS=1.
    • Local reproduction commands are documented in DEVELOPER_GUIDE.md.
    • Workflow YAML parses successfully.
    • Local fast lane reports 61 passed; local HTTP integration lane reports 8 passed; opt-in model/accuracy suites report 9 skipped without env vars.

SCIOT-020 — Replace placeholder resilience tests

  • Status: DONE
  • Priority: P1
  • Value: Verify actual client behavior rather than duplicated local examples.
  • Task breakdown:
    • Make the production HTTP client safe to import without starting its main loop.
    • Extract production helpers for local fallback and timed reconnection.
    • Preserve failed registration state instead of forcing the server state to available.
    • Replace simulated registration and timeout tests with calls to production functions.
    • Cover failed image and inference-result uploads.
    • Cover complete local inference and the server -1 local-only decision.
    • Cover reconnect throttling, failed reconnect, and successful recovery.
    • Remove unconditional placeholder assertions.
    • Isolate unrelated MQTT fixtures from focused client tests.
  • Acceptance criteria:
    • Tests import and exercise production client functions.
    • Timeout, failed upload, local fallback, and reconnection paths are covered.
    • No assertion is an unconditional placeholder.
  • Verification evidence:
    • 10 focused resilience tests pass against client.python.http_client.
    • Focused coverage reports 39% of the complete client module, including the resilience paths.
    • The combined focused regression run reports 17 passed.
    • Direct production client execution remained active for the agreed 15-second runtime check.
    • Importing the production client no longer starts main() or eagerly imports TensorFlow.

SCIOT-021 — Add end-to-end HTTP testing

  • Status: DONE
  • Priority: P1
  • Value: Protect the primary supported workflow.
  • Task breakdown:
    • Add an isolated FastAPI HTTP server fixture.
    • Add a deterministic two-layer fixture model.
    • Route the production client through an in-process requests-compatible session.
    • Verify registration and the initial safe full-local decision.
    • Verify a subsequent split decision after server measurements.
    • Upload a production-format intermediate tensor and consume the prediction response.
    • Verify complete local execution while the server is offline.
    • Verify registration recovery when the server becomes available.
  • Acceptance criteria:
    • A test starts an isolated server with a small fixture model.
    • A client registers, receives a split, uploads an intermediate tensor, and receives a prediction.
    • Local-only and server-recovery scenarios are included.
  • Verification evidence:
    • The end-to-end suite uses production client serialization and production FastAPI routes.
    • Full-local fixture inference returns [4.0, 6.0].
    • Split execution sends the layer-0 tensor [2.0, 3.0] and receives [4.0, 6.0].
    • Offline local inference and timed registration recovery pass.
    • The real edge server remained active for the agreed 15-second startup check.

SCIOT-022 — Add protocol and malformed-input tests

  • Status: DONE
  • Priority: P1
  • Depends on: SCIOT-007
  • Task breakdown:
    • Extract binary inference decoding into a model-independent module.
    • Use fixed little-endian field encoding in the production client.
    • Validate payload, device ID, tensor, timing, and layer bounds.
    • Reject truncated payloads.
    • Reject oversized declared tensors.
    • Reject invalid UTF-8 device IDs.
    • Reject tensors with an invalid model shape.
    • Reject unexpected trailing or malformed fields through the bounded reader.
    • Return structured 400 responses without internal exception details.
    • Return generic structured 500 responses for unexpected processing errors.
  • Acceptance criteria:
    • Truncated, oversized, invalid-encoding, and wrong-shape messages are rejected safely.
    • Error responses do not expose internal stack traces.
  • Verification evidence:
    • Four required malformed-payload cases return the same structured HTTP 400 response.
    • A valid golden payload round-trips through production decoding.
    • An injected internal exception returns a generic HTTP 500 without its secret message or traceback.
    • HTTP routes no longer return detail=str(exception) or print stack traces into responses.
    • HTTP end-to-end, protocol, and resilience suites report 18 passed.
    • The protocol is bounded and endian-stable; protocol versioning remains tracked by SCIOT-007.

SCIOT-023 — Add performance regression baselines

  • Status: DONE
  • Priority: P2
  • Value: Detect changes that significantly worsen startup, inference, memory, or throughput.
  • Task breakdown:
    • Define functional microbenchmarks separately from hardware benchmarks.
    • Choose stable metrics for startup, first inference, steady-state latency, throughput, and peak memory.
    • Define an environment metadata schema including CPU, OS, Python, TensorFlow, model, and commit.
    • Create deterministic tiny-model benchmarks suitable for CI.
    • Define laptop and Raspberry Pi benchmark procedures outside mandatory CI.
    • Establish initial baseline data and ownership for updating it.
    • Define generous regression thresholds and statistical comparison rules.
    • Add warm-up, repetition, timeout, and noise-control rules.
    • Produce machine-readable results and a concise human report.
    • Add CI reporting that warns or fails according to the agreed threshold policy.
    • Document how intentional performance changes update the baseline.
  • Acceptance criteria:
    • Benchmarks distinguish functional tests from hardware-dependent measurements.
    • Baseline results include environment metadata.
    • CI uses generous, documented regression thresholds where appropriate.
  • Implementation notes:
    • Added docs/PERFORMANCE_BASELINES.md with benchmark classes, stable metrics, environment metadata, generous threshold policy, and baseline-update guidance.
    • Added scripts/benchmark_report.py with report-only functional microbenchmarks and sciot-benchmark-report.v1 JSON output.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_benchmark_report.py tests/unit/test_policy_docs.py -q

Epic 7: Packaging and Repository Hygiene

SCIOT-024 — Split dependencies by use case

  • Status: DONE
  • Priority: P1
  • Value: Avoid installing camera, dataset, plotting, and testing packages for every deployment.
  • Task breakdown:
    • Inventory dependencies used by server, static client, laptop client, Raspberry Pi client, analysis, research, tests, and development.
    • Define separate uv extras for each runtime and tooling profile.
    • Remove duplicate dependency declarations from the base installation.
    • Add platform-specific TensorFlow markers.
    • Keep Picamera2 as a Raspberry Pi OS system dependency.
    • Regenerate the lockfile.
    • Verify clean base, server, static-client, and laptop-client environments.
    • Add dependency-profile regression assertions.
    • Document isolated and combined installation commands.
  • Notes: Completed by SCIOT-000; retained as a separately traceable packaging story.
  • Actions:
    • Define server, client, Raspberry Pi, benchmark, analysis, test, and development extras.
    • Remove duplicate dependency declarations.
    • Validate installation combinations.
  • Acceptance criteria:
    • A headless server installs without picamera2 or GUI-heavy analysis packages.
    • A standard client installs only its required runtime.
    • Documented uv commands exist for each supported profile.

SCIOT-025 — Separate source, generated data, and research results

  • Status: DONE
  • Priority: P1
  • Value: Make the repository easier to navigate and maintain.
  • Task breakdown:
    • Inventory tracked and untracked generated logs, plots, profiles, datasets, models, and simulation outputs.
    • Classify each path as source, fixture, generated output, research archive, or external artifact.
    • Define the canonical directory layout for runtime data and generated results.
    • Centralize output-path configuration instead of embedding paths in scripts.
    • Update .gitignore for reproducible generated outputs.
    • Preserve only small deterministic fixtures required by tests.
    • Move historical research snapshots outside the main runtime tree.
      • Completed as a non-disruptive classification: existing historical folders are documented as read-only research archives; new generated outputs are directed to data/results/, data/runtime_inputs/, and logs/.
    • Add a cleanup command or documented cleanup procedure.
    • Add CI checks that reject common generated artifacts and oversized accidental files.
    • Document the migration and where users find new outputs.
  • Acceptance criteria:
    • Generated logs, plots, profiler data, and simulation results have defined output directories.
    • Generated artifacts are ignored unless intentionally maintained as fixtures.
    • Research snapshots are archived outside the main runtime tree.
  • Implementation notes:
    • Added docs/REPOSITORY_LAYOUT.md with the path classification, canonical generated-output directories, cleanup procedure, and migration notes.
    • Added server.commons.RuntimePaths and moved evaluation/runtime-input defaults out of src/server.
    • Added scripts/check_repository_artifacts.py --changed to reject newly added generated artifacts and oversized files in CI.
    • Added tests/unit/test_repository_layout.py to keep the policy, path defaults, ignore rules, and CI guardrail from drifting.

SCIOT-026 — Normalize configuration

  • Status: DONE
  • Priority: P1
  • Value: Prevent configuration drift and hidden defaults.
  • Task breakdown:
    • Inventory every configuration file, key, default, and code reader.
    • Identify duplicated, unused, conflicting, and differently named settings.
    • Define canonical server and client configuration schemas.
    • Add typed parsing and validation with actionable field-level errors.
    • Validate model references, ports, hostnames, probabilities, delays, and transport combinations.
    • Define environment-variable and CLI override precedence.
    • Provide minimal and full example configuration files.
    • Add a migration layer or migration guide for old keys.
    • Remove committed and workflow-generated .backup configuration files.
    • Add valid/invalid configuration tests and startup error tests.
    • Update all scripts and documentation to use canonical terminology.
  • Actions:
    • Remove unused or duplicated settings.
    • Add typed parsing and validation.
    • Provide checked-in example configurations.
    • Eliminate ad hoc .backup files from normal workflows.
  • Acceptance criteria:
    • Invalid model names, ports, probabilities, and delay settings fail validation.
    • Server and client configurations use consistent terminology.
  • Notes: Addresses Issue #8 (singleton) - validation complete, pending singleton pattern implementation.
  • Links to Issues: #8

SCIOT-027 — Introduce stable command-line entry points

  • Status: DONE
  • Priority: P2
  • Value: Replace path-dependent script execution with discoverable commands.
  • Task breakdown:
    • Define the supported command set and command ownership.
    • Move script startup logic into importable main() functions.
    • Add [project.scripts] entry points in pyproject.toml.
    • Create a shared argument parser style and consistent exit codes.
    • Support explicit configuration paths.
    • Support safe CLI overrides for host, port, device ID, model, and run limits.
    • Resolve project resources independently of the current working directory.
    • Add structured startup validation and actionable command errors.
    • Add command help, invocation, exit-code, and outside-repository tests.
    • Update documentation and deprecate direct path-based script commands.
  • Supported commands:
    • sciot-server
    • sciot-client
  • Future candidate commands:
    • sciot-simulate
    • sciot-analyze
  • Acceptance criteria:
    • Commands provide --help.
    • Configuration paths and overrides can be supplied explicitly.
    • Commands work from outside the repository root.
  • Implementation notes:
    • Added the supported command set: sciot, sciot-server, and sciot-client.
    • sciot-server owns edge-server startup and validation; sciot-client owns the static-image HTTP client.
    • Simulation and analysis commands are intentionally not declared as stable commands yet.
    • Added --config and --validate-config to both runtime commands.
    • Added safe overrides for server host, port, HTTP model, transports, verbosity, client device ID, server address, max cycles, and run duration.
    • Added SCIOT_CLIENT_CONFIG support for the static client and a config_path argument for server startup.
    • README now uses stable CLI commands instead of direct path-based script execution.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_cli_entrypoints.py -q
    • .venv/bin/python -m pytest tests/unit/test_config_validation.py tests/unit/test_cli_entrypoints.py tests/unit/test_inference_protocol.py tests/integration/test_http_protocol_validation.py -q

SCIOT-028 — Make EMA alpha and offloading parameters configurable

  • Status: BACKLOG
  • Priority: P2
  • Value: Enable tuning of offloading algorithm without code changes.
  • Problem: Offloading algorithm uses hard-coded EMA alpha (0.5) and other tunable parameters.
  • Task breakdown:
    • Add offloading_algo.ema_alpha to configuration schema.
    • Add other tunable parameters (thresholds, window sizes).
    • Update config.py validation to include these fields.
    • Replace hard-coded values with config lookups.
    • Add documentation for parameter tuning.
  • Acceptance criteria:
    • EMA alpha configurable via settings.yaml.
    • All offloading parameters tunable without code changes.
  • Notes: Implements Issue #9; relates to SCIOT-031 pluggable algorithms.
  • Links to Issues: #9

SCIOT-029 — Replace print statements with structured logging

  • Status: BACKLOG
  • Priority: P2
  • Value: Enable proper log levels and observability.
  • Problem: Code uses print() statements instead of structured logging.
  • Task breakdown:
    • Audit all print() calls in source code.
    • Replace with structured_logger calls (DEBUG/INFO/WARNING/ERROR).
    • Add log level configuration support.
    • Add tests for log output format.
  • Acceptance criteria:
    • No print() calls in production code paths.
    • Log levels configurable via environment or config file.
  • Notes: Implements Issue #12; structured_logger.py exists.

SCIOT-030 — Fix type annotations in MessageData

  • Status: DONE
  • Priority: P3
  • Value: Enable static type checking and IDE assistance.
  • Problem: MessageData.get_latency return type mismatch.
  • Task breakdown:
    • Verify actual return type in src/server/communication/message_data.py.
    • Fix annotation to match implementation.
    • Add type-checking tests.
  • Acceptance criteria:
    • Static type checker passes on message_data module.
    • Return type consistent across codebase.
  • Notes: Implements Issue #13; ready for quick PR.
  • Implementation notes:
    • MessageData.get_latency() now advertises float, matching the actual duration value returned by the implementation.
    • Added regression coverage in tests/unit/test_message_data.py.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_message_data.py -q

Epic 8: Documentation and Observability

SCIOT-029 — Document the architecture and data flow

  • Status: DONE
  • Priority: P2
  • Value: Provide clear architectural documentation.
  • Problem: Architecture documentation was scattered and incomplete.
  • Task breakdown:
    • Define the current system boundary and supported deployment topologies.
    • Document client, transport, request-handler, model-manager, offloading, state, and telemetry responsibilities.
    • Add a component diagram with ownership boundaries.
    • Add sequence diagrams for registration, local-only inference, split inference, and reconnection.
    • Document model-hash matching and artifact discovery.
    • Document split selection inputs and strategy outputs.
    • Show ownership and lifetime of device state, model caches, interpreter caches, and background writers.
    • Link the binary protocol, configuration schema, and model artifact layout.
    • Record important architectural decisions and known experimental areas.
    • Add a documentation update checklist to future architecture-changing stories.
  • Acceptance criteria:
    • Documentation covers registration, model matching, split selection, inference, fallback, and telemetry.
    • Ownership of device state and model caches is shown.
    • The binary protocol and model artifact layout are linked.
  • Implementation notes:
    • Added docs/ARCHITECTURE.md with current boundaries, supported topologies, component ownership, server process topology, model-hash matching, split selection, telemetry, state/cache lifetime, and known experimental areas.
    • Added mermaid component and sequence diagrams for registration, split inference, and local-only/reconnect behavior.
    • Linked architecture documentation from README.
    • Added a documentation guard test to keep required sections and cross-doc links present.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_architecture_docs.py -q

SCIOT-030 — Consolidate metrics and dashboards

  • Status: DONE
  • Priority: P2
  • Value: Make runtime measurements consistent and useful.
  • Task breakdown:
    • Inventory every metric, unit, timestamp source, output file, and dashboard consumer.
    • Define a canonical versioned telemetry event schema.
    • Standardize units and names for acquisition, device compute, network, edge compute, total latency, memory, and temperature.
    • Include device ID, model ID/version, request ID, split strategy, and timestamps in every event.
    • Define one bounded telemetry writer and storage layout.
    • Make dashboard startup and empty-data states safe.
    • Add per-device, per-model, and time-range filters.
    • Add predicted-versus-observed offloading cost views.
    • Migrate analysis and simulation scripts to the canonical schema.
    • Remove dependence on obsolete global timing files.
    • Add schema, empty-state, multi-device, and migration tests.
    • Document retention, privacy, and dashboard startup commands.
  • Actions:
    • Define canonical metrics and units.
    • Make dashboard startup safe when no evaluation data exists. Done for empty/missing evaluation, inference-time, and image files.
    • Use per-device and per-model views. Done for device/model selectors and row windows when telemetry includes those fields.
    • Remove dependence on obsolete global timing files. Done for dashboard startup: missing legacy files render empty states instead of failing.
  • Acceptance criteria:
    • The dashboard starts before the first inference.
    • Metrics identify their device, model, timestamp, and execution strategy.
    • Analysis scripts consume the same schema used by runtime telemetry.
  • Implementation notes:
    • Added server.web.dashboard_data with safe loaders for empty, missing, corrupt, or older dashboard inputs.
    • Updated the Streamlit dashboard to render empty states before the first inference instead of failing at import time.
    • Added model and row-window filters alongside the existing device filter.
    • Added predicted-vs-observed offloading cost chart/table from offloading-decision.v1 events.
    • Documented dashboard telemetry schemas, metric inventory, startup command, retention, and privacy in docs/DASHBOARD_TELEMETRY.md.
    • Remaining work: migrate simulation scripts and remove remaining dashboard reliance on legacy global timing files.
  • Verification:
    • .venv/bin/python -m pytest tests/unit/test_dashboard_data.py tests/unit/test_policy_docs.py tests/unit/test_offloading_decision_summary.py -q

SCIOT-031 — Make offloading algorithms pluggable

  • Status: BACKLOG
  • Priority: P1
  • Value: Allow the edge server to support multiple split-selection strategies without rewriting request handling.
  • Problem: OffloadingAlgo currently mixes shared cost helpers, cache handling, and the concrete static offloading strategy in one directly-instantiated class.
  • Task breakdown:
    • Define an abstract/base offloading algorithm class that owns only shared inputs, validation, caches, and helper methods.
    • Prevent the base class from being used directly for split decisions.
    • Move the current static offloading behavior into a concrete subclass.
    • Define a stable algorithm interface, for example select_split() returning an offloading_layer_index.
    • Add an algorithm registry/factory that maps configuration names to concrete algorithm classes.
    • Add server configuration for choosing the offloading algorithm, with validation and a safe default matching current behavior.
    • Update RequestHandler to request algorithms through the factory instead of instantiating OffloadingAlgo directly.
    • Document how to add a new offloading algorithm and which methods must be implemented.
    • Add tests proving the base class cannot be selected directly, the default algorithm preserves current behavior, and an alternate test algorithm can be selected by configuration.
  • Acceptance criteria:
    • The current offloading behavior is preserved when no explicit algorithm is configured.
    • The base algorithm class cannot be instantiated/used as a runtime strategy.
    • A server config value selects the algorithm used for split decisions.
    • Adding a new algorithm requires a subclass plus a registry entry, not changes to request-handler control flow.
  • Notes:
    • This should be done before adding more strategy variants such as edge-only-inclusive, adaptive variance-aware, or experimental network-cost algorithms.
    • Likely concrete class names: BaseOffloadingAlgorithm, StaticOffloadingAlgorithm, and OffloadingAlgorithmFactory.

SCIOT-032 — Add service discovery with mDNS/Zeroconf

  • Status: IN PROGRESS
  • Priority: P2
  • Owner: Unassigned
  • Estimate: Unestimated
  • Depends on: SCIOT-026
  • Value: Let clients discover SCIoT servers on the local network without hard-coded IP addresses or subnet-wide probing.
  • Problem: The current HTTP client uses a fixed server_host by default and falls back to active /24 subnet scanning only when the host is unset. That scan can be slow, noisy on managed networks, and brittle when the local network is not a simple /24.
  • Task breakdown:
    • Define the SCIoT local service name, for example _sciot._tcp.local..
    • Add optional mDNS/Zeroconf advertisement on the server side.
    • Add optional mDNS/Zeroconf lookup on the Python HTTP client side.
    • Preserve explicit static server_host as the highest-priority configuration path.
    • Keep the current subnet scan as an optional fallback when mDNS is disabled, unavailable, or times out.
    • Add validated configuration fields such as discovery.enabled, discovery.method, discovery.timeout_seconds, and fallback controls.
    • Handle multiple discovered SCIoT servers deterministically.
    • Document network requirements, troubleshooting, and how discovery interacts with static configuration.
    • Add tests for static host precedence, mDNS success, mDNS timeout, multiple-server selection, subnet-scan fallback, and local-only fallback.
  • Acceptance criteria:
    • A client can discover a SCIoT server on the same LAN without a fixed IP address.
    • Existing static-IP configuration continues to work unchanged.
    • Discovery can be disabled for locked-down or production environments.
    • Discovery order is deterministic: static config, then mDNS/Zeroconf, then optional subnet scan, then local-only mode.
  • Notes:
    • Prefer mDNS/Zeroconf before broadcast discovery because it is a standard fit for named local services and avoids broad host probing.
    • This story depends on normalized configuration so the discovery settings are validated consistently.
  • Implementation notes:
    • Added docs/SERVICE_DISCOVERY.md with service name, discovery order, proposed config fields, deterministic multiple-server selection, and troubleshooting notes.
    • Remaining work: add validated config fields and implement server advertisement/client lookup.

SCIOT-033 — Add bounded queue to MQTT task_queue

  • Status: BACKLOG
  • Priority: P2
  • Value: Prevent unbounded memory growth under backpressure.
  • Problem: MQTT task_queue uses unbounded queue.Queue() which can grow indefinitely if producer outpaces consumer.
  • Task breakdown:
    • Define bounded queue size with backpressure behavior.
    • Add queue-full handling (drop oldest, block, or reject new work).
    • Add metrics for queue depth and dropped tasks.
    • Add tests for queue-full scenarios.
  • Acceptance criteria:
    • Queue has defined maximum size.
    • Behavior under backpressure is documented.
    • No memory leak from unbounded growth.
  • Notes: Follows SCIOT-005 MQTT repair; relates to Issue #6.

SCIOT-034 — Consolidate file I/O in offloading path

  • Status: BACKLOG
  • Priority: P1
  • Value: Reduce redundant file reads for offloading decisions.
  • Problem: Offloading path still reads device_inference_times.json and edge_inference_times.json unnecessarily in some code paths.
  • Task breakdown:
    • Audit all file reads in request_handler.py and offloading code.
    • Remove redundant JSON parsing after SCIOT-009 implementation.
    • Use in-memory device state for all inference timing queries.
    • Add performance test for repeated offloading decisions.
  • Acceptance criteria:
    • Offloading decisions do not read files after initial load.
    • Two devices maintain independent timing state.
  • Notes: Completes Issue #3 work.

SCIOT-035 — Add explicit time breakdown and profiling dashboard

  • Status: BACKLOG
  • Priority: P2
  • Depends on: SCIOT-030
  • Value: Make performance analysis accessible to developers and operators.
  • Problem: Time breakdown exists in plot_results.py but uses Italian labels and is not integrated with runtime telemetry.
  • Task breakdown:
    • Translate Italian labels to English in analysis scripts.
    • Connect profiler.py output to SCIOT-030 telemetry schema.
    • Add dashboard view for phase timing breakdown.
    • Add client-side timing breakdown for local/edge phases.
  • Acceptance criteria:
    • All labels use English terminology.
    • Dashboard shows device, network, and edge timing phases.
  • Notes: Implements Issue #16.

SCIOT-036 — Implement mobile cross-platform client architecture

  • Status: BACKLOG
  • Priority: P3
  • Depends on: SCIOT-024, SCIOT-032
  • Value: Enable mobile/tablet deployment for SCIoT clients.
  • Problem: No cross-platform mobile architecture exists; clients limited to Python-capable devices.
  • Task breakdown:
    • Define cross-platform architecture (Flutter/React Native/UniApp).
    • Implement camera abstraction layer for iOS (AVFoundation) and Android (CameraX).
    • Implement inference engine abstraction (CoreML/TFLite).
    • Add WebSocket/MQTT/HTTP transport clients.
    • Define offloading decision protocol for mobile.
  • Acceptance criteria:
    • Mobile client connects to SCIoT server.
    • Cross-platform architecture documented.
  • Notes: Implements Issue #23; see #24, #25, #27, #28, #29.

SCIOT-037 — Implement ESP32 client for SCIoT

  • Status: BACKLOG
  • Priority: P3
  • Depends on: SCIOT-024, SCIOT-036
  • Value: Enable microcontroller deployment for edge inference.
  • Problem: No ESP32 client exists; requires different toolchain and memory constraints.
  • Task breakdown:
    • Define ESP32 C++ project structure.
    • Implement TensorFlow Lite Micro inference.
    • Add camera driver (OV2640).
    • Implement HTTP transport client.
    • Add offloading decision handling.
    • Add power/battery management.
  • Acceptance criteria:
    • ESP32 captures image, runs inference, and can receive offloading decision.
    • Low memory footprint verified.
  • Notes: Implements Issue #19; see #30, #31.

Suggested Initial Iterations

Iteration 1 — Make the primary path runnable

  • SCIOT-001 — Resolve TensorFlow/Keras compatibility.
  • SCIOT-003 — Correct the standard client lifecycle.
  • SCIOT-019 — Repair the CI workflow.

Iteration goal: The primary HTTP path can start cleanly, run with bounded client lifecycle controls, and pass trustworthy CI feedback.

Iteration 2 — Protect the HTTP workflow

  • SCIOT-007 — Version and validate the binary protocol.
  • SCIOT-020 — Replace placeholder resilience tests.
  • SCIOT-021 — Add end-to-end HTTP testing.
  • SCIOT-022 — Add malformed-input tests.

Iteration goal: The primary HTTP workflow has an explicit contract and reliable automated coverage.

Iteration 3 — Support multiple devices safely

  • SCIOT-009 — Introduce per-device runtime state.
  • SCIOT-010 — Make shared state thread-safe.
  • SCIOT-011 — Define interpreter concurrency behavior.
  • SCIOT-014 — Improve network-cost estimation.

Iteration goal: Multiple clients operate concurrently without sharing measurements or corrupting state.

Iteration Planning

Current Iteration

Goal: Add iteration goal here.

Start date:
End date:

ID Item Owner Estimate Status Notes

Review Notes

  • Demonstrated:
  • Accepted:
  • Returned to backlog:
  • Metrics or evidence:

Retrospective

  • What worked:
  • What was difficult:
  • What should change next iteration:
  • Agreed experiment:

New Backlog Item Template

Copy this section when adding an item:

### SCIOT-XXX — Short action-oriented title

- **Status:** BACKLOG
- **Priority:** P0 | P1 | P2 | P3
- **Owner:** Unassigned
- **Estimate:** Unestimated
- **Depends on:** None
- **Value:** Why this matters to users, operators, or maintainers.
- **Problem:** What is currently wrong or missing.
- **Task breakdown:**
  - [ ] First concrete, independently verifiable task.
  - [ ] Second concrete, independently verifiable task.
- **Acceptance criteria:**
  - Observable outcome.
  - Automated or repeatable verification.
- **Notes:** Relevant decisions, links, risks, or constraints.

Decision Log

Record decisions that affect several backlog items.

Date Decision Reason Consequences

Risks and Blockers

Risk or blocker Impact Mitigation Owner Status
TensorFlow/Keras runtime incompatibility Primary server and tests cannot start Complete SCIOT-001 first Unassigned Open
Large committed artifacts Slow clones and difficult model versioning Complete SCIOT-018 and SCIOT-025 Unassigned Open

SCIOT-038 — Eliminate file re-read in offloading decision path

  • Status: BACKLOG
  • Priority: P2
  • Value: Reduces I/O overhead in critical offloading path.
  • Problem: Partially Complete (SCIOT-009). Remaining: remove redundant JSON parsing for offloading history.
  • Task breakdown:
    • Read offloading history from in-memory cache instead of re-parsing JSON
    • Add cache invalidation in src/sciot/offloading.py when history updates
    • Remove redundant file read in _check_offloading_needed() method
    • Add performance test in tests/performance/test_offloading_cache.py
  • Acceptance criteria:
    • No file re-read for offloading decisions
    • All tests pass (uv run pytest -q)
    • Performance improvement measured
  • Links to Issues: #3

SCIOT-039 — Add bound to MQTT task_queue

  • Status: BACKLOG
  • Priority: P1
  • Value: Prevents memory exhaustion in continuous inference scenarios.
  • Problem: Status: Partially addressed. Bounded writer added (SCIOT-015 scope) but MQTT queue.Queue() still unbounded. Low priority - overflow unlikely in practice.
  • Task breakdown:
    • Add max_queue_size parameter to MQTTClient.__init__() in src/sciot/mqtt.py
    • Replace queue.Queue() with asyncio.Queue(maxsize=N) in MQTT publish path
    • Add queue_full callback for monitoring when queue hits limit
    • Add tests in tests/unit/test_mqtt_client.py for bounded queue behavior
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #6

SCIOT-040 — Consolidate config loading into a singleton Config class

  • Status: BACKLOG
  • Priority: P1
  • Value: Prevents configuration drift and hidden defaults across codebase.
  • Problem: Status: Partially Complete (SCIOT-026). Config validation/DONE but singleton pattern incomplete. Need src/common/config.py with typed Config.get_server()/get_client() access.
  • Task breakdown:
    • Create src/sciot/config.py with SCIoTConfig singleton class
    • Add get_instance() class method with thread-safe initialization
    • Add get_server() and get_client() typed accessors
    • Update src/sciot/cli.py to use SCIoTConfig.get_instance()
    • Add tests in tests/unit/test_config.py for singleton behavior
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #8

SCIOT-041 — Make EMA alpha configurable instead of hard-coded 0.5

  • Status: BACKLOG
  • Priority: P1
  • Value: Makes variance detection tunable for different hardware profiles.
  • Problem: Status: Ready for implementation. Add offloading_algo.ema_alpha to settings.yaml schema and config validation.
  • Task breakdown:
    • Add ema_alpha field to Settings.offloading_algo in src/sciot/settings.py
    • Update VarianceDetector in src/sciot/offloading.py to use configurable alpha
    • Add validation: 0.0 < ema_alpha <= 1.0 in config schema
    • Add tests in tests/unit/test_variance_detection.py for configurable alpha
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #9

SCIOT-042 — Ensure thread safety of simulation CSV handling

  • Status: BACKLOG
  • Priority: P1
  • Value: Ensures data integrity in concurrent inference.
  • Problem: Status: Blocked by SCIOT-009. Needs per-device runtime state registry first. Then add serialization lock around CSV writers.
  • Task breakdown:
    • Add threading.Lock to shared CSV state in simulation module
    • Create tests/concurrency/test_csv_threading.py for thread tests
    • Run pytest -n auto to verify no race conditions
    • Document thread safety guarantees in module docstrings
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #10

SCIOT-043 — Review and reduce class-level mutable state in RequestHandler

  • Status: BACKLOG
  • Priority: P2
  • Value: Improves IDE support and catches type errors early.
  • Problem: Status: Blocked by SCIOT-009. Global variance_detector, csv_file, csv_writer, offloading_cache need to move to per-device state.
  • Task breakdown:
    • Analyze requirements from GitHub issue #11
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #11

SCIOT-044 — Replace print statements with structured logging

  • Status: BACKLOG
  • Priority: P1
  • Value: Enables better debugging and monitoring in production.
  • Problem: Status: Ready for PR. structured_logger.py exists but print() still used in request_handler.py and endpoint handlers.
  • Task breakdown:
    • Create src/sciot/logging.py with structured logger setup
    • Replace print() calls in src/sciot/offloading.py with logger
    • Add log level config to settings.yaml schema
    • Add tests in tests/unit/test_logging.py for log format
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #12

SCIOT-045 — Fix type annotation of MessageData.get_latency

  • Status: DONE
  • Priority: P2
  • Value: Fixes type checking warnings.
  • Problem: Status: Ready for PR. Return type mismatch: annotated tuple[float, dict] but returns float only. Check message_data.py.
  • Task breakdown:
    • Analyze requirements from GitHub issue #13
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #13
  • Implementation notes:
    • Duplicate of SCIOT-030 after the remote plan merge; completed by the same MessageData.get_latency() annotation fix and regression tests.

SCIOT-046 — Time breakdown

  • Status: BACKLOG
  • Priority: P2
  • Value: Provides visibility into performance bottlenecks.
  • Problem: Status: Partial. profiler.py exists, plot_results.py has Italian labels. Needs SCIOT-030 metrics consolidation.
  • Task breakdown:
    • Add @profile_phase(name) decorator in src/sciot/telemetry.py
    • Instrument inference phases: pre-processing, compute, network
    • Add time breakdown summary to dashboard in src/dashboard/
    • Add tests in tests/unit/test_telemetry.py
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #16

SCIOT-047 — ESP32 client

  • Status: BACKLOG
  • Priority: P0
  • Value: Enables cross-platform client development.
  • Problem: Status: Blocked. Depends on #22 restructuring and #24 ABCs (see #30, #31 subissues).
  • Task breakdown:
    • Create src/sciot/client.py with abstract SCIoTClient class
    • Define async interface for registration/inference/offloading
    • Add ESP32 stubs implementing the interface
    • Add tests in tests/unit/test_client_interface.py
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #19

SCIOT-048 — Connect the camera

  • Status: BACKLOG
  • Priority: P1
  • Value: Standardizes camera access across platforms.
  • Problem: Status: Blocked. Depends on #22. Raspberry Pi camera streaming for real-time inference.
  • Task breakdown:
    • Create src/sciot/camera.py with abstract CameraModule class
    • Add interface for frame capture and preprocessing
    • Add ESP32 and mobile stubs implementing the interface
    • Add tests in tests/unit/test_camera.py
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #21

SCIOT-049 — Restructure the code

  • Status: BACKLOG
  • Priority: P2
  • Value: Enables per-request performance analysis.
  • Problem: Status: Partially Complete. Dependency profiles DONE (SCIOT-000) but ABCs needed (see #24).
  • Task breakdown:
    • Add request_id to timing events in telemetry
    • Add time breakdown dashboard view in src/dashboard/timing.py
    • Add filtering by request_id and model_id
    • Add migration script for existing timing data
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #22

SCIOT-050 — Implement mobile application for SCIoT client

  • Status: BACKLOG
  • Priority: P0
  • Value: Enables iOS and Android client development.
  • Problem: Status: Blocked. Needs #22, #24. Flutter/React Native cross-platform architecture.
  • Task breakdown:
    • Create abstract base class SCIoTClient (shared with ESP32 work)
    • Define cross-platform interface in src/sciot/mobile_interface.py
    • Add stubs for iOS/Swift and Android/Kotlin clients
    • Add tests in tests/unit/test_mobile_interface.py
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #23

SCIOT-051 — [Sub-issue for #22] Define Abstract Base Classes and Interfaces

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #24.
  • Problem: Status: Next Priority. Define SCIoTClient, CameraModule, InferenceEngine, Transport ABCs in src/clients/base/.
  • Task breakdown:
    • Analyze requirements from GitHub issue #24
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #24

SCIOT-052 — [Sub-issue for #22] Refactor Raspberry Pi Client to use Base Interfaces

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #25.
  • Problem: Status: Blocked by #24. Migrate http_clientCAMpi.py to use ABCs.
  • Task breakdown:
    • Analyze requirements from GitHub issue #25
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #25

SCIOT-053 — [Sub-issue for #22] Consolidate Configuration Management

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #26.
  • Problem: Status: Config validation done (SCIOT-026). Singleton pattern (Issue #8) remaining.
  • Task breakdown:
    • Analyze requirements from GitHub issue #26
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #26

SCIOT-054 — [Sub-issue for #23] Mobile Core: Cross-platform Architecture Setup

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #27.
  • Problem: Status: Blocked by #24. Flutter/React Native shared SCIoT communication layer.
  • Task breakdown:
    • Analyze requirements from GitHub issue #27
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #27

SCIOT-055 — [Sub-issue for #23] Mobile: Platform-specific Camera & ML Integration

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #28.
  • Problem: Status: Blocked by #27. iOS (AVFoundation/CoreML) and Android (CameraX/TFLite) drivers.
  • Task breakdown:
    • Analyze requirements from GitHub issue #28
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #28

SCIOT-056 — [Sub-issue for #23] Mobile UI: Dashboard and Live Stream View

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #29.
  • Problem: Status: Blocked by #27, #28. Live camera streaming, inference overlays, offloading status.
  • Task breakdown:
    • Analyze requirements from GitHub issue #29
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #29

SCIOT-057 — [Sub-issue for #19] ESP32: Minimal Inference with TFLite Micro

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #30.
  • Problem: Status: Blocked by #19. TFLite Micro on ESP32.
  • Task breakdown:
    • Analyze requirements from GitHub issue #30
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #30

SCIOT-058 — [Sub-issue for #19] ESP32: Camera Capture and Offloading Client

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #31.
  • Problem: Status: Blocked by #19. OV2640 camera + HTTP/MQTT client.
  • Task breakdown:
    • Analyze requirements from GitHub issue #31
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #31

SCIOT-059 — Verify correctness of offloading algorithm

  • Status: BACKLOG
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #32.
  • Problem: Status: Ready. Needs SCIOT-031 pluggable algorithms for proper testing.
  • Task breakdown:
    • Analyze requirements from GitHub issue #32
    • Implement changes
    • Add tests for the implementation
    • Update documentation if needed
  • Acceptance criteria:
    • Changes address the issue requirements
    • All tests pass (uv run pytest -q)
    • No breaking changes to existing API
  • Links to Issues: #32

SCIOT-060 — Publish evaluation telemetry to Firestore

  • Status: DONE
  • Priority: P2
  • Value: Addresses UBICO/SCIoT_python_client issue #44.
  • Problem: Test/evaluation results should be available in Firebase/Firestore while preserving the local CSV artifacts used for analysis.
  • Task breakdown:
    • Analyze requirements from GitHub issue #44
    • Add optional Firestore publishing for inference-cycle telemetry
    • Add optional Firestore publishing for offloading-decision telemetry
    • Buffer Firestore events and publish them every five minutes
    • Keep local CSV files as the primary/fallback artifact
    • Mirror decisions to offloading_decisions_<timestamp>.jsonl
    • Add tests with a fake Firestore client
    • Document configuration, credentials, and document layout
  • Acceptance criteria:
    • Local CSV writes continue when Firestore is disabled or unavailable
    • Firestore publishing is buffered, best-effort, and does not block inference
    • Firebase/Firestore dependencies are installed with the server runtime profile
    • All relevant tests pass
  • Links to Issues: #44