Skip to content

TODO.md

Latest commit

 

History

History
415 lines (360 loc) · 17.2 KB

TODO.md

File metadata and controls

415 lines (360 loc) · 17.2 KB
Raw

Devmer - Infrastructure as Code in Rust

A Rust-based Infrastructure as Code (IaC) tool similar to Pulumi/Terraform, with flexible self-hosted state management, multi-language SDK support, and a modular provider architecture.

Vision

Devmer aims to be a fully open-source, self-hostable IaC solution that:

  • Requires no proprietary cloud service (unlike Pulumi Cloud)
  • Stores state in user-controlled backends (15+ options)
  • Supports imperative programming in multiple languages
  • Is built with Rust for performance and safety
  • Uses a modular crate architecture for cloud providers

Architecture

Devmer Core (100% Standalone)

┌─────────────────────────────────────────────────────────────────────┐
│                         Devmer CLI (devmer-cli)                     │
├─────────────────────────────────────────────────────────────────────┤
│                      Core Engine (devmer-core)                      │
│      Planner │ Executor │ Diff Engine │ Resource Graph              │
├─────────────────────────────────────────────────────────────────────┤
│  devmer-state  │  devmer-secrets  │  devmer-concurrency  │ devmer-audit │
│   S3/GCS/Azure │  Passphrase/KMS  │  Distributed Locking │ SOC2/HIPAA   │
├─────────────────────────────────────────────────────────────────────┤
│                    Language SDKs (devmer-runtime)                   │
│  Python │ TypeScript │ Go │ Rust Script (Rhai) │ Deno │ Bun        │
├─────────────────────────────────────────────────────────────────────┤
│                    Provider Crates (devmer-providers)               │
│  AWS │ GCP │ Azure │ Kubernetes │ Docker │ Cloudflare              │
└─────────────────────────────────────────────────────────────────────┘

Cloudmer Integration (Optional)

┌─────────────────────────────────────────────────────────────────────┐
│                    DEVMER (Works 100% Standalone)                   │
├─────────────────────────────────────────────────────────────────────┤
│  devmer-cloudmer (optional client library)                          │
│  Hooks into core when CLOUDMER_TOKEN is set                         │
└────────────────────────────────┬────────────────────────────────────┘
                                 │ HTTPS API (only metadata, no secrets)
                                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    CLOUDMER SERVICE (cloudmer.app)                  │
│  Optional web platform for teams and enterprises                    │
├─────────────────────────────────────────────────────────────────────┤
│  📊 Visualization │ 💰 Cost Insights │ 👥 Team Collaboration       │
│  🔒 Distributed Locks │ 📋 Compliance Dashboards │ 🔔 Alerts       │
└─────────────────────────────────────────────────────────────────────┘

What Works Without Cloudmer (Everything!)

Feature Standalone With Cloudmer
Deploy infrastructure ✅ Full ✅ + Visualization
State management ✅ 15+ backends ✅ + Dashboard
Secrets encryption ✅ Passphrase/KMS/Vault ✅ Same
Multi-user locking ✅ devmer-concurrency ✅ + Distributed
Audit logging ✅ File/Syslog/SIEM ✅ + Dashboard
Compliance reports ✅ Local generation ✅ + Dashboard
Cost tracking ❌ ✅ Real-time
Team collaboration ❌ ✅ Comments/approvals

Progress Overview

Crate Status Description
devmer-core ✅ Done Resource graph, diff engine, planner, executor
devmer-config ✅ Done TOML config, env interpolation, .env loading
devmer-di ✅ Done Dependency injection with shaku
devmer-state ✅ Done State backend trait + S3/GCS/Azure/Local mocks
devmer-secrets ✅ Done Secrets encryption, providers (passphrase, KMS, Vault, Age)
devmer-providers ✅ Done Provider trait + Mock + AWS (30+ resource schemas)
devmer-runtime ✅ Done Language host trait + Rhai runtime + registry
devmer-org ✅ Done Organizations, teams, RBAC, resource policies, approvals
devmer-rpc ✅ Done gRPC protocol, Engine/Host/Provider services
devmer-convert ✅ Done HCL to scripting language conversion
devmer-cli ✅ Done Complete CLI with all commands
devmer-audit ✅ Done Audit logging, SOC2/HIPAA/PCI-DSS compliance
devmer-concurrency ✅ Done Distributed locking, multi-user coordination, conflict detection
devmer-cloudmer ✅ Done Optional Cloudmer integration (visualization, costs, collaboration)
devmer-tui ⏳ Planned Terminal UI with ratatui
devmer-migrate ⏳ Planned Terraform/Pulumi state import

Completed Features

Core Engine (devmer-core)

  • Resource model with URN, inputs, outputs, dependencies
  • PropertyValue type system (string, int, bool, array, object, secret, output)
  • Dependency graph construction with petgraph
  • Topological sorting for deployment order
  • Diff engine (create, update, delete, replace detection)
  • Deployment planner with parallel operation support
  • Execution engine with event streaming
  • Provider trait and registry

Configuration (devmer-config)

  • TOML configuration parsing
  • Layered config (file + env + CLI)
  • Environment variable interpolation (${VAR}, ${VAR:-default})
  • .env file loading with dotenvy
  • Stack configuration support

Dependency Injection (devmer-di)

  • shaku-based DI container
  • Service interfaces (Config, State, Secrets, Runtime, Execution)
  • Provider registry integration
  • Runtime context for scripting languages

State Management (devmer-state)

  • StateBackend trait
  • State locking with LockInfo/LockStatus
  • State versioning and history
  • Mock S3 backend
  • Mock GCS backend
  • Mock Azure Blob backend
  • Local file backend

Secrets (devmer-secrets)

  • SecretsProvider trait
  • Passphrase-based encryption (Argon2 + AES-GCM)
  • AWS KMS provider (mock)
  • GCP KMS provider (mock)
  • Azure Key Vault provider (mock)
  • HashiCorp Vault provider (mock)
  • Age encryption provider (mock)
  • Secure memory handling (zeroize)

Providers (devmer-providers)

  • Provider trait with full lifecycle (check, diff, create, read, update, delete)
  • MockProvider for testing
  • AWS Provider with 30+ resource schemas:
    • S3 Bucket, Lambda Function, IAM Role/Policy/User
    • EC2 Instance/SecurityGroup/VPC/Subnet
    • DynamoDB Table, RDS Instance, ECS Cluster/Service
    • API Gateway, CloudWatch, SNS/SQS, Route53, and more
  • Resource schema validation
  • Diff computation with replacement detection

Runtime (devmer-runtime)

  • LanguageRuntime trait
  • Rhai embedded scripting runtime
  • Resource registration and tracking
  • ComponentResource support
  • StackReference support
  • Runtime context (ConfigProvider, SecretsProvider, ResourceProvider)
  • Output handling

Organization Admin (devmer-org)

  • Organization/Team/User hierarchy
  • Role-based access control (RBAC)
  • Built-in roles (Owner, Admin, Member, Viewer, etc.)
  • Custom role creation
  • Resource policies with glob patterns
  • Stack/project scoping per team
  • Resource type restrictions
  • Environment-based approval requirements
  • Approval workflows

Audit & Compliance (devmer-audit)

  • Comprehensive audit event model (50+ event types)
  • Event categories (deployment, resource, state, secret, auth, policy, approval, org, system, compliance)
  • Event severity levels (Debug, Info, Warning, Error, Critical)
  • Actor tracking (User, Service, System, Anonymous, ApiKey)
  • Resource tracking with URN support
  • Hash chain for tamper-evidence
  • Chain verification and integrity checks
  • File-based audit backend with daily rotation
  • Memory backend for testing
  • Multi-backend support (write to multiple backends)
  • Event querying with time ranges, filters, and full-text search
  • SOC2 Type II compliance controls (CC6.1, CC6.2, CC6.3, CC7.1, CC7.2, CC8.1)
  • HIPAA compliance controls (164.308, 164.312)
  • PCI-DSS compliance controls (7.1, 8.1, 10.1, 10.2, 10.7)
  • GDPR compliance controls (Art.30, Art.32, Art.33)
  • ISO 27001 compliance controls (A.9.2.1, A.9.2.3, A.12.4.1)
  • NIST Cybersecurity Framework controls
  • CIS Controls
  • Compliance checking engine
  • Report generation (Markdown, HTML, Text, CSV, JSON)
  • SIEM integration formats:
    • CEF (Common Event Format)
    • LEEF (Log Event Extended Format for IBM QRadar)
    • Syslog RFC 5424
    • Splunk HEC
    • Elasticsearch Bulk
    • CSV
    • JSON Lines

Concurrency Control (devmer-concurrency)

  • Distributed lock manager
    • Exclusive resource locking
    • TTL-based lock expiration (default 30 minutes)
    • Lock queuing with fair ordering (FIFO)
    • Heartbeat/lease renewal
    • Force-acquire for admin operations
    • In-memory backend (for single-instance)
    • Pluggable backend trait for distributed deployments
  • User session tracking
    • Track active user sessions
    • Monitor who's accessing which resources
    • Session expiration and cleanup
    • Client info tracking (hostname, IP, user agent)
  • Conflict detection
    • Pre-operation conflict checks
    • Detect concurrent modifications
    • Detect locked resources
    • Detect dependency conflicts
    • State version mismatch detection
    • Severity levels (Warning, Error, Critical)
    • Recommendations for conflict resolution
  • Operation journal
    • Full audit trail of lock operations
    • Query by resource, actor, event type
    • Time-range filtering
    • In-memory storage with max entries

Cloudmer Integration (devmer-cloudmer)

Note: All features below are optional - Devmer works 100% without Cloudmer

  • API client with device auth flow
  • State sync to Cloudmer for visualization
  • Deployment notifications
  • Cost insights retrieval
  • Integration hooks (CloudmerHooks)
    • Environment-based configuration (CLOUDMER_TOKEN)
    • Auto-detection of git info (commit, branch)
    • CI/CD system detection
  • Distributed locking via Cloudmer
    • Acquire/release locks via API
    • Queue position tracking
    • Heartbeat renewal
  • Clear documentation of standalone vs. enhanced features
  • Privacy-focused (only metadata, no secrets transmitted)

Completed Recently

CLI (devmer-cli)

  • devmer new - Create new project
  • devmer init - Initialize existing directory
  • devmer preview - Show planned changes
  • devmer up - Deploy infrastructure
  • devmer down - Destroy deployed infrastructure
  • devmer refresh - Refresh state from cloud
  • devmer stack - Stack management (ls, new, select, rm, history, output)
  • devmer config - Configuration management (get, set, rm)
  • devmer secrets - Secrets management (set, get, ls, rotate)
  • devmer state - State inspection/manipulation (export, import, unlock, delete)
  • devmer login - Cloud provider authentication
  • devmer convert - HCL to scripting language conversion (from, analyze, formats)
  • devmer version - Version information
  • Interactive prompts with dialoguer
  • Colored terminal output
  • Progress indicators
  • Workspace state persistence (.devmer/workspace.json)
  • Stack-specific config files (Devmer.{stack}.toml)
  • Passphrase-based secrets encryption

HCL Conversion (devmer-convert)

  • HCL parsing with hcl-rs
  • Resource mapping to Devmer types
  • Code generation (TypeScript, Python, Go, Rhai)
  • IR (Intermediate Representation) for resources, variables, outputs, data sources
  • Expression conversion (references, function calls, conditionals, for expressions)
  • Provider configuration conversion
  • Lifecycle settings conversion
  • Devmer.toml generation
  • Package scaffolding (package.json, requirements.txt, go.mod)
  • Module conversion (module calls -> components)

Remaining Work

Phase 1: MVP (v0.1.0)

Language SDKs

  • TypeScript SDK via napi-rs
  • Python SDK via pyo3
  • Go SDK via cgo
  • Language host gRPC protocol

TUI

  • Main dashboard view
  • Deployment progress view
  • Resource tree browser
  • Change preview with diffs
  • State browser

Real Provider Implementations

  • AWS SDK integration (replace mocks)
  • Actual S3 state backend
  • Actual DynamoDB locking

Phase 2: Extended Backends (v0.2.0)

  • PostgreSQL state backend
  • MySQL/MariaDB state backend
  • Redis state backend
  • etcd state backend
  • Consul state backend
  • Git state backend
  • Kubernetes ConfigMap/Secret backend

Phase 3: Migration (v0.3.0)

  • Terraform state parser (v3/v4)
  • OpenTofu state parser
  • Pulumi state parser
  • Migration wizard
  • Code generation from state

Phase 4: Audit & Compliance (v0.4.0) ✅

  • Audit event capture
  • File audit backend
  • Memory audit backend
  • Multi-backend support
  • Hash chaining for tamper-evidence
  • Chain verification
  • SOC2 compliance controls
  • HIPAA compliance controls
  • PCI-DSS compliance controls
  • GDPR compliance controls
  • ISO 27001 compliance controls
  • NIST CSF compliance controls
  • CIS Controls
  • Compliance report generation (Markdown, HTML, Text, CSV, JSON)
  • SIEM export formats (CEF, LEEF, Syslog, Splunk HEC, Elasticsearch Bulk)
  • Event querying and filtering
  • CloudWatch audit backend (optional feature)
  • S3 archival backend (optional feature)
  • PostgreSQL backend (optional feature)
  • Parquet archival format (optional feature)

Phase 5: Enterprise (v0.5.0+)

  • License key system
  • Usage analytics
  • Premium providers
  • Advanced compliance features
  • Priority support infrastructure

Milestones

v0.1.0 - Foundation ⏳

  • Core engine with resource lifecycle
  • Local file state backend
  • CLI with all commands
  • Basic TUI with deployment progress
  • Passphrase-based secrets
  • AWS provider (mock)
  • Rust script (Rhai) support

v0.2.0 - Cloud State & Secrets

  • S3/GCS/Azure state backends (mock)
  • State locking
  • KMS secrets providers (mock)
  • Terraform/Pulumi state import
  • Migration wizard

v0.3.0 - Language SDKs

  • Python SDK
  • TypeScript SDK (Node.js)
  • Deno & Bun support
  • Go SDK

v0.4.0 - Full TUI

  • Dashboard view
  • Resource browser
  • Interactive deployment
  • State browser
  • Theming support

v0.5.0 - Multi-Cloud

  • Real AWS provider
  • GCP provider
  • Azure provider
  • Kubernetes provider

v1.0.0 - Production Ready

  • All SDKs stable
  • Comprehensive AWS coverage
  • All state backends stable
  • Full audit & compliance
  • Complete documentation

Non-Goals (For Now)

  • SaaS offering - Focus on self-hosted only
  • GUI-first - CLI/TUI-first approach
  • Terraform HCL compatibility - New language approach
  • Automatic provider generation - Manual quality control

References