platform/api: /backup/v1/open_iaas/policies intermittent multi-minute hangs (root cause behind #339; OpenIaaS counterpart of #315)

[chrlesur]

Platform reliability issue — OpenIaaS counterpart of #315 (VPC ComputeManager) on the backup-policies read path.

Symptom

GET /api/backup/v1/open_iaas/policies intermittently hangs for several minutes before returning normally — the exact symptom that prompted #339 ("Observed live: a GET /backup/v1/open_iaas/policies hung ~3 minutes during a datasource read before manual abort").

Latest live reproduction (2026-06-29)

• Recette OpenIaaS, shiva.cloud-temple.com, tenant fc37d401-….
• During a terraform apply (provider built from main @ v1.8.0):
  • data.cloudtemple_compute_iaas_opensource_availability_zones stalled ~2 min.
  • data.cloudtemple_backup_iaas_opensource_policies was still "Still reading… [03m50s elapsed]" when the run was manually aborted.
• ~30 minutes later, a direct curl to the same endpoint returned HTTP 200 in 306 ms (5 policies, 1577 bytes; TLS 74 ms, TTFB 304 ms).
• Cross-check GET /api/backup/v1/spp/policies (VMware SPP equivalent, same proxy chain): 132 ms — slowness is isolated to the OpenIaaS backup path, not the HTTP/IAM/proxy stack.
• Therefore: a transient platform-side condition (likely a lock or queue on the OpenIaaS / Xen Orchestra policy backend) sporadically inflates this endpoint's latency by ~750× over normal.

Relationship with #339 and #315

• client: reads lack a per-request timeout and transient-error retry — a slow/flaky API hangs or fails plan/apply #339 (CLOSED 2026-06-26 via Release v1.8.0 #372, v1.8.0) ships the provider-side mitigation: CLOUDTEMPLE_HTTP_TIMEOUT (default 240 s) plus a bounded retry on idempotent GETs. It turns an infinite hang into either a transient retry success or an actionable failure. It does not fix the platform-side cause — client: reads lack a per-request timeout and transient-error retry — a slow/flaky API hangs or fails plan/apply #339 explicitly scopes that out:

  Platform-side flakiness on ComputeManager is tracked in platform/api: VPC write activities intermittently fail (ComputeManager workers unavailable, and 502 Bad Gateway) #315; this issue is the provider-side resilience that should hold regardless of the API's behaviour.

• platform/api: VPC write activities intermittently fail (ComputeManager workers unavailable, and 502 Bad Gateway) #315 (CLOSED 2026-06-27) is the platform-side analogue for VPC writes (ComputeManager workers unavailable / 502). The present issue is the OpenIaaS backup-policies counterpart of platform/api: VPC write activities intermittently fail (ComputeManager workers unavailable, and 502 Bad Gateway) #315: same posture — the provider mitigates, but the root cause is API-side and should be stabilised upstream.

Latency baseline (from #339's diagnostic + live cross-check)

• Nominal P50 ~ 4–8 s, P95 ~ 10 s, ~0.4 % HTTP 502.
• Pathological excursions: ≥ 3 min observed live in multiple incidents.
• Cross-tenant: the present incident is on tenant fc37d401-… (fsn-02), distinct from the client: reads lack a per-request timeout and transient-error retry — a slow/flaky API hangs or fails plan/apply #339 baseline tenant — the pathology is platform-wide, not tenant-specific.

Ask

Stabilise the /backup/v1/open_iaas/policies read path:

1. Identify the platform-side cause of the multi-minute excursions (locking, GC pause, queue, slow downstream Xen Orchestra call, etc.).
2. Document an expected nominal SLO for this endpoint so the provider can size its timeout/retry policy accordingly.
3. Notify when remediation is deployed so we can re-baseline CLOUDTEMPLE_HTTP_TIMEOUT and the retry parameters.

Provider-side: no change required (the #339 mitigation already protects user runs from infinite hangs). This issue tracks the upstream stabilisation only.

Refs

• Related to client: reads lack a per-request timeout and transient-error retry — a slow/flaky API hangs or fails plan/apply #339 — provider-side resilience (closed, fix in v1.8.0).
• Related to platform/api: VPC write activities intermittently fail (ComputeManager workers unavailable, and 502 Bad Gateway) #315 — VPC platform-side analogue (closed).