Migrate eventstore to LMDB and tune WoT bootstrap fetches - #97
Merged
Conversation
Switches the backend from Badger to LMDB via fiatjaf.com/nostr/eventstore/lmdb. Adds a one-shot export tool (tools/export-badger) and importer (cmd/import-jsonl) so existing deployments can migrate by exporting to JSONL and reimporting into a fresh LMDB store; documents the steps in the README. Also tunes the WoT bootstrap: drops per-batch concurrency from 20 to 5 and raises the fetch timeout from 8s to 30s for both the graph build and the profile refresh. Seed relays were rejecting a large fraction of our subscriptions with "too many concurrent requests" (each batch fans out to all 14 seed relays, so 20 batches = ~20 simultaneous subs per relay, above typical caps), and the 8s window was cutting off slow relays mid-drain. Trust network size had dropped from ~45k to ~13k keys as a result. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the now-dead -tags badger build flag from arm64.Dockerfile (no Go file references that tag anymore — it was only used by the old Badger-backed code path). Adds a Docker migration section to the README so users who run wot-relay via docker-compose can perform the Badger → LMDB migration without installing Go on the host: both the export and import tools are run inside throwaway golang:bookworm containers bind-mounted at the same ./db path that compose already uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fiatjaf.com/nostr/eventstore/lmdb) as the relay's eventstore. Formats are incompatible, so add a JSONL export/import path for existing deployments:tools/export-badgerdumps a legacy Badger DB,cmd/import-jsonlloads it into a fresh LMDB store. Migration steps documented in README.wotConcurrencyandprofileConcurrency20 → 5,wotBatchTimeoutandprofileBatchTimeout8s → 30s. Each batch already fans out a subscription to every seed relay, so 20 concurrent batches meant ~20 simultaneous subs per seed relay, tripping "too many concurrent requests" limits and silently dropping follow lists. The 8s window was also too short for slower relays to drain. Observed trust network size had fallen from ~45k to ~13k keys; this should recover most of the lost keys at the cost of a slower (but complete) refresh pass.Test plan
tools/export-badgeragainst an existing Badgerdb/and confirm JSONL output is well-formedcmd/import-jsonlinto a fresh directory and confirm the relay starts and serves the imported events🤖 Generated with Claude Code