Federated Catalog scalability: crawling strategy for large dataspaces

[mokhairymahmoud]

Hello,

With the Federated Catalog now embedded as a default control-plane feature, each participant runs its own crawler that periodically fetches the catalog of every other participant. For N participants, this means:

• Each participant sends N-1 requests and serves N-1 requests per crawl cycle
• Aggregate network load: N×(N-1) requests per cycle — O(N²)

For a dataspace with 5,000 participants and a 60-second crawl period, that's ~25 million requests/cycle network-wide, and if each catalog response is non-trivial (hundreds of offers with policies), the bandwidth adds up quickly.

The main Questions:

1. Is the per-participant crawling model intended to scale to dataspaces with thousands of participants, or is it designed for smaller
   deployments?
2. Is there a recommended deployment pattern for large dataspaces ?

I understand the design intentionally avoids a single centralized broker (for sovereignty/trust reasons), but I'm curious to understand how to deal with this scaling issue.

Thanks!

[paullatzelsperger]

Assuming that everyone always needs everyone else's latest data, then I think this is a classic poll-vs-event-driven problem. I can think of several ways to approach this:

Centralized Event System

Every connector subscribes to a (centralized) event system. Each connector emits an event when their data inventory changes: e.g. when an asset is added or updated. For this, the existing EDC event routing can be used.

Then, when a connector receives an event, that someone else's inventory has changed, it simply triggers a catalog request to fetch the newest update. To optimize even further, differential updates could be implemented, where the asset added/updated event carries the dataset ID, so that the consumer connector can do a dataset request instead of a catalog request.

The obvious elephant in the room is that this requires centralized infrastructure, namely the event system. This might not be suitable in all situations.

Consumer registers with Provider-owned messaging system

Another slightly more decentralized approach would be:

• each connector offers an "events asset" (which could be a Kafka/NATS/... endpoint) where the provider publish asset updates
• consumer connectors can register for events of provides that they are interested in
• when a provider adds/updates an asset, an event is emitted through the "events asset", automatically notifying all interested/registered consumers

This would be quite efficient in terms of resources, because the provider publishes once, and every consumer gets the update. However, the provider would need to admin the message system

Provider actively informs Consumer

A third option could be a variant of the previous one, where the provider calls an API of the consumer to notify about asset updates, instead of the consumer subscribing to a topic:

• during initial start up, the consumer connector "registers" with the provider. The provider negotiates access to the consumer's "events update" API, and stores that URL somewhere.
• when the provider has an asset update, they call each interested consumer's "events update API"

This would require no additional infrastructure, no NATS/Kafka,..., but a custom registration process must be implemented, and on every update the provider would have to call N interested consumers, but at least that could be spread out over time. Also, the publication of the event could happen over HTTP/REST, so no additional infrastructure is needed.

[ndr-brt]

For a dataspace with 5,000 participants and a 60-second crawl period, that's ~25 million requests/cycle network-wide, and if each catalog response is non-trivial (hundreds of offers with policies), the bandwidth adds up quickly.

60-second crawl period is totally overkill in every case, why would anybody need such frequency to get their catalog updated from all the participants, giving that contract offers are something that (in a sane scenario) are meant to stay there for days/weeks/months?

I think that we engineers tend to look in the "micro" scenario because we are used to test environments that need to run in short amounts of time, but, the production is different.
Once in a day could be a good starting point, with the possibility to setup (to be implemented, eventually :) ) exponential wait strategies (e.g: catalog is the same? next time wait 2 days, after then 4 days and so on).

1. Is the per-participant crawling model intended to scale to dataspaces with thousands of participants, or is it designed for smaller
   deployments?

It works the same with web scrapers: if the number of nodes you have to scrape becomes too high or you get throttled, the scrape frequency needs to be reduced accordingly.

2. Is there a recommended deployment pattern for large dataspaces ?

Just crawl what you need, not everything, typically in an use case a consumer knows its own providers, so it's just a matter to just crawl those, for all the use cases supported by that control-plane.
A "crawl everything" job could be launched like once in a month just to be aware of what's available in the dataspace but to be honest I don't see a lot of use for that.

I understand the design intentionally avoids a single centralized broker (for sovereignty/trust reasons), but I'm curious to understand how to deal with this scaling issue.

We can use some concepts from other p2p networks that have been running for decades, and the catalog crawling is not different from cough...cough... torrent indexers: any participant could fetch other offers related to specific use cases and aggregate them into a catalog that will be available to the whole network.