[Discussion] Notion To MD v4: Block Fetcher Technical Details

[souvikinator]

Block Fetcher Technical Documentation

Overview

graph TD
    A[Initialize BlockFetcher] --> B[Configure Settings]
    B --> C[Task Queue Management]
    C --> D[Rate Limited Fetching]
    D --> E[Block Organization]
    E --> F[Return Structured Data]

Loading

The BlockFetcher module serves as a specialized fetching layer that manages the complexity of retrieving nested Notion content while optimizing performance and respecting API constraints.

Core Capabilities

1. Unified Task Queue Processing
   Instead of handling different types of requests separately, the system now implements a unified task queue that processes:
   • Block fetching tasks
   • Comment fetching tasks
   • Page property fetching tasks

The system processes these tasks concurrently within configured batch sizes, maintaining optimal performance while respecting API limits.

2. Rate Limiting Management
   The component implements a decent rate limiting system that:

   • Tracks requests per second across all task types (default max: 3)
   • Implements a sliding window approach for rate limiting
   • Manages cooldown periods dynamically when limits are reached
   • Ensures fair distribution of rate limits across different task types

3. State Management
   The system maintains several optimized data structures:

   • queue: A unified task queue that tracks all pending operations (API calls)
   • blocks: A Map storing fetched blocks with their complete data
   • processedTasks: A Set tracking completed tasks to prevent duplication (maybe I overcomplicated here hehe)
   • pageProperties: Stores page metadata when requested
   • rootComments: Maintains comments associated with the root page

4. Block Tree Builder
   The component now handles block organization more efficiently:

   • Implements a two-pass tree building algorithm for O(N) complexity
   • Maintains accurate parent-child relationships across all block types
   • Associates comments directly with their respective blocks
   • Preserves block ordering while enabling parallel processing

Configuration Options

The BlockFetcher accepts these configuration parameters:

interface BlockFetcherConfig {
  fetchPageProperties: boolean;    // Controls page property fetching
  fetchComments: boolean;         // Enables comment fetching for blocks
  maxRequestsPerSecond: number;   // API rate limit (default: 3)
  batchSize: number;             // Parallel processing batch size (default: 3)
}

Detailed Implementation Flow

flowchart TD
    Init[Initialize BlockFetcher] --> Config[Load Configuration]
    Config --> Queue[Initialize Task Queue]
    Queue --> InitTasks[Add Initial Tasks]
    
    subgraph QueueProcessing[Task Processing Loop]
        InitTasks --> Empty{Queue Empty?}
        Empty -->|No| Batch[Take Task Batch]
        Batch --> RateLimit[Apply Rate Limiting]
        RateLimit --> Process[Process Tasks in Parallel]
        Process --> TaskType{Task Type}
        
        TaskType -->|Blocks| FetchBlocks[Fetch Block Children]
        TaskType -->|Comments| FetchComments[Fetch Comments]
        TaskType -->|Properties| FetchProps[Fetch Properties]
        
        FetchBlocks --> Store[Store Results]
        FetchComments --> Store
        FetchProps --> Store
        
        Store --> QueueMore{Queue Additional Tasks?}
        QueueMore -->|Yes| AddTasks[Add New Tasks]
        QueueMore -->|No| Empty
        AddTasks --> Empty
    end
    
    Empty -->|Yes| BuildTree[Build Block Tree]
    BuildTree --> Output[Return Data Structure]

Loading

Interface

interface QueueTask {
  type: "fetch_properties" | "fetch_comments" | "fetch_blocks";
  id: string;
  parentId?: string;
}

interface FetcherOutput {
  properties: PageObjectProperties;
  comments: CommentResponseResults;
  blocks: ListBlockChildrenResponseResults;
}

class BlockFetcher {
  constructor(client: Client, config?: BlockFetcherConfig);
  
  // Main entry point for fetching page content
  async getBlocks(pageId: string): Promise<FetcherOutput>;
  
  // Core internal methods
  private async processTask(task: QueueTask): Promise<void>;
  private async rateLimitRequest<T>(request: () => Promise<T>): Promise<T>;
  private buildBlockTree(rootId: string): ListBlockChildrenResponseResults;
  
  // Helper methods
  private addTask(task: QueueTask): void;
  private async fetchBlockChildren(blockId: string): Promise<ListBlockChildrenResponseResults>;
  private async fetchAllComments(blockId: string): Promise<CommentResponseResults>;
  private async fetchPageProperties(pageId: string): Promise<PageObjectProperties>;
}