Skip to content

vfs.adoc

Latest commit

 

History

History
977 lines (736 loc) · 24 KB

vfs.adoc

File metadata and controls

977 lines (736 loc) · 24 KB
Raw

VFS: Virtual File System Router

1. Overview

The Virtual File System (VFS) Router is chell’s abstraction layer that presents a unified filesystem view combining native ChRIS paths with virtual directories. It implements proper separation of concerns between data fetching and presentation, using cumin’s Result<T> pattern for robust error handling.

1.1. The Problem

ChRIS has a complex architecture where:

  • Native paths exist on the CUBE server (/home/user/feeds/…​)

  • Virtual paths provide utility abstractions (e.g., /bin for plugins)

  • API operations can fail in various ways

  • Users need consistent, filesystem-like navigation

Early implementations mixed concerns:

// Old approach - mixed concerns
async function listDirectory(path: string): Promise<void> {
  try {
    const items = await fetchItems(path); // Data fetching
    console.log(formatItems(items));      // Presentation
  } catch (error) {
    console.error(error);                 // Error handling
  }
}

Problems:

  • Can’t get data without side effects (console output)

  • Error handling tied to presentation layer

  • Hard to test data operations independently

  • Can’t compose operations (pipe, filter, etc.)

1.2. The Solution

VFS separates data layer (returns Result<T>) from presentation layer (renders output):

// New approach - separated concerns

// Data layer: returns Result<ListingItem[]>
async data_get(path?: string): Promise<Result<ListingItem[]>> {
  try {
    const items = await fetchItems(path);
    return Ok(items);
  } catch (error) {
    errorStack.stack_push("error", `Failed to list ${path}: ${error}`);
    return Err();
  }
}

// Presentation layer: renders data, handles errors
async list(path?: string): Promise<void> {
  const result = await this.data_get(path);

  if (!result.ok) {
    const lastError = errorStack.stack_pop();
    console.error(chalk.red(lastError?.message));
    return;
  }

  console.log(grid_render(result.value));
}

Benefits:

  • ✅ Data operations testable without console output

  • ✅ Errors handled with Result<T> + errorStack pattern

  • ✅ Composable: data_get() output feeds other operations

  • ✅ Type-safe: TypeScript enforces error checking

2. Architecture

2.1. Layer Separation

The VFS implements a clean separation between data and presentation:

┌─────────────────────────────────────────────┐
│  Presentation Layer (void returns)          │
│  - list(): renders directory listings       │
│  - Pops errors from errorStack              │
│  - Displays with console.log/console.error  │
└─────────────────────────────────────────────┘
                    ↓ calls
┌─────────────────────────────────────────────┐
│  Data Layer (Result<T> returns)             │
│  - data_get(): fetches and returns data      │
│  - dataVirtualBin_get(): /bin plugin listing │
│  - dataNative_get(): native ChRIS paths      │
│  - Pushes errors to errorStack              │
│  - Uses ListCache for performance           │
└─────────────────────────────────────────────┘
                    ↓ calls
┌─────────────────────────────────────────────┐
│  External APIs                              │
│  - chili: files_list(), plugins_listAll()  │
│  - cumin: session, cache, errorStack        │
└─────────────────────────────────────────────┘

2.2. Path Routing

The VFS routes paths to appropriate handlers:

async data_get(targetPath?: string, options: {...}): Promise<Result<ListingItem[]>> {
  const cwd = await session.getCWD();
  const effectivePath = targetPath
    ? path.posix.resolve(cwd, targetPath)
    : cwd;

  // Route to appropriate handler
  if (effectivePath === '/bin') {
    return await this.dataVirtualBin_get(options);  // Virtual directory
  } else {
    return await this.dataNative_get(effectivePath, options); // Native path
  }
}

Routing logic:

  1. Resolve relative paths to absolute

  2. Check against virtual mount points (currently just /bin)

  3. Delegate to virtual or native handler

  4. Return Result<T> for consistent error handling

2.3. Virtual Directories

Virtual directories don’t exist on the ChRIS server but provide utility:

2.3.1. /bin - Plugin Directory

The /bin directory lists all available ChRIS plugins as executable files:

private async dataVirtualBin_get(options): Promise<Result<ListingItem[]>> {
  try {
    const plugins = await plugins_listAll({});
    const items: ListingItem[] = [];

    plugins.tableData.forEach((plugin) => {
      const displayName = `${plugin.name}-v${plugin.version}`;
      items.push({
        name: displayName,
        type: 'plugin',
        size: 0,
        owner: 'system',
        date: plugin.creation_date || '',
      });
    });

    // Apply sorting and cache
    const sorted = list_applySort(items, options.sort || 'name', options.reverse);
    listCache_get().cache_set('/bin', sorted);

    return Ok(sorted);
  } catch (error) {
    errorStack.stack_push("error", `Failed to list plugins: ${error.message}`);
    return Err();
  }
}

Characteristics:

  • Listed via ChRIS plugin API, not filesystem

  • Names formatted as pl-name-v1.2.3

  • Type marked as 'plugin' for special rendering

  • Cached like native directories for tab completion

2.3.2. Future Virtual Directories

Potential future additions:

  • /proc - ChRIS instance metadata, statistics

  • /etc - Configuration, environment variables

  • /tmp - Temporary uploads before placement

2.4. Native Directories

Native directories are ChRIS paths backed by the CUBE server:

private async dataNative_get(target: string, options): Promise<Result<ListingItem[]>> {
  try {
    // Special handling for -d flag (show directory info, not contents)
    if (options.directory) {
      const parentPath = target.substring(0, target.lastIndexOf('/')) || '/';
      const itemName = target.substring(target.lastIndexOf('/') + 1);

      // Check cache, fetch if needed
      let parentItems = listCache.cache_get(parentPath);
      if (!parentItems) {
        parentItems = await files_list({}, parentPath);
        listCache.cache_set(parentPath, parentItems);
      }

      const item = parentItems.find(i => i.name === itemName);
      if (item) {
        return Ok([item]); // Single item array
      } else {
        errorStack.stack_push("error", `ls: cannot access '${target}': No such file or directory`);
        return Err();
      }
    }

    // Normal listing: fetch directory contents
    const items = await files_list({
      path: target,
      sort: options.sort,
      reverse: options.reverse
    }, target);

    // Inject virtual /bin at root
    if (target === '/' || target === '') {
      items.push({
        name: 'bin',
        type: 'vfs',
        size: 0,
        owner: 'root',
        date: new Date().toISOString(),
      });
      items.sort((a, b) => a.name.localeCompare(b.name));
    }

    // Cache and return
    listCache_get().cache_set(target, items);
    return Ok(items);

  } catch (error) {
    errorStack.stack_push("error", `Failed to list ${target}: ${error.message}`);
    return Err();
  }
}

Features:

  • Fetches from ChRIS API via files_list() from chili

  • Handles -d flag (show item info, not contents)

  • Injects virtual directories at appropriate paths

  • Caches results for performance

  • Returns Result<T> for error handling

3. Result<T> Pattern Integration

3.1. Data Layer Returns Result<T>

All data-fetching methods return Result<ListingItem[]>:

async data_get(path?: string): Promise<Result<ListingItem[]>>
async dataVirtualBin_get(options): Promise<Result<ListingItem[]>>
async dataNative_get(path: string, options): Promise<Result<ListingItem[]>>

On success:

return Ok(items); // Wraps ListingItem[] in Ok()

On failure:

errorStack.stack_push("error", "Failed to list directory: Network error");
return Err(); // Signals failure, error details in errorStack

3.2. Presentation Layer Handles Errors

The list() method handles Result<T> and displays output:

async list(path?: string, options): Promise<void> {
  const result = await this.data_get(path, options);

  // Handle failure
  if (!result.ok) {
    const lastError = errorStack.stack_pop();
    if (lastError) {
      console.error(chalk.red(lastError.message));
    }
    return;
  }

  // Handle empty results
  if (result.value.length === 0) {
    return;
  }

  // Render based on options
  if (options.long) {
    console.log(long_render(result.value, { human: !!options.human }));
  } else {
    console.log(grid_render(result.value));
  }
}

Error handling flow:

  1. Check result.ok

  2. Pop error from errorStack (pushed by data layer)

  3. Display error message to user

  4. Return early (no rendering)

3.3. Type Safety

TypeScript enforces proper error checking:

const result = await vfs.data_get('/home/user');

// ❌ TypeScript error - must check .ok first
console.log(result.value);

// ✅ Correct - check before access
if (!result.ok) {
  // Handle error
  return;
}
// TypeScript knows result.value exists here
const items = result.value;

This prevents accessing data that doesn’t exist due to errors.

4. API Reference

4.1. Public Methods

4.1.1. data_get(targetPath?: string, options?): Promise<Result<ListingItem[]>>

Fetches directory data without rendering.

Parameters:

Parameter Type Description

targetPath

(optional) string

Path to get data for. If omitted, uses CWD.

options.sort

(optional) enum

Sort field: 'name', 'size', 'date', 'owner'. Default: 'name'.

options.reverse

(optional) boolean

Reverse sort order. Default: false.

options.directory

(optional) boolean

Show directory info, not contents (like ls -d). Default: false.

Returns: Result<ListingItem[]>

  • Ok(items) - Success, items is an array of ListingItem

  • Err() - Failure, error details in errorStack

Example:

import { vfs } from './lib/vfs/vfs.js';
import { errorStack } from '@fnndsc/cumin';

const result = await vfs.data_get('/home/user', { sort: 'date', reverse: true });
if (!result.ok) {
  const error = errorStack.stack_pop();
  console.error(`Failed: ${error?.message}`);
  return;
}

// Use result.value - array of ListingItem
for (const item of result.value) {
  console.log(item.name);
}

4.1.2. list(targetPath?: string, options?): Promise<void>

Lists directory contents with rendering (convenience method).

Parameters:

Parameter Type Description

targetPath

(optional) string

Path to list. If omitted, uses CWD.

options.long

(optional) boolean

Long format with details. Default: false.

options.human

(optional) boolean

Human-readable sizes (requires long). Default: false.

options.sort

(optional) enum

Sort field: 'name', 'size', 'date', 'owner'. Default: 'name'.

options.reverse

(optional) boolean

Reverse sort order. Default: false.

options.directory

(optional) boolean

Show directory info, not contents. Default: false.

Returns: Promise<void> - Outputs to console, returns nothing

Example:

// Simple listing
await vfs.list();

// Long format with human-readable sizes
await vfs.list('/home/user', { long: true, human: true });

// Show directory info only
await vfs.list('/home/user/feeds/feed_123', { directory: true, long: true });

// Sort by date, newest first
await vfs.list('/bin', { sort: 'date', reverse: true });

4.1.3. virtualBinItems_get(): Promise<ListingItem[]>

Legacy method for backward compatibility. Returns /bin items directly.

Deprecated: Use data_get('/bin') instead for Result<T> pattern.

Returns: ListingItem[] - Empty array on error

Example:

const plugins = await vfs.virtualBinItems_get();
// No error handling - returns [] on failure

4.2. Private Methods

These are implementation details, not part of public API:

  • dataVirtualBin_get(options): Promise<Result<ListingItem[]>> - Handles /bin

  • dataNative_get(target, options): Promise<Result<ListingItem[]>> - Handles native paths

5. Usage Patterns

5.1. Pattern 1: Simple Listing (Presentation)

Use list() when you just want to display output:

// In builtins/index.ts
async function builtin_ls(args: string[]): Promise<void> {
  const parsed = parseArgs(args);
  await vfs.list(parsed._[0], {
    long: !!parsed.l,
    human: !!parsed.h,
    sort: parsed.sort || 'name',
    reverse: !!parsed.r,
    directory: !!parsed.d
  });
}

5.2. Pattern 2: Data Processing (Fetch Then Transform)

Use data_get() when you need to process data:

async function countTextFiles(dir: string): Promise<number> {
  const result = await vfs.data_get(dir);

  if (!result.ok) {
    return 0; // Or handle error differently
  }

  return result.value.filter(item => item.name.endsWith('.txt')).length;
}

5.3. Pattern 3: Wildcard Expansion

Combine with cache for efficient wildcard matching:

async function wildcard_expand(pattern: string): Promise<Result<string[]>> {
  const cwd = await session.getCWD();
  const listCache = listCache_get();

  // Check cache first
  let items = listCache.cache_get(cwd);
  if (!items) {
    // Use VFS to get fresh data
    const result = await vfs.data_get(cwd);
    if (!result.ok) {
      return result; // Propagate error
    }
    items = result.value;
    listCache.cache_set(cwd, items);
  }

  // Filter matches locally
  const matches = items
    .filter(item => minimatch(item.name, pattern))
    .map(item => item.name);

  return Ok(matches);
}

5.4. Pattern 4: Tab Completion

Prefetch directory data for immediate tab completion:

async function completer_prefetch(dir: string): Promise<void> {
  const result = await vfs.data_get(dir);

  if (result.ok) {
    // Cache populated by data_get() - ready for tab completion
    const listCache = listCache_get();
    const cached = listCache.cache_get(dir);
    // cached is now available for fast lookups
  }
}

5.5. Pattern 5: Composing Operations

Chain multiple data operations:

async function findLargestFile(dir: string): Promise<string | null> {
  const result = await vfs.data_get(dir);
  if (!result.ok) return null;

  const files = result.value.filter(item => item.type === 'file');
  if (files.length === 0) return null;

  const largest = files.reduce((max, item) =>
    item.size > max.size ? item : max
  );

  return largest.name;
}

6. Options

6.1. Sort Options

Control how results are sorted:

Value Description

'name'

Alphabetical by name (default)

'size'

By file size, largest first

'date'

By modification date, newest first

'owner'

Alphabetical by owner name

Note: Use reverse: true to invert sort order.

6.2. Directory Flag (-d)

The -d flag shows information about the directory/file itself, not its contents:

# Without -d: shows contents
ls /home/user/feeds/feed_123
# Output: file1.txt, file2.txt, ...

# With -d: shows feed info
ls -ld /home/user/feeds/feed_123
# Output: drwxr-xr-x  user  2025-01-15  feed_123

Implementation:

  1. Extract parent directory and item name

  2. Fetch parent directory listing

  3. Find item by name in parent listing

  4. Return single-item array: Ok([item])

This is particularly useful for:

  • Checking if a path exists

  • Getting link target information

  • Showing directory metadata without listing contents

7. Caching Strategy

VFS integrates with cumin’s ListCache for performance:

7.1. Cache on Fetch

Every successful data fetch populates the cache:

const items = await files_list({}, target);
listCache_get().cache_set(target, items); // Cache for next time
return Ok(items);

7.2. Cache Check Before Fetch

For operations that may hit cache elsewhere (like -d flag):

let parentItems = listCache.cache_get(parentPath);
if (!parentItems) {
  // Cache miss - fetch and populate
  parentItems = await files_list({}, parentPath);
  listCache.cache_set(parentPath, parentItems);
}

7.3. Automatic Invalidation

Cache automatically clears on CWD change (handled by cumin):

// In chrisContext (cumin)
await session.setCWD('/new/path');
// Automatically triggers: listCache.cwd_update('/new/path')
// Which clears cache if path changed

7.4. Performance Impact

See cumin’s ListCache documentation for performance measurements.

8. Error Handling

8.1. Error Flow

  1. Data layer encounters error (API failure, invalid path, etc.)

  2. Push to errorStack with descriptive message

  3. Return Err() to signal failure

  4. Presentation layer checks result.ok

  5. Pop from errorStack to get error details

  6. Display to user with appropriate formatting

8.2. Error Messages

VFS pushes user-friendly error messages:

// Not found
errorStack.stack_push("error", `ls: cannot access '${target}': No such file or directory`);

// API failure
errorStack.stack_push("error", `Failed to list ${target}: Network timeout`);

// Plugin listing failure
errorStack.stack_push("error", `Failed to list plugins: ${error.message}`);

8.3. Error Propagation

Callers propagate errors by checking and returning:

const result = await vfs.data_get(path);
if (!result.ok) {
  // Error already in errorStack, just propagate failure
  return Err();
}
// Continue with result.value

8.4. Testing Error Handling

describe('VFS error handling', () => {
  it('should return Err for nonexistent path', async () => {
    const result = await vfs.data_get('/nonexistent');

    expect(result.ok).toBe(false);

    const errors = errorStack.stack_search("cannot access");
    expect(errors.length).toBeGreaterThan(0);
  });

  it('should handle API failures gracefully', async () => {
    jest.spyOn(files, 'list').mockRejectedValue(new Error('Network error'));

    const result = await vfs.data_get('/path');

    expect(result.ok).toBe(false);
    const errors = errorStack.stack_search("Network error");
    expect(errors.length).toBeGreaterThan(0);
  });
});

9. Migration from Old API

9.1. Before: Mixed Concerns

Old VFS mixed data fetching with rendering:

// Old API
class VFS {
  async listVirtualBin(options): Promise<void> {
    const items = await fetchItems();
    if (items.length === 0) {
      console.log(chalk.gray('No plugins found.'));
      return;
    }
    console.log(grid_render(items));
  }
}

// Usage - can't get data without console output
await vfs.listVirtualBin({ long: true });

9.2. After: Separated Concerns

New VFS separates data from presentation:

// New API
class VFS {
  // Data layer
  async dataVirtualBin_get(options): Promise<Result<ListingItem[]>> {
    const items = await fetchItems();
    return Ok(items); // No console output
  }

  // Presentation layer
  async list(path?: string, options?): Promise<void> {
    const result = await this.data_get(path, options);
    if (!result.ok) {
      const error = errorStack.stack_pop();
      console.error(chalk.red(error?.message));
      return;
    }
    console.log(grid_render(result.value));
  }
}

// Usage - choose data or presentation
const items = await vfs.data_get('/bin'); // Just data
await vfs.list('/bin'); // Rendered output

9.3. Migration Steps

  1. Replace direct method calls with data_get() + list()

  2. Add Result<T> handling with .ok checks

  3. Pop errors from errorStack for display

  4. Update tests to check Result<T> returns

10. Design Decisions

10.1. Why Separate data_get() from list()?

Benefits:

  • Testability: Test data fetching without console output

  • Composability: Feed data into other operations

  • Flexibility: Caller decides how to render

  • Type safety: Result<T> forces error handling

Trade-off: Slight API complexity (two methods instead of one)

Verdict: Separation is worth it for architectural cleanliness and Result<T> pattern.

10.2. Why Result<T> + errorStack, Not Exceptions?

See cumin’s Error Handling documentation for full rationale.

Summary:

  • Result<T> makes errors explicit in signatures

  • errorStack accumulates user-friendly messages

  • TypeScript enforces error checking

  • Composable error handling

10.3. Why Inject /bin at Root, Not Mount System?

Current approach injects /bin when listing /:

if (target === '/' || target === '') {
  items.push({
    name: 'bin',
    type: 'vfs',
    // ...
  });
}

Pros:

  • Simple implementation

  • No mount registry needed

  • Clear that /bin is special

Cons:

  • Hard to add more virtual directories

  • Injection logic scattered

Future: Consider mount registry for multiple virtual directories.

10.4. Why Cache in Data Layer, Not Presentation?

Caching happens in dataVirtualBin_get() and dataNative_get(), not in list():

Pros:

  • Cache shared across all callers (wildcard, tab completion, list)

  • data_get() returns cached data for any use case

  • Single source of truth

Cons:

  • Data layer has side effects (cache writes)

Verdict: Pragmatic - caching is fundamental to data fetching, not presentation.

11. Future Enhancements

11.1. Mount Registry

Replace hardcoded /bin check with mount registry:

interface VirtualMount {
  path: string;
  handler: (options) => Promise<Result<ListingItem[]>>;
}

const mounts: VirtualMount[] = [
  { path: '/bin', handler: this.getDataVirtualBin },
  { path: '/proc', handler: this.getDataVirtualProc },
  { path: '/etc', handler: this.getDataVirtualEtc },
];

// In data_get()
const mount = mounts.find(m => effectivePath.startsWith(m.path));
if (mount) {
  return await mount.handler(options);
}

11.2. Streaming for Large Directories

Currently loads entire directory into memory. For huge directories:

async *getDataStream(path: string): AsyncGenerator<Result<ListingItem>> {
  for await (const chunk of fetchChunked(path)) {
    for (const item of chunk) {
      yield Ok(item);
    }
  }
}

11.3. Recursive Operations

Support recursive directory traversal:

async *walk(path: string, options): AsyncGenerator<Result<ListingItem>> {
  const result = await this.data_get(path, options);
  if (!result.ok) {
    yield result;
    return;
  }

  for (const item of result.value) {
    yield Ok(item);
    if (item.type === 'dir') {
      yield* this.walk(`${path}/${item.name}`, options);
    }
  }
}

11.4. Watch Mode

Detect external changes and invalidate cache:

async watch(path: string, callback: (result: Result<ListingItem[]>) => void) {
  // Poll or WebSocket for changes
  const changes = await detectChanges(path);
  if (changes) {
    listCache.cache_invalidate(path);
    const fresh = await this.data_get(path);
    callback(fresh);
  }
}

12. References

12.1. Related Documentation

12.2. Related Files

  • src/lib/vfs/vfs.ts - VFS implementation

  • src/builtins/index.ts - Command wrappers using VFS

  • src/builtins/wildcard.ts - Wildcard expansion using VFS + cache

Last updated: 2025-12-02