p9r.dev

Content Feeds

RSS Aggregation Engine

A unified feed ingestion pipeline that normalizes Atom, RSS 2.0, and JSON Feed formats into a single canonical stream. Built on event-driven architecture with backpressure-aware consumers.

$ p9r feed --subscribe https://example.com/rss
[OK] Feed registered: example.com
[OK] Polling interval: 300s
[OK] Format: RSS 2.0 (auto-detected)

Distributed Feed Mesh

Peer-to-peer feed synchronization across edge nodes. No central broker, no single point of failure. Each node maintains a partial view and gossips updates using a CRDT-based merge protocol.

Zero-Copy Parsing

Feed entries are parsed directly from the network buffer without intermediate allocation. Memory-mapped I/O meets streaming XML, achieving sub-millisecond per-entry throughput on commodity hardware.

Content Deduplication

SimHash-based near-duplicate detection that identifies semantically identical entries across feeds even when formatting differs. Configurable similarity thresholds with Hamming distance scoring.

Systems Architecture

type Feed struct {
    URI      string
    Interval time.Duration
    Parser   FeedParser
    Sink     chan<- Entry
}

func (f *Feed) Poll(ctx context.Context) {
    ticker := time.NewTicker(f.Interval)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            entries := f.Parser.Parse(f.URI)
            for _, e := range entries {
                f.Sink <- e
            }
        }
    }
}

Adaptive Polling

Feeds that update frequently get polled more often. Dormant feeds decay to long intervals. The scheduler uses exponential backoff with jitter, respecting HTTP Cache-Control and ETag headers.

WebSub Integration

Native PubSubHubbub/WebSub subscriber implementation. When hubs are available, polling is bypassed entirely in favor of real-time push notifications with cryptographic verification.

Plan 9 Philosophy

Everything is a file. Every feed is a filesystem. Mount your subscriptions, read entries as files, pipe them through the standard toolchain. Unix philosophy carried to its logical conclusion.

Feed Transformation Pipeline

A composable chain of entry processors: sanitize HTML, extract full text, generate summaries, tag with categories, rewrite links, normalize timestamps. Each stage is a standalone binary communicating over stdin/stdout.

$ p9r pipeline list
  sanitize   -> strip unsafe HTML tags
  fulltext   -> extract article body
  summarize  -> generate 2-sentence abstract
  categorize -> auto-tag via TF-IDF
  normalize  -> UTC timestamps, UTF-8

Storage Engine

Append-only log with LSM-tree indexing for feed entries. Write-optimized for high-throughput ingestion, with bloom filters for efficient existence checks during deduplication passes.

About p9r

Identity & Authentication

Feed subscriptions are cryptographically signed. Each node in the mesh verifies provenance before accepting updates. Ed25519 keypairs generated per-feed, stored in the local keyring.

Network Topology

The feed mesh self-organizes into a small-world graph. Discovery happens via mDNS on local networks and DHT lookup over the wider internet. No registration server, no directory, no gatekeepers.

Open by Default

Every protocol is documented. Every wire format is specified. Every algorithm is published. The feed mesh belongs to no one and everyone. Fork it, extend it, break it, rebuild it.

Export Formats

Render your aggregated feed as OPML, Atom, JSON Feed, or plain Markdown. Serve it over HTTP, write it to a file, or pipe it to another tool. Your data, your format, your choice.

$ p9r serve --port 8080 --format atom
Serving aggregated feed on :8080
  Subscriptions: 47 active feeds
  Entries: 12,841 indexed
  Uptime: 14d 7h 23m

Plugin Architecture

Extend the pipeline with custom processors. Write a plugin in any language that reads from stdin and writes to stdout. Register it in the pipeline config and it runs in sequence with the built-in stages.