Start a project

Backend case study

Trendwire

Backend editorial intelligence for Reddit. Trendwire runs as a scheduled pipeline that turns Reddit activity, historical context, and LLM analysis into structured editorial briefs.

See the architectureView all work
LLM passes
2
posts per run
50
run memory
SQLite
Trendwire generated editorial brief interface
trendwire --slot morning --email
fetch -> classify -> synthesize
write -> persist -> deliver

System overview

Built as a backend operating system for editorial signal

Trendwire is a scheduled Python pipeline for turning noisy Reddit activity into structured, reviewable briefs. It avoids a web-app dependency by running as a one-shot backend job with a lock file, clear output artifacts, and deployable Docker execution.

  1. Markdown brief

    Readable editorial output for each published run.

  2. JSON summary

    Structured run data with takeaways, candidates, and source comments.

  3. Email and Slack

    Optional delivery after primary files and database updates complete.

Pipeline capabilities

Backend System Overview

The pipeline keeps ingest, analysis, persistence, and delivery explicit so each stage can be inspected and extended.

One-Shot Scheduled Pipeline

Trendwire is intentionally built as a backend job, not a long-running web service. Cron, Docker Compose, or an operator invokes the CLI for a single run, while a non-blocking lock prevents overlapping executions.

Source-Aware Reddit Ingestion

The application supports scheduled multireddit snapshots and manual Texas-wide search runs behind a shared source strategy. It fetches posts and comments, normalizes them into internal models, and keeps source-specific output paths and run labels separate.

LLM-Assisted Editorial Analysis

Two LLM passes classify posts and synthesize a brief from classified threads, notable comments, and historical context. Deterministic rule scoring highlights firsthand, professional, and specific comments before synthesis.

Persistence and Historical Context

SQLite stores runs, posts, comments, classifications, rule scores, final flags, ranks, and snapshot metadata. Published runs can blend the current snapshot with archived hourly runs to surface persistent storylines.

Structured Output and Delivery

Each run writes validated markdown and JSON artifacts atomically. Published runs can update an index, render email text and HTML, send through Mailgun, and post Slack notifications without changing the core pipeline result.

Architecture

Architecture Flow

The system keeps source discovery, analysis, persistence, and delivery as explicit stages so each run can be inspected, retried, and extended without changing the whole pipeline.

  1. CLI

    Slot, source, cleanup, and delivery flags

  2. Reddit ingest

    Multireddit snapshots or manual Texas search

  3. Analysis

    Classification, rule scoring, and synthesis

  4. Persistence

    Runs, posts, comments, ranks, and flags

  5. Outputs

    Markdown, JSON, email, and Slack artifacts

Generated output

Delivery Surfaces

Backend work is still visual when the outputs are clear. This section shows the shape of the generated brief alongside the delivery artifacts produced by each run.

Subject: Texas Morning Brief

Top 5 takeaways

Strongest possible source

Comment context, link, and editorial note render from structured data.

Generated artifacts

  • .mdHuman-readable brief
  • .jsonStructured run payload
  • .email.htmlDelivery-ready HTML email
  • .email.txtPlain text fallback

Implementation

Implementation Details

The core keeps provider contracts narrow and orchestration readable. The snippets below show how the pipeline separates source data, analysis, and provider-specific LLM behavior.

Pipeline orchestrationpython
classifications = classify_posts_with_retry(
    provider,
    [thread.post for thread in threads],
)

analysis = run_analysis(
    provider,
    AnalysisInputs(
        threads=threads,
        historical_context=historical_context,
        classifications_override=classifications,
    ),
)
LLM provider contractpython
class LLMProvider(Protocol):
    def classify_posts(
        self,
        posts: list[IngestedPost],
    ) -> list[PostClassification]: ...

    def synthesize(
        self,
        classified_posts: list[ClassifiedPost],
        notable_comments: list[NotableCommentInput],
        historical_context: HistoricalContext | None,
    ) -> SynthesisResult: ...

Under the hood

Technology Stack

Trendwire is a Python backend application designed for reliable scheduled execution:

Runtime
Python CLI with Docker Compose support
Data Source
Reddit API via PRAW
AI Providers
Supports Claude, ChatGPT, Gemini, and local models
Storage
SQLite with run, post, comment, and persistence tables
Outputs
Markdown briefs, JSON summaries, email artifacts, and Slack notifications

Built for the real workflow

Turn noisy inputs into useful systems.

Westnode builds focused products, automations, and backend tools around the way a team actually works.

Start a projectView all work