Welcome to Issue #1

Hey there,

Welcome to the very first dispatch of my engineering newsletter! If you're reading this, you are successfully receiving emails from the custom Background Job Queue and SQLite database I just finished deploying to my portfolio.

In this issue, I want to briefly talk about why I moved away from flat-file JSON and built my own self-hosted delivery infrastructure instead of paying for a SaaS tool.

The move to SQLite

When initially prototyping the API, I relied on flat JSON files for persistence. It was fast to build, but it lacked ACID compliance. Concurrency issues meant that multiple subscribers signing up simultaneously could corrupt the file.

Instead of jumping straight to PostgreSQL and adding massive overhead to my server, I implemented better-sqlite3. Operating in WAL (Write-Ahead Logging) mode, it gives me concurrent reads and crash-recoverable writes, all inside a single local file.

How the background queue works

To prevent the web server from hanging while waiting for Google's SMTP servers to respond, I built a persistent queue. When you click "Send", the Express route simply logs a job and returns immediately:

// Add job to persistent queue
stmts.insertJob.run(
  'send_campaign', 
  JSON.stringify({ campaignId: id }), 
  'pending', 
  runAt
);

A completely separate Node worker_thread continually polls this SQLite table. If it finds a job, it pulls the subscriber list, transforms this markdown into an email-safe React component, and batches out the emails. If the server crashes mid-send, the worker simply picks up where it left off on reboot.

"Shipping great software is often just about making the boring, robust choices and executing them well."

What's next?

Next week, I'll be doing a deep dive into the React cache architecture and how to properly invalidate Server Components on demand.

Stay tuned, and thanks for being here early.

— John