Lesson 06

Streams for large payloads

Stream large files or responses instead of loading everything into memory.

streams
memory
http

Good Code

src/routes/download.ts

import { createReadStream } from "node:fs";
import { pipeline } from "node:stream/promises";
import type { ServerResponse } from "node:http";

export async function sendExport(filePath: string, response: ServerResponse) {
  response.setHeader("content-type", "text/csv");

  try {
    // pipeline coordinates backpressure and stream errors.
    await pipeline(createReadStream(filePath), response);
  } catch (error) {
    console.error({ error, filePath }, "Export stream failed");
    if (!response.headersSent) {
      response.statusCode = 500;
      response.end("Could not stream export.");
    }
  }
}

Bad Code

src/routes/download.ts

import { readFile } from "node:fs/promises";
import type { ServerResponse } from "node:http";

export async function sendExport(filePath: string, response: ServerResponse) {
  // Loading the whole export scales memory with file size.
  const file = await readFile(filePath);

  response.setHeader("content-type", "text/csv");
  response.end(file);
}

Review Notes

What to review

Good Code

The good version streams the file through pipeline, which manages backpressure and surfaces stream failures.

Bad Code

The bad version reads the entire file into memory before sending it, which can pressure memory or crash the process for large exports.

Takeaways

For large payloads, prefer pipeline-based streaming with an explicit error path.