How do I handle backpressure and large uploads?

Understand the concept

Backpressure is a feedback mechanism that slows a producer when the consumer cannot keep up. Without it, incoming chunks may accumulate in memory even though the application is using streams. Node streams expose flow-control signals and utilities that help coordinate reading, writing, cleanup and errors.

Important ideas

• Stream chunks instead of buffering the whole file.

• When write() returns false, wait for drain; pipeline helps manage stream flow and errors.

• Set upload limits and clean up partial files when an upload fails.

• Apply limits to total bytes and processing time, and do not trust only the client-provided content type.

Example — Node ESM file copy:

import { createReadStream, createWriteStream } from 'node:fs';
import { pipeline } from 'node:stream/promises';

await pipeline(
  createReadStream('input.csv'),
  createWriteStream('output.csv')
);

• pipeline coordinates backpressure and propagates stream failures; the surrounding job still needs failure reporting and partial-output cleanup.