Streaming Files in Node Without Blowing Your Memory Budget

File handling is where many Node services quietly fall over. A developer tests an upload with a small file, it works, and the code buffers the whole thing into memory. Then a user uploads a two gigabyte video, the process balloons, and the container gets killed for exceeding its memory limit. The fix is to think in streams. Node was designed around streaming: data flows through your program in chunks rather than arriving all at once, so memory stays flat regardless of file size. Streams power reading from disk, writing to network sockets, parsing multipart uploads, compressing responses, and piping data to cloud storage. Getting comfortable with readable streams, writable streams, piping, and backpressure unlocks scalable file handling. This post covers how streams work, how to parse multipart uploads safely, and how to move data to storage without ever holding a whole file in memory.

Why Buffering Fails

The naive way to handle a file is to read it entirely into a buffer, then act on it. For a small configuration file this is fine. For user uploads it is a liability. Each in-flight request that buffers a large file consumes memory equal to the file size, so a handful of concurrent uploads of large files can exhaust the heap and crash the process. Worse, buffering adds latency: you cannot start writing to disk or forwarding to storage until the entire file has arrived. Memory pressure also triggers more frequent garbage collection, which pauses the event loop and slows every request. In containerized deployments with hard memory limits, the orchestrator simply terminates a process that exceeds its allowance, turning one large upload into an outage. The lesson is that file size in production is unbounded and adversarial, so any approach whose memory grows with file size is a ticking problem.

Streams As Chunked Pipes

A stream processes data piece by piece as it becomes available, keeping memory roughly constant no matter the total size. Node has four stream types. Readable streams are sources you pull data from, like a file on disk or an incoming request body. Writable streams are destinations you push data into, like a file being written or a response being sent. Duplex streams are both, like a TCP socket. Transform streams are duplex streams that modify data passing through, like compression or encryption. The core operation is connecting a readable to a writable so chunks flow automatically from source to destination. Instead of loading a file and writing it, you connect the read stream to the write stream and let Node move the bytes in small chunks. Memory usage stays tied to the chunk size and internal buffers, not the file, which is exactly the property you want under load.

Piping And Pipeline

Connecting streams is called piping, and it is the idiomatic way to move data in Node. The classic pipe method links a readable to a writable, but it has a serious flaw: if an error occurs midstream, pipe does not automatically clean up the other streams, which leaks file descriptors and memory over time. The modern, correct tool is the pipeline function from the stream module. It connects any number of streams in sequence, propagates errors, and destroys every stream properly when the chain finishes or fails, calling you back or resolving a promise at the end. You can insert transform streams in the middle, for example piping a file through a compression stream into a response, all in one declarative chain. Always prefer pipeline over manual pipe for production code. The error handling and cleanup it provides are not optional niceties; they are what keeps a long-running server from slowly degrading.

Backpressure Explained

Backpressure is the mechanism that prevents a fast source from overwhelming a slow destination. Imagine reading from a fast local disk and writing to a slow network upload. If the reader produced chunks as fast as it could, they would pile up in memory waiting for the writer, defeating the purpose of streaming. Instead, when a writable stream's internal buffer fills, its write call signals that it is full, and a properly connected readable pauses until the writable drains and emits a drain event, then resumes. The pipe and pipeline mechanisms handle this signaling automatically, which is the main reason to use them rather than manually listening for data events and writing yourself. If you do consume a stream manually, you must respect the return value of write and the drain event, or you reintroduce the unbounded memory growth that streaming was supposed to solve. Backpressure is the quiet hero of stable file handling.

Parsing Multipart Uploads

Browser file uploads arrive as multipart form data, a format that interleaves file bytes with field boundaries and metadata in a single request body. You cannot just read the body as a file; you must parse the multipart structure. In Express, the standard tool is Multer, which parses multipart requests and exposes uploaded files. Crucially, configure it to stream rather than buffer when files can be large, and always set limits on file size and field count to reject abusive uploads before they consume resources. Multer can write directly to disk, but in cloud deployments you often want a custom storage engine that streams each file part straight to object storage. Validate the declared content type and, ideally, sniff the actual file signature, because clients lie about MIME types. Treat every byte of an upload as untrusted: enforce size limits at the parser, scan or constrain file types, and never derive a filesystem path directly from a user-supplied filename.

Streaming To Cloud Storage

In modern deployments you rarely keep uploads on the local disk, since containers are ephemeral. Instead you stream files to object storage like S3, MinIO, or a compatible service. The pattern is to take the readable stream of the incoming file part and pipe it directly into the storage client's upload, so bytes flow from the client through your server to storage without ever fully landing in memory or on disk. The AWS SDK's managed upload handles multipart uploads to S3 from a stream, splitting large files into parts automatically and retrying failed parts. For very large or untrusted uploads, a stronger pattern is presigned URLs: your server issues a short-lived signed URL and the client uploads directly to storage, bypassing your server entirely. That removes upload bandwidth and memory from your application's concern and scales effortlessly. Choose direct streaming when you must process or validate bytes, and presigned URLs when you can let storage handle it.

Downloads And Range Requests

Serving files back deserves the same streaming discipline. Read the file as a stream and pipe it to the response rather than loading it into memory, so a large download does not spike your heap. Set the correct content type and content length headers so clients and proxies behave well. For media like video and audio, support HTTP range requests: the browser asks for a byte range so it can seek and stream progressively, and your server responds with a partial content status and only the requested slice of the file. Implementing ranges means reading the file with a start and end offset and setting the content range header accordingly. Without range support, seeking in a video forces a full re-download, which feels broken. Also consider letting a reverse proxy or CDN serve static files directly, since they are far more efficient at it than your Node process and free the event loop for real work.

Hardening And Limits

Streaming solves memory, but file handling has a wide attack surface that needs explicit hardening. Always enforce a maximum file size at the parser so an attacker cannot stream an endless body and fill your disk or storage bill. Limit the number of files and fields per request. Never trust the client-provided filename: generate your own identifier, often a UUID, and store the original name separately as metadata, which prevents path traversal and collisions. Validate the real content type by inspecting the file's magic bytes rather than the declared header. Set timeouts so a slow or stalled upload cannot hold a connection open indefinitely, a classic denial of service vector. Scan uploads for malware if they will be served to other users. Finally, store files outside any web-served directory and serve them through controlled endpoints with authorization checks. Treat the upload path as a security boundary, because for many breaches that is exactly what it is.