Or how watching an image load changed the way we thought about programming.
I can’t remember exactly when I first encountered Node.js. It was probably early enough that we were still discovering the joys of asynchronous programming, which apparently involved nesting callbacks until our code resembled an archaeological excavation site.
I wasn’t using Node professionally at the time. I was just experimenting with it, and one of those experiments involved streaming an image to a browser. Nothing terribly ambitious, although I remember being absolutely fascinated by what happened next.
The image started appearing from the top down, progressively, as the data arrived. Unlike what I was accustomed to seeing with my Apache-powered applications, where the browser would sit there looking rather blank before the image suddenly appeared, this little experiment showed me something I’d never really appreciated before. The browser didn’t need the entire image before it could start doing something useful with it. I was blown away!
And before someone decides to explain how Apache was perfectly capable of streaming too, yes, we know. The interesting thing here wasn’t what the servers were capable of doing. It was what we actually did with them, and more importantly, how we thought about data processing in the first place.
That experiment got me thinking about events and streams, and I eventually wrote an entire application around them. I kept the event handlers in a single file, used named functions and organised the listeners so that you could follow what was happening without having to decipher an endless chain of anonymous callbacks. It was clean, surprisingly expressive code, particularly for a time when Node examples were frequently an exercise in spaghetti appreciation.
The funny thing is that I wasn’t a functional programmer. I didn’t even know what functional programming was, much less that some of the ideas I’d been exploring had anything to do with it. I simply liked the way Node allowed me to organise work around things happening, rather than things finishing.
That distinction stayed with me, and it eventually led to a rather peculiar analogy involving rivers, dams and immigration checkpoints. But first, let’s revisit some old code, because the history is considerably more interesting when we look at what people actually wrote.
When Node met the checkpoint
One of the things I liked about early Node was how little ceremony there was around HTTP. You could write headers, send some data and close the response whenever you were ready. Here’s an actual fragment from Node’s v0.1.90 test suite:
res.writeHead(200, {"Content-Type": "text/plain"});
res.write("The path was " + url.parse(req.url).pathname);
res.end(); Source: Node v0.1.90 HTTP tests
There’s something beautifully simple about that. write() sends data into the response, while end() signals that we’re finished. Those are separate operations, which means we could write again, or produce data in response to another event, without requiring the entire response to exist beforehand.
Of course, this didn’t mean that everyone suddenly began designing applications around continuous data flow. Quite the opposite, actually. Much of the emerging ecosystem was concerned with making Node familiar to web developers who already understood routes, controllers and templates. Enter Express. Oh, how I hated early Express.
It wasn’t that Express couldn’t use streams, and we’re not particularly interested in arguing about when it acquired this feature or that API. What bothered me was the tendency to take something fundamentally interesting about Node and wrap it in the same programming model we’d been using for years.
Here’s a small excerpt from Express 0.10.0’s HTTP client implementation:
res
.addListener('data', function(chunk){ buf += chunk })
.addListener('end', function(){ callback(null, buf, res) }) Source: Express 0.10.0 HTTP client
Look carefully at what’s happening. The HTTP response delivers data incrementally through data events, but the implementation concatenates everything into a string and waits for end before handing the result to the caller.
There’s nothing technically incorrect about this. If the caller genuinely needs the complete response, it’s a convenient abstraction. But the abstraction has made that decision on the caller’s behalf. Even if the first chunk contains something we could already process, we’re going to wait until the entire response arrives.
Express 0.10 also had a method called Request#stream(), which forwarded incoming data to the HTTP response as chunks arrived. Interestingly, the implementation called response.write() without checking its return value, so it didn’t explicitly regulate the source when the destination’s writable buffer filled up. The original implementation is still available.
That’s a subtle but rather important difference. Passing chunks along as they arrive isn’t quite the same as implementing a properly controlled streaming pipeline. The latter needs to account for what happens when the receiving end can’t keep up.
And that brings us back to my original fascination. The really interesting part of streams wasn’t merely that we could deliver something before an operation finished. It was that we could allow data to continue flowing through an application, transforming it along the way, while retaining control over how much work the system could accommodate.
To illustrate this, let’s reproduce the original idea without depending on how a particular browser renders an image. We’ll create two HTTP endpoints that consume an identical asynchronous source. One will collect the entire result before responding, while the other will deliver each chunk as it becomes available.
// server.mjs
import { createServer } from 'node:http';
import { once } from 'node:events';
import { setTimeout as sleep } from 'node:timers/promises';
async function* produce() {
for (const value of ['One\n', 'Two\n', 'Three\n', 'Four\n']) {
await sleep(400);
yield value;
}
}
async function checkpoint(response) {
const chunks = [];
for await (const chunk of produce()) chunks.push(chunk);
response.end(chunks.join(''));
}
async function river(response) {
for await (const chunk of produce()) {
if (!response.write(chunk)) await once(response, 'drain');
}
response.end();
}
async function onRequest(request, response) {
response.setHeader('Content-Type', 'text/plain');
try {
if (request.url === '/checkpoint') return await checkpoint(response);
if (request.url === '/river') return await river(response);
response.writeHead(404).end('Not found');
} catch (error) {
response.destroy(error);
}
}
createServer(onRequest).listen(3000); Run the server with node server.mjs, then compare curl -N localhost:3000/checkpoint and curl -N localhost:3000/river. Both endpoints produce the same output, but the first remains silent for approximately 1.6 seconds before delivering everything at once. The second starts producing output after roughly 400 milliseconds and continues delivering it as the source yields more data.
What’s interesting is that the source hasn’t changed. Neither has the eventual result. We’ve changed only the conditions under which data becomes available to the consumer.
Now look at the condition inside river():
if (!response.write(chunk)) {
await once(response, 'drain');
} When the writable stream reports that its internal buffer has reached its configured threshold, we stop requesting more data from the source until the destination signals that it can accept more. We haven’t imposed a requirement for the entire operation to finish. We’ve simply regulated the flow according to the capacity of the receiving system. This is called backpressure, and it’s one of the most important aspects of streaming.
Think about an actual river for a moment. A dam doesn’t exist because someone decided the river should reach the sea before its water becomes useful. It regulates how much water passes through, retaining some when necessary and releasing more when conditions permit. We can divert water into other channels or use it to generate electricity as it flows.
Now imagine constructing the Great Wall of China across that river instead. Good fucking luck.
That’s what the immigration checkpoint represents in this analogy. It’s a hard completion boundary imposed on the journey, irrespective of whether useful work could already be happening elsewhere.
A checkpoint certainly has its purpose. There are situations where an entire operation must satisfy some condition before the result can be accepted. But that’s a requirement about when something may be released, which is not necessarily the same as a requirement about when it may be processed. And once we recognise that distinction, streams become considerably more interesting than a way to serve files.
The work happens in the river
Let’s consider something more useful than four strings arriving every 400 milliseconds. Suppose we have a large collection of sensor readings stored as newline-delimited JSON. We need to inspect each record, convert its temperature from Celsius to Fahrenheit, compress the result and write it to another file.
There’s a perfectly conventional way of doing this. Read the entire file, split it into lines, parse the JSON, perform the transformations and write the completed result. For small datasets, that’s often entirely reasonable.
But suppose the source is several hundred gigabytes, or perhaps a few terabytes. Reading everything into memory stops being particularly attractive, and even if we have enough memory, there’s no compelling reason to make the first transformation wait for the final byte of the file.
The interesting thing is that each sensor reading can be processed independently once its complete record has arrived. We don’t need the entire dataset. We need only enough data to identify and validate the current record. Here’s a modern Node implementation using asynchronous generators and the native streaming pipeline:
// process.mjs
import { createReadStream, createWriteStream } from 'node:fs';
import { rename, rm } from 'node:fs/promises';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';
const MAX_RECORD = 1024 * 1024;
const temporary = 'readings.ndjson.gz.part';
const destination = 'readings.ndjson.gz';
async function* parseRecords(source) {
const decoder = new TextDecoder();
let pending = '';
function* extractLines() {
let boundary;
while ((boundary = pending.indexOf('\n')) !== -1) {
const line = pending.slice(0, boundary);
pending = pending.slice(boundary + 1);
if (line.length > MAX_RECORD) throw new Error('Record exceeds size limit');
if (line.trim()) yield JSON.parse(line);
}
if (pending.length > MAX_RECORD) throw new Error('Record exceeds size limit');
}
for await (const chunk of source) {
pending += decoder.decode(chunk, { stream: true });
yield* extractLines();
}
pending += decoder.decode();
yield* extractLines();
if (pending.trim()) yield JSON.parse(pending);
}
async function* convertTemperatures(records) {
for await (const record of records) {
if (typeof record?.sensor !== 'string' ||
!Number.isFinite(record.temperatureC)) {
throw new Error('Invalid sensor reading');
}
yield JSON.stringify({
sensor: record.sensor,
temperatureF: record.temperatureC * 9 / 5 + 32
}) + '\n';
}
}
try {
await pipeline(
createReadStream('readings.ndjson'),
parseRecords,
convertTemperatures,
createGzip(),
createWriteStream(temporary)
);
await rename(temporary, destination);
} catch (error) {
await rm(temporary, { force: true });
throw error;
} There’s quite a lot happening in those few lines, and almost none of it involves waiting for the entire source to arrive. createReadStream() supplies chunks from the file, parseRecords() identifies complete JSON records and yields them individually, and convertTemperatures() validates and transforms each record. Meanwhile, createGzip() compresses the output before it reaches the destination file.
The stages operate as data becomes available, with backpressure coordinated by the pipeline. If the destination cannot keep up, upstream consumption is regulated rather than allowing intermediate buffers to grow indefinitely.
But notice something else about parseRecords(). We’re accumulating data in pending, which might seem contradictory given everything we’ve been discussing. Not at all. A filesystem read has no obligation to return complete JSON records. It might deliver half a line, several lines or even part of a multibyte character. We need a temporary buffer to reconstruct each logical record before attempting to parse it, and TextDecoder maintains the necessary character-decoding state across chunks.
Once a complete line is available, we parse it and release the record downstream. We also reject oversized records rather than allowing malformed input to accumulate indefinitely. That’s precisely the control we’re talking about: buffer what the operation actually requires, for as long as it requires it, then allow the result to continue flowing. There’s no requirement for the entire file to be present simply because one record needs to arrive intact.
And now consider what happens if a record fails validation halfway through the file. The pipeline throws, the operation fails and our error handler removes the temporary output. We deliberately write to a .part file because we don’t want consumers mistaking an incomplete dataset for a finished one.
Only after the entire pipeline has completed successfully do we rename the temporary file to its final destination. For a rename on the same filesystem, this provides an atomic namespace change, although applications requiring crash durability need additional measures such as synchronisation. The distinction is still useful: the data has already been processed and written while it was arriving, but its final publication is conditional on successful completion.
We haven’t needed to accumulate the entire dataset in memory to achieve that. If we wanted to introduce encryption before writing to persistent storage, that would simply become another stage in the pipeline. The encryption scheme would still need to handle integrity and authentication requirements correctly, but there’s nothing about encryption that inherently requires us to abandon streaming.
We can also observe or divert data while it travels through the pipeline. A hashing transform, for example, could update a digest while forwarding the original chunks downstream. An independent output branch could produce diagnostics or audit records, provided we account for the backpressure and failure behaviour of both destinations.
This is where streams become architecturally interesting. The individual transformations can remain small and focused while the pipeline coordinates the movement of data between them. We don’t need every stage to understand the complete journey, and we certainly don’t need every stage to hold the entire dataset.
There’s one final detail worth noticing. Our example uses await pipeline(), which returns a Promise. Doesn’t that rather defeat the premise of an article comparing Promises and Streams? Actually, no. A Promise describes the eventual outcome of an operation. It allows us to represent completion or failure, which is precisely what we need when deciding whether to publish our temporary file.
The stream, meanwhile, describes the incremental movement of data during that operation. Our await doesn’t prevent the pipeline from processing records as they arrive; it simply waits for the overall operation to complete before moving on to the final commit.
Promises and Streams solve different problems, and they work rather nicely together. The trouble starts when we make completion a prerequisite for processing even when the work itself doesn’t require it.
Back to that image
Looking back, I find it rather amusing that my first real fascination with Node came from watching an image load in a browser. There was no grand architectural exercise involved, and I certainly wasn’t thinking about functional programming or data-processing pipelines.
I was simply exploring something new and noticed that the data was already useful before the operation had finished. That discovery changed how I thought about programming. I started organising applications around events and data flow, long before I knew the names for many of the concepts I was using. Years later, the same principles would remain useful when dealing with substantially larger datasets and more demanding systems.
The real lesson isn’t that we should replace every Promise with a Stream, or that buffering data is inherently wrong. Sometimes a complete result is exactly what the next operation needs, and sometimes a simple array is entirely sufficient. What matters is whether we’ve made that decision deliberately.
If a transformation can happen while data is arriving, why force the entire source to complete first? If a consumer is slower than its producer, why allow memory to grow without regulating the flow? And if the final result must be validated before publication, why should that prevent the preceding work from happening incrementally?
These are the questions that streams encourage us to ask. They give us control over the movement of data, the work performed along the way and the conditions under which results are finally released.
We’ve spent decades building software around the idea of completing one thing before starting another. Sometimes that’s unavoidable, but often we’ve simply become comfortable with the checkpoint. And that’s rather unfortunate, because the river was doing perfectly fine.
If there’s too much water, build a dam. Regulate it, divert it, even stop it deliberately when the situation demands it. The whole point is that we get to decide how the water moves and what happens while it’s moving. Just don’t build the Great Wall of China across the bloody thing. That’s not flow control. That’s immigration.
