What Is a Message Queue - and When You Actually Don't Need One
What is a message queue usually gets asked right after something timed out. A service needed another service to do some work, called it directly, and waited - and when the second service was slow or down, the first one failed with it.
A message queue puts a durable buffer between them. The producer writes a message and moves on. The consumer picks it up whenever it is ready. Neither has to be available at the same moment as the other, and neither has to run at the same speed.
That decoupling is the whole product, and it is not free. You get an extra system to operate, ordering guarantees that are weaker than you expect, and duplicate deliveries you now have to handle yourself.
This page covers the mechanism, the cases where the trade is clearly worth it, and the cases where a table in the database you already run does the same job with none of it.
| Concept | Message queue (MQ) |
|---|---|
| Where it sits | Infrastructure - sits between two services, at the messaging layer |
| In one sentence | A durable buffer that accepts work from a producer and holds it until a consumer is ready to process it. |
The 60-Second Answer
A message queue lets one service hand work to another without waiting for it.
The problem it solves: the producer and the consumer do not have to be available at the same time or run at the same rate. A slow or restarting consumer becomes a growing backlog instead of a cascade of failures upstream.
The cost it adds: a second system to run and monitor, at-least-once delivery (so your consumers must tolerate duplicates), and the loss of a straightforward answer to "did that actually work?"
How It Actually Works
The mechanism, step by step
- 1. The producer hands off and stops caring
- 2. The broker persists the message before acknowledging it
- 3. The consumer pulls when it is ready
- 4. The message is not deleted until the consumer acknowledges it
- 5. Delivery is at-least-once, so duplicates are normal
- 6. Failed messages go somewhere instead of vanishing
- 7. Ordering and parallelism pull against each other
1. The producer hands off and stops caring
Publish, get an ack, move on
The producing service writes a message to the broker and receives an acknowledgement that it was accepted. At that point its job is done - it does not know or care which consumer will handle the work, or when.
API request -> write message -> return 202 to caller
|
+-> (later) consumer processes it
This is what turns a synchronous dependency into an asynchronous one. The caller's latency is now the cost of a single write, not the cost of the downstream work.
⚠️ In production — Your API now returns before the work is done, which means the user-facing contract changed. If the caller expects a result rather than a receipt, a queue does not fit here without also building a way to check status later.
2. The broker persists the message before acknowledging it
Durability is the difference from an in-memory channel
A real queue writes the message to disk (usually replicated) before telling the producer it was accepted. That is what makes it survive a broker restart.
An in-process channel or an in-memory list gives you the same programming model with none of this guarantee - a restart loses everything in flight.
⚠️ In production — Durability is often configurable, and the fast default is sometimes the unsafe one. If persistence is asynchronous, an acknowledged message can still be lost in a crash - check what your broker actually promises rather than assuming.
3. The consumer pulls when it is ready
Rate is set by the consumer, not the producer
Consumers fetch messages at their own pace. If a traffic spike produces ten times the usual volume, the queue absorbs it and the consumer works through the backlog at whatever rate it can sustain.
This is the property people mean when they say a queue "smooths load". Nothing is faster - the work is simply spread over more time.
⚠️ In production — A backlog that never drains is not smoothing, it is a capacity problem with a longer fuse. Queue depth needs an alert on its trend, not just on an absolute number, or you find out when the messages are hours old.
4. The message is not deleted until the consumer acknowledges it
Invisible while in flight, redelivered on failure
When a consumer takes a message, the broker hides it rather than deleting it. Only after the consumer acknowledges successful processing does it disappear.
If the consumer crashes, the acknowledgement never arrives, the visibility window expires, and the message returns to the queue for someone else to take.
⚠️ In production — The visibility window has to be longer than your slowest legitimate processing time. Set it too short and a slow-but-healthy consumer gets its message stolen and processed twice while it is still working on it.
5. Delivery is at-least-once, so duplicates are normal
The redelivery guarantee has a cost you inherit
Because redelivery is how failures are handled, the same message can be processed more than once - a crash after the work but before the acknowledgement is indistinguishable from a crash before the work.
That makes duplicate handling the consumer's responsibility. In practice this means making the processing idempotent, usually via a key that identifies the logical operation.
⚠️ In production — This is the single most common thing teams skip when adopting a queue. It does not show up in testing, because tests do not crash mid-acknowledgement - it shows up as duplicate emails or double charges under real failure.
6. Failed messages go somewhere instead of vanishing
The dead-letter queue is not optional
A message that fails repeatedly must eventually stop being retried, or it blocks the consumer and burns capacity forever. Brokers handle this by moving it to a dead-letter queue after a configured number of attempts.
The dead-letter queue is where you look when something quietly did not happen.
⚠️ In production — A dead-letter queue nobody alerts on is a silent data loss mechanism with extra steps. It needs monitoring from the day the queue goes into production, not after the first incident.
7. Ordering and parallelism pull against each other
You can have strict order or many consumers, rarely both
Processing messages in a guaranteed order requires them to go through one consumer at a time. Running many consumers in parallel is what gives you throughput. These are directly opposed.
The usual compromise is ordering within a partition or key - all messages for one user or one order stay in sequence, while different keys process in parallel.
⚠️ In production — Teams frequently assume global ordering because a single-consumer test showed it. The first time the consumer scales to two instances, the ordering assumption breaks and the bug looks like a data race in business logic.
## Where You Meet It
- An e-commerce checkout that needs to send a confirmation email, update inventory, and notify a warehouse. Doing all three inside the request makes checkout as slow and as fragile as the slowest of them; publishing three messages makes checkout depend only on the broker.
- A video or image upload pipeline. The upload returns as soon as the file is stored, and transcoding happens behind a queue where a burst of uploads becomes a backlog rather than a pile of timeouts.
- A service that calls a third-party API with a strict rate limit. The queue holds the work and consumers drain it at a rate you control, instead of your traffic pattern deciding whether you get throttled.
- A nightly batch that used to run as one long job. Splitting it into messages lets the work spread across instances and lets a failure retry one item rather than restarting the whole run.
When You Actually Need It
- The caller does not need the result to answer its own request - a receipt is an acceptable response.
- The downstream work is slow, spiky, or calls something you do not control, and its latency should not be your API's latency.
- You need the work to survive a restart of the service that performs it, which rules out an in-memory background task.
- Consumers need to scale independently of producers, or you want to add a second consumer of the same events later without touching the producer.
- Traffic arrives in bursts that are much larger than your steady-state capacity, and delaying the work is acceptable.
- You are calling a rate-limited dependency and need a place to hold work while you drain it at a fixed rate.
When You Don't
- The caller needs the answer now. A queue in a synchronous request path adds a hop and gives you nothing - call the service directly.
- The volume is low and the work is short. A background thread with a retry, or a
jobstable in the database you already run, handles thousands of items a day without a new system to operate. - You have exactly one producer, one consumer, and both are the same deployment. You are adding a network hop and an operational surface to decouple things that ship together anyway.
- You cannot make the consumer idempotent. Without duplicate handling, at-least-once delivery turns a rare crash into a visible correctness bug - fix that first, adopt the queue second.
- You need strict global ordering across all messages. You will end up with a single consumer, at which point the queue is a durable buffer and not a scaling mechanism - be sure the buffer alone is worth it.
- Nobody is going to monitor queue depth, consumer lag, and the dead-letter queue. An unmonitored queue does not fail loudly; it silently accumulates work that never happens.
What People Get Wrong
"Adding a queue makes the system faster."
It makes one request faster by removing work from it. Total throughput is unchanged and end-to-end latency for that work usually gets worse. What you buy is that the slow part stops blocking the fast part.
"The broker guarantees exactly-once delivery."
Practically every queue gives at-least-once. Systems that advertise exactly-once achieve it within a bounded scope - typically their own storage in a single transaction - and it does not extend to the side effects your consumer performs on the outside world.
"A queue decouples the services."
It decouples them in time and availability, not in meaning. Both sides still have to agree on the message schema, and a breaking change to that schema fails just as hard as a breaking API change - only later, and in a place nobody is watching.
"We need a queue because we might scale later."
A database table with a status column gives you durable work, retries, and visibility, and it converts to a real queue in a day if the volume ever arrives. Adopting a broker for hypothetical scale buys operational cost now against a benefit that may not come.
Why It Was Built This Way
The design falls out of one constraint: two services almost never have the same availability and the same rate at the same time. A direct call couples both properties together - if the callee is down, the caller fails, and if the callee is slower than the caller, the caller has to wait or drop work. There is no configuration that removes this, because it is a property of the call itself.
A queue breaks the coupling by inserting something whose only job is to be durable and always accept writes. The producer's availability now depends on the broker rather than on the consumer, and the consumer's rate becomes independent of the producer's. Everything else in the design - persistence before acknowledgement, invisible-then-redelivered messages, dead-letter queues - exists to make that buffer trustworthy when either side dies at an inconvenient moment.
What was traded away is certainty and simplicity. In a direct call, the response tells you what happened. Behind a queue, the producer knows only that a message was accepted, so answering "did it work?" now requires status tracking, consumer lag metrics, and someone watching the dead-letter queue. That is the real price, and it is why the honest answer for a great many systems is that a table with a status column is enough - it provides the same durable hand-off with tooling the team already knows how to operate.
What to Read Next
- what is idempotency in apis
- what is at-least-once delivery
- what is backpressure
Reach for a message queue when the caller genuinely does not need the result and the downstream work is slow, spiky, or outside your control. Before you do, check whether a status column on a table you already run gives you the same durable hand-off - it usually does at low volume, and it comes with none of the duplicate handling, ordering surprises, or dead-letter monitoring that a broker makes your problem.
댓글
댓글 쓰기