Aug 17 2016
As software applications grow in size and complexity, it becomes even more important to identify and maintain boundaries. One way to set these boundaries is by employing microservices. This approach has its pros and cons, but I’ll leave that discussion for another post. Instead, I’d like to focus on the role of message queues in a microservices architecture.
A message queue is a software component used for passing data (messages) between services. It exposes an interface for adding, reading, and removing messages. Generally speaking, messages are persisted. Examples include RabbitMQ, Amazon SQS, and Apache Kafka.
Message queues facilitate asynchronous communication between discrete services. There can be multiple producers (services adding messages to the queue) and consumers (services taking messages from the queue). The producers and consumers are decoupled in time and space; a producer need not know when or by whom a message will be consumed. This is important, and the benefits are legion:
Message queues are not the only way to pass data between services, but they have some desirable characteristics, especially for solving certain kinds of problems.
Imagine, for a second (or two, I don’t really care how long), a user sign-up flow for a web application. A user fills out a sign-up form, the data is sent to the server, a record is created in the database, and a confirmation email is sent out. A naive server-side implementation might create a database record and attempt to send an email synchronously before responding to the user request. If there’s a temporary failure, should the user have to wait for a retry? Probably not.
One way to address this concern is by sending the email asynchronously. Upon receiving the user sign-up request, create a database record, and add a message to a queue indicating the creation of a new user. Then, implement another service that consumes messages from the queue, which would actually send the confirmation email, handling retries and failures. It’s easy to imagine extending this email service to send other types of emails, neatly isolating this functionality in the application.
As with most software components, there are tradeoffs when using message queues. They potentially add infrastructure complexity and operations overhead. Decoupled services, while in some ways simpler to reason about, can make the flow of a request through the system harder to trace. Some services would be better off exposing synchronous APIs, perhaps using REST. As always, carefully consider the requirements before adding a new tool.