This is part 1 of a 3 part series on untangling tightly coupled systems. Check out part 2 here! Plus, don't forget to register for Stephen's on-demand webinar: Why EventStoreDB Decouples your Distributed Ball of Mud: A Beginner's Guide, for more.
Data Integration Problems with Tightly Coupled Systems
If you are fortunate enough to develop a system that stores important data, you will soon come across a time when its data has to be integrated or shared with another system.
And as your system becomes more critical and/or popular, it may have to integrate with more and more external systems.
On one hand this is great because it lets other systems help the business scale without overloading your system with every technical and business concern.
However, if dependencies between systems are not managed properly, systems can become tightly coupled making them hard to change. Over time, this can turn into a Big Ball of Mud which slows down the pace of innovation. This makes your team and business less competitive in this ever changing world.
How Tightly Coupled Systems are Formed
When your system is core to the business and has to share data to other parties, an update to your system may have to update another external system.
For example, say you develop the ordering system of an e-commerce business. An order that was submitted to your system may have to alert:
- The packing department to gather inventory and pack the order
- The finance department to record the sales for its revenue report
- The product department to keep track of key metrics for their recommendation engine
The straightforward way to accomplish this is to directly update these external systems when the action is triggered (e.g. when the order is saved, update the packing database).
These direct updates are usually done with synchronous messaging.
Synchronous Messaging
Synchronous message is a call that have to wait for the other party’s process to complete before it is completed.
Most method calls in software development are synchronous. For example:
- Simple local method call (e.g. a common string concatenation function)
- RESTful HTTP GET Request
- Database row insert
Synchronous messaging are also known as request-reply messaging because it follows a pattern where the caller requests for information, and the callee replying.
How Synchronous Messaging forms Tight Coupling
Synchronous calls ensures the operation is completed before you continue and so is simple to track and understand.
However, this can subtlety create strong dependencies with external systems that may not be immediately obvious.
For example, your code base, that was supposedly only responsible for the ordering system, may have to include calls, data structure and logic that belongs to other teams, which complicates your code base.
Your system can get slowed down by these extra calls that take time to complete and may even fail sometimes due to unexpected downtime. This can bring the whole system down.
Also, your ordering system may additionally have to:
- Be aware of the table structure of the packing database ahead of time
- Categorize the order product to a specific type for the finance system
- Change the code if other teams change, say, their database field names or API parameter types
Use Asynchronous Messaging to Decouple Systems
The problem with the ordering system is that it is overburdened with responsibilities to update itself as well as many external systems at the same time.
One way to solve this is to transfer the responsibilities back to the external systems with asynchronous messaging and events.
This can be done if your system can publish a simple message to other systems to notify them that you have completed an action.
We call this kind of message an event or state transition, and it represents the action that triggered a state change to your system.
A separate program that lives in the external system, called an event handler, can be developer to listen for these events.
And when the handler receive these events, it can update their own database themselves. And when they can update themselves, your system can be decoupled from these systems and the responsibilities to update them after the event.
Event publishing is a very different form of communication compared to calling them directly through synchronous messaging.
This is also known as asynchronous messaging.
Asynchronous Messaging
Compared to synchronous messaging, asynchronous messaging does not wait for the other parties' process to complete before it is completed. It is fired off one way and does not expect any reply back.
Asynchronous messaging can be found in many places:
- Broadcasting an instant message to a group of people
- Publishing a post to social media
- Logging an event to system monitoring or data lake
These messages can be published or broadcasted to multiple recipients unlike synchronous messaging, which is typically one to one communication.
Also unlike synchronous messaging, asynchronous messaging does not require your system to have knowledge of the recipient.
Asynchronous messaging is akin to publishing a holiday event on the community bulletin board. Where synchronous message is like going door to door to advertise it.
And since asynchronous messaging doesn’t require information about its recipients, it is the ideal form of communication to help decouple systems.
Steps to Decouple Systems with Asynchronous Messaging
To decouple a system that is dependent on synchronous calls with external systems, follow these steps:
1. Locate the Post-Action Synchronous Calls
Find the synchronous call to the external system that is made after an action is completed. This is usually located after a database transaction is committed.
For example, after your system received an order submission request, it will try to save it to the database. Upon success, it will try to update the packing database or other data sources with synchronous calls.
Locate these calls.
class OrderSubmissionService
{
...
void SubmitOrder(int orderId, string productId, int quantity, ... )
{
// Perform some validation check
...
// Submit the order
MyOrderDB.OrderSubmission.Insert(orderId, productId, ...)
// vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
// Below are some post-action external synchronous calls
... // Packing system related complex logic
MyPackingDB.Order.Insert(orderId, productId, ...)
... // Finance system related complex logic
FinanceAPI.CreateJournalEntry(orderId, productType, amount, ...)
... // Product system related complex logic
ProductAPI.AddOrder(orderId, ...)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
}
}
2. Design an Event for this Action
This event has to be communicated in an asynchronous way free from external concerns so that For example, we can craft an OrderSubmittedEvent that describes the order submission:
class OrderSubmittedEvent
{
int orderId,
string productId,
int quantity,
int customerId,
...
DateTime OccurredAt
}
3. Write an Event Handler
Write a program in the external system, known as an message or event handler, that will listen and pick up this event when it is published. Extract and move the code responsible for the synchronous call to this handler.
For example, the OrderReadyForPackingEventHandler will listen for OrderSubmittedEvent and perform the same update as the synchronous call in step 1:
class OrderReadyForPackingEventHandler
{
void ListenForOrder(OrderSubmittedEvent evt)
{
... // <=== Packing system related complex logic
MyPackingDB.Order.Insert(evt.OrderId, evt.ProductId, ...)
}
}
This is repeated for the calls to the finance and product APIs as well.
4. Replace Synchronous Calls with Asynchronous Messaging
Replace the synchronous calls located in step 1 with a call that publishes an event designed in step 2.
This is an asynchronous message, an event, that will be published to the external systems.
For example, notice how the code below is simpler now than the one in step 1 with the service focused only on ordering related concerns:
class OrderSubmissionService
{
...
void SubmitOrder(int orderId, string productId, int quantity, ... )
{
// Perform some validation check
...
// Submit the order <==== This is the key action
MyOrderDB.OrderSubmission.Insert(orderId, productId, ...)
// Below replaces all the synchronous calls from before
MyMessageBus.Publish(new OrderSubmittedEvent { orderId, productId, ...})
}
}
5. Done!
With these steps, your system is now more loosely coupled from the other systems.
Synchronous vs Asynchronous Messaging
While this post has focused on the benefits of asynchronous messaging, there are many benefits to synchronous messages as well.
Asynchronous messages can take a while to reach the recipient and even be lost midway, and so data may become inconsistent.
On the other hand, Since synchronous messaging always waits for a response from the recipient, It often leads to better data consistency.
However, the trade off for this is lower availability, more complex codebase, and the slower performance due to coupling between the systems.
Choosing synchronous vs asynchronous is a matter of balancing consistency and availability.
There’s no universal right or wrong decision and it just depends on your use case.
Conclusion
We have explored how tightly coupled systems can slow down the pace of innovation of your business and how it is formed by synchronized calls to external systems.
We have examined why asynchronous messaging can help decouple these systems and show how it can be done with a few simple high level steps.
In the next part, we will explore the difficulties to implement this with traditional application patterns and how EventStoreDB can help.
Register for our webinar to learn more in the meantime: Why EventStoreDB Decouples your Distributed Ball of Mud: A Beginner's Guide.
Read part 2 in this series.