Fixing Broken MuleSoft API-Led Architecture: 3 Costly Anti-Patterns

Introduction: The Promise vs. The Reality of API-Led Connectivity

Speed, agility, and modularity are the three components of a holy architectural trinity in the current era of enterprise IT. For ten years now, MuleSoft has been promoting API-led connectivity as the ultimate architectural model to overcome the rigidity of legacy integration solutions. The architectural concept provides for a simple, layer-based model – System APIs expose and uncouple underlying systems of record, Process APIs compose and orchestrate the data from those systems into useful capabilities, and Experience APIs package those capabilities for digital experiences.

Theoretically, such three-tiered architecture offers for clear governance, reusability of assets, minimized risks of operations, and increasing speed of delivery. The more enterprises build out their applications networks, the fewer time-to-market they need to develop new solutions based on existing digital assets.

However, enter any enterprise that implemented API-led connectivity in the last three to five years, and you will witness something completely different.

Instead of building a flexible network of modules, organizations become trapped in a network of latency, exorbitant cost of infrastructure, duplicated code, and organizational paralysis. The developers complain that just adding another field to a database would require changes and deployment of not one, but three separate applications using three separate repositories. Such a situation may arise due to a lack of practice and experience in building such software, usually gained in the industry aligned mulesoft developer course online.

What has gone wrong here? The method is fine, but it is the implementation thereof that is the victim of some fundamental misapprehensions. An architectural pattern is an ideal. An anti-pattern is what happens when the ideal is taken as gospel without any appreciation of the reality of distributed computing and the associated tradeoffs.

To save an API-led architecture from falling into complacency, engineering leaders and architects need to uncover how far their implementations are off the path of original principles. Using the five most widespread anti-patterns of API-led connectivity and applying proven approaches to counter them, enterprises will be able to reinvigorate the performance and economic value that was promised by the architecture.

Anti-Pattern 1: The Pass-Through Layer ("Russian Doll" Architecture)

The Diagnosis: Bureaucracy Wrapped Up in Code

The biggest and costliest mistake that one can make when developing API-led architectures is adhering ritualistically and mindlessly to the development of all three layers in every transaction. Enterprises affected by the anti-pattern do not allow developers to deploy the System API that communicates with the end consumer or the Experience API that talks to the back-end system directly. Every integration effort results in the creation of System, Process, and Experience APIs, irrespective of any transformation and data orchestration happening there.

The pattern cannot be missed: An Experience API receives an HTTP request and immediately passes it on, unchanged, to a Process API. The Process API receives the message, performs no business logic, does not connect to any other data source, performs no computation, and passes that very message on to a System API. The System API runs a query on the enterprise database or software service, encapsulates the exact same schema received from the backend in a standard envelope, and sends it right back up the line.

This is what is known as the "Russian Doll" architecture. Each level merely contains the level beneath it without any additional semantics added in. While the architecture may resemble the layers shown in the sales presentation diagram, in practice, it is all latency, network costs, and ceremony.

Why Teams Get Stuck in This Pattern

This is usually a result of organizational dogma posing as governance. The Architecture Review Board (ARB) develops a rigid compliance checklist. A developer will not get approval for their architecture unless they design three separate API interfaces and deploy three applications.

Teams also cite this as being the result of future unknown requirements. The engineer will state that since the Process API does not do anything except forward an order payload right now, in six months time the business would like to combine the warehouse inventory information within that process. The developer "futures" this requirement and therefore builds the empty layer now, paying upfront in terms of operational and development overhead.

Consequences: Latency, Cost, and Maintenance Hell

The consequences that result from the introduction of pass-through layers are profound in three ways:

First, there is no negotiating around the laws of network physics. Each hop from an Experience API to a Process API to a System API requires a TCP handshake, TLS negotiation, JSON serialization and deserialization, traversing the network stack, and queueing through load balancers. What used to take thirty milliseconds of direct data retrieval now takes three hundred milliseconds. The extra latency introduced gets compounded by hundreds of user sessions at once and results in timeouts.

Second, infrastructure costs grow geometrically. With cloud integration platforms, each API runtime needs its own memory and processing power. Using three different runtimes to make one simple data fetch means tripling license costs, compute costs, and memory costs. Companies often reach their compute limits long before they anticipate, just because of having useless pass-through layers.

Third, change management becomes paralyzing. As the underlying system introduces an additional data attribute the consumer is dependent on, the engineering team cannot simply edit the single contract. Instead, they have to make changes to the System API, test, and deploy; changes to the Process API, test, and deploy; and finally changes to the Experience API, test, and deploy. Three lines of data become three pull requests, three automated deployment pipelines, three tests of deployment success, and thus three times the chance of failing the deployment.

The Solution: Value-Driven Tiering and Pragmatic Layer Skipping

In order to get rid of the pass-through layers, an organization must move past structural dogmatism and toward value-driven tiering:

1. Implement the Value-Add Mandate: The API layer should be there only if and only if it does something important. The System API should abstract, isolate, or normalize the underlying protocol or brittle legacy schema. The Process API should coordinate, combine, or implement stateful business logic spanning multiple dissimilar systems. The Experience API should re-format, sanitize, or combine the data for consumption by the presentation layer of a particular channel. An intermediary layer which does not do any of those things should not be there.

2. Standardize Layer Skipping: Create a formal architectural requirement that allows Experience APIs to communicate with System APIs without going through the Process layer when nothing else is needed. If a mobile app just needs to retrieve a user profile and the System API built on the customer identity repository already provides a nice and clean RESTful interface, let the Experience API connect directly to that System API.

3. Prune Speculative Runtimes: Don’t introduce speculative runtimes that serve nothing but some potential future requirement. If integration today consists solely of fetching data in a straightforward fashion, do so in as clean and simple a way as possible. Since APIs abstract away implementations from consumers, adding a Process API in the future, once orchestration becomes a necessity, can happen through an API Gateway in such a way that it does not break upstream consumers.

Anti-Pattern 2: The Distributed Monolith (System APIs as Fragile Database Proxies)

The Diagnosis: Object-Relational Mismatch in the Cloud

System APIs aim to isolate the application network from unstable backends. These protect enterprises from vendor eccentricities, arcane protocols, and strict data structures.

The second important anti-pattern emerges when System APIs merely become a poor, direct reflection of the databases in terms of HTTP. In other words, while System APIs should provide the abstraction for a domain boundary, the development team ends up creating an API wherein each table in a relational database is turned into an endpoint and each column into a JSON property.

Here, one can witness System APIs exposing functionality related to adding an order header record, adding an order line record, updating an inventory balance record, and adding an audit record – all through separate HTTP endpoints corresponding directly to different tables in the database.

Why Do Teams Get into This Anti-Pattern?

The root of this anti-pattern lies in developer patterns formed due to traditional Object-Relational Mapping (ORM) tools and scaffolding tools. While an integration engineer is assigned to open up a legacy database having three hundred tables, the easy way out is to use some automation tool to scan the schema and create three hundred CRUD (Create, Read, Update, Delete) endpoints. It seems productive and efficient on the first sprint.

Second, teams believe that by exposing their raw table structures from the database using the HTTP endpoint, they are offering maximum flexibility to the consumers of the API. Overcoming this gap in the design pattern calls for practical experience as well as mulesoft developer training on domain-driven design concepts.

Consequences: Gossip, Lack of Transactional Integrity, and Tight Coupling

Because of the transformation of database tables into endpoints, the architecture combines the worst features of monolithic databases and distributed architectures:

First of all, there will be no transactional integrity. With a compliant ACID relational database, the application opens a transaction, writes the header, writes the lines, updates stock, and commits it all at once. If anything goes wrong, the whole thing will roll back and will be done correctly. But if the tables are separated into different HTTP endpoints, the Process API component has to try to make an atomic transaction over HTTP calls. If the line-item insert fails after the header was inserted, it will fall into a corrupt state.

The second side effect is excessive chattiness. For one customer invoice, a service will need to make one request for the invoice itself, ten requests for its associated ten lines, one request to get customer information, and many more requests to get shipping information. This is the archetypical N+1 query problem, blown up from a memory optimization issue into a network disaster for the entire enterprise.

Lastly, the approach produces the total antithesis of decoupling. If a DBA renames a column, splits a table to improve indexing, or normalizes the schema, the change will immediately violate the API contract of the System API, which in turn makes all the Process APIs that relied on the raw tables of the system useless. An application network has not been created in the enterprise; rather, a very slow, highly distributed, and fragile database was built.

The Fix: Domain-Driven Design and Aggregate Roots

The fix demands switching from the exposure of data structures to the principles of Domain-Driven Design (DDD):

1. Expose Aggregate Roots, Not Database Tables: The system APIs have to be defined against the functional business domain. Do not expose tables, but aggregate roots instead. "Order" aggregate needs to receive only one self-consistent command for submitting the whole order including all lines, payments and delivery addresses in one network packet. The System API takes upon itself responsibility for starting database transactions, saving changes to the appropriate multiple internal tables, and committing or rolling back atomically.

2. Hide Backend Implementation Details: The very purpose of a System API is to hide. Whether the backend system consists of an ancient mainframe that uses only fixed width flat files or a poorly organized relational database that is spread over twelve tables in third normal form - whatever that is behind the scenes, must end at the System API implementation. The public contract should represent clear business objects. The caller has to know nothing about the backend storage structure or representation.

3. Coarse-Grained Design for Operations: Reduce network noise by designing coarse-grained APIs as a default principle. System APIs must supply all the context required for executing a particular business operation in one request-response loop, while fine-grained interactions can occur only between collocated microservices that incur zero network costs.

Anti-Pattern 3: The Omniscient Process Layer (The Monolithic Mega-Orchestrator)

The Diagnosis: Business Logic Concentrated into Fragile Monoliths

Whereas the first anti-pattern is a hollow, ineffectual Process layer, the third is its antithesis: the omnipotent Process API that attempts to solve everybody’s problems.

Here, a Process API takes on the role of the centralized brain for an entire department or enterprise journey. Think of a Process API with an innocent sounding name like “CustomerProcessAPI” or “OrderFulfillmentEngine.” After many years of development, it ends up being an enormous application full of hundreds of diverging routes, complex graphical flows, complicated scripts, and countless lines of transformation logic.

Any time any team requires the functionality for handling customers, checking credit limits, sending welcome notifications, handling address changes, calculating tax, and determining fraud scores, another diverging branch is added to this enormous application.

How Teams Get Into This Problem

Centralized development teams tend to seek out a quick fix for cross-cutting business logic since the development is centralized. Cross-cutting business requirements such as balancing the inventory before producing shipping labels must not reside within either System API due to the principle of layering. The most obvious way is to place it in the shared Process API.

Because success has been followed by acceptance, the team keeps on adding functionalities on top of this existing and successful process application. Creating a whole new microapplication appears to be a redundant operation, but adding just another way within the existing application looks insignificant. Mastering the art of balancing microservices and orchestration becomes one of the key achievements in the comprehensive mulesoft certification training.

The Consequences: Blast Radius, Resource Depletion, and Team Conflict

The existence of the Process API brings back all the worst nightmares that the service-oriented architecture was designed to eradicate:

The biggest threat is the huge blast radius. In the case where fifteen different consumer channels rely on one giant Process API, even one little bug that was added just for a particular case of an internal desktop application might unexpectedly knock out the mobile application or the online store for customers. One tiny memory leak, exception, or spike in traffic on one of the routes depletes the whole runtime container of the threads and memory, causing every unrelated capability that is running within the application to crash.

In addition to that, development slows down due to code conflict. There are multiple feature teams working on the same huge configuration files at once. Merge conflicts become common scary things. It takes hours to run automated tests, but the fact that the application has strong coupling to multiple other services makes it practically impossible to create isolated environments for integration testing.

Finally, horizontal scaling becomes highly inefficient. For a particular transformation path in the monolithic process to handle holiday traffic, it will require that all processes be scaled – even those unused transformations, connection pools, and memory overhead of the monolith – which means computing resources will be wasted on a large scale.

The Solution: Micro Choreography and Domain Bounded Process Services

In order to break up the omniscient process layer, companies need to move away from monolithic orchestration to process microservices that are clearly defined by domain:

• Ensure Single Responsibility Bounded Contexts: Break down monolithic mega-processes into bounded process services that have clear domains. Instead of an order process monolith, break up the process life cycle into distinct services such as Order Intake, Payment Settlement, and Fulfillment Dispatch Services.

• Embrace Event-Driven Choreography Over Synchronous Orchestration: The most efficient method to reduce bloated process API is to not synchronize orchestrate anymore. Rather than having a centralized engine that goes System A, wait, then System B, wait, then System C, adopt an event-driven architecture where the starting system will publish an event on the enterprise message broker stating that a certain event has occurred, such as validation of orders. Other systems will subscribe to this event and act accordingly by publishing their own events after completing what needs to be done. Choreography does away with the idea of the centralized mind that has all the knowledge.

• Isolate Flows That Are Frequently Changing and Have High Resource Consumption and High Business Criticality: Isolate the processes that are constantly changing or have a lot of resource consumption or high business criticality into their own runtime environment right away. Do not ever let high volumes of traffic which are not business critical to share an execution environment with processes that are highly valued.

Going forward: Building sustainable application networks

API-led connectivity is not a collection of dogmas inscribed on tablets of stone; it is a mindset intended to help businesses separate the dynamic needs of their consumers from static enterprise records. It takes the organization's basic assumption that the systems in a modern enterprise should function at vastly different speeds.

If teams find themselves entangled in pass-through layering, database mirroring, monolithic orchestration, edge chatty meshes, and point-to-point isolation, they are not doing so because the model is broken. They are doing so because they have confused procedural conformance for architectural design. They have focused on form rather than function – specifically, on how the architecture would look rather than the economics and engineering of scaling out distributed systems.

Reversing the effects of those anti-patterns will require a balance: Balance between governance and pragmatic delivery, beauty and latency, velocity and stewardship. By aggressively cutting unnecessary hops, respecting boundaries, choreographing events, communicating at the edges, and building a rich ecosystem of reuse, enterprises can turn their inflexible, costly integration sprawl into exactly what API-led connectivity was supposed to deliver: An adaptable, robust and self-reinforcing nervous system.

In case you need to bring together integration theory and architecture for development purposes, then you should consider the mulesoft training courses provided by OnlineITGuru. Acquiring practical experience will enable your developers and architects to develop efficient Mulesoft integrations that are not prone to architectural failures.