I've wanted to put something like this together for awhile. It was important to show that these ideas are not mutually exclusive, and are in fact very complimentary. I think there is a little more work to do, but I generally feel good about it. If nothing else we can use this as our IT Lingo Bingo card. :)
Don't be fooled. The concept of complex event processing (CEP) has been around for at least a decade. However, it is quickly being attached to terms like BPM, BAM, SOA, EDA, EAI, and BI. IT has become desensitized due to the continued onslaught from buzzword driven development (BDD), driving many to simply view CEP as an umbrella term for BPM, BAM, SOA, EDA, and EAI. In reality, these concepts and their associated technologies are finally allowing CEP concepts to be realized.
Where digital signal processing (DSP) is all about making sense of the noise at a hardware level, CEP is all about making sense of the noise at an enterprise level. Modern enterprises capture lots of information through many different mechanisms.
- Web logs feed web analytic tools for things like visitor-to-buyer conversion rates.
- EAI interactions feed BAM tools for correlating backend information flow to business processes.
- SOA gateways centrally monitory activity between actionable interfaces and systems of record.
- Human workflow tools track end-user involvement.
- RFID tags can track inventory to, and inside, stores.
- ETL tools track movement inside the data layer.
Making sense of all these events and correlating them in a meaning way for a business owner is what CEP is all about. End-to-end visibility is just the beginning. Once you add side-to-side visibility and the ability to data mine this information is truly where business insight can be gained. SOA is about agility; CEP is about moving purposefully.
In the intermediary pattern, the direct communication between a consumer and its provider is broken.
Messages are exchanged through logic that may be:
- Active or Passive - Active intermediaries may change the message’s content or reroute to another destination. Passive intermediaries do not modify the message or change its intended destination. Due to their read-only nature, they may also be configured to be out-of-band in high transaction scenarios.
- Invasive or Non-invasive - Invasive intermediaries require code changes and may be a development time or runtime construct. Non-invasive intermediaries may require configuration changes or installation of a plug-in.
- Local or Remote - Local intermediaries may be deployed in-process or out-of-process. Remote intermediaries are deployed on standalone network nodes and may be an appliance.
Although intermediaries may have different implementation and deployment models, they offer the following capabilities:
- Message Validation
- Message Translation
- Message Routing
- Instrumentation
- Monitoring
- Auditing
- Caching
- Security
A single message could traverse multiple intermediaries with different intermediaries responsible for different capabilities. These capabilities are typically associated with aspect-oriented programming (AOP). In fact, AOP can be thought of as type of in-process intermediary. The great thing about AOP is that it allows developers to separate our the two types of code that exist within their services: business logic and operational logic. The draw back up completely relying on AOP in your SOA efforts, is that AOP is entirely a development activity. In other words, it may help you develop new services more efficiently and effectively, but what about existing services or services built on another development platform? Intermediaries deployed as infrastructure entities allow these capabilities to be implemented after the service has been deployed. Agents and brokers are popular infrastructure implementations of the intermediary pattern.
Agents
Agents live on the same node as the consumer and provider, and typically use the same communication protocol. The resulting context is a highly distributed execution model, but that doesn't imply that policies can't be centrally managed and pushed to the agents. This potentially gives agents a great deal of resilience and optimized message routing options. Several agents communicating with each other using the same protocol results in a bus topology.
Traditionally, the ESB term referred to SOA products deployed over a bus topology. Meaning you have multiple agents residing on different nodes on the network. In this setup all the nodes typically speak the same protocols. Typically HTTP and SOAP, but I’ve seen customers do this over MQ. Today, many ESB vendors have added brokers to their solution to give them flexible deployment models. As a consequence, the term ESB has become overloaded and has lost meaning as it has transitioned from a product, to an architecture, and to a marketing term. Unfortunately, ESB has maintained its name recognition in the SOA marketplace. Therefore, vendors have begun appending ESB to their product name to remain relevant to those customers.
Agents are inherently more invasive and potentially may not be able to reach all your services to provide complete enterprise visibility. At a minimum, they must support the deployment platform. Agents also have difficulties with protocol transitions. This is readily apparent when you start integrating line of business applications with adapters because of configuration and cost. Sometimes this can be mitigated if the line of business application has an adapter for the bus.
Brokers
Brokers live on separate nodes from consumers and providers. When the broker is supporting more than two nodes, it is said to deployed in either a hub or gateway topology. In a hub topology, messages flow in every direction and there are often nodes that are both consumers and providers.
The resulting context is n(n-1) connection points, which greatly simplifies the integration landscape. This is why EAI products have leveraged the broker pattern over a hub topology. Many of these systems have varying degrees of performance and reliability characteristics, so often these brokers exchange messages asynchronously. Hubs are key to implementing event-driven architectures. In a gateway topology, messages are usually initiated in one direction.
Gateways are largely used as an abstraction mechanism to simply client application development by hiding the complexity of legacy systems and serving as a platform for data aggregation. To support these capabilities, gateways exchange messages synchronously. Gateways are key to implementing service-oriented architectures.
Summary
In the context of service-orientation, Service Bus refers to an agent-based SOA solutions and Service Gateway refers to a synchronous broker-based SOA solutions. My team continues to refer to the Microsoft Managed Services Engine (MSE) as a Service Intermediary, because it can be deployed as a bus or gateway. Scenarios, like complex event processing, requiring an asynchronous broker-based solution continue to be fulfilled by BizTalk Server.
I associate with a lot of folks that travel. One thing we all have in common was labeling the Detroit airport as the worst airport people have ever flown through. Today, I was pleasantly surprised to see that they have completely renovated and it is in fact very nice these days.
-Chris
Everyone that knows me, knows that I'm a loyal Marriott customer. They have always treated me well at every property. In 2007, I burned a lot of points traveling for Bri's gymnastics. Close to the end of the season, we ended up falling short on points need for a particular property and Marriott spotted me the points. I thought to myself, this could be a bug in their system, but I was happy nonetheless. I also didn't travel much during 2007 and in fact I just short of qualifying for gold status. I've enjoyed platinum for as long as I can remember, so I was definitely disappointed. Much to my surprise, I received a platinum card in the mail last week. Maybe there was a bug in their mail distribution program? So, I logged into the site I continued to show up as platinum. For my stays this week, they definitely treated me like a platinum customer. If you think I was loyal before, I'm completely fanatic now!
-Chris
PS - BTW, the new rooms are awesome.