SOA and EDA, and the Problems They Solve
These two problems are really part of the same problem: how to automate your business. Your business consists of events, tasks, and processes. When automating your processes with software, business events often trigger business processes. Business processes often generate business events.
SOA is now widely recognized as a solution to the first problem and an approach that delivers new functionality faster with greater productivity and greater flexibility to accommodate changes. EDA has been around for a while but is not widely known. It’s now becoming more widely used to solve the second problem of detecting and reacting to events.
The bad news is some SOA technologies don’t support EDA, so you could end up having to spend more money on a second architecture to gain EDA capabilities. And then every time you wanted to automate or change part of your business, you’d need to change two different systems.
The good news is SOA and EDA can be part of the same architecture. In fact, an “advanced SOA” or “event-driven SOA” is an evolution of SOA that has already been implemented by several companies. Following the very principles of SOA, these companies have been able to incrementally add on eventdriven technologies as needed.
SOA is an architecture that enables companies to break up monolithic applications into smaller components or services that can be reused to deliver or change functionality. The benefit is that by writing less new code and assembling services together using a business process-centric type of workflow, you deliver new functionality faster, increase developer productivity, and make applications more flexible to changes in the business.
Consider the following scenario: a telecommunications firm targets a particular consumer to make the switch to their service, promising a new model of phone and rate plan. The promotion proves a resounding success, and the particular customer gets swayed to change services by a telephone customer service representative.
These types of structured campaigns are supported in several telecommunications companies using SOA. Services exist to target the customers via e-mail or in call center applications, provision the new services, activate billing, and send out or hand out the phone to a customer.
This approach works for well-understood, structured business processes. But what happens if something goes wrong with the process? For example, suppose it turns out this promotion is so successful that the telecommunications company runs out of this particular brand of phone. Orders still get taken. The phones aren’t shipped. The order process may get “stuck” later on and send an alarm that the shipment hasn’t been completed. But by then it’s too late and the new customer has already become dissatisfied.
What just happened? This was not a failing of an SOA or an associated process. It’s an example of an enterprise that can’t make sense of fast-moving business events and has suffered the consequence. Ideally, the company should have detected potential stock-outs early on and gotten more phones or changed the program. But the events that would point to potential stock-outs weren’t detected or being watched by the right software. This is where EDA comes in.
EDA is an architecture in which events trigger the real-time exchange of messages between independent software applications. EDA relies on event-processing agents to detect events across an enterprise and, using a push approach, notify all other software applications that need to be notified of these events, all in real-time.
With EDA, a replenish re-order that returns with a failure is detected and identified by a complex event processing (CEP) system as a potential stock-out for the service campaign. It might lead to the service campaign manager being notified as well as the automatic modification of the campaign to let future prospects know when to expect their phones.
There are several key differences between classic SOA and EDA. Events exist in SOA as well, but they’re not “pushed” to many recipients. An SOA is a classic request/reply type architecture. It’s a one-to-one relationship implemented using web services or other protocols. In an EDA, the notification is one-to-many and usually implemented using publish-subscribe messaging. This is very different.
SOA is also often characterized as having services that are loosely coupled. Clients invoke specific services, but the services can move and the implementations can change. With EDA, the agents are strongly decoupled; they don’t necessarily know where their events go.
Finally, SOA is more about supporting structured control flow. It has the notion of service orchestration for composing services into processes. EDA is all about correlating seemingly unrelated events together to detect the right business events and patterns that may be of interest. These events or patterns may in turn trigger SOA-based processes.
Components of Event-driven SOA
What are all the components of SOA and EDA, and how can they be used together in one architecture?
The most easily understood component of SOA is services. These are software modules that represent reusable business functionality, like “process order” or “bill customer.” There are also system services, like auditing, logging, and even orchestration.
SOA is fundamentally heterogeneous, which basically means companies host services on many different platforms. The majority of companies have Java, Java EE, and .NET-based services. Most companies also service enable existing applications, exposing functionality as services using adapters or web services.
A business event is any activity that happens inside or outside your business. If you’re lucky, an event is captured somewhere in your software infrastructure either directly or indirectly.
There are thousands to millions of events that occur in most companies on a daily basis. All events somehow get noted and affect the operations of a company. Most events will not be relevant, some will have a negative impact on a business, others will have a positive impact, and a few will offer a big opportunity.
All companies are event driven, since our world is event driven. However, most IT applications and processes are geared toward predictable, repeatable events. It’s when events happen that are notable or exceptional that most businesses have trouble. Most companies also find it hard to fit an event into a larger trend that may affect their business.
A technology that has gained popularity in the last ten years is messageoriented middleware (MOM). Examples include any Java Message Service (JMS)-based product such as TIBCO Enterprise Message Service™ or messaging products such as TIBCO Rendezvous®, Microsoft Message Queue, and IBM’s WebSphere MQ. Messaging is “the other” service transport along with HTTP, and is also the foundation for EDA. Most messaging vendors now support request/reply and publish/subscribe message exchange patterns, synchronous or asynchronous communication, and different levels of reliability. Through the publish/subscribe or broadcast methodologies, events can be encapsulated into a message and sent out to an entire enterprise. As opposed to the request/reply model of services, the event is automatically sent out through some sort of event handler and is received by an event “sink” that may have no permanent connection to or prior knowledge of the event handler.
There are efforts to have web services operate in this manner, notably through the WS-Eventing and WS-Notification specifications, as well as several others. These specifications are currently moving through standards committees and will hopefully be a factor in the market.
Enterprise Service Bus: The Foundation for SOA and EDA
An ESB supports different types of protocols and formats used in SOA including web services, JMS-based and proprietary messaging, as well as other protocols like FTP and e-mail or adapter-based protocols. The definition varies, but it usually includes transformations and content-based routing. Some ESBs can be purchased standalone; others may be part of a larger suite that includes orchestration and integration.
When an ESB supports all the message exchange patterns, then it can also support EDA, making it the foundation for a combined architecture. In this case, messaging is often extensively used, both for publish/subscribe communication in EDA and for reliable request/reply communication in SOA. One of the dangers with an ESB is that different ESB and messaging products are implemented separately for EDA and SOA. The result is that events often remain trapped in different ESB silos, on top of the events already trapped in application silos that are in proprietary formats and much harder to reach. All of this leads to more development, integration, and ongoing maintenance.
Service orchestration is the graphical wiring of a set of services to automate system-to-system processes. For example, in the case of providing a new phone service for a customer, an orchestration might access data warehouse profile information, enter customer information into billing and CRM systems, provision the new service in the network, and order the new phone.
Orchestration is critical to realizing the intended benefits of SOA: greater productivity and faster time to market for new functionality. It’s the process-centric part of an application that varies by business and changes with business needs. With the right services, customers can add or change processes graphically as processes instead of writing code. Orchestration is best for structured processes; it doesn’t help you automate responses to unexpected events. It is also generally reactive. In many SOA implementations, the business events that trigger orchestrations happen outside the scope of an SOA. The most common cases are events triggered by a user action in an application or an event recorded within an application or database.
CEP has become more widespread in the last few years. CEP helps automate intelligent responses to various events, often using a graphical approach. Like orchestration, CEP is a more flexible approach than coding.
How events are related, for the most part, is a black hole of knowledge because it’s extremely complex. A CEP system identifies notable events (no matter their source), gains an understanding of what the event is, tracks it, and fits it into a larger pattern so action can be taken. That action might be the firing off of an event that notifies a person or triggers an automated process. Where SOA and orchestration automate the well-understood processes, CEP automates the intelligent correlation and decision-making done by individuals. Many observers believe a CEP type of product is necessary to complete a true EDA because while ESBs/MOMs do understand events, they understand them at a simple level that lends itself to rather reactive processes.
A CEP system usually has several parts. The first are connectors to actually capture and identify events, which typically sit on top of ESBs/MOMs, databases, and applications. They capture events, send them through an event-processing server, filter out non-useful events, and put them in a common indexable format so they can be tracked. Metadata is often part of this indexable format. While this sounds a bit like an RDBMS, CEP does not require a permanent store to process events, and deals with events in motion, where time passage is a critical element. In CEP, detected events and situations can be stored for audit and compliance requirements.
Making sense of these captured events is a bit tricky. In general, users must create business models that understand dependencies and relationships between events and other events and between events and eventual outcomes. These business models include dependencies between physical infrastructure, applications, processes, and people in an organization.
Events and trends in events need to be mapped to these business models, or correlated, forming the basis of recognizing patterns that may be occurring. Rules play an important part in the set up of business models, as well as in the correlation between events and business models, and the most important part of the CEP loop: the ability to take action to mitigate negative patterns or take advantage of opportunities. A good rules engine is part of most CEP products.
A CEP system allows users to organize seemingly random, unrelated events, find trends, and to predict outcomes. Action can then be taken to prevent a negative outcome from occurring. The same is true on the upside: a positive occurrence can be exploited more effectively. The patterns can also be analyzed to improve the underlying processes and applications so as to prevent future miscues.
So to return to the original question: are you implementing the right SOA for the long term? If you’re implementing an ESB that supports an EDA and messaging for any eventdriven communication, the answer is yes. If you’re not implementing an ESB – for example, if you’re implementing an SOA using application servers, integration brokers and code – the answer is no. You need an ESB.
This brings up the next question: when do you need CEP? The best strategy is to ask that question now and identify the first CEP projects in the same way you identified your first projects for SOA. Think about any area of the business where tracking the state of a process or handling unexpected events adds significant business value.
www.btquarterly.com/
SOA is now widely recognized as a solution to the first problem and an approach that delivers new functionality faster with greater productivity and greater flexibility to accommodate changes. EDA has been around for a while but is not widely known. It’s now becoming more widely used to solve the second problem of detecting and reacting to events.
The bad news is some SOA technologies don’t support EDA, so you could end up having to spend more money on a second architecture to gain EDA capabilities. And then every time you wanted to automate or change part of your business, you’d need to change two different systems.
The good news is SOA and EDA can be part of the same architecture. In fact, an “advanced SOA” or “event-driven SOA” is an evolution of SOA that has already been implemented by several companies. Following the very principles of SOA, these companies have been able to incrementally add on eventdriven technologies as needed.
How SOA Works
SOA is an architecture that enables companies to break up monolithic applications into smaller components or services that can be reused to deliver or change functionality. The benefit is that by writing less new code and assembling services together using a business process-centric type of workflow, you deliver new functionality faster, increase developer productivity, and make applications more flexible to changes in the business.
Consider the following scenario: a telecommunications firm targets a particular consumer to make the switch to their service, promising a new model of phone and rate plan. The promotion proves a resounding success, and the particular customer gets swayed to change services by a telephone customer service representative.
These types of structured campaigns are supported in several telecommunications companies using SOA. Services exist to target the customers via e-mail or in call center applications, provision the new services, activate billing, and send out or hand out the phone to a customer.
This approach works for well-understood, structured business processes. But what happens if something goes wrong with the process? For example, suppose it turns out this promotion is so successful that the telecommunications company runs out of this particular brand of phone. Orders still get taken. The phones aren’t shipped. The order process may get “stuck” later on and send an alarm that the shipment hasn’t been completed. But by then it’s too late and the new customer has already become dissatisfied.
What just happened? This was not a failing of an SOA or an associated process. It’s an example of an enterprise that can’t make sense of fast-moving business events and has suffered the consequence. Ideally, the company should have detected potential stock-outs early on and gotten more phones or changed the program. But the events that would point to potential stock-outs weren’t detected or being watched by the right software. This is where EDA comes in.
Enter EDA
EDA is an architecture in which events trigger the real-time exchange of messages between independent software applications. EDA relies on event-processing agents to detect events across an enterprise and, using a push approach, notify all other software applications that need to be notified of these events, all in real-time.
With EDA, a replenish re-order that returns with a failure is detected and identified by a complex event processing (CEP) system as a potential stock-out for the service campaign. It might lead to the service campaign manager being notified as well as the automatic modification of the campaign to let future prospects know when to expect their phones.
There are several key differences between classic SOA and EDA. Events exist in SOA as well, but they’re not “pushed” to many recipients. An SOA is a classic request/reply type architecture. It’s a one-to-one relationship implemented using web services or other protocols. In an EDA, the notification is one-to-many and usually implemented using publish-subscribe messaging. This is very different.
SOA is also often characterized as having services that are loosely coupled. Clients invoke specific services, but the services can move and the implementations can change. With EDA, the agents are strongly decoupled; they don’t necessarily know where their events go.
Finally, SOA is more about supporting structured control flow. It has the notion of service orchestration for composing services into processes. EDA is all about correlating seemingly unrelated events together to detect the right business events and patterns that may be of interest. These events or patterns may in turn trigger SOA-based processes.
Components of Event-driven SOA
What are all the components of SOA and EDA, and how can they be used together in one architecture?
Services: DNA of SOA
The most easily understood component of SOA is services. These are software modules that represent reusable business functionality, like “process order” or “bill customer.” There are also system services, like auditing, logging, and even orchestration.
SOA is fundamentally heterogeneous, which basically means companies host services on many different platforms. The majority of companies have Java, Java EE, and .NET-based services. Most companies also service enable existing applications, exposing functionality as services using adapters or web services.
Events: DNA of EDA
A business event is any activity that happens inside or outside your business. If you’re lucky, an event is captured somewhere in your software infrastructure either directly or indirectly.
There are thousands to millions of events that occur in most companies on a daily basis. All events somehow get noted and affect the operations of a company. Most events will not be relevant, some will have a negative impact on a business, others will have a positive impact, and a few will offer a big opportunity.
All companies are event driven, since our world is event driven. However, most IT applications and processes are geared toward predictable, repeatable events. It’s when events happen that are notable or exceptional that most businesses have trouble. Most companies also find it hard to fit an event into a larger trend that may affect their business.
Mesaging: The Comon Transport for SOA and EDA
A technology that has gained popularity in the last ten years is messageoriented middleware (MOM). Examples include any Java Message Service (JMS)-based product such as TIBCO Enterprise Message Service™ or messaging products such as TIBCO Rendezvous®, Microsoft Message Queue, and IBM’s WebSphere MQ. Messaging is “the other” service transport along with HTTP, and is also the foundation for EDA. Most messaging vendors now support request/reply and publish/subscribe message exchange patterns, synchronous or asynchronous communication, and different levels of reliability. Through the publish/subscribe or broadcast methodologies, events can be encapsulated into a message and sent out to an entire enterprise. As opposed to the request/reply model of services, the event is automatically sent out through some sort of event handler and is received by an event “sink” that may have no permanent connection to or prior knowledge of the event handler.
There are efforts to have web services operate in this manner, notably through the WS-Eventing and WS-Notification specifications, as well as several others. These specifications are currently moving through standards committees and will hopefully be a factor in the market.
Enterprise Service Bus: The Foundation for SOA and EDA
An ESB supports different types of protocols and formats used in SOA including web services, JMS-based and proprietary messaging, as well as other protocols like FTP and e-mail or adapter-based protocols. The definition varies, but it usually includes transformations and content-based routing. Some ESBs can be purchased standalone; others may be part of a larger suite that includes orchestration and integration.
When an ESB supports all the message exchange patterns, then it can also support EDA, making it the foundation for a combined architecture. In this case, messaging is often extensively used, both for publish/subscribe communication in EDA and for reliable request/reply communication in SOA. One of the dangers with an ESB is that different ESB and messaging products are implemented separately for EDA and SOA. The result is that events often remain trapped in different ESB silos, on top of the events already trapped in application silos that are in proprietary formats and much harder to reach. All of this leads to more development, integration, and ongoing maintenance.
Orchestration: A Flexible Approach to Automating Structured Proceses
Service orchestration is the graphical wiring of a set of services to automate system-to-system processes. For example, in the case of providing a new phone service for a customer, an orchestration might access data warehouse profile information, enter customer information into billing and CRM systems, provision the new service in the network, and order the new phone.
Orchestration is critical to realizing the intended benefits of SOA: greater productivity and faster time to market for new functionality. It’s the process-centric part of an application that varies by business and changes with business needs. With the right services, customers can add or change processes graphically as processes instead of writing code. Orchestration is best for structured processes; it doesn’t help you automate responses to unexpected events. It is also generally reactive. In many SOA implementations, the business events that trigger orchestrations happen outside the scope of an SOA. The most common cases are events triggered by a user action in an application or an event recorded within an application or database.
CEP: A Flexible Approach to Automating Inteligent Decision-Making
CEP has become more widespread in the last few years. CEP helps automate intelligent responses to various events, often using a graphical approach. Like orchestration, CEP is a more flexible approach than coding.
How events are related, for the most part, is a black hole of knowledge because it’s extremely complex. A CEP system identifies notable events (no matter their source), gains an understanding of what the event is, tracks it, and fits it into a larger pattern so action can be taken. That action might be the firing off of an event that notifies a person or triggers an automated process. Where SOA and orchestration automate the well-understood processes, CEP automates the intelligent correlation and decision-making done by individuals. Many observers believe a CEP type of product is necessary to complete a true EDA because while ESBs/MOMs do understand events, they understand them at a simple level that lends itself to rather reactive processes.
A CEP system usually has several parts. The first are connectors to actually capture and identify events, which typically sit on top of ESBs/MOMs, databases, and applications. They capture events, send them through an event-processing server, filter out non-useful events, and put them in a common indexable format so they can be tracked. Metadata is often part of this indexable format. While this sounds a bit like an RDBMS, CEP does not require a permanent store to process events, and deals with events in motion, where time passage is a critical element. In CEP, detected events and situations can be stored for audit and compliance requirements.
Making sense of these captured events is a bit tricky. In general, users must create business models that understand dependencies and relationships between events and other events and between events and eventual outcomes. These business models include dependencies between physical infrastructure, applications, processes, and people in an organization.
Events and trends in events need to be mapped to these business models, or correlated, forming the basis of recognizing patterns that may be occurring. Rules play an important part in the set up of business models, as well as in the correlation between events and business models, and the most important part of the CEP loop: the ability to take action to mitigate negative patterns or take advantage of opportunities. A good rules engine is part of most CEP products.
A CEP system allows users to organize seemingly random, unrelated events, find trends, and to predict outcomes. Action can then be taken to prevent a negative outcome from occurring. The same is true on the upside: a positive occurrence can be exploited more effectively. The patterns can also be analyzed to improve the underlying processes and applications so as to prevent future miscues.
Implementing the Right SOA Today
So to return to the original question: are you implementing the right SOA for the long term? If you’re implementing an ESB that supports an EDA and messaging for any eventdriven communication, the answer is yes. If you’re not implementing an ESB – for example, if you’re implementing an SOA using application servers, integration brokers and code – the answer is no. You need an ESB.
This brings up the next question: when do you need CEP? The best strategy is to ask that question now and identify the first CEP projects in the same way you identified your first projects for SOA. Think about any area of the business where tracking the state of a process or handling unexpected events adds significant business value.
www.btquarterly.com/
© Галактика, 2007
© Издание 12NEWS (ИП Маринин А.Л.), 2007
