When we talk about SOA and integration software, we typically think of enterprise SOA middleware that lives in a data centre. For example, we think of an Enterprise Service Bus implemented using messaging middleware like IBM WebSphere ESB forming the backbone for the organisation’s IT applications. All of this infrastructure is core to achieving the value proposition of SOA in enterprise terms.
To build on this foundation and put this capability into the hands of users, we need to expand our thinking to consider how we reach out beyond the data centre. This causes us to consider things like:
- Who are the users of the system and what is important to them in business terms?
- How do they need to interact with the system? For example are they in offices on a laptop or desktop PC or mobile?
- How is the organisation structured? Is it a set of large office complexes or a network of branches that interact with a central back-office (such as in retail or banking)?
All of these concerns and more are addressed in the field of what we have come to know as Extended Integration. This whole class of problems relate to extending the reach of SOA out of the data centre and into the hands of the people who need to use it. Extended Integration gives rise to conceptually to a three-tiered architecture as follows:
- The enterprise SOA platform, such as an ESB, BPM and other enterprise middleware servers.
- An intermediate integration tier translating the protocols and semantics of the enterprise SOA platform into a form that is easily consumed by the end user applications. This might be in itself an enterprise-class server like WebSphere Application Server (WAS) or a smaller footprint hub like the Lotus Expeditor micro broker. Whilst logically separate from the primary SOA platform, this tier might still be situated within the data centre or remotely, for example deployed in a retail branch.
- A consumer tier built using appropriate client technologies such as Rich Internet Application (RIA), rich desktop or mobile client or indeed solid state sensors and actuators when the application is consumed by a machine.
The distinction of the middle layer is at the heart of the Extended Integration thought, that of having an intermediate tier to translate between the service orientation of the SOA platform and the required service orientation of the consuming applications.
This way of viewing SOA solutions sharpens the focus particularly on some particular technologies we have available to us today.
Web 2.0, WOA, widgets and mashups
No surprises that any modern IT-related observation includes reference to Web 2.0. This is not a spurious reference I hasten to add, simply an illustration of how a set of technologies and architectures that fall under the Web 2.0 banner appear when viewed through this Extended Integration perspective.
Web Oriented Architecture (WOA) is about recognising that the kind of modern, flexible, heterogeneous web applications we see in the Web 2.0 world are constructed using fine-grained services that take SOA a step further. It is a web-centric view of the world consisting of networked resources and URLs, lightweight contracts between services (i.e. unlike WS-*) and the key to the kind of end-user IT systems that can respond rapidly to business change that will be most needed in these turbulent times. Where we have WS-* underpinning the high qualities of services required for Enterprise SOA, we have REST, URIs, feeds and HTTP underpinning the consumer channels where more often we require a quick, simple and short-term answer in a lightweight device such as a web browser.
Furthermore, we see far better separation of data from presentation in the WOA model, the Achilles Heel of many Web 1.0 systems, with the increased flexibility that brings. We now have an architectural pattern for the web tier constructed of data feeds in well-known formats (e.g. ATOM, JSON), on standard protocols and semantics (HTTP and REST) and to match it, a similarly modular approach for creating the user interface: widgets. Not that widgets are anything new, but the approach of designing the web UI in terms of modular, interoperable components reflects a sea change in recent times. When HTML took hold in earnest, it was considered so simple in comparison with traditional, programming-oriented desktop UI approaches that the UI was a disposable commodity. In many ways this led down the path of sprawling, point-solutions in the web tier on the grounds that they were relatively cheap and easy to extend and replace. Now when we survey the consequences ten or more years on, we realise that in many cases, significant cost savings in terms of development and maintenance can be achieved by applying the same sort of thinking in the web application tier as we apply in enterprise SOA: modularity and designing for reuse. In IBM our recommended AJAX framework is the Dojo Toolkit, not least because of its comprehensive widget component model that facilitates such reuse and controlled extension of UI functionality.
Mashups have become an increasingly important evolutionary product of this approach. Mashups mix together WOA resources from a variety of sources to produce new systems quickly. Adopting WOA principles across the board for longer-lived systems means that by developing an application on feeds and widgets, it can be effectively “mashup-enabled” at the same time. The key here is that composition is achieved with business-user tools, not programming. From a business perspective that opens the potential for a “just good enough” application without the need for a lengthy and IT-intensive development project. The key focus from an IT perspective for the mashup environment is in providing a sufficiently rich library of widgets and feeds for the business users to consume.
In Web 2.0 terms, Extended Integration is all about mapping SOA and WOA together and translating the semantics of WS-* into the simple REST endpoints that WOA applications and mashups can easily consume. This aspect of Extended Integration closely aligns with the end-user interaction with the SOA system.
Pervasive messaging and edge integration
In a similar vain, the thought of pervasive messaging has been with us now for around ten years. This flavour of messaging-oriented middleware originated from the domain of embedded computing, solid state sensors and telemetry. A particular scenario in point is Supervisory Control and Data Acquisition (SCADA) in the industrial automation field. At a high-level, pervasive messaging couples together minimal-footprint protocols, semantics to deal with expensive and/or fragile networks and small footprint client libraries designed to run within the bounds of a constrained memory and CPU. IBM’s MQTT protocol provides the means for sensors, actuators and other constrained devices to communicate directly with WebSphere enterprise SOA middleware. It should be noted that the protocol specification is published such that third-party software vendors and specialist hardware can be developed talk MQTT into IBM software.
The development of MQTT was an important first step but it soon became clear that having every device in every location transmitting data directly into the enterprise was something of a two-edged sword. For example, one RFID tag passing a reader may generate many events for an individual tag read. Reconciling is technically straightforward within the enterprise SOA tier, however it means that valuable compute resources are spent within the data centre just performing processes on behalf of the remote sites (more generally known as the edge-of-network). This might be a non-trivial amount of processing for a large retail chain, say, consisting of 1000 stores. The development of first the micro broker and more recently the Really Small Message Broker (RSMB) provide the basis for what we term edge integration. Edge integration is about providing the architectural benefits and capabilities of messaging middleware within a sufficiently small footprint that they can be run with relatively low cost alongside the remote assets concerned. Edge integration provides a focal point for communication between the edge and the enterprise systems.
Some typical requirements satisfied by edge integration include:
- Better separation of concerns between the edge and the enterprise SOA systems as discussed above.
- Flexibility and interoperability between edge applications and devices. Device and sensor hardware and software often have highly proprietary low-level APIs. The ability to decouple applications and devices using a messaging paradigm facilitates the extension of the system and replacement of hardware and software with minimal disturbance.
- The need for the remote site to continue operation when the network link to the enterprise is down. Such requirements are commonly found in retail where it is paramount that the store systems can continue processing transactions even if the store itself is offline. The micro broker acts as a focal point for the edge and allows applications to continue to send messages that are persisted on the edge infrastructure and transmitted when the link to the enterprise returns. Furthermore this bi-directional link is achieved using messaging protocols to ensure the data is delivered in a consistent and reliable fashion.
Further discussion of edge integration can be found in the IBM Redwiki Messaging from the Edge in an SOA environment using Lotus Expeditor 6.2.
Edge integration forms an integral part of the Extended Integration thought both in terms of its history as an extended set of protocols for edge-of-network and conceptually within the three-tiered architecture described above. The edge integration hub is the focal point in translating between the enterprise SOA protocols (such as WebSphere MQ) and the specialist protocols of the constrained environment at the edge of the network (such as MQTT).