Bridging to the WebSphere JMS provider from the micro broker

One of the key value propositions in the micro broker is its ability to connect the integration hub at the edge with an enterprise messaging server. The Bridge within the micro broker provides this capability in three flavours:

• MQTT (v3) direct into another micro broker or Message Broker.
• WebSphere MQ JMS – requires MQ JMS client to be packaged as a plug-in to Expeditor.
• Third-party JMS providers using JNDI – requires the third-party JMS client to be packaged as a plug-in within Lotus Expeditor.

The third flavour above is what we use to connect the micro broker to the WebSphere JMS provider found in WAS, WESB and WPS.

This post describes how one configures the bridge to make use of the WebSphere JMS provider.

Note that Expeditor must be configured to use the Device Runtime Environment (DRE) J2SE JDK since the WebSphere JMS client requires a full J2SE runtime to work properly.

Creating the WebSphere JMS administered objects for JNDI programmatically

For the JNDI flavour of the bridge, a JMS Connection Factory and the target JMS Destinations must be made available using a JNDI provider. The Bridge is then configured with the appropriate JNDI keys to the administered objects. At runtime the objects are retrieved from JNDI by the Bridge using the keys provided. The JMS flavours of the micro broker both also require a “sync-queue” in order to honour once-only persistent qualities of service.

There are two approaches to creating the JMS administered objects. One is to use the WebSphere JNDI provider (JNDI over IIOP). To make use of this capability one needs to package the WebSphere JNDI client as an Expeditor plug-in. This is a good approach when Expeditor will be in close and reliable networked proximity to the WebSphere server since the creation of the JMS administered objects can be done automatically with the Messaging Destinations in WebSphere. Similarly use of a remote JNDI repository fits the JEE administrative model in that the objects can be managed independently of the application. In scenarios where the network may be unreliable (or indeed only available at certain times), an alternative approach is to create the administered objects programmatically and bind them into the built-in JNDI provider within Lotus Expeditor. This means even without a network at start-up time, the bridge can at least start, even if it cannot successfully connect straight away.

Packaging the WebSphere JMS client in Expeditor

To do this, we need to make the WebSphere JMS client available to the Lotus Expeditor runtime in the form of an Expeditor Feature.

A reduced footprint variant of the client is available as a feature pack for WAS. In order for these classes to be found by the Expeditor runtime, you will need to create a plugin in Eclipse containing the WebSphere JMS client JAR files, exporting all the interfaces (apart from javax.jms) contained in the JAR in the MANIFEST.MF for the plug-in.

There are a couple of very important points to note:

1. You will need to make sure that the javax.jms package is NOT exported and any classes in this package are removed from the underlying JAR file. Expeditor has its own version of the JMS interfaces already exported in the platform and two plugins exporting the same interfaces cause JVM errors at runtime.
2. Note that when you package your JMS client, the feature.xml file referencing the plug-in should have the unpack attribute set to true for the plugin containing the JMS client. In order for the interfaces contained within the underlying WebSphere JMS client JAR to be exported correctly when deployed, the JAR cannot be nested inside the plugin JAR. This is due to a limitation of how the OSGi class loader mechanism works.

Creating the administered objects

The following snippet shows how the WebSphere JMS administered objects are created and bound into the Expeditor JNDI repository.

public static final String JNDI_NAME_SIB_CONNECTION_FACTORY = "jms/SIBConnectionFactory";
public static final String JNDI_NAME_SIB_OUTBOUND_QUEUE = "jms/SIBOutQueue";
public static final String JNDI_NAME_SIB_INBOUND_QUEUE = "jms/SIBInQueue";
public static final String JNDI_NAME_SIB_SYNC_QUEUE = "jms/SIBSyncQueue";
public static final String JNDI_NAME_SIB_DEAD_LETTER_QUEUE = "jms/SIBDeadLetterQueue";

...

/**
 * Use the WebSphere JMS provider's Factory APIs to create the necessary objects
 * and bind them in JNDI.
 */
private void createAndBindSibJMSObjects() {
	try {
		// Create the ConnectionFactory
		com.ibm.websphere.sib.api.jms.JmsFactoryFactory jff = com.ibm.websphere.sib.api.jms.JmsFactoryFactory.getInstance();
		com.ibm.websphere.sib.api.jms.JmsConnectionFactory cf = jff.createConnectionFactory();

		cf.setBusName("SCA.APPLICATION.domino8Node01Cell.Bus");
		cf.setProviderEndpoints("localhost:7276:BootstrapBasicMessaging");

		// Create the request queue (messages on the outbound topic will be forwarded here)
		JmsQueue inQueue = com.ibm.websphere.sib.api.jms.JmsFactoryFactory.getInstance().createQueue(
					"PDARemoteSales.MQTT_JSONoverJMS_Export_RECEIVE_D_SIB");
		// Messages expiry after 3 minutes if not consumed.
		inQueue.setTimeToLive((long)180000);

		JmsQueue outQueue = com.ibm.websphere.sib.api.jms.JmsFactoryFactory.getInstance().createQueue(
		"PDARemoteSales.MQTT_JSONoverJMS_Export_SEND_D_SIB");
		// Messages expiry after 3 minutes if not consumed.
		outQueue.setTimeToLive((long)180000);

		JmsQueue deadLetterQueue = com.ibm.websphere.sib.api.jms.JmsFactoryFactory.getInstance().createQueue(
		"PDARemoteSales.Microbroker.DeadLetter");
		// Messages expiry after 3 minutes if not consumed.
		deadLetterQueue.setTimeToLive((long)180000);
		JmsQueue syncQueue = com.ibm.websphere.sib.api.jms.JmsFactoryFactory.getInstance().createQueue(
				"PDARemoteSales.Microbroker.SyncQueue");
		// Messages expiry after 3 minutes if not consumed.
		syncQueue.setTimeToLive((long)180000);

		try {
			InitialContext ctx = new InitialContext();
			ctx.bind(JNDI_NAME_SIB_CONNECTION_FACTORY, cf);
			ctx.bind(JNDI_NAME_SIB_INBOUND_QUEUE, inQueue);
			ctx.bind(JNDI_NAME_SIB_OUTBOUND_QUEUE, outQueue);
			ctx.bind(JNDI_NAME_SIB_SYNC_QUEUE, syncQueue);
			ctx.bind(JNDI_NAME_SIB_DEAD_LETTER_QUEUE, deadLetterQueue);

			System.out.println("All JMS SIB objects bound");
		} catch (NamingException ex) {
			System.err.println("Failed to bind JMS instances.");
			ex.printStackTrace();
		}

	} catch (JMSException e) {
		System.err.println("Failed to create JMS instances");
		e.printStackTrace();
	}

}

The first portion of the code uses the WebSphere JMS client’s Factory classes to instantiate the JMS administered objects with the latter portion binding them into Expeditor’s JNDI provider (note the Expeditor InitialContext will be used by default within an Expeditor environment).

Configuring the micro broker Bridge

The following snippet shows how the Bridge is configured using the micro broker’s administrative API. In this scenario we are bridging into WPS sending messages from a topic in the micro broker into a queue in WPS and from a queue in WPS back to a topic in the micro broker. By this point we have looked up a LocalBroker instance from the micro broker BrokerFactory service.

// Obtain a handle to the broker's bridge
Bridge bridge = broker.getBridge();

// Create a pipe definition -- this is the root of all bridge links
PipeDefinition pipe = bridge.createPipeDefinition("sibPipe");

JNDIConnectionDefinition connectionDefinition = bridge.createJNDIConnectionDefinition(pipe.getName()+"_Connection");

// Set the Initial Context to be used by the bridge to retrieve the administered objects
connectionDefinition.setInitialContext("com.ibm.pvc.jndi.provider.java.InitialContextFactory"); // uses the parameterised initial context factory i.e. XPD
connectionDefinition.setConnectionFactoryKey(JNDI_NAME_SIB_CONNECTION_FACTORY);
connectionDefinition.setDeadLetterKey(JNDI_NAME_SIB_DEAD_LETTER_QUEUE);
connectionDefinition.setSyncQKey(JNDI_NAME_SIB_SYNC_QUEUE);
connectionDefinition.setURL("none"); // Not meaningful for Expeditor JNDI but *is* still required to be set.

pipe.setConnection(connectionDefinition);

// Create an outbound flow that reads from a topic called "localoutbound" and
// puts to the remote queue.
FlowDefinition outbound = bridge.createFlowDefinition("outboundFlow");
// Set the source to be a single topic, "localoutbound"
outbound.setSources(new DestinationDefinition[] { bridge.createTopicDefinition("localoutbound")});
// Set the destination to be a queue on the remote WPS, using the reference bound in JNDI
outbound.setTarget(bridge.createJNDIDefinition(JNDI_NAME_SIB_INBOUND_QUEUE));
// Add the flow to the pipe
pipe.addOutboundFlow(outbound);

// Create an inbound flow that reads from a remote queue called "outbound" and
// puts to a topic called "localinbound"
FlowDefinition inbound = bridge.createFlowDefinition("inboundFlow");
// Set the source to be a queue on the remote WPS, using the reference bound in JNDI
inbound.setSources(new DestinationDefinition[] { bridge.createJNDIDefinition(JNDI_NAME_SIB_OUTBOUND_QUEUE)});
// Set the destination to be a topic called "localinbound" on the local broker
inbound.setTarget(bridge.createTopicDefinition("localinbound"));

// Add the flow to the pipe
pipe.addInboundFlow(inbound);

// Pipe is configured, add it to the bridge
bridge.addPipe(pipe);
// Start the pipe
bridge.startAllPipes();

The pipe is now ready for use.

What is Extended Integration?

When we talk about SOA and integration software, we typically think of enterprise SOA middleware that lives in a data centre. For example, we think of an Enterprise Service Bus implemented using messaging middleware like IBM WebSphere ESB forming the backbone for the organisation’s IT applications. All of this infrastructure is core to achieving the value proposition of SOA in enterprise terms.

To build on this foundation and put this capability into the hands of users, we need to expand our thinking to consider how we reach out beyond the data centre. This causes us to consider things like:

• Who are the users of the system and what is important to them in business terms?
• How do they need to interact with the system? For example are they in offices on a laptop or desktop PC or mobile?
• How is the organisation structured? Is it a set of large office complexes or a network of branches that interact with a central back-office (such as in retail or banking)?

All of these concerns and more are addressed in the field of what we have come to know as Extended Integration. This whole class of problems relate to extending the reach of SOA out of the data centre and into the hands of the people who need to use it. Extended Integration gives rise to conceptually to a three-tiered architecture as follows:

1. The enterprise SOA platform, such as an ESB, BPM and other enterprise middleware servers.
2. An intermediate integration tier translating the protocols and semantics of the enterprise SOA platform into a form that is easily consumed by the end user applications. This might be in itself an enterprise-class server like WebSphere Application Server (WAS) or a smaller footprint hub like the Lotus Expeditor micro broker. Whilst logically separate from the primary SOA platform, this tier might still be situated within the data centre or remotely, for example deployed in a retail branch.
3. A consumer tier built using appropriate client technologies such as Rich Internet Application (RIA), rich desktop or mobile client or indeed solid state sensors and actuators when the application is consumed by a machine.

The distinction of the middle layer is at the heart of the Extended Integration thought, that of having an intermediate tier to translate between the service orientation of the SOA platform and the required service orientation of the consuming applications.

This way of viewing SOA solutions sharpens the focus particularly on some particular technologies we have available to us today.

Web 2.0, WOA, widgets and mashups

No surprises that any modern IT-related observation includes reference to Web 2.0. This is not a spurious reference I hasten to add, simply an illustration of how a set of technologies and architectures that fall under the Web 2.0 banner appear when viewed through this Extended Integration perspective.

Web Oriented Architecture (WOA) is about recognising that the kind of modern, flexible, heterogeneous web applications we see in the Web 2.0 world are constructed using fine-grained services that take SOA a step further. It is a web-centric view of the world consisting of networked resources and URLs, lightweight contracts between services (i.e. unlike WS-*) and the key to the kind of end-user IT systems that can respond rapidly to business change that will be most needed in these turbulent times. Where we have WS-* underpinning the high qualities of services required for Enterprise SOA, we have REST, URIs, feeds and HTTP underpinning the consumer channels where more often we require a quick, simple and short-term answer in a lightweight device such as a web browser.

Furthermore, we see far better separation of data from presentation in the WOA model, the Achilles Heel of many Web 1.0 systems, with the increased flexibility that brings. We now have an architectural pattern for the web tier constructed of data feeds in well-known formats (e.g. ATOM, JSON), on standard protocols and semantics (HTTP and REST) and to match it, a similarly modular approach for creating the user interface: widgets. Not that widgets are anything new, but the approach of designing the web UI in terms of modular, interoperable components reflects a sea change in recent times. When HTML took hold in earnest, it was considered so simple in comparison with traditional, programming-oriented desktop UI approaches that the UI was a disposable commodity. In many ways this led down the path of sprawling, point-solutions in the web tier on the grounds that they were relatively cheap and easy to extend and replace. Now when we survey the consequences ten or more years on, we realise that in many cases, significant cost savings in terms of development and maintenance can be achieved by applying the same sort of thinking in the web application tier as we apply in enterprise SOA: modularity and designing for reuse.  In IBM our recommended AJAX framework is the Dojo Toolkit, not least because of its comprehensive widget component model that facilitates such reuse and controlled extension of UI functionality.

Mashups have become an increasingly important evolutionary product of this approach. Mashups mix together WOA resources from a variety of sources to produce new systems quickly. Adopting WOA principles across the board for longer-lived systems means that by developing an application on feeds and widgets, it can be effectively “mashup-enabled” at the same time. The key here is that composition is achieved with business-user tools, not programming. From a business perspective that opens the potential for a “just good enough” application without the need for a lengthy and IT-intensive development project. The key focus from an IT perspective for the mashup environment is in providing a sufficiently rich library of widgets and feeds for the business users to consume.

In Web 2.0 terms, Extended Integration is all about mapping SOA and WOA together and translating the semantics of WS-* into the simple REST endpoints that WOA applications and mashups can easily consume. This aspect of Extended Integration closely aligns with the end-user interaction with the SOA system.

Pervasive messaging and edge integration

In a similar vain, the thought of pervasive messaging has been with us now for around ten years. This flavour of messaging-oriented middleware originated from the domain of embedded computing, solid state sensors and telemetry. A particular scenario in point is Supervisory Control and Data Acquisition (SCADA) in the industrial automation field. At a high-level, pervasive messaging couples together minimal-footprint protocols, semantics to deal with expensive and/or fragile networks and small footprint client libraries designed to run within the bounds of a constrained memory and CPU. IBM’s MQTT protocol provides the means for sensors, actuators and other constrained devices to communicate directly with WebSphere enterprise SOA middleware. It should be noted that the protocol specification is published such that third-party software vendors and specialist hardware can be developed talk MQTT into IBM software.

The development of MQTT was an important first step but it soon became clear that having every device in every location transmitting data directly into the enterprise was something of a two-edged sword. For example, one RFID tag passing a reader may generate many events for an individual tag read. Reconciling is technically straightforward within the enterprise SOA tier, however it means that valuable compute resources are spent within the data centre just performing processes on behalf of the remote sites (more generally known as the edge-of-network). This might be a non-trivial amount of processing for a large retail chain, say, consisting of 1000 stores. The development of first the micro broker and more recently the Really Small Message Broker (RSMB) provide the basis for what we term edge integration. Edge integration is about providing the architectural benefits and capabilities of messaging middleware within a sufficiently small footprint that they can be run with relatively low cost alongside the remote assets concerned. Edge integration provides a focal point for communication between the edge and the enterprise systems.

Some typical requirements satisfied by edge integration include:

1. Better separation of concerns between the edge and the enterprise SOA systems as discussed above.
2. Flexibility and interoperability between edge applications and devices. Device and sensor hardware and software often have highly proprietary low-level APIs. The ability to decouple applications and devices using a messaging paradigm facilitates the extension of the system and replacement of hardware and software with minimal disturbance.
3. The need for the remote site to continue operation when the network link to the enterprise is down. Such requirements are commonly found in retail where it is paramount that the store systems can continue processing transactions even if the store itself is offline. The micro broker acts as a focal point for the edge and allows applications to continue to send messages that are persisted on the edge infrastructure and transmitted when the link to the enterprise returns. Furthermore this bi-directional link is achieved using messaging protocols to ensure the data is delivered in a consistent and reliable fashion.

Further discussion of edge integration can be found in the IBM Redwiki Messaging from the Edge in an SOA environment using Lotus Expeditor 6.2.

Edge integration forms an integral part of the Extended Integration thought both in terms of its history as an extended set of protocols for edge-of-network and conceptually within the three-tiered architecture described above. The edge integration hub is the focal point in translating between the enterprise SOA protocols (such as WebSphere MQ) and the specialist protocols of the constrained environment at the edge of the network (such as MQTT).