Asynchronous Enterprise Messaging

These notes on enterprise-level messaging applications offered by IBM, Microsoft, and other vendors. This is related to my pages on enterprise development and TIBCO. Let me know what you find helpful or missing.

Topics this page:

Site Map  
About this site  

Asynchronous Message Queuing Services

Unlike electronic mail (e-mail) the messaging discussed here uses different message formats and communication protocols (handshaking mechanisms) to exchange information between software components rather than between people (using software application clients).

Like email, also called "interpersonal messaging", enterprise message queuing applications use asynchronous processing to send and receive messages. An application can send messages when the recipient is not up and running simultaneously. This allows a component to send information to another and to continue to operate without receiving an immediate response.

Email provides a "best effort" delivery method that does NOT automatically confirm receipt.

Transactional Messaging

But transactional messaging systems also assumes the responsibility to ensure reliable delivery of messages to the target queue, or, if it cannot do so, take appropriate action.

Message queuing applications are generally designed so that messages flow in a request/reply fashion. But unlike emails, a transactional messaging client does not have to request messages in order to receive them. Inter-program communication messages are delievered to a client as they arrive.

After the messaging system accepts a message from the application into a queue as an intermediate storage point. The application is then free to continue other work. This allows communicating programs to run at different times.

Message queuing provides the connectivity layer "backbone" for what IBM calls an enterprise service bus (ESB) within a service-orientated architecture (SOA).

Messaging components read and operate on the messages independently. The sending and receiving applications are uncoupled or loosely coupled. rather than tightly coupled, such as with Remote Procedure Call (RPC). The component sending messages can send messages without depending on information about other components' interfaces. Programs are insulated from network complexities (fixed message formats, communications protocols, etc.). Communication protocols are hidden from the applications. All this allows components to be easily replaced.

Most importantly, messaging applications are designed to be reliable — ensure that a message is delivered once and only once. Lower levels of reliability are available for applications that can afford to miss messages or to receive duplicate messages.

Passing SOAP messages through message queues provide visibility and auditability to interactions.

Java Messaging Service API (JMS)

The de facto interface is MQI (Message Queuing Interface) proprietary to IBM. The AMI Application Messaging Interface

Sun developed the Java System Message Queue Service (JMS) application-level messaging standard API SDK to enable J2EE peer-to-peer apps (using Java 2 SDK, Enterprise Edition, version 1.3) to create, send, receive, and read loosely coupled, reliable, and asynchronous messages.

The sample programs for the API 1.1 doc simplifies and improves programmer productivity by defining a common set of concepts and programming strategies that will be supported by all JMS technology-compliant messaging systems.

Sun's free "Platform Edition" has a Windows variant (now) but does not have the scalability, reliability and advanced security features of the for-fee enterprise edition.

JMS operates in two modes when calling a servlet at a JMS Destination:

• JMS Point-to-Point (Queue) and
• JMS Publish/Subscribe (Topic) (supported in MQ with SupportPac MA0C)

JMS messages can participate in Java Transaction API (JTA) transactions.

The workflow:

1. The servlet enqueues messages to a JMS Destination.
2. The MDB (Message Database) receives the messages, and (depending on Message Property settings) will either
   • ignore the message,
   • rollback the message, or
   • generate and enqueue a response message.
3. The servlet receives response messages, verify their correctness, and exit.

Sun's JMS Developer forum

Sun's JMS Tutorial for J2EE SDK 1.3.1 or 1.4.

IBM WebSphere MQ Series (WMQ)

IBM's Message Queue product (developed in IBM's Hurley Labs in the UK) has the largest market share of any messaging middleware solution. Its full name was "IBM WebSphere MQSeries" because it runs on WebSphere servers. IBM dropped the "Series" at v6.0. v7 debut Oct 2009.

The MQ product consist of three major components operating independently of each other:

1. MQ Server
2. MQ Queue Manager
3. MQS Clients, which maintain synchronous communication / connection with the Queue Manager. It can participate in local but not global units of work involving MQSeries resources.

Servers transmit messages communicate via channels — each a unidirectional link between a MCA (Message Channel Agent) on each queue manager at each end of the communicating channel.

MQSeries uses the Message Channel Protocol (MCP) to ensure delivery.

MQ messages can be as large as 100MB with v6.0, its latest release.

Several Queue Managers can be clustered so that each queue manager can access the queues of all other queue managers in a queue-sharing group through Cluster-sender and Cluster-receiver channels rather than remote queue definitions. Information about clusters are held in a SYSTEM CLUSTER REPOSITORY QUEUE and exchanged through a SYSTEM CLUSTER COMMAND QUEUE.

To ensure high availability and pull workload balancing, applications request web services by connecting to (targeting) a shared queue.

Instead of using a private channel definition to target a specific queue manager, a global definition in a shared repository (DB2 database available to all queue managers) specifies that all servers in the queue-sharing group listen on a generic port (e.g., 4005) for inbound network attach requests. On z/OS, this is done by a listener task started by the channel initiator on each server handling communication between WebSphere MQ on z/OS VTAM and other systems (AIX, Windows, etc.).

If outbound messages are always smaller than 63KB (including headers), a shared channel can provide high availability to communication via synchronization information stored in the SYSTEM.QSG.CHANNEL.SYNCQ queue. Private channels store synchronization data in the SYSTEM.CHANNEL.SYNCQ queue. Consistently smaller messages can take advantage of intra-group queuing such that upon receipt of a message destined for a queue on a different queue manager in its queue-sharing group, enabling messages to "hop" to the correct destination via shared queue SYSTEM.QSG.TRANSMIT.QUEUE instead of using a less efficient transmission queue and channel. Transmission queues temporarily store messages destined for a remote queue manager. To send messages directly, at least one transmission queue is defined for each remote queue manager targeted.

This is why early characterization of Message size Avg/Max is helpful in making configuration decisions.

A message queue channel is "in doubt" when the sending message channel agent is waiting for acknowledgement of receipt for a batch of messages being processed. So information needed to resolve in-doubt units of recovery must come from the coordinating system. WebSphere MQ loses its connection, upon restart it attempts to recover all inconsistent objects.

When a shared queue manager detects another queue manager has terminated abnormally (disconnecting from central z/OS Coupling Facility structures), it performs a multi-phase "peer recovery" process

The number of messages which can fit in a queue (a queue's maximum depth) is the combination of the message size and the size of the persistent z/OS page set where object definitions and message data are stored on non-shared queues. As messages are removed from a queue, space in the page set is freed for reuse.

In-flight (not committed) units of work being performed on the failed queue manager are backed out if the Backout Threshold has not been reached. ???

During the first phase, units of work that progressed beyond the in-flight phase are recovered. This involves committing messages for units of work that are in-commit and locking messages for units of work that are in-doubt. During the second phase, uncommitted messages related to in-flight units of work are rolled back, and input-exclusive information about active handles on shared queues in the failed queue manager are reset, allowing other active queue managers to open the shared queue for input. Request messages in a syncpoint (synchronization point when all the recoverable data that an application program accesses is consistent but not yet put to the response message) are put back on to the request queue and become available for another server instance to process.

IBM WebSphere Application Server Network Deployment V6.0 replaces external clustering software like IBM HACMP with a hot peer failover capability for critical services.

MQI Calls

Applications using the WMQ API (MQI) issue 13 different calls:
1. MQCONN connects an application program to a queue manager.
2. MQCONNX returns a unique connection handle.
3. MQOPEN opens the object before an inquiry, set, get, or put command could be executed on it.
4. MQPMO MQPUT to put messages on queues
5. MQGMO MQGET to take them from queues.
6. MQPUT1 places messages onto a queue that is already open.
7. MQCLOSE closes application objects.
8. MQDISC disconnects connections between an application and a queue manager.
9. MQINQ inquires on the MQSeries object settings.
10. MQSET sets values for attributes of these objects.
11. - 13. Syncpoint commands.

MQ Message Descriptors

MQ messages also contain a Message Descriptor (MQMD) which contains metadata about (properties of) the message and its application data:
• persistence,
• priority,
• message id,
• correlation id, which provides a reference to a particular topic
• report,
• feedback,
• reply to queue,
• reply to qmanager,
• expiry,
• group id,
• message sequence number,
• offset,
• format,
• encoding and
• coded character set id

Queue Manager Commands

Create QM:
crtmqm -q QMgrName
(-q specifies the default queue manager)
-lc (use circular logging),
-ll (use linear logging),
-lf (specify size of log file)
-lp (specify directory to hold log files) can be passed with the crtmqm command

Delete a queue manager:

dltmqm QMgrName

Start a queue manager:

strmqm QMgrName
If the queue manager is started for the first time, default objects are usually created.

Inquire, define, and alter a queue manager's objects:

mqsc QMgrName
this accepts standard input and standard output options

Display attributes of a queue:

DISPLY QUEUE (XXX) ALL

Altered attributes using the ALTER command.

Delete queues using the DELETE command.

Types of Queues

MQ products manage queues (resources):
1. Local queues are local to a queue manager. To create a local queue with a put and get enabled option, a persistence option and maximum depth and maximum message length specified as 1000 and 2000 respectively
   DEFINE QLOCAL (MY_QUEUE) REPLACE +
   DESCR (" A Local Queue") +
   PUT (ENABLED) GET (ENABLED) +
   DEFPSIST (YES) SHARE +
   MAXDEPTH (1000) MAXMSGL (2000)

2. Remote (Distributed) Queues
   DEFINE QREMOTE (RQ1) + RNAME (YYY) RQMNAME (QM2)
   RNAME refers to the remote queue manager named RQ1
   RQMNANE is the name of the remote queue

3. Alias Queue which references another local queue or a local definition of a remote queue can be defined using this:
   DEFINE QALIAS (XXX) TARGQ (YYY)

4. Model queues are used as a template to create dynamic queues:
   DEFINE QMODEL (ANSQ) DEFTYPE (TEMPDYN)

5. Dynamic queues can be permanent or temporary in that they do not survive a queue manager restart.
   DEFINE QMODEL (ANSQ) DEFTYPE (TEMPDYN)

6. Transmission queue
   DEFINE QLOCAL (QM1) USAGE (XMITQ)

7. A Dead Letter queue is used when either a destination queue is not available or when application data conversion fails at the sending end. There is a SYSTEM DEFAULT DEAD LETTER QUEUE.

Other topics of functionality:

1. Triggering
2. MQGMO segmentation options
3. Message data translation
4. Reporting
5. Functional Sequence
6. Message groups and Segmented Messages
7. Distribution Lists
8. MQ Security

Tek-Tips forum on MQ Series

IBM's Crash Course Series portions are on YouTube.

Training classes on MQ Series from:

• Web Age Solutions
• Verhouf

Distributed Computing with IBM(r) MQSeries (John Wiley & Sons; Bk&CD Rom edition, October 25, 1996) by Leonard Gilman, Richard Schreiber, Len Gilman

Achieving Production Quality Messaging article by Nancy Cox at Network Computing magazine

PC Networking slideshow

IBM's MQ newsgroup and HTML Forum

Message Queue User Group UK

Other members of IBM's MQSeries family of product provide specialized functionality:

• MQSeries Adapter provides the framework and tools to build and customize MQSeries adapters for existing and new, pre-packaged or custom-developed applications.

• MQSeries Integrator — IBM WebSphere Business Integration Message Broker and WebSphere Business Integration Event Broker (MQSI) — is an information broker providing for message transformation and information flow modeling across multiple and different business systems.

• MQSeries Everyplace interfaces to mobile protocols and devices.

• MQ Series Workflow enables the definition, execution and swift change of complete business processes.

IBM made MQ open for others to integrate with it (the 'glue' that bonds together many disparate platforms and technologies):

• BEA MessageQ
• MDI interface for the Python programming language

IBM's MQWare is for independent software vendors (ISVs) and Internet Service Providers (ISPs) to embed IBM's MQSeries messaging technology in shrink-wrapped Windows NT-based applications. According to a July 1997 news release: The product provides assured, "once only" message delivery between instances of a message-enabled application (e.g., it ensures information is not lost between servers using MQWare to issue "send and forget" messages between instances of their application in an NT Server-based network).

MQWare was the first product to conform to the Business Quality Messaging concept, a set of standard APIs for suppliers to guarantee messaging delivery.

Historical note:

Business Quality Messaging specifications ratified September 1997 by the now defunct non-profit BQM Forum formed April 1997 among AT&T, Compaq, IBM, Intel, Microsoft. aimed to "revolutionize" business-to-business trading by offering an inexpensive and universal method of reliably transferring information, such as SAP R/3 data, between trading partners. WARNING: Spammers and virus distributors have taken over websites formerly owned by the Message Oriented Middleware Association (bqm.org and www.moma-inc.org).

Shanky Tiwari January 2002 article at Developer 2.0

WebSphere Business Process Management 6.2.0 Performance Tuning

Momentum

Talarian

Candle's Roma product

Extended Architecture (XA) Sync Points

Message queuing provide a means of integrating business processes through message-driven processing.

Asynchronous messages owned by various resource managers (managing queues in WMQ or in database tables) can be combined into a single unit of work using the XA transaction management interface published by the X/Open Company.

XA defines a syncpoint coordinator computer subsystem that manages units of work. It determines when updates within a unit of work can be committed or backed out.

IBM's MQI protocol

• MQCMIT commits a unit of work after the program has satisfactorily completed processing. This cannot be used with external coordinators like CICS or IMS.
• MQBACK rolls back an operation to a state before the unit of work began.
• MQBEGIN begins a global unit of work.

Transaction Processing Facility (TPF)

MQ in z/OS

• Monitoring WebSphere Application Performance on z/OS

Microsoft's MSMQ Message-Oriented Middleware (MOM)

The Microsoft Host Integration Server 2000 and 2004 MQI Channels Tab is used to define MQI channels.

MSMQ was code-named "Falcon" during its development within Microsoft.

MSMQ offers message queuing functionality such as message delivery, cost-based message routing, and full support for transactions.

Components of Microsoft's Message-Oriented Middleware (MOM) offering include:

• The Primary Enterprise Controller (PEC) for the entire enterprise (with names up to 126 characters long).
• Intersite and intrasite message traffic among sites (with names up to 31 characters long).
• Connecting networks linking independent clients (laptops) with the MSMQ server and the MSMQ dependent client (warehouses/stores of data).

MSMQ is integrated with other Windows NT features such as Microsoft Transaction Server (MTS).

MSMQ interoperes with IBM's MQSeries through the Microsoft MSMQ-MQSeries Bridge and other message queuing products through products from Level 8 Systems.

Triggers

Now-defunct CommerceQuest, Inc. 813.639-6300 (which turned into Proforma, then Metastorm) used to sell MQTester add-in (first available Sep. 2003) which extends LoadRunner scripting, playback, monitoring, reporting and analysis capabilities to the IBM MQSeries infrastructure.

It is "agentless". It issues MQ commands to define triggers which notify events such as Channel Started and Channel Stopped, plus Queue Manager events such as

• Queue Manager Active
• Queue Manager Not Active Remote
• Queue Name Error
• Unknown Object Name
• Not Authorized (type 1 -4)

It also monitors performance triggered events:

• Queue Depth High Threshold exceeded
• Queue Depth Low Threshold exceeded
• Queue Depth Full
• Queue Service Interval High Threshold exceeded

Performance

WMQ vs MSMQ

Microsoft commissioned a study by NSTL [dated May 17, 2000] which claims its Message Queuing (MSMQ) product runs "10 times faster than IBM's MQ Series".