3

If a Solaris based MQ Manager is sending messages to an intermediate Windows MQ Manager that then sends on messages to a Linux MQ manager, does the CCSID need to be changed on them all to be the same? I don't think so but I am being pushed to do so...

I have an issue where a client is sending messages from an application on Solaris via an intermediate MQ Manager on Windows but the Linux MQ Manager final destination based application receives the message with CR/LF line ending characters it can't deal with. Should the receiver end group write a conversion exit program? Or an MCA?

JoshMc
  • 10,239
  • 2
  • 19
  • 38
jc303
  • 232
  • 2
  • 8
  • Is the path something like this: Solaris Application -> Solaris QMGR -> Windows QMGR -> Linux QMGR -> Linux Application? What type of application is the Solaris application (MQI? IBM MQ Classes for Java? IBM MQ Classes for JMS?). What is the CCSID set to on each QMGR? Do the Sender channels have CONVERT(YES) set? Is the application that GETs from the Linux queue manager doing a GET with CONVERT and what type of application is it. Does it run on the Linux server as well, if not what platform does it run on? – JoshMc Aug 08 '17 at 15:34
  • Yes the path is like that Solaris Application (java) -> Solaris QMGR -> Windows QMGR -> Linux QMGR -> Linux Application. The windows QM is CCSID 437, I don't currently have visibility into the sender and the receiver CCSID. Yes the Solaris to Win Channel has convert set to yes and so does the win to linux channel. I am now thinking this is unnecessary. The app doing the reading of the message is a app written in house I believe, but I have yet to get more details, I will though. – jc303 Aug 08 '17 at 22:16

2 Answers2

2

IBM MQ does not deal with line feeds. It is not like transferring a file with ftp and using ascii mode where ftp will convert from Unix LF end of line to Window CR and LF end of line. For MQ it is just another character in the character set to translate. If the sending app is sending data that includes CR/LF characters, MQ will treat them as any other character in the data. If the receiving application expecting a file with end of lines to be sent as a MQ message it would be required to deal with the end of lines.


MQ Classes for Java applications running on a Solaris server default to CCSID 819, and a IBM MQ queue manager running on Solaris also will default to CCSID 819. CCSID 819 is described as ISO 8859-1 ASCII.

I created a file called test.txt containing the single word "test" and ran unix2dos against the file.

The output below shows the ASCII characters as HEX values:

$cat test.txt | hexdump -C
00000000  74 65 73 74 0d 0a                                 |test..|
00000006

Note that in the above output the ASCII hex values 0d is the CR and 0a is the LF, these are the common Windows end of line.

If we assume that the default CCSID 819 is used for both the Solaris MQ Classes for Java application and the Solaris queue manager, then we can start out with the assumption that the above two hex values represent CR/LF at the end of each line.

You stated that your Windows queue manager has CCSID 437 which is typical for a US based Windows server. CCSID 437 is described as USA PC-DATA and is also ASCII.

Linux queue managers typically default to CCSID 1208. CCSID 1208 is described as UTF-8 with IBM PUA and is a Variable Byte character set it can have from 1 to four bytes per character. This can represent over a million characters including all ASCII characters. All 127 ASCII characters are represented in UTF-8 as the same single byte HEX values as in ASCII.

Going from ASCII to UTF-8 is loss less, going from UTF-8 to ASCII can have loss if non-ASCII characters are used since there is not a equivalent character in ASCII and MQ converts it to the default substitution character with HEX value 1A. I have seen this for example with a Euro symbol. Most if not all of the first 255 characters of UTF-8 are the same as CCSID 819.

Given the above CCSID assumptions the conversion would look like this:

Original IBM MQ Classes for Java app running on Solaris sending data with CR/LF end of line characters:

$cat test.txt | hexdump -C
00000000  74 65 73 74 0d 0a                                 |test..|
00000006

Solaris Queue Manager Sender channel with CONVERT(YES) sending to Windows Queue Manager with CCSID 437:

cat test.txt | iconv -f IBM819 -t IBM437 | hexdump -C
00000000  74 65 73 74 0d 0a                                 |test..|
00000006

As expected the output is the same since both 819 and 437 are ASCII character sets and the data was not representing any thing above the first 127 ASCII characters.

Solaris Queue Manager Sender channel with CONVERT(YES) sending to Windows Queue Manager with CCSID 437 Sender channel with CONVERT(YES) sending to Linux Queue Manager with CCSID 1208:

cat test.txt | iconv -f IBM819 -t IBM437 | iconv -f IBM437 -t UTF-8 | hexdump -C
00000000  74 65 73 74 0d 0a                                 |test..|
00000006

As expected the output is the same since both 819 and 437 are ASCII character sets and the first 127 characters of UTF-8 (1208) are normal ASCII characters.


Summary: If the sending application is sending CR/LF in the data, MQ message conversion will not change this and if the CCSIDs in use are the above listed CCSIDs it does not even change the actual HEX character value. The sending application would need to change what they are sending or the receiving application would need to accommodate these characters.


A good reference on ASCII, UNICODE, UTF-8 and more can be found in the article "Unicode, UTF8 & Character Sets: The Ultimate Guide".

JoshMc
  • 10,239
  • 2
  • 19
  • 38
  • This is amazing, thank you for taking the time to write this. The receiving side has understood (finally) that there is no option but to write an exit program for this task. – jc303 Aug 10 '17 at 15:52
  • note that the receiving app may not need a exit (at least not in the sense of the MQ term exit), I'm sure TibcoBW could handle this in some way either directly via the designer or via calling Java code to accommodate the data being sent. Is the data supposed to be a file in a MQ message or is it just badly formatted data such as XML that included line feeds? – JoshMc Aug 11 '17 at 16:04
  • The sender finally agreed to update their sending app and it now properly specifies MQSTR in the messages inbound, this basically fixed everything. Wasted about 2 weeks of investigation. – jc303 Aug 11 '17 at 17:04
0

I smell a bad setup in your MQ environment and anyone who says to make them all the same doesn't know or doesn't understand MQ (& should have zero opinion about the configuration).

Also, setting the sender channel CONVERT attribute to YES is a VERY bad idea and ONLY should be used in extreme cases. see IBM Best Practices for more information. The ONLY time data conversion should be done is when the application issues an MQGET with Convert.

Since, Solaris has no clue about CR/LF, I going to guess at what is going wrong:

  • The message's Format field of MQMD has the value 'MQSTR' set.
  • The sender channel between the Solaris queue manager has the attribute CONVERT set to YES
  • The receiving application on Linux did NOT issue and MQGET with Convert.

Questions:

  1. Did the Solaris application actually put the message with CR/lF?
  2. Did the Solaris application set the MQMD Format field to 'MQSTR'?
  3. Why are the messages hoping through the Windows queue manager? Why not make a direct connection between Solaris queue manager and the Linux queue manager?
  4. What value does the CONVERT attribute of the Solaris sender channel have? i.e. Is it Yes or No?

The simplest solution is to have the Linux application issue an MQGET with Convert assuming the MQMD Format field is set to 'MQSTR'. MQ will automatically convert the message data to the correct platform.

If the MQMD Formet field is not set to 'MQSTR' then MQ will NOT convert the message data between platforms. That would imply that the Solaris application put the CR/LF in the message. I find this hard to believe. If a developer did this then they truly do not know MQ (and should not be programming it).

Roger
  • 7,062
  • 13
  • 20
  • I will get more info about the particulars but I think you are correct - the supplier of the message said that they've never had to add any additional changes to their messages to dozens of other parties. Our side was receiving data in HEX. We asked them to add the MQSTR and we started receiving data in ASCII, but there were CR and LF in the data. In fact we also had them change the channel to convert yes, to noavail. I think internally the devs for the app haven't properly considered receiving raw messages with MQMD.Format=MQFMT_NONE set and that they are responsible for MQGET with convert. – jc303 Aug 08 '17 at 22:21
  • The receiving application is Tibco Business Works – jc303 Aug 10 '17 at 15:53