The Case For Unicode

The common misconception is that Unicode is only required if your SAP system need to have multiple languages that are not supported by the standard code page. There are much more to it, click "Read more" to understand all the reasons why your SAP systems need to be running Unicode, written in non-techno speak. Truth of the matter is Unicode has become the Lingua Franca of inter application communication. This is an industry standard, not an SAP standard. Data passed in interfaces between applications needs to be interpreted correctly in the target application, especially if there are interfaces between your SAP system and external vendors. SAP realized that if it wants to interface effectively with other applications in the enterprise, interface with the rest of the world through common standards of XML, and using JAVA technologies, that they have to get to the common, industry standard way of data representation. It is hard enough for data to be converted between application formats in interfacing. You don't want to add character representation/translation as a complicating factor into an already complex mix. The Unicode standard is a standards based representation of characters so that interfacing between different applications becomes easier and more consistent. Middleware solutions depend on a common data representation format for consistent interfacing between applications on different platforms. SAP's Service Oriented Architecture (SOA) requires your SAP system to be Unicode enabled for this reason as well. When other applications that are Unicode enabled (SAP Portals, MDM, etc....) send data to an SAP system that is not Unicode enabled, you risk what is called "silent data loss" when the Unicode/Non-Unicode conversion is made during transit. This happens because Unicode's character set represents over 107,000 characters, whereas standard ASCII represents 94 printable characters. SAP does the translation from Unicode to non-Unicode by looking at the codepage in the target system, as well as the logon language. If the Unicode character being sent matches one of the 94 printable characters represented by code page 1100 (covering western European and English languages - assuming the target system is represented by one of these languages), then the characters get translated correctly. If the Unicode character being sent is not found in the standard character set of the target system's logon language/code page combo (eg. the Euro symbol), then a '#' typically gets substituted losing the meaning of that character. This affects the integrity of the record. This error may not be reported by the standard validation routines because the '#' is not an invalid character for an alphanumeric field. This does not necessarily mean that all you have to is look for '#' in your data records. You can also have silent data loss because the encoding translation looked at only one byte of a two byte source character. The system does not detect this and therefore does not report it as an error - this is referred to as silent data loss.

Different platforms support different character encodings. Sometimes the code that represents a character in one coding (think external interfaces) may represent a different character with the same coding number in the target. This results in data corruption as well. Visit the Unicode Consortium's website for further explanation of this phenomenon.

How YOUR non-Unicode SAP system can potentially be affected

If you have any Unicode system in your SAP landscape interacting with your nonUnicode SAP system, you may risk uncontrolled data loss. Think of: SAP Portals (Runs on a JAVA stack that is Unicode), consider your CRM, SRM, MDM, XI/PI, BW or any other Unicode enabled SAP system in your landscape. HTTP/HTML. If you have a web facing (internet OR intranet) application interacting with your non-Unicode SAP system, you may suffer silent data loss because the source (the HTTP frontend), will allow a Unicode character to be entered. See SAP note 838402 SOA (Service Oriented Architecture) - SAP recommends that SOA not be implemented in new projects that involve a non-Unicode system, because the conversion may result in silent data loss. SOA was designed with the intent to provide standardized and reliable business processes between different components. See SAP note 1358929 MS-Office integration - Microsoft products are fully Unicode compliant. If your nonUnicode SAP system accepts data from Excel for example, you risk silent data loss because Excel allows Unicode character input. See SAP note 838402

SAP Products Requiring Unicode Enabled R/3 Systems

Most, if not all, of the supporting software around SAP is already Unicode enabled. This includes Unix, Windows, Oracle, MaxDB, DB/2, JAVA, XML, LDAP etc... In order to ease in the Unicode Conversion for their customer base SAP put a lot of tools in place to defer the conversion. There are Unicode/Non-Unicode conversion routines built into the RFC code for example so that you can communicate from your SAP portals to your R/3 backend seamlessly. This backward compatibility and conversion is not only becoming a headache, but also increasingly difficult to prevent hidden data loss due to character set differences in character representation. Hence SAP's increasing recommendation for their clients to do the conversion. SAP products that require your SAP systems to be Unicode enabled include:

Process Integration (PI) Service Oriented Architecture (SOA) Master Data Management (MDM)

But That Is Not The Only Reason To Upgrade/Convert!

You know you might need to upgrade to Unicode if:

Your CEO says you are expanding the business to China, you will need Unicode. For Chinese legal reporting and business reasons. Your HR Department says that you will need to roll out ESS/MSS to your European subsidiaries, you need Unicode You're planning to install a new ECC 6.0 system, you need Unicode. You can't install a new SAP system with non-Unicode anymore. Ever. Your business users are evaluating new functionality that includes PI or MDM, you will need Unicode

Realistically, what is your risk if you defer converting to Unicode?

Read more...