Implementation Conventions, or Why EDI Is Such a Pain

I once attended a local EDI user group meeting and heard a presentation by a leading EC/EDI consultant. He was normally quite informed and informative, but I could not believe my ears when he suggested that one way to solve the problems of EDI would be to outlaw implementation conventions. I'll give him the benefit of the doubt and assume that what he really meant was not what he said, because what he suggested would be impossible to accomplish. Implementation Conventions are a fact of life in X12 and EDIFACT, they will never go away, and they will always cause as many problems as they solve. Here is a brief explanation of why.

Standards are Generalized

The X12 and EDIFACT standards for EDI evolved over time to include in them the ability to convey every piece of data that anyone would ever want to exchange. An X12 standard for a specific transaction set (a document) is much more a library or a tool kit from which you can build your own document rather than a specific representation of a document. The standards also continue to evolve as business processes evolve. Consequently, the standards have more in them than any single implementation could ever need or use. Very few bits are mandatory, and most are optional. No one ever uses the full standard, so some way of specifying the part that you do use is absolutely necessary. These specifications may be informally scribbled on the backs of napkins, formally documented as implementation guidelines or conventions, or left for trading partners to figure out from the test EDI data they receive. In the last several years these specifications have become formally referred to as Implementation Conventions (ICs), or Implementation Guides (IGs).

We can illustrate the concepts and problems around ICs by using Venn diagrams from elementary set theory. Let us consider all the defined segments and elements of a typical X12 Transaction Set, say an 850 Purchase Order, as a universe. Let us also consider the subset of that universe that is mandatory, that one must use in order to comply with the standard. This situation is depicted in Figure 1, with the shaded circle representing the mandatory pieces of the 850.

Figure 1 - Mandatory Bits

The shaded piece in itself almost never encompasses enough information to fully describe the desired business transaction. So, we must add other pieces to complete the definition of a typical implementation of the 850. For example, the 850 requires that one send a purchase order number, a purchase order date, and at least one line item among other things. However, it does not require that you send a part or catalog order number for that line item. So, if you and your trading partner need this information (which you probably do), you need to specify it in your IC. This is shown in Figure 2, with the open circle representing the pieces defined in the IC.

Figure 2 - Typical Implementation

This is the primary source of the common complaint that that the EDI standards are not "standard", and the reason why managers can't understand why a "standard" program should be so much trouble to implement.

Because standards are generalized, there are many different ways to implement them, even when the same basic information is exchanged. These leads to our next problem.

Different Implementations of the Same Business Processes

Another problem with the current standards is that they provide so much latitude that there are several ways to represent the same information. In other words, data that is semantically equivalent can be expressed in completely different representations. This is shown in Figure 3 representing two implementations using the same business data, and Figure 4 the different representations of that data. The two open circles represent different ICs, and their intersection is the common pieces. Again, the mandatory pieces are in the inner shaded circle. This is a somewhat extreme case, with little in common between the EDI implementations.

Figure 3 - Same Business Data

Figure 4 - Different Implementations of the same process

Some of the primary sources of this latitude in representation are:

• Equivalent code values - The value in one element is often used to describe the kind of content that appears in another element. For example, PO106 in the purchase order line item segment contains a coded value that tells the type of part number is in PO107. "VP" means it is a Vendor Part Number, "PN" just means it is a Part Number. I can use VP, you can use PN, but we can both mean the same thing.
• Different locations for the same data - Semantically equivalent data can be put in different segments and elements in different implementations. For example, I may put part description in PO107 with a PO106 qualifier of "PD" for part number description. You might instead choose to put it in a PID segment (a child of the PO1 segment) in PID05.

Things get even more complicated when the business processes differ.

Different Business Processes, same Transaction Set

Things get even more complicated when the business processes differ. Another way that the standards are very generalized is that they can represent several slight variations in the same basic business process. The basic business data may be slightly different, and therefore the EDI implementations may also be different. This is shown in Figures 5 and 6, using the same depictions as previous. In Figure 5 dealing with the data, there are no mandatory pieces.

Figure 5 - Different Business Data

Figure 6 - Different Implementations due to different Processes

There may be several sources of these differences in business data:

• Different Transaction Set Purposes - The same transaction set may be used for different business purposes. For example, the 850 Purchase Order can be used to send both a release against a blanket PO, as well as an original, one time PO. In the latter case, one may want to explicitly specify payment and delivery terms. In the former case, these have probably already been negotiated face to face and there is no need to exchange them.
• Same Purpose, but differing business cycles - Documents may be exchanged as just one part of an exchange of several documents in an overall business activity. For example, in one instance a request for quote and a quotation may be exchanged before the purchase order. The buyer and seller may then already know and have stored in their application systems several things about each other and the transaction. Therefore, they don't have to transmit things like contract terms and street addresses. In another case, the P.O. might be the first document exchanged and the two parties might need to exchange that data.
• Same Purpose, but different internal application or processing requirements - The basic business transaction may be the same. However, due to differences that are entirely internal to the trading partners, they may ask for different types of data. For example, one customer may require that vendors include their assigned vendor ID in their business data, while others might derive it from the sender address of the EDI message. One customer may not care about detailed cost tracking on expenses and only wish to have the total amount due on an invoice. Another customer may wish detailed tracking and require detailed break down of charges as well as their internal cost centers for the recipients of the goods or services.

There also is one more case to consider within a single standard, and that is representing different business data in the same fashion.

Same Representation, Different Data

We may also have a case where semantically different data, i.e., data that will be processed differently by a business application, may appear in the same data element in a transaction set in two different implementations. This is not quite as common as the other cases, but it is not unheard of. This situation is not quite as well illustrated using Venn diagrams, but an example should be sufficient. When invoicing customers using the X12 810 Invoice,  one customer might want their invoices sent with the part description for an alphanumeric pager in PID05, with a qualifier of "F" for free form text in PID01. A different customer may want the same PID segment with a qualifier of "F" in PID01, but instead have the name of the person who uses the pager placed into PID05.

Finally, we may have to deal with different versions of the standards.

Different Versions of Standards

ANSI ASC X12 issues a new major version of the X12 standard each year, and two minor releases. In most cases the changes may be minor, such as including new values for coded elements. However, sometimes new segments are added and sometimes transaction sets are heavily revised. Some segments may be eliminated or revised. The X12 Design Rules and Guidelines specify that standards should be upwardly compatible (i.e., data coded to comply with an old standard should comply with a new one). In set terms this means that the old version should be a subset of the new, and that there should be an "onto" correspondence between the old version and the new version (i.e., everything in the old version can be mapped "onto" a corresponding item in the new version). However, this is not always the case. Take an extreme case of an 850 Purchase Order for X12 versions 2001 (circa mid 1980) and X12 version 4010 (1997), and we are dealing with two different universes that don't exactly intersect as neatly as we would like. Again, the mandatory pieces are shaded, and there may be things mandatory in the new version which aren't in the old.

Figure 7 - Different Standards Versions

Some companies stay on the same X12 version forever, while others may have a policy to upgrade to the latest version every year. The result is that a company may have to support several standards versions of the same transaction set.

Things can get even more complex when we deal with entirely EDI different standards. For example, one might trade with companies in Europe or Asia who use the EDIFACT standards rather than the X12 standards used in North America. In such a case the universes would be completely different with no intersection.

The Golden Rule

Not the one from the Bible, but the one that goes "He who has the Gold makes the Rules". This means that you do what it takes to make your customers happy. So, you use their Implementation Convention rather than yours. If you have at least two customers, you probably have at least two different Implementation Conventions that you need to support. Instant complexity. If you happen to have the gold you can usually make the rules and impose only one IC on all of the companies you trade with. Most of us aren't so fortunate.

What a Typical Company Must Deal With

The end result of all of this is that for even a simple example of a company with only a few customers and three versions of the standards, the picture can get fairly complicated, as shown in Figure 8. (I've left out the mandatory shaded circle because these diagrams are getting beyond my skill with Word Art!)

Figure 8 - Typical Company

It is easy to imagine how messy this gets as the number of trading partners grows.

The Impact

The impact of this is that vendors can't effectively hard code application software to accommodate all of these variations. The variations are just too complex. Stand alone general purpose EDI software has evolved to become highly configurable so that the variations may be handled. Most EDI software packages have "mapping" facilities that allow them to be programmed to encode business data into or decode it from its representations in the EDI standard. For outbound data, the source is typically a file or database table that has been exported or populated from the business application, with the destination being an EDI "interchange" of encoded data. For inbound data, the source is the interchange and the destination is an application import file or database table.

Specialized software packages are even available to help one produce Implementation Convention documents based on the standards. These hard copy manuals are then exchanged with trading partners. In addition, an EDIFACT message called IMPDEF has recently been developed to allow an IC to be represented in an encoded form that may be processed by an EDI software package. That encoded IC is then used as to define the EDI source or destination rather than the complete standard.

Can't we do any better than this?

When I wrote the first version of this tutorial nearly ten years ago, there were hopes that new approaches to business process analysis and new techologies (first distributed object technology such as CORBA and DCOM, then XML, and now XML Web Services) were going to fix these problems. Reality has shown us that many of these problems remain as intractable as ever.

Some of the new technologies may reduce the complexity of integrating EDI facilities with business applications. Rather than rely on stand alone EDI translation and communication software, applications may include XML parsers and seralizers or even support communications and security, as can be done with XML Web Services. However, XML has its own unique set of problems, not the least of which are a lack of standardized schemas for common business messages (still waiting since 1998), or even a standard, universally accepted library of common element and attribute names with their associated semantics.

However, while on the whole the technology is improving, some of the problems are not going to go away very easily. Even if the technology could some day be completely standardized, it would still be difficult to accommodate the varying business processes and differing implementations imposed by different trading partners. If this problem is ever solved it will be solved by industry assocations and business people, not by technologists.

About  |  Services  |  Resources  |  Contact

©Rawlins EC Consulting 2007