EDI 101-B – Standards and Syntax

EDI 101 – part II – The Basics of Standards and Syntax

So, you’ve decided to come back for more, eh?  Glutton for punishment, I guess.

This time around, we’re going to cover the concepts of the “STANDARDS” and also the SYNTAX and the Content of your EDI Document.  Now, again, I’m coming from a background in retail and using the ANSI/ASC-X12 standard.  And we use version 4010, which is, arguably, a few versions behind, but that’s not truly important.  I know that UN/EDIFACT and TRADACOMS have their own standards and documents, but, again, I’m just dealing with what I know – X12.

For each industry that uses EDI and the standards, there are different forms that can be used.  The book on My desk for the X12, v 4010, is the size of a dictionary.  And it’s printed on that same super thin paper in tiny little type-face.  And it’s almost 1800 pages of that tiny type.  But not every document is used in every industry that may use EDI and use the X12 standard.  Some are strictly for retailers; some for real estate; some for insurance, for banking, for hospitals.  Some of the documents MAY be used across the industrial lines, but some are very specific and specialized.

 Within that X12 standard, there are literally HUNDREDS (at least 300 by My count) of documents that can be traded – from the 850 Purchase Order, the 810 Invoice, the 860 PO Change, the 852 Activity Data to the 262 Real Estate Information Report, the 255 Underwriting Information Services and 249 Animal Toxicological Data.

Wow…  Who knew?

With TRADACOMS (the Standard used in the United Kingdom for most retailers), there are a couple of dozen.  I’m not sure how many documents are in use for the UN/EDIFACT standards, but I’m sure there are a few.

For each document, there are then a series of hierarchy loops – levels, basically – of the information structure.  These levels – the hierarchies – lay the data out in a defined pattern, so that you can have similar data “grouped” with similar data.  Within those levels, you will have the SEGMENTS and the ELEMENTS we touched on last time.  And you can have segments in multiple levels and even repeated within a level, as needs require.

Still there and with Me?   Good.

When you come to the hierarchies, they’re going to – GENERALLY – follow a structure or a pattern.  Kind of like the e-mail analogy I used last time, where we had a TO, FROM, SUBJECT, BODY and CLOSE, the hierarchies will follow a similar kind of pattern.   For example, an 856, the ADVANCED SHIP NOTICE – or ASN – will follow a particular pattern.  A very common pattern is called SOPI.   SOPI stands for SHIPMENT, ORDER, PACK, and ITEM.

The SHIPMENT hierarchy is all about just what it says – the SHIPMENT information and data.  In this hierarchy loop (or level), you’ll find information about the ASN Number, shipment date information, some ship to or ship from information, a bill of lading or tracking number and more.  You can specify the kind of container that is being used (corrugated cardboard) and the name of the shipping company, the weight of the shipment, the number of cartons, and so much more information about the SHIPMENT.

Following SHIPMENT, you’ll generally find the ORDER hierarchy loop.  This contains information and data, as it pertains to the order information.  You’ll find some date references – order date, ship date, arrival/anticipate date, the Purchase Order Number, maybe vendor identification (number, etc.).  Again, this hierarchy loop is all about the ORDER information.

Next up, you’ll generally have a PACK loop.  Most times, this is a pretty small bit of data.  In the ASN spec I use, it’s all about the marks and numbers – the carton label number – for that box.  That’s pretty much it.  In here, however, there could be any data that refers to the packaging of the products ordered.

Then we’ll see the ITEM hierarchy loop.  This is where you’ll find all the data, as you guessed, about the ITEM being shipped in the ASN.  Widgets…  Shoes…  Apples…  Whatever…  This is all about the goods being ordered and shipped.  Everything that’s in that shipment should be listed on the ASN and this is where the item specific detail goes: colors, sizes, quantities, UPCs, SKUs, the works.

Within each hierarchy loop, there are a number of SEGMENTS that contain the elements and the data.  Each segment has a name – an identity.  Within the ASC X12 standards, it’s generally a 2 or 3 character code that identifies what data should be contained in the SEGMENT.  For example, there’s the TD1, TD3, TD4 and TD5 segments.  This is where you would – generally – find the information pertaining to the CARRIER DETAIL.  Things like who the trucking company is, any routing transit time, special handling, hazardous materials information and more.  Or there can be the SN1 segment.  This is all about the item detail – the shipment.  This segment is where you put in the information – the details – about the item being shipped.  Here’s where you can have UPCs, Item Numbers, SKU numbers, Item Descriptions and more – as long as it’s all about the item being shipped.

The SEGMENTS are further split up into DATA ELEMENTS.  This is the nitty-gritty detail of the shipment.  This is where your content really comes into play.  And the STANDARDS also come in here, as the STANDARD lays out what SEGMENTS fall into which hierarchy loops or levels and what elements and data can be included in the segment. 

The ELEMENTS are all about the actual detail of the shipment: quantities, PO numbers, costs, UPCs, item numbers, carton sizes, and more, are all displayed in the ELEMENTS in the SEGMENTS.  This is the level where you really need to have a keen eye for details, as there may be any one of a dozen possible elements to use to identify the data being sent.

Let’s assume you’re working at the ITEM level and the LIN (Line Item Detail) segment.  And you’re trying to get across a VENDORS STYLE NUMBER or designation.  There are a number of choices – looks like 4 in the copy of the X12 Standard I use.  You can use VA (Vendor’s Style Number) or you can use VC (Vendor’s Catalog Number); or how about VP (Vendor’s Part Number), VN (Vendor’s Item Number) or even VU (Vendor’s Basic Unit Number).  Hey!  That’s five!  Of course, then I also see XA (Preferred Part Number), the MG (Manufacturer’s Part Number) and more and more and more.

In this same SEGMENT, you can also have all the information related to OTHER numbers and information related to the item being shipped – the UPC, the SKU, and so forth.  Truly, however, this qualifier (known as the Product/Service ID Qualifier) could be for use in many documents and many segments.  It could be used for financial records, medical records, educational records… 

This can be where many people who create EDI translation documents have to be really careful.  Since there are a lot of codes and qualifiers that could be used to relay the data and information you’re trying to get across, you need to be sure of what you and your trading partners will recognize.

In a previous blog, I talked about the concepts of EDI being replaced by XML; how there’s the DTD/Schema that tells you want the data being transmitted is.  Well, that DTD/Schema basically functions as the formal “STANDARD” of the document, even though there isn’t any formal STANDARD with XML…  The only “RULE” in XML is that you have a set of tags around each bit of data you’re sending.  The DTD/Schema then tells the receiver what it is that this TAG means.  Think of the TAG as the ELEMENT QUALIFIER in the SEGMENT of an X12 document.

Even with all of the potential for confusion that can be found in any of the standards, having that standard and set of rules makes EDI something that’s not exceptionally difficult.  It can be easy to master, as long as you pay attention to the details and work with your trading partners on the documents you’re trading – from syntax to content – to be sure that the data you’re trading – sending back and forth – is clean, reliable and usable.

Author: Craig Dunham – EDI Coordinator
Read more about Craig here:
http://editalk.com/contributors/

Ch-ch-changes…

Ch-ch-changes… Turn and face the strange…” – David Bowie belted out back in 1972.  Well, 1971, actually, but the song wasn’t released until 1972.  And, as usual, I digress in the details.  But still, some of the lyrics are quite appropriate today.  And especially in the world of EDI.  Good ol’ Ziggy Stardust (aka Bowie) sang out:

                                   “Still don’t know what I was waiting for..

That’s something we hear a lot in the EDI world – once somebody finds out how well EDI can help them.  They don’t know what they waited for – or balked against – when given the option of EDI.  Once they’ve seen-the-light, it suddenly becomes a no-brainer.  But at the time, it was strange and unknown and a change.  And we all know what people can be like when it comes to change.. Don’t we?

  • Change is hard!
  • What’s wrong with the way we’ve always done it?
  • Oh, great!  Now what do I have to learn?

Right now, I’m working with our Accounting group in getting them to embrace and accept the 810 EDI Invoice.  And, for the most part, I’m lucky that they’re open and willing to “face the strange” and go with it..  However, where it’s making My life a living hell is that they expect everything to be done.  Now.  2 Minutes ago.  Yesterday.   ASAP.  Jump!  Jump!  JUMP!!!

Think about the time that you first began to become a part of the EDI world  You probably came from some kind of MIS position – either an operator or a programmer or an analyst or .. Or, you came from another group that your EDI program touches – either the accounting group or the buying group or the warehouse group  or .. Well, you get the picture.

                                        Embrace the change..

And think about the changes (Ch-ch-changes) that you encountered along the way.  Think of how you had to ch-ch-change the concepts that you held and others kept of the way things were and how they were going to be.  Think of how you and others in your organization had to ch-ch-change the way you did things – things that had been done “that-way” for years (or even longer?)..

Some of the pods of flesh on this planet are pretty adept at change.  Others – well, not so much.  No, they’re like the stubborn mule in the old Western-Comedy, leaning back, digging in their heels and not budging.  It takes a lot of force to get that immovable object to take that step forward and “embrace the change”..

Then you sometimes have to try and keep up with those changes..  In recent articles, we’ve touched on many of the changes coming to and infecting EDI as a concept.  Things like AS2, XML, E-Catalogs..  Ch-ch-changes, indeed. 

But are any or all of these changes going to help or hurt you..?

And how good are you at accepting and going with change..?  How good are you at accepting change and working with it and finding the solution to the newest ch-ch-change coming at you..?  Think about your daily commute to and from work.  There’s an accident at this highway and that street.. or the road is closed because of “police activity”..  Or there’s some guy protesting ________ (the war in Iraq, China’s hosting of the Olympic Games, gays in the military, our government’s failed policies, the new Wal*Mart coming to town, whatever – fill in the blank) from that bridge, hanging a sign over the highway..  How quick are you to think – “hmmm.. I can detour here at Main Street, go down 3 blocks to Fifth Avenue, hang a left and be back on the freeway beyond that problem”..?  Or do you just sit there with a bunch of other commuters, waiting for your turn to squeeze through the half open lane to pass by the wreck, not willing to deviate from the norm?

How well you handle change means a lot – both professionally and personally.  Change is an integral part of life.  It’s something that creeps up on us on little tiny quiet feet or comes barrelling into the china shop and disrupting lives all around.  But change is inevitable – just like death and taxes.

And change is big in EDI – no matter how static and stable the platform and concept may be.  There are – and will always be – changes to the way we do things.  Standards are often being updated.  Segments are added or deleted from the document specs.  Suppliers and buyers are often requesting new information to be sent or received.  New applications are added to your back-end systems and now you have to map this segment/element to this other file and record over there.   The PO box you use to receive payments or invoices has been altered, and the data in your documents (POs, Invoices) must reflect that new alteration.  You’ve adjusted your factor or payment “lock-box” location or service provider.  You’ve signed up with a new VAN/Network and have a new qualifier and ID..  All of these are ch-ch-changes.

               “I watch the ripples change their size But never leave the stream..

These are just a few examples of the ch-ch-changes you may face.  And there will be many more, too.  I’ve had our EDI program up and running – well – WE’VE had our EDI program up and running since the very late 90s.  About 5 years ago, we changed our translator (upgrade) and then added a new document (the ASN) and added and expanded our trading partner count by .. well .. multitudes.  Then we added some information to our PO (requested by some of our suppliers) and changed a terms code and .. well, you get the idea.

Ch-ch-changes are important and everyday.  Expect them, plan for them and implement them.  And do not be afraid of them. 

Author: Craig Dunham – EDI Coordinator
Read more about Craig here: http://editalk.com/contributors/

EDI vs. XML – how readable is XML?

Over on the Inovis blog, Bill Chessman ponders about the human readability of XML vs. EDI.  A different take on an old debate. 

Ah, the old “EDI vs XML” debate rears it’s enormous jumbo head of dissent again…  *sigh…..

XML is simply a language for possibly trading information in a human readable format.  Much like HTML (the language of the Internet), you use “tags” that tell you what the next piece of information is.  So you can have a simple tag of <NAME> and then a person’s name and then the closing tag </NAME>….  You can continue to use these simple kinds of tags for the rest of the information you’re sending (sending maybe an address, a 2nd line of address, a city, a state, a zip and more) and the human eye can read it as

John Smith
123 Main Street
Suite 123
Anytown
ST
92345

In his post at the Inovis blog, however, Bill makes an excellent point – and one I’ve touted before in other comments – in that XML may be easier (in the layout of tags and direct information) to read then EDI formats, but it’s all dependent upon the DTD and the schema.  It’s easier because there is no rigid, solid standard to follow.  “Data A” can be ANYWHERE in the document, as long as it’s labeled as “Data A”.  The DTD and the schema are a lot like the translation specs we use in EDI – in that it tells you what tags will contain what data and what it all means.

But it’s that same lack of a standard that makes XML so much more complicated and so much less a solid answer.  There’s no specific spot for that “Data A” mentioned above.  It can be ANYWHERE… and it’s the job of the DTD or schema to tell you where it is and what it means.

Kay Dietz (of Inovis) offers a great course (through the training roadshow) on XML 101 – giving you the most basics of basics – kind of like a sample when you’re at the store…  “Here, try this cookie.  Isn’t it the best? Aisle 3B, 4.99 a dozen!”  XML is a great thing – offering a lot of possibilities for the transmission of data between partners.  And it’s supposed to be human-eyes readable.

But that’s where the problem comes in….  If (as Bill mentions) you don’t know what this tag is and means, or you don’t know what the abbreviation or code I’m using is, how can you interpret the data.  And sure, the schema or dtd can tell you what that is, but you have to read it and/or format the document to tell you what it means….

XML is, at its most basic, a language.  Just like Cobol and C++ and HTML and Chinese and French and English and…. Well, you get the point.  But let’s take that comparison even deeper….  XML could be very similar to Chinese or English – in that you will have different variations on the same language, all depending upon who is speaking.  There are many different dialects in both languages.  In the Chinese language, if you have two people speaking two different dialects, then there is a lot that they cannot communicate.  In the English language, at least if there are two people and two dialects, most of it can come across.  This isn’t a failing in the Chinese language at all.

XML can be a lot like English or a lot like Chinese.  It can be similar and yet different or it can be different and different.  Let’s say I’m sending you an order.  I can send it with the tag <ORDER> and then the number and the closing tag (don’t forget that closing tag!)…  But what if you use <PURCHASE> in your own system.  Sure, that schema or DTD will let you know that I’m sending you the purchase order number in that <ORDER> tag, but you will have to look at the schema for that meaning.

In EDI, since there is a standard (whether ANSI/X12 or TRADACOM or ________), you know that if I’m sending an order, it will be in the BEG segment at location BEG03.  Always.  If you’re fluent in the “language” of EDI – the standard you’re using – you know exactly what information is in what segment and where to look for it and what format it will be in and all of that.  Just as if you’re fluent in English, you will know that in the Southern Dialect, “y’all” generally means “all of you” in a more formalized version of the language.

In My example above (about the name, address, etc., tags), the same information is carried in the N1, N3 and N4 segments.  And it also gives you a bit more information by the element designator (ST = Ship To, BT = Bill To, etc.) so you can also have the same information – but in a different format:

N1*ST*John Smith~
N3*123 Main street*Suite 123~
N4*Anytown*ST*92345~

And there you have the same information.  Sure, there’s the added symbols of * as an element separator and the ~ as the segment terminator, but it’s still readable, IF YOU KNOW THE LANGUAGE.

One of the things that Bill mentions (in the posting) is that EDI was not really supposed to be “human readable” – as it is/was a way of getting data from a source machine or system to a destination machine or system.  It was transferring data from My systems to yours.  If you want “human readable”, you create a report.  It prints out and has all the “tags” set up as field names (name, address, address, city, state, zip) followed by the appropriate data.

Hmmm… that’s just like the XML format of the data above – in that it’s now suddenly human readable.  But instead of having to have the user (aka the human) try and figure out what they mean – the report tells them exactly what they mean, because the data was transmitted and then translated via the standard spec.

Look at it this way – suppose you’re a maker/seller of … gum…  Just looking at the pack of Trident White on My desk, there are 12 pieces of gum to a pack.  Now, let’s say that you put 12 packs in a box.  Then there are 12 boxes to a carton.  And 12 cartons to a pallet.  That’s a lot of twelves…

Now I send you an order…

XML – <QTY>144</QTY>

EDI – PO1*1*144*EA*UP*445600456729~

In the XML version, I order 144.  But 144 what?  Each? Boxes? Cartons?  Pallets?  In the EDI version I give you My UOM (unit of measurement) in the line item detail.  To get that information to you in the XML format, I’d have to add another set of tags – like <UOM>EACH</UOM> or similar or tell you what that quantity means in the DTD or schema.  As a matter of fact, each element in the EDI segment has to have its OWN SET of tags for EACH piece of information…..  So now I’m sending all of this information to you – think about any kilo-character limits or charges you may have – and that simplicity of XML – that “freedom” now is costing you extra.

The XML for just that simple EDI line would look something like this:

<LINE>1</LINE>
<QTY>144</QTY>
<UOM>EA</UOM>
<UPC>445600456729</UPC>

Each piece of information for the item being ordered has to have a set of tags to designate it.  And for EACH and EVERY order I send you, you’re getting that same set of tags and data over and over and over again.  For each line item (style) I’m ordering, you’re getting all of those characters that make up those tags – over and over and over and over and….  Whereas with EDI, you get – hey, look – a separator (*)….  So, let’s just do a quick character count of the EDI PO1 segment I’ve got up there…. 29.  29 characters…  The same for an XML version of that line…?  Let’s see .. 14 for the qty line .. um .. carry the one .. divide by .. add .. Where’s that damn calculator..?!?  It came to 64 characters, btw.

 And the more “detailed” I make My tags (use QUANTITY instead of QTY, UNIT OF MEASURE instead of UOM) and you can see how it can REALLY start to add up.  Oh, and then you’re also sending that DTD or schema with EACH and EVERY document.  Imagine if you had to send your translation spec with EVERY document you send..

So, yeah, XML has some benefits in the simplicity department.. and in the “human interface” area.  But with no standards to abide by, no rules to play by, that human readability goes right out the window with whatever dialect you want to speak.

Author: Craig Dunham – EDI Coordinator
Read more about Craig here: http://editalk.com/contributors/