Creating your own EDI Schema for BizTalk - Part I

First off, you'll notice that the title of this blog includes Part 1.  I don't have any other parts...yet.  But from what I've seen so far in terms of paucity of documentation and general gotchas, I fully expect there to be.

So I'm create an EDI schema for a 204 transaction.  Not a very large transaction, so it figures to be relatively easy to put together.  As it turns out, there are a couple of loops within the transaction.  Since the 204 transaction is a load tender, there is a header/detail relationship between some of the segments.  So after create the header segments, I define an element called DetailLoop.  Then, further down in the schema, I define DetailLoop as containing, among other segments, an S5 segment.

Once the schema has been built, I build the project that contains it.  A build error is generated. Specifically, the error is:

Segment name [DetailLoop] does not start with value of tag [S5]

The problem is that the EDI schema validator gives more weight to the loop name than I originally thought.  It is expecting that the name of the loop start with the name of the first tag in the loop. In other words, when I changed the name of the loop from DetailLoop to S5Loop, the error went away.

Authenticating senders and recievers in BizTalk

Once more back into the EDI/BizTalk breach.

While testing an BizTalk EDI-document orchestration that I created, I ran into a couple of problem.  Specifically, an EDI document was rejected because BizTalk was unable to authorize the sender and receiver.  The specific error message (as found in the Event Log) was “The recipient of the document is not recognized” or “The sender of the document is not recognized“. The documentation on exactly what is being done here is sparse, so let me describe what BizTalk is looking for.  To start with, consider the following fist couple of lines in the EDI document.  These lines are before the transactions contained within the document and are used by EDI subsystem authenticate and route the document.

ISA*00*          *00*          *ZZ*CONTOSO        *12*4162164603     *050328*0852*U*00401*000004393*0*P*>
GS*OW*9055551212*5198581234*20050331*1052*3733*X*004010

When an EDI document is received through the pipeline, the subsystem opens the document to find the send.  In the above code, the sender is indicated by the 'ZZ' and 'CONTOSO' blocks in the first line.  As well, the sender id is found in the third field ('9055551212') in the second like.  These data are combined to form a party identifier. The party identifier looks like EDI://CONTOSO:ZZ:9055551212. Once the identifier is created, the EDI Subsystem examines the list of Parties to find the sender.  If not found, then an entry is put into the event log saying that the sender could not be authenticated.

The second part of the authentication process is to authenticate the receiver. The identifier for the receiver is constructed in a similar manner.  For the above example, the recipient party identifier is EDI://4162164603:12:5198581234.  The information used to create this come from the first line (segments 6 and 7) and  the second line (segment 4).  Not only does the EDI Subsystem look for the sender party identifier in the Parties list, but also the recipient party identifier.

What this means is that in order for the EDI Subsystem to accept an EDI document, both the sender and recipient need to be defined as a Party.  To do this, open up the BizTalk Explorer, expand the view to reveal the Parties node.  Right click on Parties and select Add Party. In the dialog box that appears, give a reasonable name for the party.  In the Properties list that appears, add a property called EDI, with a Qualifier of EDI and a Value of the sending party identifier (for our example, EDI://CONTOSO:ZZ:9055551212).  Save this new party.  Make sure that a second party is created using the receiving party identifier.  With this combination of parties, the incoming EDI documents can be correctly authenticated and fed in to the processing pipeline.

New Article - Optimizing the Downloading of Large Files in ASP.NET

Just to let everyone know, I have posted a new article on techniques that can be used to optimize the downloading of large files in ASP.NET.  It discusses some of the architectural issues that impact download speed if you need to push multi-megabyte files to a browser client. If you're interested, you can find it here.  As always, comments and suggestions are appreciated.

XSD2EDI Failure locks the repository

As a result of the problems that I described in my last post, I ran into another little documented (there seem to be a number of these surrounding EDI and BizTalk) problem.  When I rebooted my system after the Validate Schema process hung my system, I immediately tried to run it again (by definition, insanity is trying the same thing over and over, expecting a different result, isn't it?).  This type around I got an error message saying that my respository was locked.  Specifically (and for Google), the message is:

Schema validation failed: Repository was locked on host [hostname] by process [xsd2edi], processid [nnn]

Sympton: Build output:
XSD2EDI failed to convert XSD: Can't lock repository because it is already locked

It turns out that the correction for this is quite simple.  In the BizTalkEDIDb, there is a table called parame.  This table contains a single record (at least, that's what it has on my system).  One of the first in this table is called repolock.  In that field is the name of the host, process and process id that is updating (I assume) the repository.  It is set by XSD2EDI and, naturally, the information survives a system crash and restart.  To unlock your respository and be able to validate your schemas again, set the value of repolock to null using your favorite method. 

BizTalk EDI and Virtual PC

As you might have figured out (from my recent posts), I'm working on a BizTalk project.  Specifically, I've been dealing with the joys of trying to get the base EDI adapter working the way that I want/expect.  Yesterday, it was trying to get the 850 Schema validated. There are two items that I want to mention that got in my way.

First, because of all the different software I try out, I'm a firm believer and user of Virtual PC.  So the BizTalk I'm using is installed on a VPC running 2K3.  The system on which it's running has 1GB of RAM, around 650MB are allocated to the VPC.

So back to the story.  In the BizTalk project in VS.NET, you right click on a schema and select Validate Schema.  This basically runs the XSL2EDI.exe process to validate the schema and convert it into the BizTalkEDIDb tables.  On my system, I was finding that running this process would freeze my machine after about 10 minutes of running.  And by 'freeze', I don't just mean the VPC.  My entire system would seize up, requiring a hard reboot to get started again.  After having this happen a couple of times, I looked at what might be the cause.

Turns out that the project in which the schema I was validating still had an orchestration running.  Once I stopped the orchestration, the freezing problem seems to be solved.  Or, to put it a slightly more pragmatic way, it hasn't happened since and I've validated more than one schema since then. It's only empirical evidence, but it's all that I have.

XSD2EDI Failed to convert XSD

I've been spending some time on BizTalk Server 2004 these past few weeks.  And for the next few, just so that you know the source of future posts.  I've run into this situation a couple of times, so I'm posting as much as a reminder to me as for any other reason.

My current project has me working EDI (after a seven or eight year absense) integration with an existing application.  As a result, I get the chance to work with BizTalk's EDI Adapter. From a configuration perspective, it's quite simple to get set up, but I was receiving a strange error on my build.

XSD2EDI: Failed to convert XSD: Compiling repository failed

According to a support article (http://support.microsoft.com/?kbid=883549), this is caused by an authentication issue with the BizTalk Base EDI Service.  But, in my case, that wasn't the problem. Instead, what appears to be happening is that this server, even though it is marked as being Automatic, wasn't starting.  The first time I saw this error, the service actually Disabled, so you need to address that particular issue.  But even after that, I would still see the error the next time I rebooted the server.  Nothing in the event log to indicate a problem, so at the moment I'm both puzzled and looking for clues.  I'll keep you posted if I find something.

Using C# in BAT files

Personally, writing BAT files has never been my thing.  And with the experience I have in C#, the ability to script and automate in that language would be quite useful.  Thanks to this post from Jason Clark, I now have the ability to run C# (and for the VBers among you, VB.NET) as part of a BAT file.  Incredibly cool.

Getting Schema information from a Database

Yesterday in my class I was going over the new GetSchema API in Whidbey and I learned from one of my students about another technique that I wasn't aware of - SQL ANSI 92 Information Schema Views. They are also supported in Oracle.

There are almost too many ways of getting database schema information but I'll try to summarize here to see how they compare. Depending on what you want to do, one of these will be more appropriate for you.

  • A common technique is to execute a SQL statement and describe the result set. Execute a “Select * FROM known_table_name”. Some API's would have you add a “WHERE 1=2“ if you can't just “describe“ the result set without incurring row retrieval payload. You can take advantage of this technique using the DataReader's GetSchemaTable method. In this example, you are really getting the schema of a query and not per se the underlying table structure. As such, it doesn't have to be constrained to a “*“ or 1 table query either.
  • Query the SQL Server System tables - this is close to the metal - which can burn you, but if you don't mind built in fragility. This technique is going to likely become obsolete in SQL Server Yukon with the new “Sys.*“ views.
  • Slightly better than using the system tables would be using system stored procedures such as sp_tables
  • Use the SQL-92 ANSI Standard Information Schema views such as “SELECT * FROM Northwind.INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = N'Customers'. Much nicer, and generic across anything that implements this standard (SQL Server 7.0 and up, and Oracle 9i? and up)
  • In .NET 1.0, you could also use the GetOleDbSchema method of an OleDbConnection. This is somewhat hard coded to a RDBMS and requires an OleDb driver.
  • In .NET 2.0, you can use new GetSchema method on the connection object. In theory, the idea is you get schematic information specific to the provider - so things that are specific to the provider can be made available/discoverable. But I'll warn you, it's a pretty abstract API. It's also remotely similar at the root level to the MSXML 4.0 “GetSchema“ method in that a) the methods have the same name, and b) they accept an argument of “Schema“ to return. In theory, you'll get more accurate schema information for the thing you are connecting too. Collections can be defined by the provider which differs from GetOleDbSchema which normalizes all db objects into a known set of objects. That may not work so well for new providers, Object Relational databases, or who knows what next.

More Class Designer Productivity Potential: Batch Editing.

Daniel Moth says that he's not excited about the properties box in the class designer and would prefer to use the code editor to make those kinds of changes. It may not be obvious but one of the things you can do with that properties pane that you can't do in the code editor is make multiple changes across several class or several members at the same time.

Select all of the items that you want to make a mass change to, and any common properties are show in the properties dialog. I find this useful for decorating properties of my own components with custom attributes. Perhaps I want to change a bunch of methods to Static.

Daniel mentions another limitation. There is no full signature support on the model surface in the class designer. This makes it impossible to see the differences between your overloads. In fact, overloads are all grouped together and a count is shown.

Another mass editing scenario would be to change the XML comments on a bunch of methods - for example several overloads. You can't see the individual overloaded methods - just one of them with a “+1 overloads“ next to them. Furthermore, when you change the comment for a method that is overloaded (and shown as “+2 overloads“) one would hope the comment would be applied to all of the overloads, however the comment is only applied to the first one. Hopefully this is a bug and will be fixed. I've logged it with MS in the Product Feedback Center.

Installing Visual Studio Team System Dec CTP

I'm getting a fair amount of questions about this topic lately so worth a blog entry.

The best way to install any beta (or even more so CTP's) is to use Virtual PC. This will save you from having to reformat your entire machine a few times. I don't think I've ever know a VS.NET beta release that uninstalled properly.

So if you are going to use Virtual PC - the best way to get started is to find a friend who has already done the install successfully and get them to give you a copy of their Virtual Machine's.

In general with VPC's, you get better performance if the VHD files are located on a drive other than what your host OS is installed on. If you have a 2nd internal drive, great, otherwise, a good 7200 RPM external USB 2.0 drive will give a good performance boost. You'll also get best performance if you don't use a differential drive or an undo disk. To save memory and CPU cycles, turn off any unessential services and running programs in both the host and guest operating systems.

Yes, you need 2 machines for VSTS - one for the server/data tier, and a second for the client. You can't currently install everything on one box - that is not a supported scenario - at least for now. Your server should also be a domain controller. Unless you have 2 GB of ram, you'll likely want to host each one of those VPC's on a separate box. I've had good results having the server/data tier hosted in Virtual Server. You also have no real need to log in/have a UI open for the server box once it's all installed and configured.

There is a good document here with more detailed installation instructions: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvs05/html/vstsinstallguide.asp

Having said all of that, the next beta is due out this month or early next month so you might want to wait to get a much better experience. As always, keep in mind that beta's are flaky and CTP's are worst than that.