BizTalk 2004 and 2006 -> Processing a large message and a faster, less CPU intensive splitter pattern.

The following is a comparison of splitter orchestration patterns using BizTalk 2004 SP1 and BizTalk 2006 Beta 1. You can download the samples discussed below at the end of this blog entry and install and test on a BizTalk 2004 installation and/or BizTalk 2006 installation. Included in the download is a large test message (approximate size of 133 MB).

Some details of the tests are as below:

1) All orchestration implementations split or debatch the same message.
2) The xml message to split is approximately 133 MB in size (on disk)
3) There are a total of 80570 production orders in the message.
4) An abbreviated message with just one complete production order looks like the below.

5) Each production order in the message contains a production order id. For example: sch_prod_order_id = 10031
6) There are 51 distinct production order ids in the message.
7) In each splitter pattern implementation, 51 messages are split out from the main message, as defined by the 51 distinct production order ids. Each split message contains all the productions orders with one distinct production order id.
8) The size of the split messages range in size from 111KB to 23MB.
9) Four Different Orchestration Splitter pattern implementations were tested. The patterns were tested on a BizTalk 2004 SP1 installation with 1GB of memory and a BizTalk 2006 Beta 1 installation. All BizTalk databases (i.e. MessageBox etc.) ran on the same physical machine as the BizTalk servers. My BizTalk 2006 Beta 1 installation was installed on a VPC image. The image had only 700 MB of memory assigned to it, and needed more memory for a fair comparison with the BizTalk 2004 SP1 installation. Therefore the results for the BizTalk 2006 installation are incomplete.

Implementation One : Orchestration Splitting with a Map

The details on this splitter pattern can be found HERE. To quickly recap, a large production order message is processed by an orchestration and first run through a map utilizing custom XSLT to produce an xml message with the distinct production order ids as below:

<ns0:DistinctProductionOrders xmlns:ns0="http://SqlSplitterHelper">
  <ProductionOrder ProductionOrderId="10001" />
  <ProductionOrder ProductionOrderId="10003" />
  <ProductionOrder ProductionOrderId="10002" />
</ns0:DistinctProductionOrders>
  
Each iteration of a loop in the orchestration is used to read a distinct production order id. Additionally, in each iteration of the loop, a map is used  to filter out a split message containing productions orders with that iterations distinct production order id.

Implementation Two : Orchestration Splitting using a SQL Server database

One of the problems faced with Splitting the large messsage using a map, is the CPU utilization of the BizTalk server. During the life of a splitter map orchestration instance, the CPU is maxed out at 100% (Note: This was true with the BizTalk 2004 installation but did not seem to be with the BizTalk 2006 installation). If there is only one BizTalk Server in the group, this limits the number of orchestration instances that can be running simultaneously. Therefore why not use something else to split the message and not rely on CPU resource intensive XSLT? One approach is to use a SQL server database to help split the large message. Querying 80000 or more rows is fast and efficient in Sql server and each message can easily be split using a Select Statement (with a For XML Auto or XML Explicit clause). One of the problems is how to get the 133 MB message from a BizTalk Orchestration into a Sql Server database efficiently (explained below). Below are the steps done in the orchestration to get this pattern to work:

1) The large production message comes into the Sql splitter orchestration.
2) The large message is assigned to a variable of type : System.XML.XMLDocument

varXMLDomMessageToProcess = msgProductionOrdersToSplit;
 
3) A .Net helper component was called from the orchestration passing as one of the parameters the OuterXML of System.Xml.XmlDocument variable as below:

varstrReturnedXML =  varObjSqlHelper.GetDistinctProductionOrders(varXMLDomMessageToProcess.OuterXml,varStrSqlConnectString,varGuidOrchestrationInstanceID);

Note: Ultimately a stored procedure is called by the .Net helper code to process the large production order message. A .NET helper component (called within an Atomic Scope shape) is used to call the stored procedure in place of Send/Receive shapes and a Send port using a Sql Adapter for the following reasons:
a) To cut down on the number of items that have to be persisted.
b) Passing the large string of XML to the stored procedure via a Send Port and Sql Adapter did not seem to work with the large message (Note: I did 
not try this with BizTalk 2006 only BizTalk 2004).

4) The .Net Helper method, receives the string of XML (Note: This is the entire 133 MB message) and calls a stored procedure passing as one of the parameters the string of XML (Note: The stored procedure is called only once with the entire 133MB message). The Stored procedure then inserts all the production orders into a Sql Server relational table. It also returns back a string of XML with a list of distinct production order ids. The code for the .Net helper component that is called from the orchestration is as below:

public string GetDistinctProductionOrders(string xmlProductionOrderMessage,
      string connectionString,
     System.Guid orchestrationIdentifier)
{
 string distinctOrdersXML;
 // Set up SqlConnection,SqlCommand and Parameters to call Stored procedure -> InsertProductionOrders
 SqlConnection sqlConnect = new SqlConnection();
 sqlConnect.ConnectionString = connectionString;
 SqlCommand sqlCommand = new SqlCommand();
 sqlCommand.CommandType = System.Data.CommandType.StoredProcedure;
 // Below is name of the Stored Procedure that will be called
 sqlCommand.CommandText = "[InsertProductionOrders]";
 // Add the Parameters needed to call the stored procedure
 sqlCommand.Parameters.Add(new System.Data.SqlClient.SqlParameter("@xmlString", System.Data.SqlDbType.Text));
 sqlCommand.Parameters.Add(new System.Data.SqlClient.SqlParameter("@orchestrationIdentifier", System.Data.SqlDbType.UniqueIdentifier));
 // Set the XML passed to the method to the @xmlString Parameter that will be receive by the stored proc.
 sqlCommand.Parameters["@xmlString"].Value = xmlProductionOrderMessage;
 // Set the UniqueIdentifier (Guid) so calls to splitter proc will get correct records for that orchestration instance
 sqlCommand.Parameters["@orchestrationIdentifier"].Value = orchestrationIdentifier;
 sqlCommand.Connection = sqlConnect;
 // This will take a few minutes, so set to 0 to wait indefinitely
 sqlCommand.CommandTimeout = 0;
 // open connection to DB
 sqlConnect.Open();
 // Use an XmlReader to get back the Distinct XML of ProductionOrdersIds from the Stored Procedure
 System.Xml.XmlReader xmlReader = sqlCommand.ExecuteXmlReader();
 // Now use the XmlReader to get the XML returned by the stored procedure
 xmlReader.MoveToContent();
 string xmlProductionOrderIdNode; 
 System.Text.StringBuilder sbForReturnedXML = new System.Text.StringBuilder();
 while (!xmlReader.EOF)
 {
   xmlProductionOrderIdNode = xmlReader.ReadOuterXml();
   sbForReturnedXML = sbForReturnedXML.Append(xmlProductionOrderIdNode);
 }
 // Get all the XML from the String Builder
 distinctOrdersXML = sbForReturnedXML.ToString();
 xmlReader.Close(); 
 sqlConnect.Close();
 // Return the Distinct list of Production orders in XML Format
 // Note: The calling orchestration will add the Root node and targetnamespace to this xml document.
 return distinctOrdersXML;

5) The .NET helper component from above passes the large message down to a Sql Server Stored procedure (all 133MB in one call to the stored procedure).
A Sql Server 2000 database was created with one table and two stored procedures to perform the initial insert of the XML into a relational table and to 
split out the messages. The objects inside of the database are:

Table -> InsertNewProductionOrders. To hold the contents of the production order XML in a relational table.
Stored Procedure -> InsertProductionOrders. Uses an OpenXML statement to take the XML and insert into the InsertNewProductionOrders table.
Stored Procedure -> GetSplitProductionOrderMessage. Is called to split out each individual message.

The Stored procedure (InsertProductionOrders) inserts the XML message into relational table (InsertNewProductionOrders) using an OpenXML statement. It 
then executes a select statement (with a For XML clause) to return the list of distinct orders in the table. Other benefits of using OpenXML with BizTalk can be found HERE and HERE.

Stored procedure -> InsertProductionOrders is as below:


Create Procedure InsertProductionOrders

-- Will take string of passed XML and insert into a table

@xmlString Text, -- This is the String of XML for all the production orders
@orchestrationIdentifier uniqueidentifier  -- Identifier for matching records to correct orchestration 
instance

as

DECLARE @iDocProdOrders int

-- Use OpenXML to insert the orders into the table

EXEC sp_XML_PrepareDocument @iDocProdOrders OUTPUT, @xmlString, 'xmlns:ns0="'">http://SqlSplitterHelper"/>'
if @@error <> 0 goto err_handler

Insert Into InsertNewProductionOrders
Select *, @orchestrationIdentifier 
From OpenXML (@iDocProdOrders,'//ns0:ProductionOrder',1)
              With (trk_unit_id int '@trk_unit_id',
 pro_product_id int '@pro_product_id',
 actual_grade varchar (80) '@actual_grade',
 actual_basis_weight float '@actual_basis_weight',
 actual_length_at_turnup float '@actual_length_at_turnup',
 actual_length float '@actual_length',
 actual_weight_at_turnup float '@actual_weight_at_turnup',
 actual_weight float  '@actual_weight',
 required_width float  '@required_width',
 actual_width float  '@actual_width',
 required_diameter float '@required_diameter',
 actual_diameter float  '@actual_diameter',
 actual_core varchar (80) '@actual_core',
 actual_property_1 varchar (255) '@actual_property_1',
 actual_property_2 varchar (255) '@actual_property_2',
 actual_property_3 varchar (255) '@actual_property_3',
 update_timechain smallint '@update_timechain',
 update_time datetime '@update_time',
 update_user_id int  '@update_user_id',
 position_index int  '@position_index',
 comment varchar (255) '@comment',
 required_length float '@required_length',
 required_weight float '@required_weight',
 actual_mfg_grade varchar '@actual_mfg_grade',
 actual_moisture float '@actual_moisture',
 actual_caliper float '@actual_caliper',
 actual_colour varchar '@actual_colour',
 actual_finish varchar '@actual_finish',
 set_number int  '@set_number',
 position_percent int '@position_percent',
 tare_weight float  '@tare_weight',
 user_scale_weight float '@user_scale_weight',
 wire_side char (3) '@wire_side',
 trkc_length_adjust_type_id int '@trkc_length_adjust_type_id',
 actual_compression float  '@actual_compression',
 actual_hardness float  '@actual_hardness',
 sch_prod_order_id int  '@sch_prod_order_id',
 trk_set_item_id int '@trk_set_item_id',
 trk_unit_id_package int '@trk_unit_id_package')  
if @@error <> 0 goto err_handler  

EXEC sp_XML_RemoveDocument @iDocProdOrders
if @@error <> 0 goto err_handler

-- Now return back the list of Distinct Production Orders from the
-- table. The orchestration instance id is used to return back the correct Production orders
-- if two or more orchestration instances are running simultaneously.

Select distinct sch_prod_order_id as 'ProductionOrderId'
From  InsertNewProductionOrders as ProductionOrder
Where  OrchestrationIdentifier = @orchestrationIdentifier
For XML Auto

Return 0

err_handler:

Return -1


6) As with the map splitter orchestration, each iteration of a loop shape processes a distinct ProductionOrderId. (Note: the list of distinct ProductionOrderId's was returned by the InsertProductionOrders stored procedure discussed above). In each iteration of the loop, stored procedure GetSplitProductionOrderMessage is called to retrieve a split message containing productions orders with one distinct production order id. A Send and  Receive Shape are used in the orchestration that use a Static Solicit - Response Send port to call stored procedure GetSplitProductionOrderMessage,passing it the following parameters:

@ProductionOrderID int,  -- To filter out the production orders for that message.
@orchestrationIdentifier uniqueidentifier, -- GUID of orchestration instance.
@deleteIndicator  int = 0 -- To indicate whether to delete this orchestrations instances records,
-- so as not to bloat table InsertNewProductionOrders with unecesarry records.

Stored procedure GetSplitProductionOrderMessage, then uses a For XML Auto clause to return back the split message as below:
 
Create Procedure GetSplitProductionOrderMessage

@ProductionOrderID int,
@orchestrationIdentifier uniqueidentifier,
@deleteIndicator  int = 0

as

Select *
From  InsertNewProductionOrders As  ProductionOrder (nolock)
Where sch_prod_order_id = @ProductionOrderID
and OrchestrationIdentifier = @orchestrationIdentifier
For XML Auto -- , XMLData
if @@error <> 0 goto err_handler

if @deleteIndicator = 1
Begin
 Delete InsertNewProductionOrders
 Where OrchestrationIdentifier = @orchestrationIdentifier
 if @@error <> 0 goto err_handler
End


Return 0

err_handler:

Return -1

Implementation Three and Four : Singleton Orchestrations Instances of the Map Splitter and Sql Helper Splitter

For those not familiar with BizTalk, multiple instances of the same orchestration type can be running simultaneously. For example, if the Map Splitter Orchestration subscribes to a receive port that has just accepted two large production order xml messages, two separate running instances of the Map Splitter Orchestration will be launched simultaneously to process the two messages (Note: this is not always true and is dependent on server load). Both the Splitter Orchestration using a map and Splitter Orchestration using a Sql Helper Database are resource intensive at certain point(s) during their lives as orchestration instances. If multiple orchestration instances are running simultaneously on the same BizTalk host instance machine, then at some point the host instance may run out of memory or CPU cycles. Details about when and why these patterns are resource intensive will be discussed below in the results section. To help alleviate the problem, one solution is to configure an orchestration to limit itself to one running instance. One straightforward method is as below:

1) A correlation type in an orchestration is created as below. CorrelationTypeForSingletonReceive is configured to correlate for all incoming messages on a physical receive port that an orchestrations logical receive port is bound to.

2) A Correlation Set is created in the orchestration that derives from the correlation type that was discussed above:

3) The first Receive Shape in the orchestration will initialize the above correlation set. No more orchestration instances of this type will be launched until this orchestration has completed.


4) An outer loop (Loop for Singleton) is then invoked with an expression of 1 == 1. This never ending loop will process all incoming production order messages, but will process only one at a time. Two nested loops are contained within the outer loop. Loop for Distinct Orders (collapsed in the below image) contains the necessary shapes to split and send out messages. Loop For Subsequent Messages will then pick up a new production order message that is ready for processing, but will only do so when the previous message has been completely processed and split into the 51 separate  messages. A receive shape in this loop has its Follow Correlation Set property set to the Correlation set that was derived in step 2) and initialized by the first receive shape of the orchestration. This will pick up a new production order message to process.

5) Once a new incoming message is picked up, it will then go to the top of the outer loop to be split into separate messages.

6) One problem with the above singleton orchestration, is that new production order messages may be received and queued up in the messagebox while the singleton orchestration is processing another production order message. These new production order messages are subscribed to by the running singleton orchestration instance. But if this orchestration instance fails, then these queued up unprocessed messages will be suspended, in an unconsumed state. These Zombie messages will then have be dealt with. In this implementation, a Catch block is used to catch any errors in the orchestration. Inside the catch block is a loop shape. A receive shape in this loop has its Follow Correlation Set property set to the Correlation set that was derived in step 2) and initialized by the first receive shape of the orchestration. This loop will then pick up any of the unconsumed messages. These unconsumed messages could then be submitted back to the singleton  orchestration. Additionally in BizTalk 2006, Suspended messages can be subscribed to by orchestrations, so it is possible that this technique could be used.

Results of Splitter Tests

Note: for all tests a series of System.Diagnostics.Debug.WriteLine were used in the orchestrations to capture the progress and processing times of the orchestration instances.
DebugView, was used to view the WriteLine statements in real time.

BizTalk 2004 Results

The following hardware was used: Laptop with 2.0 GHz processor and 1GB of memory.
Operating System :  Windows XP SP2
This machine hosted the BizTalk server and a Sql Server with all the relevant BizTalk databases. The SplitterHelper database that is used by the orcSplitterUsingSql.odx and orcSplitterUsingSql_Singleton.odx orchestrations was installed on the BizTalk Server machine, but separate tests were also done with a remote SplitterHelper database installed on a Laptop with a 866 MHz processor and 512MB of memory. The BizTalk server machine and the remote Sql Server database were connected over a LAN.

Tests for processing One Message:

In the below result table, only one 133 MB production order message was processed at a time. Only one orchestration was enlisted and started for each test. Total times begin when an orchestration instance starts and ends when the entire message has been split. They do not include the initial publication of the message into the messagebox. The split messages were not sent out through a send port.
Note: The orchestrations can be configured to send out the split messages.
Note: Orchestrations orcSplitterUsingMap_Singleton.odx and orcSplitterUsingSql_Singleton.odx are not included in the results because only one message  is being processed. They will have near identical results to orcSplitterUsingMap.odx and orcSplitterUsingSql.odx.

Below is a legend for the tests:

All times are in minutes:seconds, for example 3:45

Orchestration Name:
The name describes the pattern used.

Local SplitterHelper DB:
For the Map pattern orchestrations, Not Applicable
For the Sql pattern orchestrations, YES if a local SplitterHelper database was called on the BizTalk machine, NO if the SplitterHelper database was  called on the remote Sql Server SplitterHelper database.

Get Distinct Orders:
For the Map pattern orchestrations, time to generate the distinct production order xml message using a map with a Custom XSLT template.
For the Sql pattern orchestrations, time when the orchestration instance starts to when the .Net helper component returns the distinct production order xml message to the calling orchestration. This time includes the SplitterHelper Database processing.

Time to Split:
For the Map pattern orchestrations, time to split out all 51 messages using a map.
For the Sql pattern orchestrations, time to split out all 51 messages using the SplitterHelper database GetSplitProductionOrderMessage stored procedure.

Total Time:
For all patterns, total time to process the entire message.

Orchestration Name Local SplitterHelper DB Get Distinct Orders Time to Split Total Time
orcSplitterUsingMap.odx N/A 8:36 13:49 22:15
orcSplitterUsingSQL.odx Yes 3:01 3:54 6:55
orcSplitterUsingSQL.odx No 4:23 1:20 5:43


Discussion of Results

orcSplitterUsingMap
The CPU was pinned at 100% the entire time this orchestration was running. The map to get the distinct list of production orders and the map to split out the orders are responsible for the CPU utilization.

The good:
a) The whole splitting operation is contained within this orchestration. This simplifies administration and maintenance operations.

The bad :
a) With The CPU pinned at 100%, this limits and/or slows down other operations on the BizTalk server.
b) The total time for the orchestration to complete is not stellar at over 22 minutes.

orcSplitterUsingSql (local SplitterHelper DB)

While the above orcSplitterUsingMap is CPU intensive, the orcSplitterUsingSql is memory intensive, for the following reasons:
a) The operation to load the message in an XML Dom in the orchestration, uses a large amount of memory as the OuterXml is passed to the .NET helper component.
b) The OpenXML call in stored procedure -> InsertProductionOrders , uses a large amount of memory  (basically the same reason as above) and is also CPU intensive. 

The good:
a) This pattern is much faster. Taking only approximately 5-6 minutes to complete.
Note: Sometimes this pattern took longer, up to 12 minutes, because BizTalk was hogging all the memory and  would not give up enough memory to Sql Server and the SplitterHelper database. But the remote Splitter Database had much better results. See below.
b) The CPU is not pinned at 100%.

The bad:
a) Uses quite a bit of memory. If more than one orcSplitterUsingSql instance is running simultaneously on the same server instance, then there will be out of memory issues. (This will be discussed in more detail below)
b) The Sql OpenXML call is memory and CPU intensive. (But the SplitterHelper database can be installed on another machine than the BizTalk Server. See below )
c) The SplitterHelper database is just something else to maintain. Even though it is only a temporary/scratch database, for a fully fault tolerant  solution, this database should be installed in a clustered environment.
d) The structure of this production order message is simple and the Sql OpenXML call is relatively straightforward to write. But with a more complicated message, there would be more T-SQL to write in the stored procedures.
e) The split messages in this case are simple in structure and a For XML Auto clause is sufficient to return the proper xml. For more structurally complex split messages, a For XML Auto clause might not be able to return the desired XML. To get more control of the returned XML, a For XML Explicit clause can be used.   

Note: The new XML features in Sql 2005 will help tremendously in processing XML messages in Sql Server.

orcSplitterUsingSql (remote SplitterHelper DB)

The above results for orcSplitterUsingSql (local SplitterHelper DB) can be applied here also. The only difference is that the SplitterHelper database is located on a different physical server than the BizTalk Server. When the Sql  OpenXML call is executed, the high memory and CPU needs of this operation will be applied to this remote sql server instead of the BizTalk Server.  This helps the BizTalk server a great deal as this heavy Sql OpenXML processing is delegated to another server. Comparing times, orcSplitterUsingSql (remote SplitterHelper DB) in general has better times (4-5 minutes)  compared to the (5-6) minutes for the orcSplitterUsingSql (local SplitterHelper DB). Obviously having the SplitterHelper database on another physical machine helps tremendously. The Sql Server installed on the BizTalk machine has to process the persistence points and other BizTalk operations. Therefore having the SplitterHelper database installed on another machine really helps.
Note: Better times can be realized, the remote lightweight machine hosting the SplitterHelper database was a laptop with a only a 866MHz processor and 512MB of memory.

Test for processing Two Simultaneous Messages:

The same hardware set up was used as the tests for processing one message.

For more of a real world scenario, two identical 133 MB large messages were submitted simultaneously to be processed at the same time. Only one orchestration type was enlisted and started for each test. Only the total time to split the two messages are recorded in the below table. Total times begin when the first orchestration instance starts and ends when the two messages have been split. They do not include the initial publication of the message into the messagebox. The split messages were not sent out through a send port. A file receive location was used to submit the two messages at the same time. The receive location was first disabled, two 133MB messages were copied into the folder. The receive location was then enabled.

Orchestration Name

Local SplitterHelper DB

Total Time To Process
orcSplitterUsingMap.odx

N/A

45:23
orcSplitterUsingMap_Singleton.odx

N/A

42.11
orcSplitterUsingSQL.odx

Yes

22:49
orcSplitterUsingSQL.odx

No

22:46
orcSplitterUsingSQL_Singleton.odx

Yes

13:54
orcSplitterUsingSQL_Singleton.odx

No

11:38


Discussion of Results


orcSplitterUsingMap

Two instances of this orchestration were launched simultaneously. While watching the debug window, the two orchestrations ran neck and neck, basically splitting the exact same message at the same time. The memory on my machine was just about maxed out. If I added a third message to be processed, I do believe the machine would have run out of memory. The time to split two messages basically doubled, from splitting one message.

Goods and Bads: Basically the same comments as with the one message tests (see above)

orcSplitterUsingMap_Singleton

As explained above, this orchestration is configured to only ever allow one running instance of itself. Therefore it had similar results to the single message tests. The times were just doubled as two large messages were processed sequentially. From above, two instances of the orcSplitterUsingMap could be running at the same time, but any more and the orcSplitterUsingMap_Singleton will have to be used.

Goods and Bads: Basically the same comments as with the one message tests (see above)

orcSplitterUsingSql (local SplitterHelper DB)  and orcSplitterUsingSql (remote SplitterHelper DB)

Grouped these two together because they had very similar results. This was fun to watch in the debug window. Two orcSplitterUsingSql orchestration instances would be started up at almost exactly the same time. I then had debug statements to see when the .Net helper component was called by the orchestration instances. Both orchestration instances would then try to call it a the same time, but there was just not enough memory to go around, and one instance eventually won out and started to call the stored procedure on the SplitterHelper Database. In HAT, I could see the losing orchestration instance dehydrating and rehydrating for another retry at the atomic scope shape. The losing instance upon rehydration, would then start at the top of the orchestration (last persistence point). This pattern repeated a few times, with the dehydration and rehydration of the loser orchestration instance. The winning orchestration instances call to the .Net helper component completed and now the losing orchestration instance started again to call the .Net helper component. The winning orchestration instance was now in a position to start splitting messages. The winning instance would split between 0 and 17 messages, and the loser instance would repeat the cycle of trying to call the .Net Helper  component, dehydrate and then rehydrate. Eventually, nothing was happening, but then the BizTalk host instance automatically recycled itself (see below  error messages and warnings from the event log). After the host instance recycled and started up, the loser orchestration instance successfully called  the .Net Helper component, and then the two orcSplitterUsingSql instances happily started calling the Splitter Database at the same time to split their respective messages.

Below are some of the errors from the event log before the automatic recycling of the BizTalk host instance:

The Messaging Engine received an error from a transport adapter while notifying it with the BatchComplete event.

The Messaging Engine failed while writing tracking information to the database

The Messaging Engine encountered an error while deleting one or more messages

The Messaging Engine failed to retrieve the configuration from the database

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location:  "SQL://redlake/SplitterHelper/" Reason: Exception of type System.OutOfMemoryException was thrown

The Messaging engine failed to process a message submitted by adapter:SQL Source URL:SQL://redlake/SplitterHelper/. Details:Could not find a matching  subscription for the message. This error occurs if the subscribed orchestration schedule or send port has not been started, or if some of the message properties necessary for subscription evaluation have not been promoted. Please refer to Health and Activity Tracking tool for more detailed information on this failure

Finally two warning messages appeared in the event log, announcing the recycling of the BizTalk server instance:

All receive locations are being temporarily disabled because either the MessageBox or Configuration database is not available. When these databases become available, the receive locations will be automatically enabled.

An error has occurred that requires the BizTalk service to terminate. The most common causes are an unexpected out of memory error and an inability to connect or a loss of connectivity to one of the BizTalk databases.  The service will shutdown and auto-restart in 1 minute. If the problematic database remains unavailable, this cycle will repeat.

Note: No split messages were lost or duplicated when the host instance was recycled. BizTalk maintains the state of the orchestration instances with  persistence points.

Note: I sometimes became impatient waiting for the host instance to recycle itself. I then went into the Windows Task Manager and clicked on the Processes Tab. I would them find BTSNTSvc.exe process, right mouse buttoned on it to launch the pop up menu and clicked on End Process to terminate  the BizTalk host instance (Note: This is a real nasty thing to do and I do not recommend it). After manually terminating the BTSNTSvc.exe process the host instance automatically restarted itself and then finished off processing the two orchestration instances.

Note: As explained above, the two orchestration instances were able to successfully complete after the host instance was recycled. One of the problems (I think), was that the winning orchestration instance was still holding onto a good chunk of memory (even after it called the .Net Helper component). But after the host instance recycled, the losing orchestration instance was able to get enough memory to process.

Note : As explained above the .Net Helper component is called within an Atomic Scope shape. A property on the Atomic Scope shape is called Batch and can be set to either True or False. If this property was set to True, and two orcSplitterUsingSql instances are running, only one instance is allowed to call the .Net Helper component and the other has to wait (I observed this by having a Debug statement inside the  .Net Helper component). If it is set to False, then by watching the debug statements, both can call it simultaneously. Either way, true of false, the host instance had the same behavior of recycling itself. Setting it to True might be more beneficial in this case. HERE is a scenario where you might want to set it to False.

Note: Sometimes both orchestration instances dehydrated and rehydrated a number of times before one successfully called the .Net Helper component. The different patterns of dehydrating and rehydrating were observed when the batch property on the atomic scope shape was changed and when the two orchestration instances were launched at different times.

Note: HERE is another scenario similar to the above on processing and mapping a large message.

The good:
a) The good news is that no messages were lost, due to persistence points, and both orchestrations instances successfully completed. But other than that, there is really nothing good here.

The bad:
a) Recycling of BizTalk Hosts instances on a regular basis, is not acceptable in a production environment.
b) This is not a good pattern for multiple simultaneous large messages to be processed. If there was multiple instances running in the Host, then they could share the load and take on one large message each. But at some point there might be more large  messages to process then available host instances (also you might run out of money buying more Enterprise versions of BizTalk Server). More memory on the BizTalk machine might help also. Moving the BizTalk databases to another physical machine would also help.
Note: One fix is to allow only one running singleton instance of this orchestration (see below).

orcSplitterUsingSql_Singleton (local SplitterHelper DB)  and orcSplitterUsingSql_Singleton (remote SplitterHelper DB)

As explained above, this orchestration is configured to only ever allow one running instance of itself. Therefore it had similar results to the single message tests. The times were just doubled as two large messages were processed sequentially.

The Good:
1) This pattern can handle simultaneous large  messages, by processing one at a time.
2) This pattern is much less CPU intensive then the map patterns.
3) Other orchestrations and operations can be running on the BizTalk server, while the sql splitter orchestration instance is splitting messages.
4) The host did not recycle, because only one orcSplitterUsingSql_Singleton orchestration instance is ever running.

The Bad:
Basically the same comments as with the one message tests (see above)

The overall winner:

If more than one large message is being submitted in the same timeframe, then the singleton orchestrations must be used. This is somewhat dependent on hardware for each BizTalk instance and the number of BizTalk instances assigned to the host running the orchestrations. The performance and time winner is clearly the orcSplitterUsingSql_Singleton (remote SplitterHelper DB), but the downside to this method is that a SplitterHelper database must be maintained and the extra code (.Net and T-Sql) that must be written to get this to work. Additionally, performance will be improved if the SplitterHelper database is installed on a dedicated machine.

BizTalk 2006 Tests

Some of the same tests were performed against the BizTalk 2006 installation. BizTalk 2006 was installed on a VPC Windows 2003 image (running on a Windows XP host) with only 700MB assigned to the VPC image. The BizTalk 2006 installation was handicapped compared to the BizTalk 2004 installation and not all the tests were run. Until the tests using BizTalk 2006 can be performed on the same hardware as the BizTalk 2004 installation, the comparisons are not accurate.

The patterns I do have results for are as below (one message being processed):

orcSplitterUsingMap -> 48:36
orcSplitterUsingSql (remote SplitterHelper DB) -> 17:01

Some of the observations I did make are:
The map to get the distinct list of production orders was much quicker on BizTalk 2006. 2 minutes compared to over 8 minutes on BizTalk 2004. When Splitting the messages using a map, the CPU was not pinned, but this mapping operation was very slow. I am attributing it to the fact that BizTalk 2006 being installed on a VPC image with only 700MB of memory assigned to it. 

Final Thoughts

From reading -> Large messages in BizTalk 2004, what's the deal? it discusses 
performance problems processing large messages with BizTalk 2004. Therefore, something to consider is using Sql Server to help with message processing in BizTalk:
1) This post has discussed, a BizTalk splitter pattern that has used Sql Server to help process a large message.
2) This POST discusses using Sql Server to help with a BizTalk aggregator pattern.
3) This POST discusses processing a large flat file using a Sql Server BCP operation to help split the message. 
4) A future title of a post could be: Extending BizTalk Mapping and Transformations using Sql Server 2000/2005. A good demo for this future post, would be passing two large XML messages down into a Sql Server stored procedure, then using T-Sql to perform the actual mapping of the messages.
5) As mentioned above. Sql 2005 has quite of number of improvements and enhancements for processing XML.

I really would like to try the tests discussed in this entry on a proper BizTalk 2006 installation (I just do not have the time or spare machine to do so at the moment).  From reading the documentation and from my few observations, there are real performance improvements with large messages and BizTalk 2006. 

Please read all the posts in the below blogs. These are written by the BizTalk team members at Microsoft. Having an understanding of how BizTalk works internally is a must before you start your first BizTalk project.
BizTalk Performance Blog:
Core Engine Blog

The below whitepaper is a good read for performance and BizTalk
Microsoft BizTalk Server 2004 Performance Characteristics Whitepaper

You can download the samples discussed above HERE LINK and try on a BizTalk 2004 and/or BizTalk 2006 installation. Read the ReadMe.Txt before installing and running.