The Structure of a SOAP Message

In order to gain a complete understanding of services-oriented architecture (SOA), one of the basic pieces of knowledge is the structure of a SOAP messages.  At a high level, SOAP is a lightweight protocol for exchange of information in a decentralized, distributed environment.  But that description gives short shrift to the true utility of SOAP, that being the transmission of a request for service between two heterogeneous and networked applications.  In this article, we look at the structure of a SOAP message in great detail, as well as put the SOAP message into the context of the HTTP request.  This second part has to potential to be a fleeting reference, as there is nothing in the SOAP specification that requires that HTTP be the conduit.  However the ubiquity of HTTP has made it the de facto standard for transmission and it will probably be a few years before it is displaced.

 

For the purposes of this document, SOAP is a Remote Procedure Calling protocol that works over HTTP.

The body of the request is in XML. A procedure executes on the server and the value it returns is also formatted in XML.

Procedure parameters and returned values can be scalars, numbers, strings, dates, etc.; and can also be complex record and list structures.

Request example 

Here's an example of a SOAP request:

POST /examples HTTP/1.1
User-Agent: Radio UserLand/7.0 (WinNT)
Host: localhost:81
Content-Type: text/xml; charset=utf-8
Content-length: 474
SOAPAction: "/examples"



   
      
         41
         

      

   


Header requirements 

The format of the URI in the first line of the header is not specified. For example, it could be empty, a single slash, if the server is only handling SOAP calls. However, if the server is handling a mix of incoming HTTP requests, we allow the URI to help route the request to the code that handles SOAP requests. (In the example, the URI is /examples, telling the server to route the request to the "examples" responder.)

A User-Agent and Host must be specified.

The Content-Type is text/xml. The charset may be specified, if not, the default is US-ASCII. Other acceptable charsets are UTF-8 and UTF-16. UTF-8 is recommended for maximum interop. See the note on UTF character encodings, below.

The Content-Length may be specified, if it is, it must be correct.

As with the URI in the first line, SOAPAction is used to route the request to a handler on the server that will respond. It's entirely up to the application to determine how this header element is used. In many implementations the URI and the SOAPAction header will have the same value.

Payload format 

The payload is in XML, a single element.

All the attributes of the are required as shown in the example above.

The must contain a single element, which contains a single element which is the procedure call. The name of the procedure is the name of this element. Note that the procedure name must be a valid XML element name.

The elements contained within the procedure call are the parameters to the procedure. The names of the parameters is significant, the order of the parameters are not. Parameter type is indicated by the xsi:type attribute.

For example, the procedure name could be the name of a file containing a script that executes on an incoming request. It could be the name of a cell in a database table. Or it could be a path to a file contained within a hierarchy of folders and files.

A procedure call may take no parameters, if so, the procedure element must not contain sub-elements.

Scalar values 

The following scalar value types are supported by this subset of SOAP 1.1.

Type value Type Example
xsd:int 32-bit signed integer -12
xsd:boolean a boolean value, 1 or 0 1
xsd:string string of characters hello world
xsd:float or xsd:double signed floating point number -12.214
xsd:timeInstant date/time 2001-03-27T00:00:01-08:00
SOAP-ENC:base64 base64-encoded binary eW91IGNhbid0IHJlYWQgdGhpcyE=


Structs 

A value can also be a struct, which is specified by an XML element that contains sub-elements. structs can be nested, and may contain any other type, including an array, described below.

Here's an example of a two-element struct:


   18
   139
   

The names of struct elements are significant, the order of the elements is not.

Arrays 

A value can also be an array, which is specified by an XML element with a SOAP-ENC:arrayType attribute whose value begins with ur-type[, followed by the number of array elements, followed by ].

Here's an example of a four-element array:


   12
   Egypt
   0
   -31
   

The order of array elements is significant, the names of the elements are not.

You can mix types as the example above illustrates.

If the array elements are of a single type, the value of the array element's SOAP-ENC:arrayType specifies the type of the array's sub-elements, for example, SOAP-ENC:arrayType="xsd:int[4]" means an array of four xsd:int elements.

For mixed-type arrays, the SOAP-ENC:arrayType attribute always specifies xsd:ur-type.

For single-type arrays, the xsi:type attribute is optional for array item sub-elements, but its inclusion is recommended.

Null values 

A value can also be a null, which is specified by an XML element with an attribute, xsi:null, whose value is 1 as follows: .

Response example 

HTTP/1.1 200 OK
Connection: close
Content-Length: 499
Content-Type: text/xml; charset=utf-8
Date: Wed, 28 Mar 2001 05:05:04 GMT
Server: UserLand Frontier/7.0-WinNT



   
      
         South Dakota
         

      

   


Response format 

Unless there's an error, return 200 OK.

The Content-Type is text/xml.

Content-Length may be specified, if it is, it must be correct.

The body of the response is in XML, a single element.

The must contain a single element, which contains a single element which is the returned value of the procedure.

The single element contained in the has an arbitrary name which must match the name of the procedure that was called, with the word "Response" tacked on to the end of its name. Let's call this element the wrapper for the response. Even this is not the actual value of the response, which is contained in the single optional sub-element of the wrapper and must be a valid parameter.

The namespace of the wrapper element should match the namespace in the request.

If the wrapper has no sub-elements then the procedure did not return a value

Fault example 

HTTP/1.1 500 Server Error
Connection: close
Content-Length: 511
Content-Type: text/xml; charset=utf-8
Date: Wed, 28 Mar 2001 05:06:32 GMT
Server: UserLand Frontier/7.0-WinNT



   
      
         SOAP-ENV:Client
         Can't call getStateName because there are too many parameters.
         

      

   


Fault format 

If there's an error return 500 Server Error.

The may contain a which must contain two elements, and .

is intended for use by software to provide an algorithmic mechanism for identifying the fault. is intended to provide a human-readable explanation of the fault.

A may not contain both a and a wrapper (returned value).

When to fault 

If you can't process a request, for any reason, you must generate a fault.

Examples of fault conditions include:

1. An element or attribute in a request that is in an XML namespace which has not been declared.

2. A header in the request with a mustUnderstand="1" attribute that you don't understand.

3. A parameter is of a type that you don't support.

re: Gotta Love Misinformed Authors Spreading FUD

I have no problem with the opinion that Steve Gillmor shares both in his original eWeek article or in his subsequent comments in Drew's blog on the topic.  Specifically, I appreciate his willingness to stand behind his views.  To quote Mr. Gillmor,

“As for Robert Scoble, he is certainly capable of standing behind his own views when he wants to. I certainly stand behind mine.”

I take issue, however, with exactly what he is standing behind.  Is it the view that XAML stands for the Transaction Authority Markup Language?  Is it the opinion that Longhorn “veers-away” from some unstated (and, to the best of my knowledge, non-existant) standard that separates the layout and logic portions of an application?  Could it be that he stands behind the idea that developers create applications using the XSD 'language'?  Or that programmers will have to 'unlearn' XSD once Longhorn arrives. If so, it might be better to find a more stable platform on which to make your stand.

The idea that I'm going to have to wait until 2006 before getting access to some of the functionality that will be available in Longhorn is frustrating to me as a developer.  Sometimes ignorance can be bliss.  But I balance my frustration with the ability to better plan my current application architectures.  Do I expect to include XSD/XML functionality in my current code?  Certainly, assuming that I want my applications to be able to exchange data with not only other Microsoft products but the rest of the computing world.  Do I expect that arrival of Longhorn to change my use of XSD/XML?  Only in the integration that my apps have with the file system on which they are running.  The rest of the world speaks XML and I don't think that Longhorn will change that significantly.  Nor is the release of Longhorn likely to make my life as a developer more difficult, as Mr. Gillmor suggests.  If I take advantage of the information that is being made available by Redmond now, I should be ready for it, whenever it gets here.

Defining the Longhorn Battleground

Along with Sam Gentile and Drew Marsh (and others, I'm sure), I had a chance to read the eWeek opinion piece Can Microsoft's 'Longhorn' Pull it Off?.  Is this article an accurate representation of the facts that surround Longhorn?  Not even close.  Does the article display a lack of understanding by the author of the basic technologies used in Longhorn?  Yes.  But even if you completely eliminate the chaff, there is still an interesting element that can be gleaned from the approach taken by Mr. Gillmor.  What you are seeing is the initial 'shot across the bow' - the angle that Microsoft bashers will take as they attempt to capture mind- and developer-share.

The problem is that, without knowledge that can only be gained by playing with XAML and WinFS or reading articles and documentation on the topic, there is no way for the casual developer to tell the different between eFact and real facts.  I guess that means it's time to fire up the old Remington and produce some content more directed to the general masses.

Identifying New Records

While the use of AutoNumber and Identity fields in tables is greatly applauded by database designers the world over, they can be annoying for developers to use. The problem being that, when we add records to the database, there are many cases where we need to know the key of the record that has just been added. Fortunately, there are a number of techniques that are available to assist us in our task. In this article we look at the methods for both SQL Server and Access, with an eye to the pros and cons of each approach. As always, our goal is to give you enough information to make up your own mind

SQL Server

SQL Server 2000 offers three, count 'em, three options for retrieving the Id of a recently added record. The 'best' method depends on what you're doing and what else might be going on in your database. In all cases, the Id is a field marked as the primary key of a table and that has the Identity property set to True.

@@IDENTITY This is an intrinsic variable in SQL Server that contains the Id of the record that was last created on this connection, regardless of the table.
IDENT_CURRENT('table') This function returns the ID of the record that was last created in the specified table.
SCOPE_IDENTITY This variable contains the Id of the last record that was created within the current scope, regardless of the table.

So what are the ramifications of using each of these options? With @@IDENTITY, you get the most recent identity on the current connection. That means that if, as part of the INSERT statement, triggers get executed that in turn INSERT records into other tables, @@IDENTITY contains the last Id that was generated. Regardless of the table into which the record was inserted. Now if you created all of the stored procedures, triggers and INSERT statements yourself, that is fine. You can control when the @@IDENTITY value is being checked. But what happens if, sometime in the future, another developer, who is unaware that @@IDENTITY is being used, adds a trigger that creates an Audit log record. All of a sudden, @@IDENTITY returns a different id.  And not the one for the just added record.  As a result, your application breaks even though 'nothing' has changed. And that is the kind of bug that we all just love to track down.

The IDENT_CURRENT function is best used when you can control who has access to the database and when. By specifying the table as part of the function, you can eliminate the issues associated with @@IDENTITY. At least as far as the addition of records in triggers goes,   However IDENT_CURRENT works at the table level, not the connection level.  It returns the most recently created id, regardless of who created it. Unfortunately, in a busy environment, developers can't be sure between the execution of the INSERT command and the retrieval of IDENT_CURRENT, a different record isn't inserted by another user.

The SCOPE_IDENTITY instrinsic variable addresses some of the issues raised with the other two methods. Its value is the last Id created within the current scope. Athough technically the scope is the current connection, practically, the scope is more typically the currently executing stored procedure. Now you don't have to worry about the possibility of future enhancements 'breaking' your code, nor do you have to deal with other activity impacting the Id that is returned. If you perform more that one INSERT in a stored procedure, you do need to use the SCOPE_IDENTITY between each statement in order to retreive all of the created Id's. But again, that is within your sphere of control.

Unfortunately, if you are using SQL Server 7.0 or earlier, then the @@IDENTITY method is the only choice available to you. Both IDENT_CURRENT and SCOPE_IDENTITY were introduced with SQL Server 2000.

Microsoft Access

With Access, you are limited to basically a single technique. On a positive note, the same technique works all the way back to Access 97

First of all, I assume that we are inserting a record into a table where the primary key has an AutoNumber type. The addition of the record must be accomplished by using the AddNew and Update methods of the ADO Recordset object. Then, once the record has been added, store the absolute position of the new record and perform a Requery. Finally, set the cursor back to the bookmarked record and read the Id. Seem like a lot of work? And inflexible to boot? Such is the joy of Access

Set cn = CreateObject("ADODB.Connection")
Set rs = CreateObject("ADODB.Recordset")

cn.Open "DSN=MyDSN;"

rs.CursorLocation = adUseClient
rs.Open "SELECT CustNo, CustomerName, Contact", cn, adOpenStatic, adLockOptimistic

rs.AddNew

' CustNo is the AutoNumber field
rs.Fields("CustomerName").Value = "ObjectSharp"
rs.Fields("Contact").Value = "Bruce Johnson"
rs.Update

' The record has been inserted, but rs.Fields("CustNo").Value is zero
bookmark = rs.absolutePosition
rs.Requery
rs.absolutePosition = bookmark

' Voila
MsgBox rs.Fields("CustNo").Value

A Spoonful of SOA's Alphabet Soup

Last year, American corporations spent over $4 billion integrating heterogeneous applications (otherwise known as Enterprise Application Integration or EAI). Why? Because one of the best ways to squeeze more profits out of a company is to reduce the costs of doing business. Among the many other possibilities of boosting the bottom line(some of which have now been identified as illegal) is finding ways to streamline the flow of information through the corporation.

But what does that salute to business motherhood and apple pie have to do with web services? Because the pressure to lower costs is the reason that almost every developer should pay attention to the innovations and standards that are at play in the web services arena. Even with all of the hype surrounding the technology, it seems likely that web services will play a large role in EAI for the next few years at least. It is for this reason that I'm creating this series of articles. My goal is to walk developers through the process of designing and implementing a commercial-grade web service and supporting architecture.

To make sure that we're starting on the same page, the rest of this article will focus on some definitions. If you're already familiar with XML, SOAP, WSDL, UDDI and other various web service acronyms, feel free to skip to the next article. Otherwise, read on to learn about the alphabet soup that surrounds web services

What is a Web Service

To get our discussion started, let's define a web service. One of the more formal definitions that I've seen is "loosely coupled, reusable software components that semantically encapsulate descrete functionality and are distributed and programmatically accessible over standard Internet protocols". If you break apart each of the terms in this complicated rambling, you'll find the kernal of what most people expect a web service to be. That is a set of functions that can be accessed remotely using TCP/IP as the transportation medium. This broad definition covers almost all of the instances for which web services are suited, allowing us to explore the most common deployment options.

Ignoring the technical for a few seconds, let's consider the problem that web services are intended to address. That would be the need to access functionality provided by a remote server through the Internet. One of the most common examples is a stock quote function. If you're developing a corporate site, you don't want to focus on the details of retrieving quote information about your company. It's not worth it to dig into stock market streaming for such non-critical information. So instead you search the Internet until you find a company that specializes in providing stock market information. Fortunately for you they have implemented a technique that allows you to retrieve the needed data by making a call to their web site, leaving the formatting of the result up to you. And you have just experienced the power of a web service.

But the utility of a web service does not require that a third party be brought into the equation at all. Many companies are starting to deploy pieces of functionality as a web service within their own walls. This allows companies to experiment with web services without becoming dependent on an outside service. Not to mention avoiding connectivity, speed and security issues.

All web services start with a request being created. The source can be a browser or an application. Regardless, the request is formatted into an XML document and transmitted across the Internet to the web server. The server has a process that listens on a given port (usually, port 80, the HTTP port). When a request arrives, the XML document is parsed to determine what components needs to be instantiated and what methods are called. Finally, the result is bundled back up into an XML document and sent back to the calling application.

Now we have glossed over many of the issues that make developing web services challenging. This includes user authentication, transactions (grouping multiple SOAP requests into a single unit) and security. We will be dealing with these areas later in our series.

SOAP

Since SOAP is one of the focal points of the hype associated with Web Services, it makes sense to start there. And as much as SOAP might appear to a complicated entity, its basic purpose is quite simple. The Simple Object Access Protocol defines the XML format for messages that are sent between two processes. That's it, that's all. Nothing magical at work here.

But as with most technical subjects, the devil is in the details. Just sending an XML message using SOAP is by no means sufficient to have two applications communicate with one another. Each application needs to understand the context of the messages in order for SOAP to be effective. And that leads to the requirement to define data types (Section 5 of the SOAP specification) or RPC function call formats (Section 7 of the specification). Not to mention the techniques that utilize the HTTP port to send messages, something that, while not in the required portion of the SOAP specification, is supported by almost every SOAP implementation.


Figure 1 - SOAP Message Structure

The format of a SOAP message is quite straightforward. The message is delivered in an envelope. The envelope contains two parts. The Header contains information that describes how the recipient should process the message. The Body contains the payload of the message (in other words, the contents) and an optional section called SOAP Fault. This optional section contains error or status information.

UDDI

As we all know, building a better mousetrap is not sufficient to get the world to beat a path to your web site. As with any idea, you need to publicize the functionality that your web service makes available. Fortunately, there is a 'yellow pages' for web services and a set of functions that can be used to create, manipulate and search the entries. So when someone needs to find that stock market web service, they could browse this repository, searching for the functionality that they require.

The format used to add entries to these yellow pages is called the Universal Description, Discovery and Integration standard or UDDI. There are currently two main repositories for this information, one at http://uddi.microsoft.com/inquire and the other at http://www-3.ibm.com/services/uddi/inquiry.api. Ultimately, the goal of this evolving standard (and, when you get right down to it, web services in general) is to eliminate the human part of business transactions. The ultimate vision is that when a customer places an order with your company through the web service whose interface you have exposed through UDDI, the service will be able to search the directory to see that client has a web service that allows an invoice to be submitted electronically. Utopian vision? Currently, yes. Pipe dream? I don't think so. Give the IT community enough time, and this functionality will make its way into enough of the mainstream to be considered a requirement for 'serious' business.

WSDL

Next up in our alphabet soup is Web Services Description Language or WSDL. For those of you like to pronounce your acronyms, that would be "whiz-dull". As you might guess, the purpose of WSDL is to describe the web service that is being exposed. More specifically, it provides the details about the functionality that is implemented, what messages are used to request the service and the sequence in which the messages are exchanged. In other words, everything that an intrepid developer needs to know in order to use our service in their application.

Now defining the WSDL file for a web service is not a requirement for it to be used. However, not having one is like selling an ActiveX component without providing documentation, on-line or otherwise. And even though the format of a WSDL document is convoluted (it is an XML-based standard, after all), most development environments (including Visual Studio) provide tools that will automatically generate WSDL for you. So there is no reason (other than laziness) for not creating a WSDL document for any web service that you develop.

XML

The idea that I can describe XML adequately in a couple of paragraphs is ludicrous. The subject is complex enough that books are devoted so just the basic structure and usage. In my opinion, the reason for this difficulty it that XML is both flexible and extensible. So it can be used in almost any situation where data needs to be stored or transferred. Still, I'm brave (or stupid) enough to give a brief definition a try.

My two second description of XML has always been 'comma-delimited files on steroids'. My reason? The purpose of a CSV file is to store information so that another application can read/import it. This works fine, so long as the receiving application is aware of the structure of the incoming file. The best that CSV files can offer in this area is to place field names at the top of each column. So the possibility for problems of interpretation (or at least situations that require programmer-to-programmer discussions) exists. XML improves on this by applying a more rigorously enforced structure to the data. The order in which the data is physically laid out in the document is irrelevant, but each data field is named, so that it is easier to find. And when you combine XML data with DTDs or Schemas, the type and range of data are now subject to validation. Certainly a superior mechanism than CSV files.

For the most part, our use of XML will be to package up the data that needs to be send to the web service for processing and to receive back the results. As such, our XML documents will be relatively straightforward. But your complexity may vary, depending on the solution that you're trying to provide.

So much for the common protocols/acronyms.  These standards are enough to get you started in the web services world.  But anything more than a cursory look at the technology reveals many more TLAs or FLAs (three- and four-letter acronyms).  For completeness (actually, there is no hope for completenesss: the number of acronyms increases too quickly), here are some additional standards that are commonly used.

WS-Security

One of the main roadblocks to more universal acceptance of web services in production applications is the concerns raised about the security of messages as they fly across the Internet.  The WS-Security standard is a set of SOAP headers which are intended (and actually succeed, when used correctly) to define the authentication, encryption and signing that are used within a particular message.

While extensible enough to allow for custom security to be implemented, WS-Security natively supports a wide variety of security models, including Public Key Infrastructure (PKI) and Kerberos tokens.  And when combined with the Web Services Enhancements (WSE) toolkit, developers can easily integrate X.509 certificates, role-based security, WS-Addressing (see below for an explanation) and DIME attachmements (again, more information below). WS-Security is both mature and versatile enough to address all but the most stringent security requirements.

DSIG - Digital Signatures

Another aspect of the security associated with web service messages is known as non-repudiation.  In plain terms, this means that the recipient of the message is absolutely certain of who sent the message. As well, the recipient is absolutely certain that the content of the message has not been changed. If both of these conditions are met, then the message has the potential (not that I say potential here, not the reality...yet) of being a legally binding document. 

The typical approach accomplishing non-repudiation in a SOAP message is through digitial signing. Without getting into the dirty details, it involves hashing the message so that the change to a single byte will be detected by the processing.  For those who are interested in the details, check out here. But ignoring the specifics, the WSE toolkit makes it simple to digitally sign SOAP messages.

DIME - Direct Internet Message Encapsulation

Using jargon, DIME is a binary message format that is designed to encapsulate one or more payloads into a single message.  In real world terminology, the DIME protocol is used to include attached files to a SOAP message.

In order for DIME to be truly useful, any type of file must be attachable to a message.  This causes some temporary concerns.  Not every file is suitable for transmission in an XML format.  Consider for a moment the problems that would arise if the attached file contained characters that looked l ike an XML tag.  Also, binary formats, such as audio, video or images, can cause a problem when embedded in an XML document.  And since DIME is associated with the XML-based SOAP message, the underlying technology used to transmit the attachments must be compatible.  This means base64. 

Base64 is a encoding mechanism that allows arbitrary binary information to be translated into a format that is appropriate for both XML documents and e-mail.  The basic approach is to break the source data into 24-bit blocks.  Each block is translated into four characters with each character representing 6-bits of the 24-bit block.  The characters (there are 64 possible choices, hence the name base64) in the translation are present in all variants of both ASCII and EBCDIC, making it completely compatible with XML.

WS-Policy

One of the issues associated with services is determining which methods are exposed and which parameters. That problem is handled through WSDL.  Next up is determining the requirements and capabilities provided by a particular web service.  That information is made available through the Web Services Policy Framework, otherwise known as WS-Policy.

A WS-Policy document defines a set of assertions about the policy supported by a particular web service.  Possible assertions include encryption methods, the types of digital signatures which are recognized, and the presense or absense of particular WS-Security information. The assertions can be queried in real-time before constructing the service request.  The requester can then tailor the encryption, authentication and signing used on the message to match the needs of the web service.

WS-Routing

Workflow is an up and coming issue associated with SOAP messages.   At the moment, many web services are simple point-to-point solutions. In that situation, web services closely resemble Remote Procedure Calls (RPC). But the real future of SOA involve the processing of documents. And when a SOAP message contains a document, there is an implication that the document will need to pass through a number of steps before it has finished processing.  Say, for example, the document is a sales order. That document will need to be processed through the incoming sales service, the manufacturing service and the shipping service. And to keep track of the services in the process flow, a standard is required.

WS-Routing is a protocol that is used to route SOAP messages from service to service in a work flow.  It supports all of the common types of flow, including request/response, peer-to-peer and one-way messaging. At the moment, there are no tools to assist with implementing WS-Routing.  But if you can wait as as yet undermined amount of time, it would appear that this functionality is addressed in Indigo, Microsoft's upcoming web services middleware application.

WS-Referral

While WS-Routing handles the basic flow of messages, the interim steps are static.  Once defined, the message does what it has been programmed to do.  The WS-Referral protocol allows for the modification of the flow while the message is in transit.  As a result, this protocol and WS-Routing are usually found hand-in-hand.  Which also means that there are no currently available tools to help with implementing the protocol.  So you will either need to grit your teeth and bear it or wait for Indigo.

WS-Addressing

The WS-Addressing protocol is used to identify the endpoints of a web service.  The endpoint of a web service looks like the URL that is specified in the proxy.  It is the place to which the service request is made. Internally (at least to IIS) it defines the server and code that implements the methods exposed by the service. 

While in most circumstances, the endpoint of a web service is static (at least, it is most of the time), there are times when the actual endpoint needs to be more flexible. This is particularly true when the request passes through various routers and gateways.The WS-Addressing specification provides a means for redefining the endpoint of a web service as the message is in transit.

GXA - Global XML Web Service Architecture

While the term GXA still appears in a great deal of the older document from Microsoft regarding web services, the term has fallen out of general use.  It referred to a collection of extensible, orthogonal specifications related to providing require functionality for web service requests.  The standards included in GXA were WS-Security, WS-Routing, and WS-Referral.

SOA - Service Oriented Architecture

Service Oriented Architecture is the next great thing in web services.  Put simply, it is a style of designing applications based on the loosely coupled architecture that is forced by using services.  A service is a piece of software that exposes a set of methods that can be invoked (called an interface).  Because the only way to interact with the service is through the interface, the actual implementation and even the location of the service is irrelevant to the calling application.

But utimately, the benefits associated with using SOA, along with the details involved with designing and implementing a SOA-based are beyond the scope of this article.  The purpose of this article was to introduce you to some of the concepts behind web services and to explain some of the acronyms that inhabit this world. Future articles will dive more deeply into the concepts, benefits, issues and implementation details of SOA in the real world. I hope that you'll follow me on my journey down the rabbit hole.

Identifying SOAP Requests in Http Modules

This might seem a little on the simplistic side, but given the difficulty that I had finding this information, I'm posting it in the hope that it helps others.

I have created an HTTP Module whose job it is to raise EIF (Enterprise Instrumentation Framework) events on the receipt and response to SOAP requests.  The key here is the SOAP requests.  I don't want to pay any attention to the non-SOAP requests that come through.  So I needed to find a way to separate the two classes of messages using the information that was available.

The answer is to use the HTTP_SOAPACTION header that is included with SOAP requests (but not with the normal GETs and POSTs that a web site sees).  For example, the following code simply skips processing any non-SOAP requests.

private void Application_BeginRequest(Object source, EventArgs e)
  {
   HttpApplication application = (HttpApplication)source;

   // Don't do anything for non-SOAP requests
   if (application.Context.Request.ServerVariables["HTTP_SOAPACTION"] == null)
       return;

   }

Naturally, I have tied this procedure into the BeginRequest event for the HttpApplication object in the Init method for the HttpModule.

If someone has a better idea (or a reason why/when this won't work), I'm open to suggestions.  But for what I'm trying to accomplish it did the trick.

Stored procedures vs Dynamic SQL

If you're locked in a battle with a DBA over the 'benefits' of using stored procedures, check out this blog entry by Frans Bouma.  Cogent argument in favor of dynamic SQL combined with the passion of his beliefs.  Quite interesting reading, even if it goes too far in the other direction. My own personal belief is that there are instances where SPs are better than dynamic SQL in terms of overall performance (such as when a complex calculation is being performed on a large set of data across a relatively slow network connection).

EIF Presentation at CTT User Group Meeting

For all of you fans out there, I will be giving a presentation on the Enterprise Instrumentation Framework at 6:30 on November 26.  If you're interesting in reading the abstract or registering for it, visit the CTTDNUG web site.

Putting Attributes to Use

Whether you know it or now, attributes are already part of your development life. Creating a web service using ASP.NET?  Attributes are used to define the methods that are exposed.  Make a call to a non-COM DLL?  Attributes are used to define the entry point for the DLL, as well as the parameters that are passed.  In fact, closer examination shows that attributes play a large part in many cool (the programmer’s code word for ‘advanced’) features. For this reason, as well as the possibility of putting them to use in your own environment, a deeper understanding of attributes is worth gaining (a writer’s code phrase for ‘topic’).

Why are Attributes Useful

One of the starting points for a discussion about the why and when of attributes is with objects. Not a surprising starting point, given the prevalence of objects within the .NET Framework. But before believing that objects are the be all and end all of programming, consider the effort that a developer goes through while creating a typical object oriented application.  The classes that are part of the hierarchy lay strewn about the workspace.  When the situation calls for it, the developer grabs one of the classes and inserts it into their project.  They then surround the class with the code necessary to integrate it into the application and perform all of the common functions.  To a certain extent, this code is basically the mortar that holds the various classes together.

But think about the problems associated with this development process.  The common functions alreadya mentioned include logging the method calls, validating that the inbound parameters are correct and ensuring that the caller is authorized to make the call.  Every time that a class is created, all of the supporting code needs to be included in each method. 

This problem results from the fact that the common functions don’t really follow the same flow as the class hierarchy.  A class hierarchy assumes that common functions move up and down the inheritance path.  In other words, a Person class shares methods with an Employee class which in turn shares methods with an HourlyEmployee class. However, the functions described in the previous paragraph apply across all of the classes in a model, not just those in the same inheritance path.  Which means that the necessary code might very well have to be implemented more than once.

Is there a solution within the object-oriented world?  Yes and no.  The solution requires that a couple of things come together.  First, all of the classes in the hierarchy would need to be derived from the same ancestor.  Then the procedures required by the common functions would need to be baked into the ancestor.  Finally, each of the methods that want to implement the common functions (which could very well be all of them) would call the procedures in the ancestor.  Do you find this beginning to sound like the gluing together of pieces that we’re trying to avoid in the first place?  Me too.

Separation of Concerns

The solution, at least at a conceptual level, revolves around separating concerns. When used in this context, a ‘concern’ is a set of related functions.  Going back to the examples mentioned in the previous section, the functions that log a method call would make up one concern. The validation of inbound parameters would be a second. By keeping the implementation of the concerns separate, a number of different benefits ensue.

First of all, keeping the concerns separate increases the level of abstraction for the entire application.  In general, the functionality required by the concern will be implemented in a single class.  So while in development, the methods can be implemented and tested without regard to the other elements of the application.

The second rationale for separation is also a justification for using an object-oriented architecture.  The level of coupling when a concern is maintained in a separate class is quite low.  Only the public interface is used by the various classes, meaning that the details of the implementation are hidden.  It also means that the details can be changed at the whim…er…judgment of the developer. 

To be completely up front, a concern does not have to be a set of functions that can be applied to multiple classes.  The functions in a single class can also be categorized as a concern.  The difference, as has already been noted, is that the method logging and parameter validation apply to different classes throughout the hierarchy.  For this reason, they are known as ‘cross-cutting concerns’.

One more thing before we get into how concerns are actually put together in .NET.  In many articles on this topic, the term ‘aspect’ is used to describe what we are calling a concern.  For the most part, ‘aspect’ and ‘concern’ can be used interchangeably.  And from this, the more commonly heard term ‘aspect oriented programming’ is derived. 

Creating Concerns in .NET

Actually, it’s not the creation of a concern that is the difficult part.  After all, a concern is just a collection of related functions.  In .NET (as well as in pretty much every object-oriented language), that is just a fancy term for a class.  The difficult part is in integrating the functionality provided by the concern into existing classes without impacting the class itself.  To accomplish this feat requires the use of Application Domains

In the .NET Framework, an application domain is a logical boundary that the common language runtime (CLR) creates within a single process.  The key to the application domain concept is one of isolation.  The CLR can load different .NET applications into different domains in a single process.  Each of the applications runs not only independently of one another, but cannot directly impact one another.  They don’t directly share memory.  They each have their own process stack.  They even can load and use different versions of the same assembly.  For all intents and purposes, the applications are isolated. 


It might not immediately be apparent why an application domain is required.  Even though application domains are isolated from one another, programs running in one can still invoke methods running in another.  This is accomplished through a technique not that dissimilar to .NET Remoting.  And since we take advantage of this remoting to implement a cross-cutting concern, understanding it is worth a few more words.

 

Figure 1 – Cross-Domain Method Invocation Flow

The diagram shown in Figure 1 shows the flow of a call between the client class and the receiving object.  The assumption built into the diagram is that the client and the recipient are in different application domains.  First, transparent proxy is created.  This proxy contains an interface identical to the recipient, so that the caller is kept in the dark about the ultimate location of the callee.  The transparent proxy calls the real proxy, whose job it is to marshal the parameters of the method across the application domain.  As it turns out, before the receiving object sees the call there are zero or more message sink classes that get called.  These classes can perform a variety of functions, as well shall see shortly.  The last sink in the chain is the stack builder sink. This sink takes the parameters and places them onto the stack before invoking the method in the receiving object.  By doing this, the recipient remains as oblivious to the mechanism used to make the call as the initiator is.

A review of this flow reveals that chain of message sinks.  These are classes that get called every time a cross-domain call is made.  And they would seem to be the perfect place to implement our cross-concern functionality. So the question becomes one of how do this message sinks get created and how do we tell the CLR to use them.

The answer is two-fold.  First, in order to indicate that an object should be created in a separate application domain, the recipient class needs to be inherited from ContextBoundObject.  So much for the easy part.  The second step is to create an attribute which is used to decorate the recipient class. 

Context Attributes

The attribute class which decorates the recipient class is one that we create ourselves.  It allows us to provide the message sinks which will ultimately implement the cross-cutting concern. The class itself must be derived from the ContextAttribute class.  The ContextAttribute class requires that two methods be implemented.  First, the IsContextOk function returns a Boolean value indicating whether the current application domain is acceptable for this instance of the class.  Since the purpose of our attribute is to force the object to be placed into a separate domain, this function should always return a true. 

_

Public Class TraceLogger

   Inherits ContextAttribute

   Public Overloads Function IsContextOK(ByVal ctx As Context, _

ByVal ctorMsg As IConstructionCallMessage) As Boolean _

Implements IContextAttribute.IsContextOK

      Return False

   End Function

End Class

The second function that is required by the ContextAttribute base class is one called GetPropertiesForNewContext.  The purpose of this method is to allow information about the new context.  It is in this method that we indicate the message sinks that are to be included in the method invocation.

Public Overl"

      End Get

    End Property

    Public Overloads Sub Freeze(ByVal NewContext As Context) _

      Implements IContextProperty.Freeze

      Return

    End Sub

End Class

I know it seems like a long journey, but we’re almost at the end.  The TraceLoggerObjectSink class used in the GetObjectSink method is where the cross-cutting concern function actually gets implemented.  In order to be included in the chain of message sinks, this class needs to implement the IMessageSink interface.  As well, at least one constructor in the class needs to accept the next message sink in the chain as a parameter. 

Public Class TraceLoggerObjectSink

    Implements IMessageSink

    Private fNextSink As IMessageSink

    Public Sub New(ByVal ims As IMessageSink)

        fNextSink = ims

    End Sub

End Class


The other two methods required by IMessageSink are SyncProcessMessage and AsyncProcessMessage.  These methods are called when the recipient method is invoked synchronously and asynchronously respective.  For simplicity, we’ll just focus on SyncProcessMessage. The diagram shown in Figure 2 illustrates how SyncProcessMessage comes into play.

Figure 2 – Process Flow within SyncProcessMessage

As part of the Message Sink chain, the SyncProcessMessage method of this class is called.  Within the method, any preprocessing of the request is performed.  The method then invokes the SyncProcessMessage method for the message sink provided in the constructor.  In its turn, each of the message sinks perform the same logic until the recipient object is called.  On the way back, the flow is reversed.  From the perspective of just this one sink, however, the flow is return through the same procedure.  So after the call to SyncProcessMessage, any post processing of the response is done and control allowed to pass back up the chain.

Public Overloads Function SyncProcessMessage(ByVal Message As IMessage) _

   As IMessage Implements IMessageSink.SyncProcessMessage

   Dim RetValue As IMessage

   PreProcess(Message)

   RetValue = _NextSink.SyncProcessMessage(Message)

   PostProcess(Message, RetValue)

   Return RetValue

End Function

Within the PreProcess and PostProcess methods, any manner of processing can take place.  The actual name of the method, the parameter values and the return values are all available to be view and modified.  And it is in these methods that the cross-cutting concern functions, such as tracing, validation and authentication can be performed.

Summary

Is the use of a ContextBoundObject and cross-domain calls the best way to implement a cross-cutting concern.  Probably not.  The restriction that the class must be inherited from ContextBoundObject means that this technique is not suitable for every situation.  For example, if you wanted to implement method tracing for an object that was being used in COM+, you have a problem.  The COM+ class needs to inherit from System.EnterpriseServices and there is no way to have a single class inherit from multiple bases.  However for relatively straightforward situations, this technique is not only effective, but also useful.  After all, being able to add functionality without altering the underlying class is always a good way of improving programmer productivity and lowering the rate of defects.  Quite a noble goal in its own right.

Using a DLL from Web Service

I had the distinct pleasure of trying to incorporate a non-COM compliant DLL into a web service yesterday.  Along with the issues associated with marshalling parameters (and which I'll mention in a separate blog entry), I also had to get the web service to find the DLL that needed to be loaded.  I would have thought that simply placing it into the bin directory under the virtual root would be sufficient.  Apparently not. Even after doing so, I still got a could not load DLL message

The correct answer is that the DLL needs to be someplace in the directories listed in PATH.  And by default, the bin directory is not in this list. 

For those who are unfamiliar with the process, the PATH is build from a combination of sources.  First, the System Path as defined in the System Environment Variables in the System administration applet is included.  Then the User path as defined in the same place is appended.  Finally the directories added to the PATH through Autoexec.bat are included. 

So if you plan on using a DLL in a web service, make sure that either the DLL is installed someplace along your PATH or your PATH is modified to include the web service's bin directory.