I love Lego.
To be fair, the number of people who don’t fall into that category is probably fairly small. There is nothing like the joy of taking the slightly differently shaped blocks and creating something bigger and better. And I’m not a big fan of all of the custom kits either.If a piece only has one purpose (like as the nose for an X-wing fighter), then it’s not for me.
I also love Star Trek. Well, not love, but greatly appreciate and enjoy the various forms over the years. And I have referenced Star Trek in various presentations, not to establish my geek cred (of which I have very little), but because of how software works ‘in the future’.
And yes, Lego and Star Trek are related in this way.
The key to Lego block is the simple and consistent interface. Doesn’t matter what the shape of the block is, the fact that every block has the same interface allows them to be connected. And it is through the various connections that a much bigger whole can be created.
Star Trek takes the Lego idea and applies it to software. Every wonder how Geordi and Data were so quickly able to create new and complex software? Because all of the different components had the same interface. Or at least similar enough interfaces so that the components could communicate with one another. And connected software allows you to create a much bigger whole.
Now let’s move back to the here and now. What’s missing from our current software environment that would prevent Geordi from creating the application that saves the Enterprise? Two things, really. The lack of a standard set of interfaces and the inability of most software functionality to be ‘connected’. And this is the next phase in software development.
If you’re a company that provides services to others, then you need to think about enabling access to your service from the cloud. Want to allow people to easily buy your services or products? Give them an interface on the Web that allows them to do so. Not a Web page, but an API. Have some information that others might find useful? Give them an interface to access it. Create the Lego blocks that I was talking about earlier. Find standard interfaces for the type of data/service you offer and expose them on top of your services. In other works, provide Lego blocks for others.
One of the benefits of doing so is that you let others build out functionality based on your services. If your service or data is compelling enough, others will build your front-end for you. You have already seen this happen with a number of the social sites that are out there. People combine information from Foursquare, Twitter, Facebook, Google+, etc. to create interesting apps for others to use. The engagement level of people with apps that run on their phones are high and likely to move higher. Finding ways to integrate your service/data with that ecosystem can only be beneficial.
So what’s the downside? Well, you have to implement and/or design the interface. Technical yes, but not beyond the scope of what most companies can do. And you need to provide the infrastructure for surfacing your API. This is where virtualization comes into play. I’m a fan of Azure and the new functionality it offers, but speaking generically, virtualize where it makes the most sense. If you’re a Microsoft shop, I believe you’ll find the biggest bang for your efforts with Azure.
But the technology is not the key here…it’s the concept. Look at the products you offer to your clients. Find ways to expose those products to the Internet. Be creative. The payoff for your efforts have the potential to be significant. But more importantly, you take the first step towards what will be the development and integration paradigm for the next decade…Lego.
One of my current tasks for a client has been to facilitate the integration between a Java client and a WCF service. The mechanism for doing this is JSON and I have been quite grateful that WCF makes it relatively easy to implement this type of communication. However, there is one area that has been causing some grief for me (and my Java colleagues) that I finally create a workaround for yesterday.
The source of my problem starts with the contract used by the WCF service. A number of the methods include parameters that are abstract data types. Consider the following class declaration.
[KnownType(ButtonCommand)]
[KnownType(MouseCommand)]
[DataContract]
public abstract class AbstractCommand
The AbstractCommand class is just a base class for a couple of concrete classes (ButtonCommand and MouseCommand). The WCF method is declared as accepting an AbstractCommand type, but the value passed into the method will be either a ButtonCommand or a MouseCommand.
[OperationContract]
[WebInvoke(BodyStyle=WebMessageBodyStyle.Wrapped,
RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json)]
StatusCode ExecuteCommand(AbstractCommand command);
Naturally, when the client invokes this method, the expectation is that a concrete class instance will be passed along. However, if all you do is pass in the JSON for one of the concrete classes in, you get an exception saying that an abstract class cannot be instantiate. The reason for the error can be found in how WCF dispatches message. When the request arrives, WCF examines the message to determine which method is to be invoked. Once identified, the parameters for the method (found in the JSON) are created. This creation involves instantiating an object using the default parameterless constructor for the type and then assigning property values as found in the JSON. However, the AbstractCommand type cannot be instantiated (it is abstract, after all). And there is no immediately apparent mechanism to determine which concrete type should be used.
To address this final problem, Microsoft introduced the idea of a type hint. The JSON for a concrete type passed into the ExecuteCommand method would look something like the following:
{"command":{"__type":"ButtonCommand:#Command.Library", "prop1":"value", "prop2":"value"}}
The name/value pair of “__type” is used by WCF to determine the type that should be instantiated before performing the property assignment. This is conceptually similar to how types are provided through SOAP. This mechanism does have the hard requirement that the “__type” value be the first pair in the JSON list.
Enter Java into the picture. In order to communicate with my WCF service, the client using the Java JSONObject class. This class takes a collection of name/value pairs and converts it into a JSON-formatted array. This array is then sent to my method, where it goes through the previously described process. However the JSONObject seems to take the portion of the JSON specification where it says the name/value pairs are ‘unordered’ to heart. There is (apparently) no way to force a particular name/value pair to be emitted as the first pair. In other words, the JSONObject cannot reliably produce JSON for a concrete type that is being passed into a parameter of an abstract type.
Crap.
Well, it was actually stronger language than that once I figured this out.
The solution was not nearly as complicated as identifying the problem. I created a WCF endpoint behavior extension. In the AfterReceiveRequest event handler, I look at the body of the request (which has already been converted into XML, as it turns out..a surprise, albeit an easier choice format for the next step). If the “__type” property is the first pair in the JSON array, it is converted to a “__type” attribute in the XML element. Otherwise, it appears as a regular child node. So the transformation consists of finding the XML elements that have a name of “__type” and moving the value up into an attribute of the parent node. The modified message is then injected back into the WCF pipeline. And now the request is processed as desired.
As you might have guessed from the name of my blog, I have a long-term love of distributed applications. My infatuation with the technology actually goes back to earlier forms of COM+ and Web services. I believe that a well designed distributed application can be used to address the scalability, reliability and even identity issues that spring up in most large scale business applications. The biggest problem with designing distributed application is knowing which of the many available technologies is the best choice for the situation.
This is the reason that I was looking forward to the release of the Windows Communication Foundation (WCF). If nothing else, WCF provided a common interface to the multiple technologies used to build distributed applications. Well, WCF has been released for a few months now (and was actually in a 'go live' state for a lot longer), which means that interest in how it can be used in real-life is growing. To help people learn and use this very cool technology, ObjectSharp is introducing a new course on the best practices surrounding WCF. The course is being developed by Peter Madziak, with more information about the content and thinking behind the course found here. If you have any interest in distributed applications in general and WCF at all, you owe it to yourself to check out the course description.
A comment made by Udi Dehan on my most recent article (the Benefits of a Service-Oriented Architecture) got me thinking. In the comment and on his blog he has been contemplating the differences between OO and SOA and, more importantly, how programmers can adjust.
First of all, I believe that the idea that the debate is an either/or proposition is wrong. Not everything should be represented as a service. Nor should everything be represented as an object. There are times when the interface represented by a service layer is the better way to go. However, even for the more devout SOA believers, that idea that the implementation of a service won't contain objects is ludicrous.
That having been said, I believe that designing using SOA is more of a mindset than a programming style. And the trick to becoming proficient is to be aware of the differences between OO and SOA, at the conceptual level. At first glance, it might seem that the main difference devolves into an argument about Remote Procedure Calls (RPC) versus document-based method calls. However, I believe that the distinguishing characteristic is actually more likely to be statefulness than the calling technique.
In a traditional object-oriented application, the typical sequence of steps to invoke a method is as follows:
- Instantiate an object
- Optionally set some property values
- Call the method, including any necessary parameters
- Wait for a response
Inherently, these steps are stateful. An instance of the object must exist before the method can be called. Properties in the object can be set, either through the constructor or by direct assignment. The fact that these properties are available for use within a method means that the object maintains state.
But what about static methods. It is possible to define a method on a class that doesn't require an instantiated object to be used. The methods don't have access to the instance-level properties and methods. How could they? No instance has been created.
However, even static methods can be stateful. Classes can also define static properties. Static properties can be assigned values, values that can be used by static methods. Therefore the mere inclusion of static methods is not a sufficient condition for statelessness.
“But wait a second”, you say. “What if the static method didn't use any static properties? What if all of the external values required by the method are included in the parameter list? Surely that's not stateful!”
Of course, you'd be right. If all of the required information is available to a method through the parameter list at the time of the call, it is no longer a stateful operation. But here's the real kicker. Convert the parameter list into an XML document. Pass the document into the static method and you have just created the skeleton of a service. A message (the XML document) is passed into a service provider (the static method) which intrinsically knows how to process it.
And therein lies the difference between OO and SOA. It's not that the two aren't related. It's that SOA is much more demanding about how programmers interact with it. Kinda nice when knowledge sneaks up like that, isn't it?
Even if you don't have your ear to the ground, the far-off rumble is a reminder that services are coming. First, there was the web services wave. Recently, Microsoft created another groundswell with the introduction of Indigo at the last PDC. Although the future may still be a year or more off, there is no denying it: the programming world is about to enter the era of services.
Currently, the big architectural push is towards a service-oriented architecture (SOA). While the term sounds impressive, it is important to understand both what an SOA is and how it can benefit a development team. There are a number of different links available that describe the components and concepts behind SOA, including at ondotnet.com, xml.com, and microsoft.com. Given this wealth of information, it would be redundent to cover it here (but perhaps in another article ;). Instead, I'd like to focus on some of the benefits that accrue to those who choose to follow the SOA path.
Even with that slough-off of responsibility for the concepts of SOA, it is still necessary that we have a common reference as the jumping off point for a discussion of benefits. So briefly, a service-oriented architecture is a style of application design that includes a layer of services. A service, from this limited perspective, is just a block of functionality that can be requested by other parts of the system when it is needed. These services have a published as set of methods, known as an interface. Usually, each interface will be associated with a particular business or technical domain. So there could be an interface used for authentication and another interface used to create and modify a sales order. The details of the provided functionality are not important for this article so much as the relative location of the code that implements the functions.
A word of warning to developers who are new to the services architecture (or an object-oriented environment as well). SOA is as much about a mind set as it is about coding. In order to reap any of the benefits of SOA, the application must be designed with services in mind. It is difficult, if not impossible, to convert a procedural application to services and still achieve anything remotely close to what can be gained from baking services in from the beginning. That having been said, there are usually elements of procedural applications which can be 'servicized'. This is the equivalent of refactoring the applications looking for common functionality.
And so, on to the benefits. In each case, we are talking about development teams that embrace SOA to the point that most of the development effort is focused on the creation and utilization of services.
Loosely Coupled Applications
Strictly speaking, coupling is the level of impact that two modules have on one another. From a developer's perspective, it is the odds that a change in one module will require a change in another. We have all seen tightly coupled systems. These are applications were developers are afraid to touch more procedures than necessary because they might cause the result of the application to break. The goal, whether we know it or not, is to create a loosely coupled environment. And services go a long way towards making such an environment a reality.
Consider for a moment what I consider to be the epitome of loosely coupled systems: a home entertainment system (Figure 1).
Figure 1 - A Loosely Coupled Home Entertainment System
Here are a whole bunch of different components (the exact number depends on how into electronic gadgets you are), frequently created by different manufacturers. The while the user interface can get complicated, between the devices the connections are quite simple. Just some wires that get plugged into standard sockets. Need to replace one of the element? No problem. Remove the wires connecting Component A to the system. Take Component A away. Insert Component B. Put the wires into the appropriate sockets. Turn things on and go. Talk about loosely coupled.
In software, a loosely coupled environment allows for changes to be made in how services are implemented without impacting other parts of the application. The only interaction between the application and the services is through the published interface. Which means that, so long as the interface doesn't change, how the functions are implemented are irrelevant to the calling application. Strict adherance to such an approach, an approach which is strongly enforced by SOA, can significantly reduce development by eliminate side-effect bugs caused by enhancements or bug fixes.
Location Transparency
Another benefit accrued through the use of interfaces and the loosely coupled structure imposed by SOA is location transparency. By the time we're done, it will seem like many of the benefits achieved with SOA will come from this same place (interfaces, that is). In this instance, location transparency means that the consumer of the service doesn't care where the implementation of the service resides. Could be on the same server. Could be on a different server in the same subnet. Could be someplace across the Internet. From the caller's perspective, the location of the service has no impact on how a request for the service is made.
It might not be immediately obvious why location transparency is a good thing. And, in the most straightforward of environments, it doesn't really make a difference. However as the infrastructure supporting a set of web service becomes more complicated, this transparency increases in value. Say the implementing server needs to be moved. Since the client doesn't care where the service is, the change can be made with no change required on the client. In fact, the client can even dynamically locate the implementation of the service that offers the best response time at the moment. Again, not something that is required for every situation, but when it's needed, location transparency is certainly a nice tool to have in your arsenal.
Code Reuse
Functional reuse has been one of the holy grails for developers since the first lines of code were written. Traditionally, however, it has been hard to achieve for a number of reasons. It is difficult for developers to find a list of the already implemented functions. Once found, there is no document that describes the expected list of parameters. If the function has been developed on a different platform or even in a different language, the need to translate (known technically as marshaling) the inbound and outbound values is daunting. All of these factors combine to make reuse more of a pipe dream than a reality.
The full-fledged implementation of a SOA can change this. The list of services can be discovered dynamically (using UDDI). The list of exposed methods, along with the required parameters and their types, are available through a WSDL document. The fact that even complex data structures can be recombined into a set of basic data types held together in an XML document makes the platform and language barrier obsolete. When designed and implemented properly, SOA provides the infrastructure that makes reuse possible. Not only possible, but because it is easy to consume services from either .NET or even the more traditional languages, developers are much more likely to use it. And ultimately, the lack of developer acceptance has always been the biggest barrier of all.
Focused Developer Roles
Almost be definition, a service-oriented architecture forces applications to have multiple layers. Each layer serves a particular purpose within the overall scope of the architecture. The basic layout of such an architecture can be seen in Figure 2.
Figure 2 - Some Layers in a SOA
So when designing an application, the various components get place into one of the levels at design time. Once coding starts, the someone is actually going to create the component. The skills necessary to develop each of the components are going to differ and the type of skill required will depend on the layer in which the component is placed. The data access layer requires knowledge of data formats, SQL, ADO.NET. The authentication layer could utilize LDAP, WS-Security, SHA or other similar technologies. I'm sure you get the idea enough to put it in play in your own environment
The benefit of this separation are found in how the development staff can be assigned to the various tasks. Developers will be able to specialize in the the technologies for a particular layer. As time goes by, they will be experts in the complex set of skills used at each level. Ultimately, by focusing the developers on a particular layer, it will even become possible to include less skilled programmers in a paired or a mentoring environment. This doesn't eliminate the need for an overarching technical lead, but it certainly decreases the reliance of a company on experienced generalists.
Greater Testability
It has been a few paragraphs since they were mentioned, but the fact that services are only access through their interfaces is the reason for this benefit. The fact that services have published interfaces greatly increases their testability.
One of the nicest tools in recent years for assisting with the testing process is NUnit. A .NET version of the JUnit, NUnit allows for the creation of a test suite. The test suite consists of a number of procedures, each of which is designed to test one element of an application or service. NUnit allows the suite to be run whenever required and keeps track of which procedures succeed and which fail. My experience with NUnit is that it makes the process of developing a test suite incredibly simple. Simple enough that, even with my normal reticence for testing, I no longer had an excuse not to create and use unit tests.
The fact that services publish an interface makes them perfect for NUnit. The test suite for a particular service knows exactly what methods are available to call. It is usually quite easy to determine what the return values should be for a particular set of parameters. This all leads to an ideal environment for the 'black box' testing that NUnit was developed for. And whenever a piece of software can be easily tested in such an environment, it inevitably results in a more stable and less bug prone application.
Parallel Development
There is another side benefit that arise both from the increased testability (and tools such as UNint) and the division of components into layers. Since the only interaction that exists between services is through the published interfaces, the various services can be developed in parallel. At a high level, so long as a service passes the NUnit test for its interface, there is no reason why it would break when the various services are put together. As such, once the interfaces have been designed, development on each of the services can proceed independently. Naturally, there will need to be some time built into the end of the project plan to allow for integration testing (to ensure that the services can communicate properly and that there was no miscommunication during development). But especially for agressive project timelines, parallel development can be quite a boon.
Better Scalability
This particular benefit refers to the ability for a service to improve its response time without impacting any of the calling applications. It is an offshoot of the location transparency provided by SOA. Since the calling application doesn't need to know where the service is implemented, it can easily be moved to a 'beefier' box as the demand requires. And if the service is offered through one of the standard communication protocols (such as HTTP), it is possible to spread the implementation of the service across a number of servers. This is what happens when the services is place on a web farm, for example. The reasons and techniques for choosing how to scale a service are beyond the scope of this article (although they will be covered in a future one). It is sufficient, at this point, to be aware that they are quite feasible.
Higher Availability
Location transparency also provides for greater levels of availability. Read the section on scalability and web farms for the fundamental reason. But again, the requirement to utilize only published interfaces brings about another significant benefit for SOA.
Building Multi-Service Applications
I've always described this benefit as the “Star Trek Effect“. The idea can be seen in many Star Trek episodes, both new and old. In order to solve an impossible problem, the existing weapon systems, scanning systems and computer components need to be put together in a never-before seen configuration.
“Cap'n. If we changed the frequency of the flux capacitor so that it resonated with the dilithium crystals of that alien vessel, our phasers might just get through their shields. It's never been done before, but it might work. But it'll take me 10 seconds to set it up“
Try reading that with a Scotish accent and you'll get my drift. Here are people working with services to the level that no programming is required to be able to put them together into different configurations. And it is the goal that most service-oriented architects have in mind, either consciously or subconsciously, when they design systems. And, although we are not yet at the Star Trek level of integration and standardization, SOA provides a good start.
Summary
And that's all SOA is, after all. A good start. But it is a start that is necessary to gain the benefits I've just run through. Sure it takes a little longer to design a service-based application. Not to mention creating all of the infrastructure that needs to surround it, assuming that it will be production quailty. However in the long run, the benefits start to play out.
Now I'm not naive enough to believe that we haven't heard this song before. It is basically the same song that has been playing since object-oriented development was first conceived. We heard it with COM. Then it was DCOM. Now it's web services and SOA. Why should be believe that this time will be different from the others?
I can't offer any reason in particular in answer to that very astute question. Honestly, SOA feels different than COM and OO. Perhaps that is because we've already seen the problems caused by COM, CORBA, etc and they are addressed by web services. Perhaps it is because there is now a common language that can be used across platforms (XML). Perhaps it is the level of collaboration between the various vendors regarding service-based standards. But to me at least, all of these items make me much more optimistic that this time we are heading down the right path.
In order to gain a complete understanding of services-oriented architecture (SOA), one of the basic pieces of knowledge is the structure of a SOAP messages. At a high level, SOAP is a lightweight protocol for exchange of information in a decentralized, distributed environment. But that description gives short shrift to the true utility of SOAP, that being the transmission of a request for service between two heterogeneous and networked applications. In this article, we look at the structure of a SOAP message in great detail, as well as put the SOAP message into the context of the HTTP request. This second part has to potential to be a fleeting reference, as there is nothing in the SOAP specification that requires that HTTP be the conduit. However the ubiquity of HTTP has made it the de facto standard for transmission and it will probably be a few years before it is displaced.
For the purposes of this document, SOAP is a Remote Procedure Calling protocol that works over HTTP.
The body of the request is in XML. A procedure executes on the server and the value it returns is also formatted in XML.
Procedure parameters and returned values can be scalars, numbers, strings, dates, etc.; and can also be complex record and list structures.
Request example 
Here's an example of a SOAP request:
POST /examples HTTP/1.1
User-Agent: Radio UserLand/7.0 (WinNT)
Host: localhost:81
Content-Type: text/xml; charset=utf-8
Content-length: 474
SOAPAction: "/examples"
41
Header requirements
The format of the URI in the first line of the header is not specified. For example, it could be empty, a single slash, if the server is only handling SOAP calls. However, if the server is handling a mix of incoming HTTP requests, we allow the URI to help route the request to the code that handles SOAP requests. (In the example, the URI is /examples, telling the server to route the request to the "examples" responder.)
A User-Agent and Host must be specified.
The Content-Type is text/xml. The charset may be specified, if not, the default is US-ASCII. Other acceptable charsets are UTF-8 and UTF-16. UTF-8 is recommended for maximum interop. See the note on UTF character encodings, below.
The Content-Length may be specified, if it is, it must be correct.
As with the URI in the first line, SOAPAction is used to route the request to a handler on the server that will respond. It's entirely up to the application to determine how this header element is used. In many implementations the URI and the SOAPAction header will have the same value.
Payload format
The payload is in XML, a single element.
All the attributes of the are required as shown in the example above.
The must contain a single element, which contains a single element which is the procedure call. The name of the procedure is the name of this element. Note that the procedure name must be a valid XML element name.
The elements contained within the procedure call are the parameters to the procedure. The names of the parameters is significant, the order of the parameters are not. Parameter type is indicated by the xsi:type attribute.
For example, the procedure name could be the name of a file containing a script that executes on an incoming request. It could be the name of a cell in a database table. Or it could be a path to a file contained within a hierarchy of folders and files.
A procedure call may take no parameters, if so, the procedure element must not contain sub-elements.
Scalar values
The following scalar value types are supported by this subset of SOAP 1.1.
| Type value |
Type |
Example |
| xsd:int |
32-bit signed integer |
-12 |
| xsd:boolean |
a boolean value, 1 or 0 |
1 |
| xsd:string |
string of characters |
hello world |
| xsd:float or xsd:double |
signed floating point number |
-12.214 |
| xsd:timeInstant |
date/time |
2001-03-27T00:00:01-08:00 |
| SOAP-ENC:base64 |
base64-encoded binary |
eW91IGNhbid0IHJlYWQgdGhpcyE= |
Structs A value can also be a struct, which is specified by an XML element that contains sub-elements. structs can be nested, and may contain any other type, including an array, described below.
Here's an example of a two-element struct:
18 139 The names of struct elements are significant, the order of the elements is not.
Arrays A value can also be an array, which is specified by an XML element with a SOAP-ENC:arrayType attribute whose value begins with ur-type[, followed by the number of array elements, followed by ].
Here's an example of a four-element array:
- 12
- Egypt
- 0
- -31
The order of array elements is significant, the names of the elements are not.
You can mix types as the example above illustrates.
If the array elements are of a single type, the value of the array element's SOAP-ENC:arrayType specifies the type of the array's sub-elements, for example, SOAP-ENC:arrayType="xsd:int[4]" means an array of four xsd:int elements.
For mixed-type arrays, the SOAP-ENC:arrayType attribute always specifies xsd:ur-type.
For single-type arrays, the xsi:type attribute is optional for array item sub-elements, but its inclusion is recommended.
Null values A value can also be a null, which is specified by an XML element with an attribute, xsi:null, whose value is 1 as follows:
.
Response example HTTP/1.1 200 OK
Connection: close
Content-Length: 499
Content-Type: text/xml; charset=utf-8
Date: Wed, 28 Mar 2001 05:05:04 GMT
Server: UserLand Frontier/7.0-WinNT
South Dakota
Response format Unless there's an error, return 200 OK.
The Content-Type is text/xml.
Content-Length may be specified, if it is, it must be correct.
The body of the response is in XML, a single
element.
The must contain a single element, which contains a single element which is the returned value of the procedure.
The single element contained in the has an arbitrary name which must match the name of the procedure that was called, with the word "Response" tacked on to the end of its name. Let's call this element the wrapper for the response. Even this is not the actual value of the response, which is contained in the single optional sub-element of the wrapper and must be a valid parameter.
The namespace of the wrapper element should match the namespace in the request.
If the wrapper has no sub-elements then the procedure did not return a value
Fault example
HTTP/1.1 500 Server Error
Connection: close
Content-Length: 511
Content-Type: text/xml; charset=utf-8
Date: Wed, 28 Mar 2001 05:06:32 GMT
Server: UserLand Frontier/7.0-WinNT
SOAP-ENV:Client
Can't call getStateName because there are too many parameters.
Fault format
If there's an error return 500 Server Error.
The may contain a which must contain two elements, and .
is intended for use by software to provide an algorithmic mechanism for identifying the fault. is intended to provide a human-readable explanation of the fault.
A may not contain both a and a wrapper (returned value).
When to fault
If you can't process a request, for any reason, you must generate a fault.
Examples of fault conditions include:
1. An element or attribute in a request that is in an XML namespace which has not been declared.
2. A header in the request with a mustUnderstand="1" attribute that you don't understand.
3. A parameter is of a type that you don't support.
Last year, American corporations spent over $4 billion integrating heterogeneous applications (otherwise known as Enterprise Application Integration or EAI). Why? Because one of the best ways to squeeze more profits out of a company is to reduce the costs of doing business. Among the many other possibilities of boosting the bottom line(some of which have now been identified as illegal) is finding ways to streamline the flow of information through the corporation.
But what does that salute to business motherhood and apple pie have to do with web services? Because the pressure to lower costs is the reason that almost every developer should pay attention to the innovations and standards that are at play in the web services arena. Even with all of the hype surrounding the technology, it seems likely that web services will play a large role in EAI for the next few years at least. It is for this reason that I'm creating this series of articles. My goal is to walk developers through the process of designing and implementing a commercial-grade web service and supporting architecture.
To make sure that we're starting on the same page, the rest of this article will focus on some definitions. If you're already familiar with XML, SOAP, WSDL, UDDI and other various web service acronyms, feel free to skip to the next article. Otherwise, read on to learn about the alphabet soup that surrounds web services
What is a Web Service
To get our discussion started, let's define a web service. One of the more formal definitions that I've seen is "loosely coupled, reusable software components that semantically encapsulate descrete functionality and are distributed and programmatically accessible over standard Internet protocols". If you break apart each of the terms in this complicated rambling, you'll find the kernal of what most people expect a web service to be. That is a set of functions that can be accessed remotely using TCP/IP as the transportation medium. This broad definition covers almost all of the instances for which web services are suited, allowing us to explore the most common deployment options.
Ignoring the technical for a few seconds, let's consider the problem that web services are intended to address. That would be the need to access functionality provided by a remote server through the Internet. One of the most common examples is a stock quote function. If you're developing a corporate site, you don't want to focus on the details of retrieving quote information about your company. It's not worth it to dig into stock market streaming for such non-critical information. So instead you search the Internet until you find a company that specializes in providing stock market information. Fortunately for you they have implemented a technique that allows you to retrieve the needed data by making a call to their web site, leaving the formatting of the result up to you. And you have just experienced the power of a web service.
But the utility of a web service does not require that a third party be brought into the equation at all. Many companies are starting to deploy pieces of functionality as a web service within their own walls. This allows companies to experiment with web services without becoming dependent on an outside service. Not to mention avoiding connectivity, speed and security issues.
All web services start with a request being created. The source can be a browser or an application. Regardless, the request is formatted into an XML document and transmitted across the Internet to the web server. The server has a process that listens on a given port (usually, port 80, the HTTP port). When a request arrives, the XML document is parsed to determine what components needs to be instantiated and what methods are called. Finally, the result is bundled back up into an XML document and sent back to the calling application.
Now we have glossed over many of the issues that make developing web services challenging. This includes user authentication, transactions (grouping multiple SOAP requests into a single unit) and security. We will be dealing with these areas later in our series.
SOAP
Since SOAP is one of the focal points of the hype associated with Web Services, it makes sense to start there. And as much as SOAP might appear to a complicated entity, its basic purpose is quite simple. The Simple Object Access Protocol defines the XML format for messages that are sent between two processes. That's it, that's all. Nothing magical at work here.
But as with most technical subjects, the devil is in the details. Just sending an XML message using SOAP is by no means sufficient to have two applications communicate with one another. Each application needs to understand the context of the messages in order for SOAP to be effective. And that leads to the requirement to define data types (Section 5 of the SOAP specification) or RPC function call formats (Section 7 of the specification). Not to mention the techniques that utilize the HTTP port to send messages, something that, while not in the required portion of the SOAP specification, is supported by almost every SOAP implementation.
Figure 1 - SOAP Message Structure
The format of a SOAP message is quite straightforward. The message is delivered in an envelope. The envelope contains two parts. The Header contains information that describes how the recipient should process the message. The Body contains the payload of the message (in other words, the contents) and an optional section called SOAP Fault. This optional section contains error or status information.
UDDI
As we all know, building a better mousetrap is not sufficient to get the world to beat a path to your web site. As with any idea, you need to publicize the functionality that your web service makes available. Fortunately, there is a 'yellow pages' for web services and a set of functions that can be used to create, manipulate and search the entries. So when someone needs to find that stock market web service, they could browse this repository, searching for the functionality that they require.
The format used to add entries to these yellow pages is called the Universal Description, Discovery and Integration standard or UDDI. There are currently two main repositories for this information, one at http://uddi.microsoft.com/inquire and the other at http://www-3.ibm.com/services/uddi/inquiry.api. Ultimately, the goal of this evolving standard (and, when you get right down to it, web services in general) is to eliminate the human part of business transactions. The ultimate vision is that when a customer places an order with your company through the web service whose interface you have exposed through UDDI, the service will be able to search the directory to see that client has a web service that allows an invoice to be submitted electronically. Utopian vision? Currently, yes. Pipe dream? I don't think so. Give the IT community enough time, and this functionality will make its way into enough of the mainstream to be considered a requirement for 'serious' business.
WSDL
Next up in our alphabet soup is Web Services Description Language or WSDL. For those of you like to pronounce your acronyms, that would be "whiz-dull". As you might guess, the purpose of WSDL is to describe the web service that is being exposed. More specifically, it provides the details about the functionality that is implemented, what messages are used to request the service and the sequence in which the messages are exchanged. In other words, everything that an intrepid developer needs to know in order to use our service in their application.
Now defining the WSDL file for a web service is not a requirement for it to be used. However, not having one is like selling an ActiveX component without providing documentation, on-line or otherwise. And even though the format of a WSDL document is convoluted (it is an XML-based standard, after all), most development environments (including Visual Studio) provide tools that will automatically generate WSDL for you. So there is no reason (other than laziness) for not creating a WSDL document for any web service that you develop.
XML
The idea that I can describe XML adequately in a couple of paragraphs is ludicrous. The subject is complex enough that books are devoted so just the basic structure and usage. In my opinion, the reason for this difficulty it that XML is both flexible and extensible. So it can be used in almost any situation where data needs to be stored or transferred. Still, I'm brave (or stupid) enough to give a brief definition a try.
My two second description of XML has always been 'comma-delimited files on steroids'. My reason? The purpose of a CSV file is to store information so that another application can read/import it. This works fine, so long as the receiving application is aware of the structure of the incoming file. The best that CSV files can offer in this area is to place field names at the top of each column. So the possibility for problems of interpretation (or at least situations that require programmer-to-programmer discussions) exists. XML improves on this by applying a more rigorously enforced structure to the data. The order in which the data is physically laid out in the document is irrelevant, but each data field is named, so that it is easier to find. And when you combine XML data with DTDs or Schemas, the type and range of data are now subject to validation. Certainly a superior mechanism than CSV files.
For the most part, our use of XML will be to package up the data that needs to be send to the web service for processing and to receive back the results. As such, our XML documents will be relatively straightforward. But your complexity may vary, depending on the solution that you're trying to provide.
So much for the common protocols/acronyms. These standards are enough to get you started in the web services world. But anything more than a cursory look at the technology reveals many more TLAs or FLAs (three- and four-letter acronyms). For completeness (actually, there is no hope for completenesss: the number of acronyms increases too quickly), here are some additional standards that are commonly used.
WS-Security
One of the main roadblocks to more universal acceptance of web services in production applications is the concerns raised about the security of messages as they fly across the Internet. The WS-Security standard is a set of SOAP headers which are intended (and actually succeed, when used correctly) to define the authentication, encryption and signing that are used within a particular message.
While extensible enough to allow for custom security to be implemented, WS-Security natively supports a wide variety of security models, including Public Key Infrastructure (PKI) and Kerberos tokens. And when combined with the Web Services Enhancements (WSE) toolkit, developers can easily integrate X.509 certificates, role-based security, WS-Addressing (see below for an explanation) and DIME attachmements (again, more information below). WS-Security is both mature and versatile enough to address all but the most stringent security requirements.
DSIG - Digital Signatures
Another aspect of the security associated with web service messages is known as non-repudiation. In plain terms, this means that the recipient of the message is absolutely certain of who sent the message. As well, the recipient is absolutely certain that the content of the message has not been changed. If both of these conditions are met, then the message has the potential (not that I say potential here, not the reality...yet) of being a legally binding document.
The typical approach accomplishing non-repudiation in a SOAP message is through digitial signing. Without getting into the dirty details, it involves hashing the message so that the change to a single byte will be detected by the processing. For those who are interested in the details, check out here. But ignoring the specifics, the WSE toolkit makes it simple to digitally sign SOAP messages.
DIME - Direct Internet Message Encapsulation
Using jargon, DIME is a binary message format that is designed to encapsulate one or more payloads into a single message. In real world terminology, the DIME protocol is used to include attached files to a SOAP message.
In order for DIME to be truly useful, any type of file must be attachable to a message. This causes some temporary concerns. Not every file is suitable for transmission in an XML format. Consider for a moment the problems that would arise if the attached file contained characters that looked l ike an XML tag. Also, binary formats, such as audio, video or images, can cause a problem when embedded in an XML document. And since DIME is associated with the XML-based SOAP message, the underlying technology used to transmit the attachments must be compatible. This means base64.
Base64 is a encoding mechanism that allows arbitrary binary information to be translated into a format that is appropriate for both XML documents and e-mail. The basic approach is to break the source data into 24-bit blocks. Each block is translated into four characters with each character representing 6-bits of the 24-bit block. The characters (there are 64 possible choices, hence the name base64) in the translation are present in all variants of both ASCII and EBCDIC, making it completely compatible with XML.
WS-Policy
One of the issues associated with services is determining which methods are exposed and which parameters. That problem is handled through WSDL. Next up is determining the requirements and capabilities provided by a particular web service. That information is made available through the Web Services Policy Framework, otherwise known as WS-Policy.
A WS-Policy document defines a set of assertions about the policy supported by a particular web service. Possible assertions include encryption methods, the types of digital signatures which are recognized, and the presense or absense of particular WS-Security information. The assertions can be queried in real-time before constructing the service request. The requester can then tailor the encryption, authentication and signing used on the message to match the needs of the web service.
WS-Routing
Workflow is an up and coming issue associated with SOAP messages. At the moment, many web services are simple point-to-point solutions. In that situation, web services closely resemble Remote Procedure Calls (RPC). But the real future of SOA involve the processing of documents. And when a SOAP message contains a document, there is an implication that the document will need to pass through a number of steps before it has finished processing. Say, for example, the document is a sales order. That document will need to be processed through the incoming sales service, the manufacturing service and the shipping service. And to keep track of the services in the process flow, a standard is required.
WS-Routing is a protocol that is used to route SOAP messages from service to service in a work flow. It supports all of the common types of flow, including request/response, peer-to-peer and one-way messaging. At the moment, there are no tools to assist with implementing WS-Routing. But if you can wait as as yet undermined amount of time, it would appear that this functionality is addressed in Indigo, Microsoft's upcoming web services middleware application.
WS-Referral
While WS-Routing handles the basic flow of messages, the interim steps are static. Once defined, the message does what it has been programmed to do. The WS-Referral protocol allows for the modification of the flow while the message is in transit. As a result, this protocol and WS-Routing are usually found hand-in-hand. Which also means that there are no currently available tools to help with implementing the protocol. So you will either need to grit your teeth and bear it or wait for Indigo.
WS-Addressing
The WS-Addressing protocol is used to identify the endpoints of a web service. The endpoint of a web service looks like the URL that is specified in the proxy. It is the place to which the service request is made. Internally (at least to IIS) it defines the server and code that implements the methods exposed by the service.
While in most circumstances, the endpoint of a web service is static (at least, it is most of the time), there are times when the actual endpoint needs to be more flexible. This is particularly true when the request passes through various routers and gateways.The WS-Addressing specification provides a means for redefining the endpoint of a web service as the message is in transit.
GXA - Global XML Web Service Architecture
While the term GXA still appears in a great deal of the older document from Microsoft regarding web services, the term has fallen out of general use. It referred to a collection of extensible, orthogonal specifications related to providing require functionality for web service requests. The standards included in GXA were WS-Security, WS-Routing, and WS-Referral.
SOA - Service Oriented Architecture
Service Oriented Architecture is the next great thing in web services. Put simply, it is a style of designing applications based on the loosely coupled architecture that is forced by using services. A service is a piece of software that exposes a set of methods that can be invoked (called an interface). Because the only way to interact with the service is through the interface, the actual implementation and even the location of the service is irrelevant to the calling application.
But utimately, the benefits associated with using SOA, along with the details involved with designing and implementing a SOA-based are beyond the scope of this article. The purpose of this article was to introduce you to some of the concepts behind web services and to explain some of the acronyms that inhabit this world. Future articles will dive more deeply into the concepts, benefits, issues and implementation details of SOA in the real world. I hope that you'll follow me on my journey down the rabbit hole.
Whether you know it or now, attributes are already part of your development life. Creating a web service using ASP.NET? Attributes are used to define the methods that are exposed. Make a call to a non-COM DLL? Attributes are used to define the entry point for the DLL, as well as the parameters that are passed. In fact, closer examination shows that attributes play a large part in many cool (the programmer’s code word for ‘advanced’) features. For this reason, as well as the possibility of putting them to use in your own environment, a deeper understanding of attributes is worth gaining (a writer’s code phrase for ‘topic’).
Why are Attributes Useful
One of the starting points for a discussion about the why and when of attributes is with objects. Not a surprising starting point, given the prevalence of objects within the .NET Framework. But before believing that objects are the be all and end all of programming, consider the effort that a developer goes through while creating a typical object oriented application. The classes that are part of the hierarchy lay strewn about the workspace. When the situation calls for it, the developer grabs one of the classes and inserts it into their project. They then surround the class with the code necessary to integrate it into the application and perform all of the common functions. To a certain extent, this code is basically the mortar that holds the various classes together.
But think about the problems associated with this development process. The common functions alreadya mentioned include logging the method calls, validating that the inbound parameters are correct and ensuring that the caller is authorized to make the call. Every time that a class is created, all of the supporting code needs to be included in each method.
This problem results from the fact that the common functions don’t really follow the same flow as the class hierarchy. A class hierarchy assumes that common functions move up and down the inheritance path. In other words, a Person class shares methods with an Employee class which in turn shares methods with an HourlyEmployee class. However, the functions described in the previous paragraph apply across all of the classes in a model, not just those in the same inheritance path. Which means that the necessary code might very well have to be implemented more than once.
Is there a solution within the object-oriented world? Yes and no. The solution requires that a couple of things come together. First, all of the classes in the hierarchy would need to be derived from the same ancestor. Then the procedures required by the common functions would need to be baked into the ancestor. Finally, each of the methods that want to implement the common functions (which could very well be all of them) would call the procedures in the ancestor. Do you find this beginning to sound like the gluing together of pieces that we’re trying to avoid in the first place? Me too.
Separation of Concerns
The solution, at least at a conceptual level, revolves around separating concerns. When used in this context, a ‘concern’ is a set of related functions. Going back to the examples mentioned in the previous section, the functions that log a method call would make up one concern. The validation of inbound parameters would be a second. By keeping the implementation of the concerns separate, a number of different benefits ensue.
First of all, keeping the concerns separate increases the level of abstraction for the entire application. In general, the functionality required by the concern will be implemented in a single class. So while in development, the methods can be implemented and tested without regard to the other elements of the application.
The second rationale for separation is also a justification for using an object-oriented architecture. The level of coupling when a concern is maintained in a separate class is quite low. Only the public interface is used by the various classes, meaning that the details of the implementation are hidden. It also means that the details can be changed at the whim…er…judgment of the developer.
To be completely up front, a concern does not have to be a set of functions that can be applied to multiple classes. The functions in a single class can also be categorized as a concern. The difference, as has already been noted, is that the method logging and parameter validation apply to different classes throughout the hierarchy. For this reason, they are known as ‘cross-cutting concerns’.
One more thing before we get into how concerns are actually put together in .NET. In many articles on this topic, the term ‘aspect’ is used to describe what we are calling a concern. For the most part, ‘aspect’ and ‘concern’ can be used interchangeably. And from this, the more commonly heard term ‘aspect oriented programming’ is derived.
Creating Concerns in .NET
Actually, it’s not the creation of a concern that is the difficult part. After all, a concern is just a collection of related functions. In .NET (as well as in pretty much every object-oriented language), that is just a fancy term for a class. The difficult part is in integrating the functionality provided by the concern into existing classes without impacting the class itself. To accomplish this feat requires the use of Application Domains
In the .NET Framework, an application domain is a logical boundary that the common language runtime (CLR) creates within a single process. The key to the application domain concept is one of isolation. The CLR can load different .NET applications into different domains in a single process. Each of the applications runs not only independently of one another, but cannot directly impact one another. They don’t directly share memory. They each have their own process stack. They even can load and use different versions of the same assembly. For all intents and purposes, the applications are isolated.
It might not immediately be apparent why an application domain is required. Even though application domains are isolated from one another, programs running in one can still invoke methods running in another. This is accomplished through a technique not that dissimilar to .NET Remoting. And since we take advantage of this remoting to implement a cross-cutting concern, understanding it is worth a few more words.
Figure 1 – Cross-Domain Method Invocation Flow
The diagram shown in Figure 1 shows the flow of a call between the client class and the receiving object. The assumption built into the diagram is that the client and the recipient are in different application domains. First, transparent proxy is created. This proxy contains an interface identical to the recipient, so that the caller is kept in the dark about the ultimate location of the callee. The transparent proxy calls the real proxy, whose job it is to marshal the parameters of the method across the application domain. As it turns out, before the receiving object sees the call there are zero or more message sink classes that get called. These classes can perform a variety of functions, as well shall see shortly. The last sink in the chain is the stack builder sink. This sink takes the parameters and places them onto the stack before invoking the method in the receiving object. By doing this, the recipient remains as oblivious to the mechanism used to make the call as the initiator is.
A review of this flow reveals that chain of message sinks. These are classes that get called every time a cross-domain call is made. And they would seem to be the perfect place to implement our cross-concern functionality. So the question becomes one of how do this message sinks get created and how do we tell the CLR to use them.
The answer is two-fold. First, in order to indicate that an object should be created in a separate application domain, the recipient class needs to be inherited from ContextBoundObject. So much for the easy part. The second step is to create an attribute which is used to decorate the recipient class.
Context Attributes
The attribute class which decorates the recipient class is one that we create ourselves. It allows us to provide the message sinks which will ultimately implement the cross-cutting concern. The class itself must be derived from the ContextAttribute class. The ContextAttribute class requires that two methods be implemented. First, the IsContextOk function returns a Boolean value indicating whether the current application domain is acceptable for this instance of the class. Since the purpose of our attribute is to force the object to be placed into a separate domain, this function should always return a true.
_
Public Class TraceLogger
Inherits ContextAttribute
Public Overloads Function IsContextOK(ByVal ctx As Context, _
ByVal ctorMsg As IConstructionCallMessage) As Boolean _
Implements IContextAttribute.IsContextOK
Return False
End Function
End Class
The second function that is required by the ContextAttribute base class is one called GetPropertiesForNewContext. The purpose of this method is to allow information about the new context. It is in this method that we indicate the message sinks that are to be included in the method invocation.
Public Overl"
End Get
End Property
Public Overloads Sub Freeze(ByVal NewContext As Context) _
Implements IContextProperty.Freeze
Return
End Sub
End Class
I know it seems like a long journey, but we’re almost at the end. The TraceLoggerObjectSink class used in the GetObjectSink method is where the cross-cutting concern function actually gets implemented. In order to be included in the chain of message sinks, this class needs to implement the IMessageSink interface. As well, at least one constructor in the class needs to accept the next message sink in the chain as a parameter.
Public Class TraceLoggerObjectSink
Implements IMessageSink
Private fNextSink As IMessageSink
Public Sub New(ByVal ims As IMessageSink)
fNextSink = ims
End Sub
End Class
The other two methods required by IMessageSink are SyncProcessMessage and AsyncProcessMessage. These methods are called when the recipient method is invoked synchronously and asynchronously respective. For simplicity, we’ll just focus on SyncProcessMessage. The diagram shown in Figure 2 illustrates how SyncProcessMessage comes into play.
Figure 2 – Process Flow within SyncProcessMessage
As part of the Message Sink chain, the SyncProcessMessage method of this class is called. Within the method, any preprocessing of the request is performed. The method then invokes the SyncProcessMessage method for the message sink provided in the constructor. In its turn, each of the message sinks perform the same logic until the recipient object is called. On the way back, the flow is reversed. From the perspective of just this one sink, however, the flow is return through the same procedure. So after the call to SyncProcessMessage, any post processing of the response is done and control allowed to pass back up the chain.
Public Overloads Function SyncProcessMessage(ByVal Message As IMessage) _
As IMessage Implements IMessageSink.SyncProcessMessage
Dim RetValue As IMessage
PreProcess(Message)
RetValue = _NextSink.SyncProcessMessage(Message)
PostProcess(Message, RetValue)
Return RetValue
End Function
Within the PreProcess and PostProcess methods, any manner of processing can take place. The actual name of the method, the parameter values and the return values are all available to be view and modified. And it is in these methods that the cross-cutting concern functions, such as tracing, validation and authentication can be performed.
Summary
Is the use of a ContextBoundObject and cross-domain calls the best way to implement a cross-cutting concern. Probably not. The restriction that the class must be inherited from ContextBoundObject means that this technique is not suitable for every situation. For example, if you wanted to implement method tracing for an object that was being used in COM+, you have a problem. The COM+ class needs to inherit from System.EnterpriseServices and there is no way to have a single class inherit from multiple bases. However for relatively straightforward situations, this technique is not only effective, but also useful. After all, being able to add functionality without altering the underlying class is always a good way of improving programmer productivity and lowering the rate of defects. Quite a noble goal in its own right.