The Case for Unit Testing

A posting on Roy Osherove's blog taking issue with a post by Joel Spolsky on the failure of methodologies like MSF got me thinking.  In particular, I (like Roy) took issue with the following section (which is actually written by Tamir Nitzan, but agreed to by Joel).

Lastly there's MSF. The author's complaint about methodologies is that they essentially transform people into compliance monkeys. "our system isn't working" -- "but we signed all the phase exits!". Intuitively, there is SOME truth in that. Any methodology that aims to promote consistency essentially has to cater to a lowest common denominator. The concept of a "repeatable process" implies that while all people are not the same, they can all produce the same way, and should all be monitored similarly. For instance, in software development, we like to have people unit-test their code. However, a good, experienced developer is about 100 times less likely to write bugs that will be uncovered during unit tests than a beginner. It is therefore practically useless for the former to write these... but most methodologies would enforce that he has to, or else you don't pass some phase. At that point, he's spending say 30% of his time on something essentially useless, which demotivates him. Since he isn't motivated to develop aggressively, he'll start giving large estimates, then not doing much, and perform his 9-5 duties to the letter. Project in crisis? Well, I did my unit tests. The rough translation of his sentence is: "methodologies encourage rock stars to become compliance monkeys, and I need everyone on my team to be a rock star".

The problem I have is that the rock stars that are being discussed are not at all like that. Having had the opportunity to work with a number of very, very good developers, I found that they embrace unit tests with enthusiasm.  And their rationale has little to do with the fact that they might be create a bug that needs to be exposed.  While that might be part of the equation, it is not the real reason for creating a unit test. 

The good developers that I've encountered are lovers of good code.  They believe that well crafted code has a beauty all of its own.  They strive to write elegant, performant classes because, well, that's what craftsmen do.  But when put under the time constraints of business, it is not always possible to create the 'best' solution every single time.  An imminent deadline might require that 'good enough' classes be checked into production.  Such is the life of a paid developer.

But you and I both know these good developers are many times more productive than their 'average' conterparts.  As a result, they frequently have time within a project schedule to refactor previously completed classes.  In fact, they enjoy going back to those 'good enough' classes and improving on them.  This attitude is one of the things I've found separates the good developers from the pack.  They are actually embarrassed by some of the 'good enough' code and feel the need to make them right.

This is where the unit tests come in.  If the completed classes are supported by a solid set of unit test, then this refactoring process can be done with a low risk to the project.  The developers know that when a modified class passes the unit tests, it is much less likely to introduce further bugs. So, rather than thinking them a waste of time, the good developers I know relish the idea of creating unit tests. Perhaps this is one of the characteristics that the rest of us would do well to emulate.

Improving the Performance of Asynchronous Web Service Calls

Notice that the name of this post doesn't say that web service performance can be improved through asynchronous calls.  That is on purpose.  This particular post deals with a limitation that impacts applications that utilize async web service methods.  Actually, it has the potential to impact any application that interacts with the Internet.  This is actually one time I wish my blog was more widely read, because I can pretty much guarantee that there are thousands of developers who are unaware that the nice async design they've implemented isn't having the performance boosting effect that they expected.  And thanks to Marc Durand for pointing this out to me.

The limitation I'm talking about is one that is buried in the HTTP/1.1 specification (RFC2616).  The spec says that, to prevent a single computer from overrunning a server, the limit to the maximum number of connections that can be made to a server is two.  What this means is that if your application makes three async calls to the same server, the third call will be blocked until one of the first two is finished.  I know that this came as a big surprise to me.

Fortunately, there is a configuration setting that can adjust this number without requiring any coding changes.  In the app.config (or machine.config) file, add a connectionManagement tag.  Within connectionManagement, the add tag is used to specify the optional server and the maximum connections allowed.  The following example allows up to 10 simultaneous connections to be made to 216.221.85.164 and 40 to any server.

<configuration>
  <system.net>
    <connectionManagement>
      <add address="216.221.85.164" maxconnection="10" />
      <add address="*" maxconnection="40" />
    </connectionManagement>
  </system.net>
</configuration>

For those who like to program, you can accomplish the same behavior on a temporary basis using the ServicePointManager.DefaultConnectionLimit property.  But I was never big into adding code when I didn't have to.

MSN 7 Beta

HANDWRITING:

  • Thanks to Jamers for the official update.  I will keep this short since most people on the internet can't read apparently.
  • Go here - http://messenger.msn.com/Beta/Default.aspx Bottom of the page.  Basically - Install Journal Viewer from Windows Update and all other updates.  Which should have been done anyways.

     

  • Minimum requirements for handwrite capabilities are as follows:
    * Microsoft Windows XP or Microsoft Windows 2000 with Service Pack 3 or above.
    * Microsoft Windows Journal Viewer 1.5. ( http://g.msn.com/2meen_us/2_4371096_01?FamilyID=fad44098-8b73-4e06-96d4-d1eb70eacb44 )
    * Security Update for Windows Journal Viewer (KB886179). ( http://g.msn.com/2meen_us/2_4371096_02?FamilyID=dbf1ea4e-72bd-4359-9f93-7c232ed2dcd3 )


     


    I recently installed the latest beta release of MSN Messenger 7.  I really hoped for a couple of features that I felt were lacking in MSN Messenger 6. Well I was very glad to see that must of these features were added/fixed.  Ever tried to invite another user to an existing conversation?  It is a pain, you have to click the Invite button then go through your contact list, if this wasn’t bad enough the contact list is sorted alphabetically regardless of groups so no one is where you expect them to be.  This has been resolved, you can finely drag and drop like you tried to do a million times before.  The next thing they fixed was just a really minor annoyance, my picture was too small.  I like my pictures, they are informative and witty and I wish to see them as much as I wish others to see them.  Now the picture for yourself is the same size as your conversation partner (with the option to put it back to small if you are not a narcissist like me).   MSN toast also has the person’s picture in it, which is interesting, but just another distraction as you notice the person you never speak to has changed their picture. 

     If I could stop here I would be a happy MSN 7 user, however there is more.  Winks and nudges are the devils spawn.  I’m not really sure when it started, but flashy useless functionality has become the opiate for the masses, and as expected they apparently love it.  MSN 7 has added two little pieces of functionality that I highly doubt they had overwhelming requests for.  The aforementioned winks and nudges.  Winks are little flash cartoons that people can send you and play on your screen.  Cool the first time, but by the second time the novelty has already worn off and you are searching the options to turn them off. Can’t stop them entirely but you can turn off the Auto-Play function.  The next is nudge, which basically violently shakes your conversation window when you receive one.  Clearly this annoying, and MSN stops you from sending more than one of these in a certain time span, which leads me to believe that they KNEW it was annoying.  Yet it continues to exist.  I turned this off too.  Microsoft I beg of you, get rid of these features. Cut the bloat.

    The last feature is not something I have any use for, but I assume the tablet pc and pocket pc users of the world will enjoy it.  You can hand write your messages (no character recognition available in this beta).  This feature also seems to be backwards compatible with MSN 6 and 4, but not  for Mac users, sorry.

    Other functionality worth mentioning:

    -         Can now set your status before you login

    -         Feature packs.  You can now buy sets of emoticons/backgrounds/etc

    -         History from your last conversation shows up when you start a new conversation.  Very helpful for those long drawn out conversations.

     

    UPDATE: The Beta is now public and can be downloaded from MSN.com
  • FileNotFoundException in ASP.NET Web Services

    As you work with web services in ASP.NET, it is likely that you will eventually come upon a weird file not found exception.  The actual text of the exception will look something like “File or assembly name jdiw83ls.dll, or one of its dependencies, was not found".  Now, since it unlikely that you have a project with a name of jdiw84ls in your solution, the source of this exception and it's resolution will be hard to see.

    To understand where the message is coming from requires a little detail about how ASP.NET services moves complex objects across its boundary.  If you pass a complex object to a web service, the client side serializes the class into an XML document.  On the receiving side (the server), the inbound XML document is deserialized back into a class.  This knowledge is part of Web Services 101.

    However, what happens when the class

    <configuration>
        <system.diagnostics>
             <switches>
                <add name="XmlSerialization.Compilation" value="4"/>
             </switches>
        </system.diagnostics>
    </configuration>

     

    BizTalk 2004 Map, Getting Distinct Values using an Inline XSLT Call Template

    Need to get a distinct list of items from an XML message.

    For example the XML message containing Order Items is as below:

    (Note: That the Order ids are not sorted)

    From the above message produce a distinct list of order ids such as below:


    The custom XSLT code to get the distinct list of values looks like this:


    To use an Inline XSLT Call Template inside of a BizTalk 2004 Map,
    a) Create a new Map inside of your BizTalk project
    b) Choose the source and destination schemas
    c) Drop a Scripting Functoid on the Map, then in the properties window with
    the Scripting Functoid selected,  press the -> Configure Functoid Script button.
    d) For the Script type choose -> Inline XSLT Call Template. Place the XSLT into the Inline Script Buffer
    window.

    The map to get the distinct Order Ids looks like the below:

    Note : The scripting functoid inside of the map has no incoming or outgoing links.
    The Custom XSLT will handle the complete transformation. Ignore any warnings about no incoming
    or outgoing links when building the project.

    Download the Complete solution HERE

    Web Services and Proxy Servers

    I'm currenting working with a team that is putting the finishing touches on a commercial application that uses web services as a major component of the infrastructure.  As part of putting the client portion through its paces, an environment that included a proxy server was tested.  The initial result was not surprising to anyone who has significant experience with web services:  a connection could not be made exception was thrown. The reason for this problem is that the method for defining a proxy server to the web service proxy class on the client side is a little unexpected.  And I apologize in advance for having to use the same word (proxy) to describe to separate items (the server and the web service class). 

    If you are running Internet Explorer, then the place where the the proxy server settings are configured is in the Tools | Internet Options dialog.  Within the dialog, click on the Connections tab and the LAN Settings button.  Here you can specify whether proxy server settings are required and which server and port are used.  You can also identify whether local addresses are exempt from the proxy process.  This is fairly well known information, especially if you normally operate from behind a proxy server.

    The disconnect occurs when you make calls to web services from this same machine.  Under the covers, the HttpWebRequest class is used to transmit the call to the desired server.  However, even when the proxy server information has been specified as just described, those details are NOT used by HttpWebRequest.  Instead, by default, no proxy is used to handle the request.  This is what causes the connection exception. I know the first time I ran into this issue was a surprise to me.

    There are two possible solutions to this problem.  The first is to assign a WebProxy object to the Proxy property on the web service proxy class.  You can programmatically specify the proxy server information, but that is very difficult to modify.  Instead of creating the WebProxy object on the fly, use the GetDefaultProxy static method on the System.Net.WebProxy object.  This method reads “the nondynamic proxy settings stored by Internet Explorer 5.5” and returns them as a WebProxy object that can be assigned to the Proxy property .  This technique works fine assuming that your client has IE installed and that none of the scripts run at login time have modified the proxy settings.

    There is an alternative that is a little more flexible.  There is a app.config element called defaultProxy that is part of the System.Net configuration section.  Within this element, the details of the proxy server can be specified independently of the IE settings.  The basic format of the defaultProxy element can be seen below.

    <system.net>
       <defaultProxy>
          <proxy
             proxyaddress = "http://proxyserver:80"
             bypassonlocal = "true"
          />
       </defaultProxy>
    </system.net>

    The proxy element also includes an attribute called usesystemdefault which, when set to true, causes the IE settings to be used.  The benefit of this approach is that no additional coding is required.  The settings defined in defaultProxy are use by HttpWebRequest (and, therefore, web service calls) with no change to the application required.

    As an added benefit, this element can be added to machine.config, which allows it to be applied to every HttpWebRequest made from the client machine regardless of the application.  Keeping in mind that an uneducated developer could assign the Proxy object in code, overriding this nice, easily configurable default.

    Trip to Ottawa and Requirements Traceability

    I just got back from Ottawa, where last night I was speaking to the Ottawa .NET community about Visual Studio Tools for Office. (more on that later).

    I wasn't surprised by the Grep Cup weekend inflated hotel rates, but I was surprised to find a “2.8% DMF Fee” on my hotel bill (on top of the 15% worth of federal and provincial taxes). Upon request, I was informed that this was a “Destination Marketing Fee” which goes to fund marketing efforts to promote tourism in the area. Various Ontario tourism centers (including Hamilton - go figure?) have been lobbying the provincial governments since post 9/11 in an effort for them to allow a tax (a real tax, not a fee) for the same purpose. This past summer however, the hotels decided that this was going nowhere so they decided to start collecting (on a voluntary basis) a fee (not a real tax).

    Maybe it's just me, but I'm thinking the best way to attract people to your city is not to put a sneaky “DMF Fee” charge on those same people's hotel bill when they come to visit you and hope they don't ask about it. Even worst, because it's a fee charged by the hotel, and not a real tax - guess what - you pay tax on the DMF Fee. Icarumba! It turns out it's voluntary fee and not hotels collect it. The front desk staff sensed I was not pleased about being asked to pay for marketing fees on top of my room rate so they quickly waived the fee. But I wonder how many people willing pay this?

    This all reminds me very much about requirements management and software development. Often, people, usually too close to the problem, design features into software that doesn't meet the requirements of the user. Take for example those goofy glyphs on the Lotus Notes login window. What about clippy? Is that satisfying anybody's requirements - or is it just pissing you off? With all of our best intentions, it is extremely important that we take the time to perform reality checks on what we are building against the requirements of our users.

    Now to bring it all home. Do users really want to do their work in a web browser? Browsers are great for wandering and finding stuff, but do they want to see the value of their stock portfolio in a browser? You need to find the best environment for the job your users are trying to accomplish. If somebody is accustomed to using Excel to aggregate a bunch of their financial information, then maybe Visual Studio Tools for Office is the right tool for that job. While writing applications in Excel isn't exactly new, with VSTO you have the integration with the .NET Framework, Web Services, and the smart client deployment model, you can apply all professional development skills you have at your disposal to creating applications with Word & Excel. And don't worry, I have yet to see clippy show up in Visual Studio Office projects.

     

    Deserializing Objects

    Another post about the deserialization of objects after a web service call.  In this particular scenario, the web service method was returning an ArrayList.  On the client side, I was expecting to see a number of different types of objects.  But one of the objects, a custom class, was being returned as an array instead of an instance of the expected class.  And inside that array was an XmlAttribute and two XmlElement objects.  These were, it turned out, the XML nodes that were being returned from the web service method.  It's just that they weren't being converted into the desired object.

    After a couple of hours of digging, I found out the cause.  Many of you are probably aware of the XmlInclude attribute that is used to decorate web methods to indicate that the schema for a particular class should be included in the WSDL for a web service.  That is not, strictly speaking, it's only purpose.  To be precise, it is used by the XmlSerializer to identify the types that are recognized during the serialization and deserialization process. It's this second piece that was the solution to my problem.

    Because the web service being developed is intended to be used in an intranet environment, the same assembly was available on both sides of the fence.  To support this, the generated proxy class was modified to reference the shared assembly instead of using the class that is automatically included by WSDL.EXE.  It also meant that we didn't want to perform an Update Web Reference when the web service was modified...it would have meant making the manual changes to the proxy class again (Rich Turner might not remember me from the MVP Summit, but I was the one who was most excited when he described the ability to customize the proxy class generation being available come Whidbey).  The down side of this manual process is that, when objects of this type were added to the ArrayList that was returned, the proxy was not updated with the appropriate XmlInclude attribute. 

    That's right.  The proxy class gets the same XmlInclude attribute.  WSDL.EXE generates the class with the appropriate attributes.  But if you're like me, then it's something you need to be aware of the next time an array of XmlElements appears unexpectedly on the client side.

    Moving DataSets across Web Service Boundaries

    A situation last week had me looking deep into the bowels of how DataSets are marshaled across web service methods.  The situation is as follows.

    For programming ease, the typed DataSet that was automatically generated was modified to include a custom object as one of the column types.  If you've ever taken a close look at the code that gets generated when a DataSet is created from an XSD diagram, you'll realize that this is not a difficult thing to accomplish.  And, for the most part, there is little reason to modify the generated code. But that being said, there are times when doing so can make the life of the DataSet user a little easier.  And this isn't really the point of the blog, only the motivation for digging into DataSet marshaling.

    The modified DataSet was being returned to the client by a web service method.  When the hacked column on the returned object was accessed on the client side, a StrongTypingException is thrown.  Another look at the DataSet generated code shows that the cause of this exception is a casting error in the property.  Further examination showed that the name of the class associated with the column was being returned.  As a string.  Naturally when the literal “This.Namespace.Class” is converted to an object of type This.Namespace.Class, casting exceptions are the result.

    This behavior begged the question “why wasn't the object being XML serialized with the rest of the DataSet”.  After all, though I haven't mentioned it yet, This.Namespace.Class is completely serializable.  In particular, why was the name of the class being returned???  That didn't make any sense.

    At this point, I broke out .NET Reflector and took at look at the details of the DataSet.  First of all, if a DataSet is returned across a web service boundary, it is not XML serialized in the same manner as other objects.  Instead of converting the contents of the DataSet to XML, a diffgram of the DataSet is generated.  More accurately, the WriteXml method is called with a WriteMode of Diffgram. Within the DataSet class, there is a method called GenerateDiffgram.  This method walks across each of the tables in the DataSet, followed by walks, in turn, through the DataTables and DataRows.  When it gets to an individual column, it loads up a DataStorage object.  More accurately, since DataStorage is an abstract class, it instantiates a DataStorage derived class, using the column type to pick the appropriate class.  A veritable DataStorage factory.  When the column type is one of the non-intrinsic types, the ObjectStorage class is used.  On the ObjectStorage class, the ObjectToXml method is called.  This is where the unexpected happens. At least, it was unexpected for me.  The ObjectToXml method does not XML Serialize the object!

    In the case of the ObjectStorage class, what actually happens is that a check is performed to see if the object is a byte array.  If it is a base64 encoding of the array is returned.  Otherwise the ToString() method on the object is called. When it comes to re-hydration on the client side, a similar process occurs.  The difference is that the XmlToObject method in the ObjectStorage class instantiates the desired object by passing the string representation of the object (as emitted by the ObjectToXml method) into the constructor.

    So what is the point of all this?  First, it explains why the name of the class was appearing in the generated XML.  Unless overridden, the output of ToString() for an arbitrary class is the name of the class.  It also explains why no object was being created on the client side, as the class I was working with didn't have the appropriate constructor.  My solution, which I freely admit is a bit of a hack, is to give the diffgram generation process what it's looking for.  I overloaded ToString() to return an XML document containing the values of the class, a poor man's version of the WriteXml method that is part of the IXmlSerializable interface.  I also created a constructor that took a string as a parameter and repopulated the properties of the class (the ReadXml portion of the process).  Problem solved.  But still, I have to wonder why ToString was used and not the IXmlSerializable interface methods in the first pace.  Here's hoping someone more knowledgeable than me will provide some insight.

    Creating/Modifiying Schemas in BizTalk 2004. Quick Tip.

    When BizTalk 2004 is installed on a developer machine, one of the many things that
    is installed is the BizTalk Editor. The BizTalk Editor is used to
    create and modify .xsd schemas in VS2003 that will eventually be used in a BizTalk process.

    The BizTalk Editor is broken up into three parts:
    1) The left hand side displays a tree structure to add new nodes to the specification.
    2) The centre displays the xml markup representation of the schema.
    3) The right hand side displays the properties pane, to edit the properties of the currently
    highlighed node on the left hand side tree structure.

    This is illustrated below.


    One of the limiting factors of the BizTalk Editor is that the centre xml markup representation
    is not editable.


    To be able to edit the xml markup representation of the .xsd schema do this:
    1) Right mouse button on the schema and choose Open With... on the pop up menu.
    2) A dialog such as the one below will appear -> In this dialog, choose XML Schema editor.


    3) The .xsd schema will now display in the XML Schema Editor. This editor is more object based,
    displaying each element in a box. New attributes can be added to the Invoice, Invoices and InvoiceItem
    elements. The properties window, can also be used to edit individual attributes.

    4) While in the editor, at the bottom choose the XML tab, as below.

    5) Now the below schema can be edited directly.