Layered Design and DataSets for Dummies

Scott Hanselman does a nice 30 second intro into layered design. If any of this is new to you, run quickly to read this.

Scott does a quick bash at Datasets (although doesn't say why) and in my new role as DataSet boy I have to disagree with him and evangelize how simple datasets can make a lot of the code written by the typical programmer: CRUD stuff for example. He even mentions “Adapter“ in describing a data access layer - come on, use the DataAdapter - don't be afraid. In general, if anybody tells you to never do something, you need to question that a bit and dig into the reasons why a technology exists. Of course things may just end up being rude and the answer is indeed never - but always try and get the why.

We've been running a developer contest at the end of some of our training courses (the big 3 week immersion ones). The competition has developers build a solution build on a services oriented architecture which includes smart client, web services, enterprise services/com+ and of course data access. It's only a 1 day event, but the teams are built up of 5-6 people each. Inevitably, if one team decides to use datasets/dataadapters and the other team doesn't, the team that choose the dataset wins. This competition isn't judged or skewed for datasets by any means. But inevitably this is the thing that gives the other team more time to work on the interesting pieces of the application (like business logic: features and functions).

I over heard Harry Pierson tell a customer last week that they shouldn't use datasets in a web service because they aren't compatible with non .NET platforms. This isn't true. A dataset is just XML when you return it out of a dataset. And you probably more control over the format that is generated via the XSD than most people realize. If you want a child table nested, no problem. You want attributes instead of elements, no problem. You want some columns elements and others attributes, no problem. You want/don't want embedded schema, no problem. You don't want the diffgram, no problem. Somebody in the J2EE world has actually gone to the extent of creating a similar type of base object in Java that can deserialize the dataset in most of it's glory. (Link to come - can't find it right now).

In February I posted a “Benefits of Datasets vs. Custom Entities“ which has generated some excellent feedback. It's still in my plans to write the opposite article - when Customer Entities are better than Datasets but I'm still looking for the best template or example entity. Everyone somebody has sent me to date is somewhat lacking. To be fair, I end up always comparing them to a dataset. The things typically missing out of a custom entity are the ability to deal with Null values and the ability to track original values for the purposes of optimistic concurrency. The answer to the question of “When to use a Custom Entity over a Dataset?“ is of course when you don't need all the stuff provided for you by a dataset. So is that typically when you don't care about Null Values or Optimistic Concurrency? Perhaps. But I know there is more to it than that.

I will say there is some crummy binary serialization in the dataset (it's really XML). This is really a problem if you are doing some custom serialization or need to do some .NET remoting. But you can change the way they are serialized (and indeed it's changed in Whidbey). There are some good examples here, here, here, here, here and here.

I'm working on an article making the cases for the custom entity, but in the meantime, datasets represent a good design pattern for entities that is easy and quick to implement by the average developer - and scalable too.

TechEd (Day 3): Hands On Lab Manuals downloads available to the public

No need to have a TechEd commnet password. You can download ALL the pdf's for the plethora of topics. Some good stuff to see how the newly announced stuff (Team System, etc.) works.

Update These links are broken, give this a try: http://www.msteched.com/TechEdLabManuals.aspx

TechEd (Day 1): Balmer's Keynote & Announcing VS 2005 Team System

It's official. I'll post more thoughts and analysis about this as time permits, but, things you should know.

  • Microsoft now has a new Team version of Visual Studio to be delivered “Next Year“ according to Balmer.
  • new source control - more details to follow.
  • Project Management - so dev's will be able to see “Work Items“ in their IDE. There is also supposed to be a sharepoint portal of some kind that dev's & pm's can go to see a dashboard view of a project, milestone's, etc. integrated with MS Project Server.
  • Unit Testing - yes, a very NUnitish thing built right into visual Studio.
  • Code Coverage - yes in the editor you can see what code was executed and what was not.
  • Static Code Analysis - a la fxCop integrated right inside of visual studio.
  • Check in Source control process policy, so a manager type can say “if you check in something, all tests must pass, all static analysis rules must pass, and your code coverage must be 100%“.
  • Also showed was some Load testing stuff that is going to be better than Application Center Test - more on that later.

Of course whitehorse class modeling & SOA designer were showed quickly. Nothing new to announce yet on that front that wasn't covered at PDC....although the guy doing the demo kept saying “Services Oriented APPLICATION” designer. Is this new? Is he changing the acronym from Architecture?

TechEd (Day 0): BOF36: Integrating Unit Testing Tools and Practices Into the Software Development LifeCycle

This BOF went pretty well and a huge thanks to Jim Newkirk for assisting in the delivery. He's a real authority on the practices around NUnit and a good guy to have a round. If you buy his new book on Test Driven Development with Microsoft .NET onsite at TechEd, you can probably catch him at the MS Pavillion to sign a copy.

Some interesting points discussed:

  • Using Unit Tests to drive “example code“ for a framework or class library would be a nice to have.
  • While Code Coverage statistics may satisfy external parties that we've tested what we've developed, percentages are not an accurate measure of code quality.
  • If you write your tests after you do you coding, you already have too much information about your classes that negatively affects how you test.
  • Testing first can really influence (positively) the design of your classes.
  • Developers will work aggressively against source-code check in policies that stipulate a % of code as been covered in unit tests, and that the tests pass, and that they pass static code analysis.
  • It's difficult to test User Interface code, and for a bunch of reason's, not a really good idea or worthwhile investment because the only person who can see your application from the outside in, is through the eyes of a specific user - and you'll never be able to approach that.
  • At the end we also got into some of the difficulties of testing against a database and a bit about Mock objects. That would probably be a good bof on it's own.

Jim might have more comments, but the general feeling I got was that people still need more education about automating unit tests and that not a lot of people are doing it today, let alone Test First. Jim also mentioned that he didn't think it was possible to lecture to somebody and convince them about Test First, but more that it was something that they just really needed to see for themselves. I agree.

TechEd (Day -1): ObjectSpaces = Longhorn, 3rd time charm?

It was bumped from The initial 1.0 release, and then as of last PDC slated for Whidbey. Now it looks like we'll have to wait until Longhorn.

It's not all bad news however. ObjectSpaces is being re-orged into the WinFs file system. When you think about there is an awful lot of correlation to those technologies. I'm sure it's not terribly unrelated to the fact that the Microsoft Business Framework(MBF) that was to build on ObjectSpaces was also pushed off to Longhorn/Orcas. MBF is also to rely on an orchestration engine (Biztalk light?) features going into Longhorn so it all makes sense.

Some people will be disappointed - but this is a good rationalization of the way too many data access/storage visions within Microsoft. Both of these technologies have a common thread about objects/applications and data and breaking down the wall. Sure, MS could have released ObjectSpaces first, but do we really need that legacy and all the effort attached to YADAA (yet another data access api.

Microsoft has taken a lot of criticism (including from me) about the seemingly constant churn of all things data. So this is a good sign that MS is not going to do things, just be cause they can, but do them right. Just ask a Java developer what they think of EJB's. It's important to get it right

http://msdn.microsoft.com/data/

TechEd (Day -1): Pieces of ADO.NET 2.0 not going to make Whidbey

Highlights in this exchange of newsgroup posts...

  • Paging is cut
  • Server cursors cut (SqlResultSet class)
  • Async Open is cut, but not excute
  • SqlDataTable class cut (Good - that was a bad design pattern to propogate)
  • Command sets are cut, but you can still do batch DataAdapter updates,

http://communities.microsoft.com/newsgroups/default.asp?icp=whidbey&slcid=us

More to come, stay tuned.

NUnit 2.2 beta released

http://sourceforge.net/forum/forum.php?forum_id=377077

Things I like:

  • Assert.AreEquals support to compare arrays of the same length, type and values.
  • You can now put a Category attribute on your fixtures AND methods....and then use that as a filter when you go to run tests. Thoughts on categories? Functional Tests,  Performance Tests
  • On a similar note, there is an explicit attribute that will cause a fixture or method not to run unless explicitly selected by the user. You can now put check boxes on the tree to select multiple fixtures/methods.
  • They fixed a problem with background threads that when they raise exceptions, they weren't showing up as a problem in NUnit. Seems they've done some refactoring of how things are loaded in the AppDomain. I'm hopeful that this fixes some issues I've seen when own dynamic loading and Fusion get's lost...but only during the NUnit tests, not the production execution.

Looks exciting.

OpenSource Project for Testing Microsoft Software

Over the past few months, when I question how something works in the .NET Framework (or when somebody asks me).....I have been creating NUnit tests to verify the behaviour of some class and/or methods in the .NET Framework. Initially it is just to observe the behaviour or verify some assumptions, but by the time I'm finished, I usually inject various Assertions into my tests to tighten them up. These now serve as a test bed for me moving to a new version (or even old versions) of the .NET Framework. I can answer the question: Are any of my assumptions about how the 1.1 framework works broken in 1.2? 2.0? 9.0? etc.

I'm building up a nice collection and I might publish my work. But it struck me that this could be an open source project. In fact, I think it should be an open source project and I think it should be started by Microsoft....and not necessarily for the .NET Framework alone - but that would be an easy place to start.

Microsoft has faced increasing pressures over security and quality of their software - to the point that they've actually made windows source code available to key customers, governments and MVP's. I think that's a bit risky if you ask me. I think it is also a bit hypocritical to point the finger at Linux for being “more hackable because source code is available“ but at the same time make your own source code available to the chinese government.

But why not publish the source code to unit tests (say NUnit fixtures) in an open source format for the community to contribute to. When one of these security firms finds a hole in some MS software, they could create an NUnit test to expose it and submit it to Microsoft to fix, and then make the code for that NUnit test part of the open source project.

Instead of publishing source code, which is really meaningless to give people any kind of comfort in the code, publishing unit tests is publishing assumptions and expectations about what software is supposed to do and how it is supposed to behave. I would think this would become more important over time especially moving towards WinFx and Longhorn.

Delegation through NUnit Testing and TodoFixtures

Usually I'm the guy who all the other developers are wiating on to create some reusable framework widget or other. I usually have 10, 000 things on my plate so when somebody asks me to do something or reports a bug with some of my code, I need to find a good way to delegate.

But if you are the subject matter expert (SME), it's tough to delegate the task of making the fix or adding the feature. If you flip that on it's head though, when you find yourself in this situation, by definition you are NOT the SME of the “feature request” or “bug“. Those are external to the actual implementation which is probably what you are really the SME for. The SME for the request or bug, is of course, the finder of the bug or the requestor of the feature. So in the spirit of getting the right person for the job (and delegating a bit at the same time), get the requestor to create the NUnit test that demonstrates the bug or explains (with code - better than english can ever hope to) the request or how the requestor expects the feature to be consumed.

Case in point:

Random Developer A: Barry, there is a bug in the foobar you created. Can you fix it? I know you are busy, want me to give it a go? Do you think it's something I could fix quickly?

Barry: That's a tricky bit but I think I know exactly what I need to do to fix it, it will take me 10 minutes - but I have 3 days of 10 minute items in front of you. Why don't you create an NUnit test for me that demonstrates the bug, and  I'll fix it. Then it will only take me 2 minutes.

I also find NUnit tests a great way for people to give me todo items.

Random Developer B: Hey, I need a method added to FoobarHelper that will turn an apple into an orange, unless you pass it a granny smith, in which case it should turn it into a pickle.

Barry: Sounds reasonable. I can do that - but just to make sure I got all of that spec correct, would you mind creating an NUnit test that demonstrates that functionality you require? Thanks.

I do have to admit though that this requires a certain amount of charisma. On more than one occassion this technique has been met with some friction and unusual and jestures and mumbling. :)

Resuable (almost automatic) Transactions

Can't afford the high costs of COM+ performance overhead in the distributed transaction co-ordinator, but still want somewhat automatic transactions? Same connection? Same Transactions, but different Dac's?

DacBase dacs = new DacBase[3];

dacs[0] = new OrderDac();
dacs[1] = new CustomerDac();
dacs[2] = new EmployeeDac();


trans = DbHelper.BeginTrans();
for i = 1 to dacs.length
{
 dacs[].update(trans)
}

trans.Commit();


OrderDac.Update(entity, trans)
CustomerDac.Update(entity, trans)