VSTS ClassCleanup Method

A colleague and friend of mine Andre who in my opinion is a guru of Test Driven Development discivered this issue.

If you're used to Nunit and how it fires methods in your test Fixture you may assume VSTS fires the equivalent methods in a similar way. Well think again. I am specifically talking about the method you decorate with the ClassCleanup attribute.

Lets assume you have some code in your ClassSetup and ClassCleanup methods you would expect it to setup and cleanup for all the tests in that class. According to the comments generated by MS when a test class is created automatically, you would assume these methods execute before the first test runs and after the last test in that class runs.

Here is the comment you see in the Additional Test Attributes Region of a auto generated Unit Test.

//Use ClassInitialize to run code before running the first test in the class
//Use ClassCleanup to run code after all tests in a class have run
 

The Comments seem clear enough, or do they?

The Nunit equivalents would fire in the order you would expect from the above comments:

Class1.ClassInitialize
 Class1.TestInitialize
  Class1.Test
 Class1.TestCleaup
Class1.ClassCleanup

Class2.ClassInitialize
 Class2.TestInitialize
  Class2.Test
 Class2.TestCleaup
Class2.ClassCleanup

In VSTS they fire in this order:

Class1.ClassInitialize
 Class1.TestInitialize
  Class1.Test
 Class1.TestCleaup
Class2.ClassInitialize
 Class2.TestInitialize
  Class2.Test
 Class2.TestCleaup
Class1.ClassCleanup
Class2.ClassCleanup

Do you see the difference? The cleanup from class1 is not done until after class2 tests have been executed. All the class cleanups are done at once.

What if I wanted to set something up and tear it down for each class? Do I have to keep track of what order people will be running the tests and make sure I perform the setup only once in the first class to execute a test. What if I only want to run the tests in Class 2.

To me this is a design error, intentional or not.