We've heard of code coverage which is normally used by the developer's to verify how much of the code they've written was tested by their Unit tests. Microsoft took it a step beyond developers and gave the QA the ability to run code coverage against our tests. You can with any test type including Manual Tests have code coverage running during execution and see how much of the code is being tested. It's a great way to know whether or not the QA test suites created are covering all the code. With both developers and testers running code coverage against a Build we get a real good idea of how good that Build really is.
Check out Eric Sink's blog on Code Coverage at -> http://www.ericsink.com/articles/Code_Coverage.html
A fellow by the name of Paul Stovell did some creative thinking and came up with a tool called Scenario Coverage Analyser for TFS, which basically is an MSBuild task. Scenario Coverage Analyser allows you to summarize your code coverage statistics in terms of the Scenarios they cover. We are assuming you’re using the work item Scenario to document your requirements in VSTS.
A quick explanation of how this works, you modify the Team Builds to call Scenario Coverage Analyser. You run your Team Build and Unit Tests with code coverage Scenario Coverage Analyser does three things;
- Looks at the code coverage results generated
- Maps the individual code coverage results to the work items that the code was written for
- Produces an HTML report called Traceability Matrix
This add-in tool only runs with Builds so it does not work with the QA Test execution, but it gives the QA Team a report on how much of each Scenario (requirement) was tested by the Unit Tests.
Thoughts to comment on:
If a Scenario shows that the unit test tested 90% of its related code how much more testing of that Scenario would QA really need to do?
What % of Scenario Coverage would be acceptable?
Would this be a report you'd share with your customer?
Is there a like tool that could do the same analysis of code coverage run against the QA Tests?
Wouldn't that be a nice add-in for the QA Team? [:D]
To see more on this great add-in tool here is a link to Paul Stovell's blog ->
Paul’s how to set up Scenario Coverage Analyser ->
To download the tool -> http://readify.net/Default.aspx?tabid=269
Try it out and let us know how it works for you - add a comment to my blog with your thoughts.
Thanks to Paul for this great addition to VSTS.
Another sunny day ... Testa [A]