Agile.org has some create webcasts on subjects concerning Agile, Scrum. You can either sign up to them or go to the archive listing. Couple of interest are:
Agile Practices in a Traditional Organization
Adopting Test-First Development
Release Duration and Enterprise Agility
On May 15th is Agile and Quality: It is not an Oxymoron but a Necessity
Click here to check out the public webcast series.
If your reading this blog you likely understand what functional testing and you may use the term system testing.
Wikipedia defines these terms as:
“System testing of software or hardware is testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic”
“Functional testing is a type of black box testing that bases its test cases on the specifications of the software component under test. Functions are tested by feeding them input and examining the output, and internal program structure is rarely considered.”
If you have read Agile Software Engineering with Visual Studio (by Sam Guckenheimer & Neno Loje) you will have heard about “reducing waste”. Identified as tasks that reduce waste are functional and system testing. These two tasks can be done during code development through unit tests reducing the cost of bug fixes, bug analysis, creating a suite of automated tests and automated regressions tests and reducing the number of people involved in testing and the bug. In some teams developers and testers have been testing the same thing one through unit tests and then again later by a testers. This is duplication of work effort that is a expensive waste.
In Visual Studio there are unit testing tools and third party add in tools that developers can use to create very robust unit tests. You maybe thinking “the developers do not test the same “stuff” that testers do”. Your right, I agree. However since the team is encouraged to make changes to reduce waste why don’t we testers help them. In Visual Studio we can create test cases that are associated to the requirement/user story work item that describes what needs tested. We can pair up with developers to help them write robust unit tests to cover all the testing including boundary, error and data testing.
There are people that believe the future of the “software testers” is about to make a big change. Testers will need to be able to write and execute unit tests themselves therefore requiring the basics of coding and the ability to add assertions (validation) to the unit tests. (Check out MSDN’s Verifying Code by Using Unit Tests topics. ) I believe this will be a reality in the future but I also believe it will evolve. If you want to start now pair up with your developers to help them create unit tests that execute both the “happy path” and boundaries of an individual method, class or component. Help them to create system integration tests. Getting expose to how unit tests are designed and coded will help you move into the future. In addition having knowledge of what has been unit tested reduces test duplication later. TFS and Visual Studio help us with all this through work item traceability.
Example of work item traceability:
Example of a Test Case and Associated Automation:
Visual Studio has the tools that will help us move into the future with confidence and the security we’ll need. Humans in general are not adapt to change but change we must. I am one of those people that embraces change and excels in change but then I have had Visual Studio in my pocket!
- Kent Beck
The role of professional testing will inevitably change from “adult supervision” to something more closely resembling an amplifier for the communication between those who generally have a feeling for what the system must do and those who will make it do.
Kent Beck author of Test-Driven Development (Addison Wesley 2002), 86.
Visual Studio – my companion, my mentor, my stability, my aid, my reporter, my success
(stay tuned, next blog I will show you how easy it is to create a unit test!)
Andre King & I presented “Never Test Alone” at TASSQ in Toronto yesterday evening. After the presentation Joe Larizza (TASSQ President) addressed the group saying that with the economy businesses are looking for ways to reduce costs. Outsourcing testing will be one of them. People need to be proactive in finding ways to reduce cost and produce quicker. Joe encouraged people to talk to their management teams, present the concept outlines in “Never Test Alone” and volunteer to give it a try. See how it works, what it saves and in the end you may be saving more then you expect.
In Never Test Alone testers are involved in the Requirements & Design reviews and validations. This allows the test team to prepare for testing early, creating test cases, test scenario's and other documentation at the start of the project. Plus testers add value to the process from their experience and perspective. This reduces the cost of the test group weeding through requirements later to get an understanding that may not even be a correct one of the system. Enhancements that testers log during testing will come out during the requirement review phase. If deemed required can save money by being incorporated before development starts versus later.
Team up testers with developers during the creation of unit tests, the more tested at this stage the less cost in fixing, less bugs later, and we can reduce the duplication of testing being done between the two groups. Testers now know what has been tested therefore will not have to retest later. A testers perspective in what needs tested will differ from a developers so the combination of the two creates very rigorous unit tests.
If you are doing continuous builds (http://en.wikipedia.org/wiki/Build_automation) the unit tests are run against each build becoming regression tests. So we now have during unit testing, the functional testing and automated regression testing happening very early in the project.
Unit tests can be used in load testing. One example is login where UserID and Password are entered. The unit test can be build to retrieve from a listing of many UserID/Password combinations to execute login many times. What is the savings in knowing during development that the system is not going to have problems with 1,000 concurrent logins? Add this test to the build regression tests and you know throughout your project how well it is standing up to load. Why leave load testing to the end, do it early and save time, money and possibly the project.
Andre demonstrated a tool for automated white box testing, Microsoft Research PEX (http://research.microsoft.com/en-us/projects/Pex/). Very impressive and it can be run against code by testers saving development time for development. Check out the link above for a white paper on PEX. Maybe Microsoft will make a version of PEX that testing can use which does not require us to have access to the actual source code.
This is a list of some effects from following this presentation:
testing a lot earlier
saving money on bug management
saving money on system changes late in the project
building a team relationship
- great feeling of accomplishment
And you are not testing alone, it is a group effort and it is the route to success.
Link to TASSQ
Bruce Johnson and I attended the KWSQA Conference yesterday in Waterloo. We teamed up doing a session at the conference. Our topic was "Never Test Alone - the Route to Success". Bruce wowed the crowd with a demo of his test-driven development skills and the Microsoft Research tool Pex (Automated Whitebox .Net testing tool). As we promised the presentation slides can be downloaded see link below. Congratulations to the KWSQA group for a great job.
Never Test Alone presentation slides
Research Microsoft Pex
If we can be of help or answer any questions please don't hestitate to contact us.
Tuesday Jan. 29th, 2008 at the Toronto Association of Systems and Software Quality (TASSQ) meeting join myself and Barry Gervin for an evening presentation on VSTS Edition for Testers. Click the link for details about this presentation including a link for registration.
Hope to see you there ... Testa
I have blogged about testing early, getting your test team involved at the start of the project one item I neglected to include was Load testing. So let’s catch up and talk about when load testing should be done. A lot of projects wait till either UAT or just prior to implementation to perform Load testing, does that make sense?
What is Load Testing? -- It is the process of creating demand on a system or devise and measuring its response. It is modeling the expected usage of a software program by simulating multiple users accessing the program’s services concurrently.
With VS Team System Load Tester you are grouping and running your existing unit and web tests. When a build is ready the unit tests are run against that build. We are verifying the build quality and regression testing that any fixes made did not break something else. The QA team gets a quality build and run their web and manual tests and report any bugs. If the bug’s reported are not high severities then let’s put the unit and web tests into a load test and run it. You’ll have to do some planning ahead like a clean environment and data set up for the load testing. An important factor during load testing is having an appropriate data size to find issues like missing indexes, to many indexes, locks, bad query plans, the list goes on and on.
Load testing early will find performance issues sooner which in most instances are due to architectural issues in the application. Finding these just before implementation can turn into a No Go decision. Finding them early allows lots of time to analyze and make the necessary changes, and you can retest the fix without delaying other testing, development or implementation.
This is a best practice that with VS Team System allows us to do so easily it’s crazy not too.
Quote by Andy Warhol (1928-1987): “They always say time changes things, but you actually have to change them yourself.”
Lets change Load Testing start early don’t leave it till the end.
Enjoying a sunny summer … Testa J
We've heard of code coverage which is normally used by the developer's to verify how much of the code they've written was tested by their Unit tests. Microsoft took it a step beyond developers and gave the QA the ability to run code coverage against our tests. You can with any test type including Manual Tests have code coverage running during execution and see how much of the code is being tested. It's a great way to know whether or not the QA test suites created are covering all the code. With both developers and testers running code coverage against a Build we get a real good idea of how good that Build really is.
Check out Eric Sink's blog on Code Coverage at -> http://www.ericsink.com/articles/Code_Coverage.html
A fellow by the name of Paul Stovell did some creative thinking and came up with a tool called Scenario Coverage Analyser for TFS, which basically is an MSBuild task. Scenario Coverage Analyser allows you to summarize your code coverage statistics in terms of the Scenarios they cover. We are assuming you’re using the work item Scenario to document your requirements in VSTS.
A quick explanation of how this works, you modify the Team Builds to call Scenario Coverage Analyser. You run your Team Build and Unit Tests with code coverage Scenario Coverage Analyser does three things;
- Looks at the code coverage results generated
- Maps the individual code coverage results to the work items that the code was written for
- Produces an HTML report called Traceability Matrix
This add-in tool only runs with Builds so it does not work with the QA Test execution, but it gives the QA Team a report on how much of each Scenario (requirement) was tested by the Unit Tests.
Thoughts to comment on:
If a Scenario shows that the unit test tested 90% of its related code how much more testing of that Scenario would QA really need to do?
What % of Scenario Coverage would be acceptable?
Would this be a report you'd share with your customer?
Is there a like tool that could do the same analysis of code coverage run against the QA Tests?
Wouldn't that be a nice add-in for the QA Team? [:D]
To see more on this great add-in tool here is a link to Paul Stovell's blog ->
Paul’s how to set up Scenario Coverage Analyser ->
To download the tool -> http://readify.net/Default.aspx?tabid=269
Try it out and let us know how it works for you - add a comment to my blog with your thoughts.
Thanks to Paul for this great addition to VSTS.
Another sunny day ... Testa [A]
ObjectSharp is now offering a Team System Quality Assurance Best Practices course of which I blogged about a few blogs ago.
The course and labs cover the following topics:
- QA Role in the Software Development Life Cycle
- Overview of Visual Studio Team System
- Team Communications
- Working with Visual Studio
- Version Control
- Manual Testing
- Web Testing
- Other Testing
- Load Testing
- Defect Tracking
For more details on the course go to: http://www.objectsharp.com/training/CourseDetail.aspx?id=5030
Link to the public course schedule: http://www.objectsharp.com/training/ClassSchedule.aspx
We offer a public course in Toronto or we can host your team’s private, customized course at our fully equipped facility. We also offer onsite, customized training with our portable laptops.
I want to also share with you a great place for a weekend away. I was at a family wedding this past weekend and stayed at the “Kingbridge Centre” which is a conference centre that is open for anyone to stay. I highly recommend it, great place with lots to do and it’s just 45 minutes from downtown TO. Check it out at www.kingbridgecentre.com . The wedding was great too!
On the phone in our room at Kingbridge was a quote by Alexander Graham Bell it says alot about what I believe is the key to success in IT projects.
“Before anything else, preparation is the key to success”
Talk to you later … Testa
Best Practice in any project using any tool is involving the QA Team, the project community at the beginning of the project planning stage. I've blogged about this previously but now I'd like to share how the QA Team can help and what they would be doing.
During project planning the QA Mgr and/or Lead should be involved so that during planning a comprehensive understanding of what is involved in the overall project can be obtained. The QA representative can help in planning from knowledge of previous projects for instance test timelines, test needs, resource needs, and has the opportunity to share ideas of how QA can assist other teams.
During preliminary project planning the QA Mgr/Lead can then start planning out the QA Team so that they have the right people sourced and available to start as soon as deemed required. This eliminates running around trying to hire QA resources a week prior to testing, which normally tends to be a disaster.
Now you have a QA team with a manager or lead, what do we do, get involved with requirements. I attended a web cast recently called Elicit Effective Requirements While Building Trust. It seems a lot of people are talking about "Requirements" these days the same points keep being repeated, see the list below:
Requirements development to include:
· plan to involve the stakeholders
· creation of a product vision
· establish a common glossary
· use multiple elicitation methods
· involve the entire project community
· use multiple model techniques both text and diagrams
· identify scope early - drive modelling from scope
· verify as you iterate
· prioritize continually
The next step is to involve the entire project community including stakeholders (the business) to create a vision, document the scope then analyze and verify requirements at the same time identifying priorities and risk. A great way to do this is by conducting workshops where requirement development is conducted in an organized, well planned atmosphere. I'll share tips I got from the web cast for a successful workshop. From experience this does work great in fact during a consultant gig I did where planning was started without the stakeholder’s I suggested to the project manager we do some workshops with the stakeholders. We basically followed this same pattern and what came out of it was the needs of the stakeholders had not been fully understood and what has being planned in development was about 70% incorrect. The workshops with the stakeholder early in the project saved the day in that situation.
· Use a planning team.
· Mind you’re "P’s": purpose, participants, principles, products, place, process.
· Uncover hidden agendas.
· Expose decisions.
· Establish and use ground rules.
· Make decision-making transparent.
· Openly acknowledge conflict.
· Build in serious play and prototypes.
· Enrich the work with multi-models and wall work.
· Conduct a sponsor show-and-tell.
· Facilitate a workshop retrospective.
· Share outcomes with the entire community.
· Use feedback to improve the process.
How does VSTS help in all this?
You can use templates that come with VSTS or upload your own for gathering and documenting high level requirements. With the ability to share using VSTS Team Explorer or the website Project Portal or for those that don’t have VSTS use Team Plain free (with a TFS license). I’ll blog about Team Plain another day but it gives your project community full edit/update access to work items with out having to have VSTS installed. During the workshop you can use the VSTS templates, Vision, the Persona, Requirements List, Issues, Project Checklist, work items Scenario and Quality of Service Requirements to translate the work into documents. Use work item Task to document additional work even assign it, work item Risk to identify areas that have been identified as a risk to the project. All this is stored in TFS and accessible to the project community. An easy format in workshops to get the documents written and updated is to assign them to specific people then have others proof read and a signoff process. Keep in mind one persons interpretation may not be another’s so it has to be a team effort. During the workshop there maybe times when you have to agree to disagree this is a good time to use the Triage List to track these instances and go back to them later with additional input from the community for clarification.
Once all requirements have been reviewed, analyzed and fully documented present them to the project sponsor’s, the entire community, get feedback, make revisions and finally get signoff.
The last step before the individual teams break out to start the next cycles have another workshop meeting where you plan out Builds, what will be in each build, estimate timelines, identify testing needs for each build, arrange tester/developer unit test design, regression testing of the builds, what happens if a build does not pass validation, logging of bugs found in builds, development load testing any work effort that can be identified and planned involving builds. Attendee’s at this workshop should include a Release Manager, Development, QA and Project Management builds affect all these areas’s and therefore needs to be a collaborative effort. All the build information should be documented, published and stored in VSTS TFS.
To conclude planning, requirement development workshops and build planning workshops help all teams in the project community to enable better team planning creates a well informed project community, the left hand knows what the right hand is doing, and it’s a gigantic step towards a successful project. VSTS aids the project community in organizing the documentation that comes from planning and workshops and allows easy communication to everyone. The requirements become work items that will be tracked as the project progresses, tasks identifies other work to be done, risk identifies stuff to watch and may even require further analysis, this all is the initial setup of your VSTS project.
As for the QA Team they now have a good understanding of what will need tested and can plan out their testing, make sure they have the right resources, they can work with development in creating better more comprehensive unit tests, plan for early load testing with unit tests, start getting tests cases/scenario’s written for the planned builds, they can determine what they will need to test thereby eliminating test duplication within the project community. Bring it on we’re ready!
Barry Gervin an ObjectSharp partner commented once; If QA is responsible for the quality of the product how come they aren’t involved from the start of the project. How come they aren’t involved in the unit testing, build verification, requirement verification? It is a good question and one that all Software Projects should be asking themselves.
If you have any questions contact me though comments. If you have just comments please they are welcome.
Quote of the day: We don't accomplish anything in this world alone ... and whatever happens is the result of the whole tapestry of one's life and all the weavings of individual threads from one to another that creates something.
Have a good day … Testa
Duration: 3 days
Description: This 3 day hands-on class will equip students with the techniques and best practices for using Visual Studio Team Edition for Testers and Team Foundation Server to manage the quality of software within the development life cycle.
This course is intended for Software Quality Assurance Professionals (a.k.a. “Testers, Leads & QA Managers”) who are either using or interested in learning about Team Foundation Server and Visual Studio Team System Edition for Testers.
This course will train students how to use the tools to author and execute various types of testing supported by Team System, and how to effectively use the Team Foundation Server defect tracking features as part of a holistic approach to software development. Students will learn about work items, builds, version control, managing tests, code coverage, executing test runs, reporting and how all these items are part of an integrated solution for Quality Assurance teams and the whole development life cycle process.
This course goes beyond teaching the basics of automated testing and defect management. It covers quality assurance best practices related to analyzing requirements, understanding the best testing approach, authoring effective tests (both automated and manual) and the integration of the quality assurance role within the larger development team.
The course content is a combination of lectures, slides, demonstrations, hands-on labs, group discussions and exercise from instructors who are not only experienced with the Team System tools, but also seasoned veterans of the quality assurance discipline.
ObjectSharp now has a Facebook Group check it out. If your not a Facebooker join in on the fun. It’s a great way to share stuff with friends, family, find old school chums, join groups of interest to you.
A love affair with knowledge will never end in heartbreak. by Michael Garrett Marino
Have a good weekend … Testa [:D]