Is testing 5 users enough? CHI 2003 Session Notes

The panel on  sample size in usability tests at CHI 2003 was thought provoking and entertaining.  Commentary follows the notes:

• Jared Spool
• Points out that testing content dramatically increases the size of the usability problem set, requiring way more than 5 participants.
  
• CUE-2 (Roif Molich)
  
• Test of hotmail.com. 75% of problems identified by one team.
• Must be goal driven.
• Sell usability to colleagues
• Drive iterative development
• Find problems
• Rebuttal: (Carol Barnum)
  
• Not followed "discount model"
• Open ended scenarios, a set of 57
• No user profile (diverse users)
• No interaction with sponsor/client
• Gilbert Cockton
  
• Five users generates inaccurate problem frequency measures.
• How many users is the wrong question.
• System complexity is an obvious but neglected aspect.
• Dennis Wixon
• Goal is not to find errors, but to get them fixed (and verified)
• Test-fix-iterate
• Ethnography to assess generalizability
  

One of the strongest points was that sample size is directly related to the way in which your team chooses to feed results back into the development process.  At Microsoft, they do rapid small N iterations, testing, fixing, and testing the fixes.  In many cases, it is impractical for an organization to work this way -- and almost certainly in the case of hiring a UIE or N&N to do testing on your product. 

|||
Posted at 7:17, Published in: Web Usability Testing