Предисловие:

5 июня в Риге состоялась 10-я ежегодная конференция специалистов в области тестирования ПО. Предлагаемый материал это презентация сотрудника компании  Exigen Services  Олега Стасюка о методике исследовательского парного тестирования, основанного на сессиях. Этот материал показался нам интересным для публикации. К сожалению, презентация не позволяет  передать весь авторский "закадровый" текст. Однако, мы надеемся она будет полезна нашим читателям.

 

Session Based Exploratory Pair Testing over Skype

 

 

Theory: Pair programming techniques

• Side-by-side (Driver + Observer)
• Remote pair programming
• Ping pong pair programming

Side-by-side (Driver + Observer) (two programmers work together at one keyboard. One types in code while the other reviews each line of code as it's typed in.)
Remote pair programming (is the practice of pair programming where the two programmers comprising the pair are in different locations, working via a collaborative real-time editor, shared desktop, or a remote pair programming IDE plugin.)
Ping pong pair programming (In this pattern, the observer writes unit tests for given user stories and tries to fail unit tests for the code written by the driver, while the driver writes code to pass the observer's unit tests. This loop continues as long as the observer is unable the write unit tests to fail the code for given user stories.)

http://en.wikipedia.org/wiki/Pair_programming

Theory: Pair testing techniques

• Tester + Tester
• Tester + Developer
• Tester + Business Analyst
• Tester + Advanced tester
• Tester + Tester + Skype (the SBEPT over Skype)
• Paired or Multiple testing technique
• Tester after tester (Documentation testing)

Theory: Exploratory Testing

Exploratory testing is unscripted, unrehearsed testing. Its effectiveness depends on several intangibles: the skill of the tester, their intuition, their experience, and their ability to follow hunches.

Why is the ET disliked by managers?

Because they are in the habit of using Scripted Testing. Afraid of risks introduced by implementation new strategy.

"Some companies spend 90% of the test time for test documentation preparation and only 10% on execution"

"Writing test cases upfront takes 4 time more then writing test cases while testing"

 

Theory: Session Based Test Management

A session is an uninterrupted block of reviewable, chartered test effort.

• chartered
• uninterrupted
• reviewable

Session duration - 45 -120 minutes

Session Report:

• Session charter (includes a mission statement, and areas to be tested)
• Tester name(s)
• Date and time started
• Task breakdown (the TBS metrics*)
• Data files
• Test notes
• Issues
• Bugs

*the "TBS" metrics - test design and execution, bug investigation and reporting, and session setup
http://www.satisfice.com/sbtm/

Project Specific data

• SCRUM
• Daily builds
• No detailed requirement
• Often requirement changes

How does "the SBET over Skype" workDuration:
~ 90 minutes - for testers to decide when to finish testing

What is tested?
The same piece of application.

Who tests?
2 or more testers sit by their computers using Skype chat.

What happens to the problems found?
All found problems are First reported in Skype to acknowledge other testers.

Benefits of ‘ the SBEPT over Skype' over ‘Exploratory testing'

2 or more testers sit by their computers using Skype chat
Concurrency (competition)

All found problems are First reported in Skype to acknowledge other testers
Quality of a bug

More Ideas
Communication between testers enforces bringing new and better test ideas
"When one tester must phrase his thoughts to another tester, that simple process of phrasing seems to bring the ideas into better focus and naturally triggers more ideas" - want to delete it.

"SBEPT over Skype" after session and what benefits does it bring to testers

What happens to problems that were found?

The list of problems is sent to developers(s) for check and debriefing.

The problems are divided (with developer's help) into following categories:

Green: Easy bug that can be fixed in minutes. They are expected to be fixed by next build. And are not registered in bug tracking system, but are retested using previous session report.
Defect "time to market" is shortened. Time is not wasted on defect creation but yet retested according to the list. Testers get better picture earlier.

Orange: Already existing defect that was found earlier while testing other part of application. Link to a defect is added the reprot. Defect updated with new info.
Existing defects are fixed by one developer ideally in all places at once and retested in all places at once.

Red: New Defect to be logged as the fix will take more then several minutes. Link to a defect included.

Grey: Not yet implemented, but planned functionality. Or functionality that questions appeared for. Or functionality that was fixed during testing as was found by developers before session started.
Testers get information about the functionality they don't need to look into yet.

Strikethrough: Not a defect as per requirements. After clarifications it was decided that behavior is correct.

 

Duration of the problem
review:
15 - 60 minutes - depends on complexity of problems, is it eye-to-eye communication with possibility of hands-on-problem explanation or over Skype without possibility of connecting using remote desktop.

Some statistics

• Test Session report template preparation ~ 20 minutes (1 person)
• Session time - 70-90 minutes (all participants)
• Problem discussion - 15-60 minutes (all participants)
• Defect creation - 90-120 minutes (all participants)
• Test Session report completion - 30 minutes (1 person)

Why "the SBEPT over Skype" is disliked by management?

What is tested?

The same piece of application/code

Waste of time?
Waste of human resources?
Double (triple) work?

For 2 testers = 20 + (90 + 60 + 120) * 2 + 30 = 590 man-minutes
For 1 tester = 20 + (90 + y) + 60 + (120 * 2) + 30+ z = 440 + y +z man-minutes

y = time to find all defects that were found by the 2nd tester (>90)
z = time to explain other team members

Combining scripted and exploratory testing

It took approximately 80 man/hours to keep Test cases up-to-date with newest requirements and changed requirements. And still defects were missing.

~30 defects were found by exploring the part of application that were not covered by scripted test cases

Defect examples:

On the 4th time when entering incorrect password login is cleared, snarks are brought to the styles ((необъяснимый сбой компьютера) snark)

Possible to change password to something containing tab (by paste from clipboard). However "Forget password" feature fails to send this symbol

Enter all fields -> "Save" button becomes enabled
Clear some fields -> "Save" button becomes disabled
Enter the same data into the cleared fields -> "Save" button is still disabled
If the newly entered values differ from previously entered, "Save" button becomes enabled

I've seen a lot of templates and even test case management tools built-in assumtion that tests must be documented as a plain list of Do-Check things. Do this, validate that. Auditors loves common style tests cases  . But this is not the best way in all cases.