Download as PDF, PPTX
Uploaded bySTAG Software Private Limited
Hypothesis Based Testing: Power + Speed.
The document discusses hypothesis based testing (HBT) as a powerful methodology for improving software testing speed and effectiveness by leveraging intellect and lean principles. It outlines the structured process of HBT, focusing on cleanliness criteria, potential defect types, and quality levels while providing a case study that demonstrates the effectiveness of HBT over conventional approaches. Key benefits include increased test coverage, reduced post-release defects, and improved efficiency in testing processes.
More Related Content
Similar to Hypothesis Based Testing: Power + Speed.
Hypothesis Based Testing: Power + Speed.
- 1. Hypothesis Based Testing Power+ Speed T Ashok Founder & CEO, STAG Software Architect-HBT in.linkedin.com/in/AshokSTAG ash_thiru Webinar: 16 Feb 2012, 1400-1500 EST
- 2. Tools and processhave been significant drivers to speed up testing. Adoption of lean principles is seen as enabling focus & continuous evolution. Our appetite for speed is yet to be satiated. So where do go from here? Leveraging intellect is key to more power and speed. Hypothesis Based Testing (HBT) is a personal scientific test methodology that focuses on leveraging one’s intellect by enabling sharp goal focus, providing tools for scientific thinking to rapidly assess cleanliness of software. © 2012. STAG Software Private Limited. All rights reserved.
- 3. The “Tsunami” effect... 4 3 2 1 Speed up Leverage Asset Optimize process execution Intellect reuse Power + Speed © 2012. STAG Software Private Limited. All rights reserved.
- 4. The “Tsunami” effect... 4 Cleanliness criteria 3 2 Potential Defect Types 1 Quality Levels Speed up Leverage Asset Optimize execution Intellect reuse process Complete test cases Cleanliness Index Validation suite Power + Speed © 2012. STAG Software Private Limited. All rights reserved.
- 5. Clarity of Purpose Requirements, Features, System User story to satisfy Clear Goal may have Cleanliness criteria Potential Defect Type impacts “What to Test” & “Test for What” © 2012. STAG Software Private Limited. All rights reserved.
- 6. Expectations = CleanlinessCriteria Expectations “Properties of the system” Needs Features Environment Expectations delivered Behavior by Needs (Requirements) Structure via Features that display Behavior Material constructed from Materials in accordance to a Structure in a given Environment © 2012. STAG Software Private Limited. All rights reserved.
- 7. Quality Growth -Nine staged filter Objective Issues That user expectations are met L9 End user value User flows, experience That it deploys well in the real environment L8 Clean Deployment Compatibility, migration That the stated attributes are met L7 Attributes met Performance, security, volume, load... That it does not mess up the environment L6 Environment cleanliness Resource leaks, Compatibility... That end-to-end flows work correctly L5 Flow correctness Business flow conditions, Linkages That the functional behavior is correct L4 Behavior correctness Functionality L3 Structural integrity That the internal structure is robust Internal structural issues That the user interface is clean L2 Input interface cleanliness UI issues That inputs are handled well L1 Input cleanliness Input data handling © 2012. STAG Software Private Limited. All rights reserved.
- 8. HBT - Personalscientific test methodology Powered by STEMTM - Defect detection technology EIGHT disciplines of THINKING D8 SIX stages of DOING S6 D1 S1 powered by S1: Understand expectations D1: Business value understanding S2: Understand context D2: Defect hypothesis S3: Formulate hypothesis D3: Strategy & Planning S4: Devise proof D4: Test design S5: Tooling support D5: Tooling S6: Assess & Analyze D6:Visibility D7: Execution & Reporting D8: Analysis & Management Power + Speed © 2012. STAG Software Private Limited. All rights reserved.
- 9. D1 Business valueunderstanding D2 Defect hypothesis D3 Test strategy & planning Landscaping EFF model Orthogonality principle Viewpoints (Error-Fault-Failure) Tooling needs assessment Reductionist principle Defect centricity principle Defect centered AB Interaction matrix Negative thinking Quality growth principle Operational profiling Orthogonality principle Techniques landscape Attribute analysis Defect typing Process landscape GQM D4 Test design D5 Tooling Reductionist principle Automation complexity Input granularity principle 32 core assessment Box model Minimal babysitting Behavior-Stimuli approach concepts principle Techniques landscape Separation of concerns Complexity assessment Tooling needs analysis Operational profiling D6 Visibility D7 Execution & Reporting D8 Analysis & Management GQM Contextual awareness Gating principle Quality quantification Defect rating principle Cycle scoping model © 2012. STAG Software Private Limited. All rights reserved.
- 10. Hypothesis Based Testing(HBT) S1 Cleanliness criteria Potential defect types S3 S2 Staged & purposeful Expectations detection S4 Complete test cases S6 Goal directed measures Sensible automation S5 © 2012. STAG Software Private Limited. All rights reserved.
- 11. Cleanliness criteria Potential defect types Clear Baseline S1, S2 Staged & purposeful detection Expectations Set a clear goal for quality Complete test cases Example: Clean Water implies Goal directed measures Sensible automation 1.Colorless 2.No suspended particles 3.No bacteria 4.Odorless What information(properties) can be used to identify this? ...Marketplace,Customers, End users ...Requirement(flows), Usage, Deployment ... Features, Attributes ...Stage of development, Interactions ... Environment, Architecture ... Behavior, Structure © 2012. STAG Software Private Limited. All rights reserved.
- 12. Cleanliness criteria Potential defect types A goal focused approach S3 to cleanliness Staged & purposeful detection Identify potential defect types that Expectations can impede cleanliness Complete test cases Example: Data validation Timeouts Goal directed Sensible automation Resource leakage measures Calculation Storage Presentation Transactional ... Scientific approach to hypothesizing defects is about looking at FIVE Aspects - Data, Logic, Structure, Environment & Usage from THREE Views - Error injection, Fault proneness & Failure Use STEM core concepts > Negative thinking (Aspect) > EFF Model (View) “A Holmes-ian way of looking at properties of elements” © 2012. STAG Software Private Limited. All rights reserved.
- 13. Cleanliness criteria Potential defect types Levels, Types & Techniques - STRATEGY Staged & purposeful detection S4 NINE levels to Cleanliness Expectations L9 End user value Complete test cases L8 Clean Deployment Goal directed L7 Attributes met Sensible automation measures L6 Environment cleanliness Quality Levels L3 L5 Robustness PDT7 PDT6 L4 Behavior correctness L2 PDT5 Test Techniques (T1-T4) PDT4 L1 PDT3 TT5 T4 L3 Structural integrity PDT2 PDT: PDT1 Potential Defect Types TT4 L2 Input interface cleanliness T3 TT3 L1 Input cleanliness Test Types PDT7 TT5 PDT6 TT2 T2 TT4 PDT5 PDT4 TT1 T1 TT3 PDT3 TT2 PDT2 TT: TT1 PDT1 Test Types “Fractional distillation of bug mixture” © 2012. STAG Software Private Limited. All rights reserved.
- 14. Cleanliness criteria Potential defect types Countable test cases & Fault coverage Staged & purposeful detection Expectations Use STEM Core concepts > Box model Complete test cases > Behavior Stimuli approach S4 > Techniques landscape Goal directed > Coverage evaluation Sensible automation measures to - Model behavior - Create behavior scenarios - Create stimuli (test cases) Test Scenarios/Cases R1 PDT1 Irrespective of who designs, #scenarios/ TS1 TC1,2,3 R2 PDT2 cases shall be same - COUNTABLE TT R3 TS2 TC4,5,6,7 PDT3 Requirements & Fault traceability That test cases for a given requirement shall have the ability to detect specific types of defects Guarantee test adequacy. FAULT COVERAGE Guarantee implies that the means to the end is rational & provable © 2012. STAG Software Private Limited. All rights reserved.
- 15. HBT Test CaseArchitecture Organized by Quality levels sub-ordered by items (features/modules..), Level segregated by type, ranked by importance/priority, Item sub-divided into conformance(+) and robustness(-), classified by early (smoke)/late-stage evaluation, Type tagged by evaluation frequency, linked by optimal execution order, classified by execution mode (manual/automated) Priority Focus Stage A well architected set of test cases is like a effective bait that can ‘attract‘ defects in the Frequency system. Order It is equally important to ensure that they Mode are well organized to enable execution optimisation and have the right set of information to ensure easy automation. © 2012. STAG Software Private Limited. All rights reserved.
- 16. Cleanliness criteria Potential defect types Focused scenarios + Good Automation Architecture Staged & purposeful detection Expectations Level based testcriteria Cleanliness scenarios yield Complete test cases shorter scripts that are more flexible for change and S5 easily maintainable. Goal directed Sensible automation measures L9 End Expectations user value L8 Clean Deployment L7 Attributes met L6 Environment cleanliness L5 Robustness L4 Behavior correctness L3 Structural integrity L2 Input interface cleanliness L1 Input cleanliness © 2012. STAG Software Private Limited. All rights reserved.
- 17. Cleanliness criteria Potential defect types “Cleanliness Index” - Improved visibility Staged & purposeful detection L4 Expectations PDT10 TT8 Complete test cases PDT9 TT7 S6 L3 PDT9 TT6 Cleanliness PDT8 TT5 Goal directed Sensible automation measures PDT7 TT4 L2 PDT6 PDT5 TT3 L1 PDT4 PDT3 TT2 PDT2 PDT1 TT1 Stage Quality report CC1 CC2 CC3 CC4 R1 Met R2 Not met R3 Partially met R4 R5 © 2012. STAG Software Private Limited. All rights reserved.
- 18. HBT - ACase Study Two Teams, one using HBT & The other conventional approach Test case details Test case details (HBT) Module HBT Normal Increase Module Total Positive Negative M1 100 28 257% M1 100 59 41 M2 85 52 63% M2 85 68 17 M3 95 66 44% M3 95 67 28 M4 132 72 83% M4 132 112 20 M5 127 28 354% M5 127 85 42 M6 855 116 637% M6 855 749 106 TOTAL 1394 362 285% TOTAL 1394 1140 254 Nearly 3x increase in #test cases increasing probability of higher defect yield 2x improvement in negative cases increasing probability of better defect yield Defect details Effort details (person-hours) HBT Normal Increase Stage HBT Normal #Defects 32 16 100% Test analysis & 30 20 Design 20 (Major), 12(Minor) Out of these 32 defects, few were Front loading of effort resulted residual defects, one being critical to in lowering support cost corrupt the entire data. © 2012. STAG Software Private Limited. All rights reserved.
- 19. HBT Results 50%-1000% reduction in post-release defects Re-architecting test assets increases test coverage by 250% 30% defect leakage reduction from early stage Smart automation - 3x reduction in time Deskilling - Less experienced staff do better, faster ramp up, lower cost ‘Holes’ found & fixed at requirement stage Test assessment accelerates integration, de-risks deployment © 2012. STAG Software Private Limited. All rights reserved.
- 20. Summarizing... 4 Cleanliness criteria 3 2 Potential Defect Types 1 Quality Levels Speed up Leverage Asset Optimize execution Intellect reuse process Complete test cases Cleanliness Index Validation suite Power + Speed © 2012. STAG Software Private Limited. All rights reserved.
- 21. Thank you! Follow us @stagsoft Check out our blog at www.stagsoftware.com/blog © 2012. STAG Software Private Limited. All rights reserved.