1st Edition
Software Test Attacks to Break Mobile and Embedded Devices
Address Errors before Users Find Them
Using a mix-and-match approach, Software Test Attacks to Break Mobile and Embedded Devices presents an attack basis for testing mobile and embedded systems. Designed for testers working in the ever-expanding world of "smart" devices driven by software, the book focuses on attack-based testing that can be used by individuals and teams. The numerous test attacks show you when a software product does not work (i.e., has bugs) and provide you with information about the software product under test.
The book guides you step by step starting with the basics. It explains patterns and techniques ranging from simple mind mapping to sophisticated test labs. For traditional testers moving into the mobile and embedded area, the book bridges the gap between IT and mobile/embedded system testing. It illustrates how to apply both traditional and new approaches. For those working with mobile/embedded systems without an extensive background in testing, the book brings together testing ideas, techniques, and solutions that are immediately applicable to testing smart and mobile devices.
Setting the Mobile and Embedded Framework
Objectives of Testing Mobile and Embedded Software Systems
What Is Embedded Software?
What Are "Smart" Handheld and Mobile Systems?
Why Mobile and Embedded Attacks?
Framework for Attacks
Beginning Your Test Strategy
Attacks on Mobile and Embedded Software
If You Are New to Testing
An Enlightened Tester Makes a Better Tester
Developer Attacks: Taking the Code Head On
Attack 1: Static Code Analysis
Attack 2: Finding White-Box Data Computation Bugs
Attack 3: White-Box Structural Logic Flow Coverage
Test Coverage Concepts for White-Box Structural Testing
Not e of Concern in Mobile and Embedded Environments
Control System Attacks
Attack 4: Finding Hardware System Unhandled Uses in Software
Attack 5: Hardware-to-Software and Software-to-Hardware Signal Interface Bugs
Attack 6: Long-Duration Control Attack Runs
Attack 7: Breaking Software Logic and/or Control Laws
Attack 8: Forcing the Unusual Bug Cases
Hardware Software Attacks
Attack 9: Breaking Software with Hardware and System Operations
Attack 10: Finding Bugs in Hardware-Software Communications
Attack 11: Breaking Software Error Recovery
Attack 12: Interface and Integration Testing
Attack 13: Finding Problems in Software-System Fault Tolerance
Mobile and Embedded Software Attacks
Attack 14: Breaking Digital Software Communications
Attack 15: Finding Bugs in the Data
Attack 16: Bugs in System-Software Computation
Attack 17: Using Simulation and Stimulation to Drive Software Attacks
Time Attacks: "It’s about Time"
Attack 18: Bugs in Timing Interrupts and Priority Inversions
State Modeling Example
Attack 19: Finding Time-Related Bugs
Attack 20: Time-Related Scenarios, Stories, and Tours
Attack 21: Performance Testing Introduction
Supporting Concepts
Completing and Reporting the Performance Attack
Wrapping Up
Human User Interface Attacks: "The Limited (and Unlimited) User Interface"
How to Get Started—the UI
Attack 22: Finding Supporting (User) Documentation Problems
Attack 23: Finding Missing or Wrong Alarms
Attack 24: Finding Bugs in Help Files
Smart and/or Mobile Phone Attacks
General Notes and Attack Concepts Applicable to Most Mobile-Embedded Devices
Attack 25: Finding Bugs in Apps
Attack 26: Testing Mobile and Embedded Games
Attack 27: Attacking App–Cloud Dependencies
Mobile/Embedded Security
The Current Situation
Reusing Security Attacks
Attack 28: Penetration Attack Test
Attack 29: Information Theft—Stealing Device Data
Attack 30: Spoofing Attacks
Attack 31: Attacking Viruses on the Run in Factories or PLCs
Generic Attacks
Attack 32: Using Combinatorial Tests
Attack 33: Attacking Functional Bugs
Mobile and Embedded System Labs
Introduction to Labs
To Start
Test Facilities
Why Should a Tester Care?
What Problem Does a Test Lab Solve?
Staged Evolution of a Test Lab
Simulation Environments
Prototype and Early Development Labs
Development Support Test Labs
Integration Labs
Pre-Product and Product Release (Full Test Lab)
Field Labs
Other Places Labs Can Be Realized
Developing Labs: A Project inside of a Project
Planning Labs
Requirement Considerations for Labs
Functional Elements for a Developer Support Lab
Functional Elements for a Software Test Lab
Test Lab Design Factors
Lab Implementation
Lab Certification
Operations and Maintenance in the Lab
Lab Lessons Learned
Automation Concepts for Test Labs
Tooling to Support Lab Work
Test Data Set-Up
Test Execution: For Developer Testing
Test Execution: General
Product and Security Analysis Tools
Tools for the Lab Test Results Recording
Performance Attack Tooling
Basic and Generic Test Support Tools
Automation: Test Oracles for the Lab Using Modeling Tools
Simulation, Stimulation, and Modeling in the Lab Test Bed
Continuous Real-Time, Closed-Loop Simulations to Support Lab Test Environments
Keyword-Driven Test Models and Environments
Data Collection, Analysis, and Reporting
Post-Test Data Analysis
Post-Test Data Reporting
Wrap Up: N-Version Testing Problems in Labs and Modeling
Final Thoughts: Independence, Blind Spots, and Test Lab Staffing
Some Parting Advice
Are We There Yet?
Will You Get Started Today?
Advice for the "Never Ever" Tester
Bug Database, Taxonomies, and Learning from Your History
Lessons Learned and Retrospectives
Implementing Software Attack Planning
Regression and Retest
Where Do You Go from Here?
Appendix A: Mobile and Embedded Error Taxonomy: A Software Error Taxonomy (for Testers)
Appendix B: Mobile and Embedded Coding Rules
Appendix C: Quality First: "Defending the Source Code So That Attacks Are Not So Easy"
Appendix D: Basic Timing Concepts
Appendix E: Detailed Mapping of Attacks
Appendix F: UI /GUI and Game Evaluation Checklist
Appendix G: Risk Analysis, FMEA, and Brainstorming
References
Glossary
Index
Biography
Jon Hagar is the principal (CEO/CTO) and senior software test engineer at Grand Software Testing. For over 30 years he has worked on systems and software engineering, specializing in testing/verification and validation. He is the lead editor on ISO/IEC/IEEE29119 Software Test Standard, a member of the IEEE1012 V&V Plan working group, and co-chair on the OMG UML testing profile standard. Jon holds a patent on web test technologies and has published numerous articles on software reliability, testing, test tools, formal methods, and embedded systems. He has a B.S. in mathematics with a specialization in civil engineering and software from Metropolitan State College in Denver, Colorado, and an M.S. in computer science with a specialization in software engineering and testing from Colorado State University.
"Using the framework of attacks popularized by James Whittaker’s books, Jon Duncan Hagar describes those that are relevant here and extends the approach with new attacks specifically for mobile and embedded systems. He provides detailed information and guidance on how to test more effectively and efficiently in the mobile and embedded world. … Jon shows you what to test and how to test, giving ideas that you can use to do better testing of mobile devices now and save yourself serious trouble later on.
Jon’s extensive experience (much of it in the embedded world), his thorough research, and his deep knowledge give this book a solid foundation and provide helpful guidance and steps to take in applying testing attacks to mobile and embedded devices."
—From the Foreword by Dorothy Graham, Software Testing Consultant"Every tester who wants to keep current needs to read this book, and you can read with confidence knowing you are being guided by the best in this business. … Learn by doing, with this book as your guide."
—From the Foreword by Lisa Crispin, Agile Testing Coach and Practitioner
