|
|
1 | (62) |
|
1 The Internet of Things, V&V, and Testing |
|
|
3 | (8) |
|
|
3 | (2) |
|
Understanding Chaos, Security, and Other Product Qualities |
|
|
5 | (1) |
|
Why Test IoT Beyond the Device Itself |
|
|
5 | (1) |
|
Agile vs. Traditional vs. "Who Cares About It" Development |
|
|
6 | (1) |
|
|
7 | (1) |
|
|
8 | (2) |
|
Reference Standards, Books, and the Internet: Context Matters |
|
|
8 | (1) |
|
Valuable References to Start Your IoT Test Library |
|
|
8 | (2) |
|
|
10 | (1) |
|
|
10 | (1) |
|
|
10 | (1) |
|
2 IoT Technology in Time and Space |
|
|
11 | (14) |
|
|
11 | (2) |
|
IoT Market Segments -- Where Is IoT Now and in the Future? |
|
|
13 | (2) |
|
A Sampling of IoT Challenges in Development and Testing |
|
|
15 | (3) |
|
IoT Test Team Responses to the Development History of "Testing Is Dead" |
|
|
18 | (4) |
|
Examples of How the IoT Devices Can Impact Testers |
|
|
19 | (3) |
|
|
22 | (1) |
|
|
23 | (2) |
|
|
23 | (2) |
|
3 Big Picture Lessons Learned in IoT Project Test Planning |
|
|
25 | (12) |
|
High-Level Issues at a Glance |
|
|
25 | (3) |
|
#1 Creating and Releasing Software |
|
|
25 | (1) |
|
#2 Understanding the IoT Lifecycle |
|
|
26 | (1) |
|
|
26 | (1) |
|
#4 Avoid Impacts of Rushing in Competition |
|
|
26 | (1) |
|
#5 Advanced IoT Device Challenges |
|
|
27 | (1) |
|
#6 Testing in the Complex World of IoT Systems and Systems of Systems |
|
|
27 | (1) |
|
Project Test Risks: What Are They for IoT? |
|
|
28 | (1) |
|
Getting Started with IoT Test Plans and Strategy |
|
|
29 | (6) |
|
Introducing IoT Verification, Validation, and Testing Concepts and Standards |
|
|
33 | (2) |
|
|
35 | (1) |
|
|
35 | (2) |
|
|
35 | (2) |
|
4 Factors Driving IoT Testing / V&V Selection and Planning |
|
|
37 | (20) |
|
Factor 1 Using Integrity Levels to Drive V&V/Test Planning and Strategy |
|
|
37 | (3) |
|
Factor 2 Risk-Based DevOps and Product Maturity Testing |
|
|
40 | (4) |
|
Risk Analysis Process for Testers |
|
|
41 | (3) |
|
Factor 3 Organizational Ability Impacts on IoT Test Planning |
|
|
44 | (5) |
|
"Newbie" Companies -- Level 1 |
|
|
44 | (1) |
|
Companies with Experience Moving into IoT -- Level 2 |
|
|
45 | (1) |
|
Companies with Hardware (Electronics) Experience -- Level 3 |
|
|
45 | (1) |
|
Companies with Software Experience -- Level 3 |
|
|
45 | (1) |
|
Company with Hardware-Software Experience -- Level 4 |
|
|
46 | (1) |
|
Companies with Systems, Hardware, and Software Experience -- Level 5 |
|
|
46 | (1) |
|
Government Organization -- No Level but a Special Case |
|
|
46 | (1) |
|
Experienced Companies with Only Consumer Product History -- Level 2 |
|
|
47 | (2) |
|
Factor 4 IoT Project Size and Complexity Impacts on Testing |
|
|
49 | (6) |
|
Composite Total Scoring Factors (If You Need to Do That) |
|
|
51 | (1) |
|
How IoT Projects Should Mix and Match Factors |
|
|
52 | (1) |
|
Combining IoT Factors for Better Test Planning |
|
|
53 | (1) |
|
IoT Test/V&V Cost Estimation |
|
|
54 | (1) |
|
|
55 | (1) |
|
|
56 | (1) |
|
|
56 | (1) |
|
5 Beginner Keys for Starting IoT Test Planning |
|
|
57 | (6) |
|
IoT Key 1 Have a Ubiquitous User Interface (UI) |
|
|
57 | (2) |
|
IoT Key 2 Learning from Data Analytics |
|
|
59 | (2) |
|
IoT Key 3 Unique and Specialized Hardware Working to Be a System with the Software |
|
|
61 | (1) |
|
IoT Key 4 Level of V&V/Test Need for "Good Enough" IoT |
|
|
61 | (1) |
|
IoT Key 5 Remaining Agile |
|
|
62 | (1) |
|
IoT Key 6 Testing IoT, Systems, and Large/Complex Software (LCS) |
|
|
62 | (1) |
|
|
62 | (1) |
|
|
62 | (1) |
|
Part 2 IoT Planning, Test, Strategy, and Architecture -- Team Leadership |
|
|
63 | (88) |
|
6 IoT Test Plan: Strategy and Architecture Introductions |
|
|
65 | (18) |
|
|
65 | (2) |
|
Additional Considerations |
|
|
66 | (1) |
|
|
67 | (3) |
|
Good Enough IoT Software and Devices |
|
|
68 | (2) |
|
|
70 | (2) |
|
General Test Planning Outlined by Organization Classification |
|
|
72 | (5) |
|
The Pure Startup, Single Device, and Small Team Who Are Trying to Stay Alive |
|
|
72 | (1) |
|
Mature Groups or Growing Teams Targeting Growth of Sales |
|
|
73 | (1) |
|
IoT Test for a System with More Devices |
|
|
74 | (2) |
|
IoT Test Planning for Procuring Organizations: Governments, Large Corporations, and Others |
|
|
76 | (1) |
|
Impact of AI, Data, and Analytics on IoT Test Planning |
|
|
77 | (3) |
|
Product and Development Lifecycle Impacts on Test Planning -- DevOps and Agile |
|
|
79 | (1) |
|
|
80 | (1) |
|
|
81 | (2) |
|
|
81 | (2) |
|
7 IoT Test Planning and Strategy for Hardware and Software |
|
|
83 | (32) |
|
Traditional Testing Overview |
|
|
83 | (2) |
|
Planning in the Agile/DevOps IoT Test Lifecycle |
|
|
85 | (4) |
|
What Should Development and Test Do During Agile and DevOps? |
|
|
87 | (1) |
|
Ops Team People Skilled in Agile Test Thinking |
|
|
88 | (1) |
|
Planning Using a Hybrid Agile Test Lifecycle |
|
|
88 | (1) |
|
IoT Test Plan Budgeting, Estimating, and Scheduling |
|
|
89 | (7) |
|
Estimating Schedules -- A Brief Introduction to Support Cost Estimates |
|
|
90 | (1) |
|
Estimation of Testing Size |
|
|
91 | (1) |
|
Quality, Verification, Validation, and Testing |
|
|
92 | (1) |
|
|
92 | (1) |
|
Verification and Validation Activity |
|
|
93 | (3) |
|
IoT Test Planning: OTS Hardware and Software |
|
|
96 | (3) |
|
Management Activities of the Coil |
|
|
96 | (1) |
|
Engineering Using the Coil |
|
|
97 | (1) |
|
Software Dev-Test-Sec-Ops Team in the Coil |
|
|
97 | (2) |
|
IoT Test Planning: Security and Critical Quality Factors |
|
|
99 | (1) |
|
IoT Test Planning: Introduction to Software Test Architecture (STA) |
|
|
100 | (8) |
|
IoT Test Planning: IoT Strategy |
|
|
101 | (1) |
|
|
101 | (1) |
|
|
102 | (2) |
|
What Is a Checklist for Strategy Selection? |
|
|
104 | (1) |
|
What Is a Strategy for the Individual IoT Tester? |
|
|
105 | (1) |
|
IoT System V&V Planning: Start with a Combination of Test Strategies |
|
|
105 | (1) |
|
Hardware V&V Strategy (Chapters 8, 12, and 19) |
|
|
106 | (1) |
|
|
106 | (1) |
|
|
107 | (1) |
|
Hardware Test/V&V Planning (IEEE 1012) |
|
|
108 | (1) |
|
Test Planning for Software Systems |
|
|
109 | (1) |
|
IoT System Integration Test Planning |
|
|
110 | (1) |
|
IoT System and System of Systems V&V/Test Planning -- A Conceptual Overview |
|
|
111 | (1) |
|
|
112 | (1) |
|
|
113 | (2) |
|
|
113 | (2) |
|
8 Planning for the IoT Tester on Environments and Testing Details |
|
|
115 | (22) |
|
Assuring the Test Environment |
|
|
115 | (4) |
|
Selecting the Right Test Environment |
|
|
116 | (1) |
|
Planning for Automation in IoT Tool Environments |
|
|
117 | (1) |
|
Data Analytics with Tools |
|
|
118 | (1) |
|
IoT Detailed Project Test Planning (After the Master Test Plans) |
|
|
119 | (3) |
|
Hardware Planning a Tester Should Know |
|
|
119 | (1) |
|
Detailed Software Testing |
|
|
120 | (1) |
|
System Test Plan Patterns a Tester Should Know |
|
|
120 | (1) |
|
Planning Individual Tests (What All Testers Should Do Daily) |
|
|
121 | (1) |
|
Test Planning from Operations to the End Product Life |
|
|
122 | (13) |
|
Test Operations (Ops) Impacts on Test Planning |
|
|
123 | (1) |
|
How Does Ops and Test Planning Change Over Time? |
|
|
123 | (1) |
|
System Maintenance, Security, and Retirement |
|
|
124 | (1) |
|
Ops Maintenance in Test Planning |
|
|
125 | (1) |
|
Planning Retirement and Disposal of an IoT System |
|
|
125 | (2) |
|
Testing Integration Factors |
|
|
127 | (1) |
|
Planning Test IoT Operations and Maintenance (O&M) with Data Analytics and the Edge |
|
|
128 | (2) |
|
Test Planning for Release Deployment |
|
|
130 | (1) |
|
Factory Production: A Very Brief Introduction to Testing IoT Hardware |
|
|
131 | (2) |
|
Risk and Opportunity Management on an IoT Test Project |
|
|
133 | (1) |
|
Last but Not Least, Test Documentation |
|
|
134 | (1) |
|
|
135 | (1) |
|
|
136 | (1) |
|
9 System Engineering Concepts in IoT Test Planning |
|
|
137 | (14) |
|
Reviewing Basic Software Engineering (SE) Concepts |
|
|
137 | (2) |
|
Critical Test Enabling Engineering Support Processes |
|
|
139 | (6) |
|
|
139 | (1) |
|
|
139 | (4) |
|
System Architecture and Design -- Test Top-Level Support |
|
|
143 | (1) |
|
System Planning Trade Study -- Decision Analysis |
|
|
144 | (1) |
|
Designing with Safety for IoT |
|
|
145 | (1) |
|
Hardware Design Considerations |
|
|
146 | (1) |
|
Software Design Considerations |
|
|
146 | (1) |
|
|
147 | (2) |
|
Agile and DevOps Development Impacts |
|
|
149 | (1) |
|
|
149 | (1) |
|
|
150 | (1) |
|
|
150 | (1) |
|
Part 3 IoT Test Designs and Security Assessments |
|
|
151 | (84) |
|
10 IoT Test Design: Frameworks, Techniques, Attacks, Patterns, and Tours |
|
|
153 | (12) |
|
|
155 | (1) |
|
|
156 | (1) |
|
Example 1 Planning Pattern for IoT [ 8, 9] |
|
|
156 | (1) |
|
Example 2 Mind Maps -- Test Patterning Tool for Process Selection for IoT |
|
|
157 | (2) |
|
Example 3 Attacks for IoT -- High-Level Test Design Pattern |
|
|
159 | (2) |
|
Example 4 Test Meta-design Pattern -- Tours for IoT |
|
|
161 | (1) |
|
Applying the Specific Tours to Project Factors |
|
|
162 | (1) |
|
Example 5: Frameworks -- The Top Level of Test Design Planning |
|
|
162 | (1) |
|
|
163 | (1) |
|
|
163 | (2) |
|
11 Classic IoT V&V/Test Concepts, Techniques, and Practices |
|
|
165 | (14) |
|
Moving from Simple Testing to Providing Information on Quality |
|
|
165 | (1) |
|
Techniques, Practices, Levels, and Types of Testing to Apply to IoT |
|
|
166 | (1) |
|
Functional Test Design Techniques |
|
|
166 | (5) |
|
|
167 | (1) |
|
Boundary Value Analysis (BVA) |
|
|
168 | (1) |
|
Scenario (a.k.a. Use Case and Stories) Testing |
|
|
169 | (1) |
|
Syntax of IoT Command Structure Testing |
|
|
170 | (1) |
|
|
171 | (1) |
|
|
172 | (2) |
|
Industrial Test Practices |
|
|
174 | (2) |
|
Math-Based Testing for IoT System and Software |
|
|
174 | (1) |
|
Model-Based Development and Testing/V&V -- A Highly Automated and Integrated Test Practice |
|
|
175 | (1) |
|
IoT Levels of Testing Related to Lifecycle Phases |
|
|
176 | (2) |
|
|
178 | (1) |
|
|
178 | (1) |
|
12 Test Approaches and Quality Assessments for IoT Agile/DevOps |
|
|
179 | (8) |
|
Working Without a Formal IoT Test Plan (or Any Other Plans) |
|
|
179 | (2) |
|
Test Planning for the Agile Teams |
|
|
181 | (2) |
|
Agile Grows and Evolves (and Will Continue to Change) |
|
|
183 | (1) |
|
|
184 | (1) |
|
|
184 | (1) |
|
Agile Quality V&V Assessment |
|
|
184 | (1) |
|
|
185 | (1) |
|
|
186 | (1) |
|
|
186 | (1) |
|
13 IoT Software Security Test Attacks and Designs |
|
|
187 | (22) |
|
Before We Get Started -- Security Data Points and Warnings |
|
|
187 | (1) |
|
Definitions Used in This Chapter |
|
|
188 | (1) |
|
Using Attack Surfaces in Security Test Planning |
|
|
188 | (3) |
|
Planning Security Tests: IoT System Level Inspection and Assessment |
|
|
191 | (1) |
|
Key Activity: Risk and Vulnerability Analysis |
|
|
192 | (2) |
|
IoT Security Threat Risks and How to Find Them |
|
|
193 | (1) |
|
Cybersecurity Team: Test/V&V Supports Developer-Operations (DevOps) Security |
|
|
194 | (10) |
|
Key Activity: Cyber Kill Chain Model |
|
|
194 | (2) |
|
Key Activity: Using a Zero-Trust Security Architecture Model to Support IoT Testing |
|
|
196 | (1) |
|
Key Activity: Software Security Testing Cycle |
|
|
197 | (7) |
|
Historic Security Books You Should Have and Know |
|
|
204 | (1) |
|
Security Test Tool Categories |
|
|
205 | (1) |
|
Tools for Software Security Testing |
|
|
205 | (1) |
|
IoT Testing Tools for Hardware |
|
|
205 | (1) |
|
Software-Defined Security and AI -- An IoT Future |
|
|
205 | (1) |
|
Security Finding Reporting |
|
|
206 | (1) |
|
|
206 | (1) |
|
|
207 | (2) |
|
14 Security OWASP IoT Information Pointer and Logging Events |
|
|
209 | (8) |
|
Intro to OWASP Top Ten Threats (As of 2022) |
|
|
209 | (6) |
|
|
215 | (2) |
|
15 Internal Security Team Penetration Test Process |
|
|
217 | (6) |
|
Pen Test Process: A Beginning |
|
|
217 | (1) |
|
|
217 | (1) |
|
Pen Security Attack and Risk-Based Test Planning for Systems |
|
|
218 | (2) |
|
Full-Scale Penetration of the System |
|
|
220 | (1) |
|
Simplified Process to Perform a Pen Test |
|
|
220 | (2) |
|
|
222 | (1) |
|
|
222 | (1) |
|
16 IoT Test Environment Introduction |
|
|
223 | (12) |
|
|
223 | (1) |
|
|
224 | (1) |
|
IoT Lab/SIL Planning and Requirements |
|
|
225 | (1) |
|
Preferred Test Environment with Full Integration of a Complex IoT System |
|
|
226 | (2) |
|
The Field Test Environment, Analytics, and SIL Working Together |
|
|
228 | (1) |
|
Deep Dive on Test Environment Cost and Schedule Introduction |
|
|
229 | (1) |
|
Test Tool Introduction for IoT |
|
|
230 | (2) |
|
Test Hardware Setup for SIL-Chaos Engineering Support with ZIF Connectors |
|
|
232 | (1) |
|
|
233 | (1) |
|
|
233 | (2) |
|
|
233 | (2) |
|
Part 4 IoT Architectures, Environments, and Integrated Independent Testing |
|
|
235 | (82) |
|
17 Architectures Critical to Project Success |
|
|
237 | (6) |
|
|
237 | (2) |
|
Environments and Architectures for IoT |
|
|
239 | (1) |
|
Architecture Definition of Terms for This Book |
|
|
240 | (1) |
|
Historical Reference: Architecture in Engineering and Literature |
|
|
240 | (2) |
|
|
242 | (1) |
|
|
242 | (1) |
|
18 Overview of IoT Software Architectures: Products and Testing Support |
|
|
243 | (16) |
|
A Quick Look at IoT Architectures |
|
|
244 | (2) |
|
Overview of IoT Support Architectures |
|
|
246 | (1) |
|
Mind Maps of IoT Environments |
|
|
247 | (2) |
|
IoT Software Test Architecture (STA) Introduction |
|
|
249 | (3) |
|
IoT Software Test Architecture (STA) Details |
|
|
252 | (6) |
|
Major Element: Test Plan (a.k.a. STAp) |
|
|
252 | (1) |
|
Major Element: System/Software Under Test (SUT) |
|
|
252 | (1) |
|
Major Element: Test Environment (STAe) |
|
|
253 | (1) |
|
Major Element: Test Model (STAm) |
|
|
254 | (1) |
|
Major Architecture Element: Views, Viewpoints, and Containers (STAv) |
|
|
254 | (4) |
|
|
258 | (1) |
|
|
258 | (1) |
|
19 IoT STA System: Software Integration Lab (SIL) Environments |
|
|
259 | (22) |
|
Environment: Development Team Testing and Integration Support |
|
|
260 | (1) |
|
Environment: Hardware Team Testing and Integration |
|
|
261 | (2) |
|
Environment: Full Hardware-Software-System Integration |
|
|
263 | (3) |
|
A Simple Integration STA SIL |
|
|
263 | (1) |
|
A More Advanced IoT STA SIL with Rapid Integration Reconfiguration |
|
|
264 | (2) |
|
Test Environment Supports: Simulation, Modeling, and Emulation |
|
|
266 | (6) |
|
Full System IoT STA SIL with Simulation and Modeling |
|
|
270 | (2) |
|
Environment: Real-World Full System Software Test |
|
|
272 | (2) |
|
Special Environment: Security Test Sandbox |
|
|
274 | (5) |
|
IoT Chaos Engineering "Live" in the Real World |
|
|
276 | (1) |
|
Level of SIL Environment vs. Project Factors |
|
|
276 | (1) |
|
Bringing STA to a Large-Scale Software Test Architecture/Environment |
|
|
277 | (2) |
|
|
279 | (1) |
|
|
279 | (2) |
|
|
279 | (2) |
|
20 Tools for the Software System Integration Lab (SIL) |
|
|
281 | (6) |
|
|
281 | (1) |
|
Modeling and Requirements Management Tools |
|
|
282 | (1) |
|
Automate, Automate, Automate |
|
|
282 | (1) |
|
Simulation Test Tools Needed to Support IoT |
|
|
283 | (1) |
|
Evolve into Automation Tooling for IoT Success |
|
|
283 | (1) |
|
Evolve IoT Testing with AI Tools |
|
|
284 | (1) |
|
|
284 | (1) |
|
|
285 | (2) |
|
21 Environments for Independent Testing and IV&V on Large IoT Systems |
|
|
287 | (8) |
|
Getting the Most Out of Independent Testing and IV&V |
|
|
287 | (2) |
|
|
288 | (1) |
|
|
289 | (1) |
|
N-Version Testing Supporting Independence and IV&V |
|
|
289 | (1) |
|
Lessons Learned in Testing, in Independence, and IV&V |
|
|
290 | (2) |
|
Tooling Example for Testing Large IoT Systems and IV&V |
|
|
292 | (1) |
|
When to Consider IoT Test Independence and IV&V Environments |
|
|
293 | (1) |
|
|
293 | (1) |
|
|
293 | (2) |
|
22 Self-Organizing Data Analytics (SODA): IoT Data Analytics, AI, and Statistics |
|
|
295 | (22) |
|
|
295 | (1) |
|
|
296 | (1) |
|
SODA Implementation Options Using Stats, Taxonomies, and AI |
|
|
297 | (5) |
|
Use Case Example for SODA and Edge Data Flow |
|
|
302 | (1) |
|
Leveraging SODA for IoT Testing |
|
|
303 | (1) |
|
AI SODA -- A Near Future for IoT/Edge/Cloud Data Analytics |
|
|
304 | (1) |
|
|
305 | (1) |
|
|
305 | (2) |
|
A IoT Supporting Interface, Hardware, Platform, and Protocol Standards |
|
|
307 | (2) |
|
|
309 | (2) |
|
C IoT Testing Startup Checklist |
|
|
311 | (4) |
|
Getting Started with IoT Testing |
|
|
311 | (1) |
|
|
311 | (1) |
|
Compatibility and Integration Testing |
|
|
312 | (1) |
|
|
312 | (1) |
|
External Connectivity Issues |
|
|
312 | (1) |
|
|
313 | (1) |
|
|
313 | (1) |
|
|
313 | (1) |
|
|
313 | (2) |
|
D Example of an IoT System of Systems |
|
|
315 | (2) |
Index |
|
317 | |