Explore each pillar inside the System Test Framework. Every tile reflects the new structure and links to a dedicated chapter ready for your content.
Click any tile to read the full chapter content.
Quality is the foundation of System Test and the link to development.
Quality is something everyone naturally strives for. The term 'quality' originates from the Latin word 'qualitas', coined by the Roman philosopher Cicero in the 1st century BCE. He derived it from the Latin adjective 'qualis', meaning 'of what kind'.
The origin of systematic quality efforts lies in guilds, which are professional associations. Craftsmen were responsible for producing quality themselves. If they did not meet the quality requirements of customers, they were excluded from the guild.
Nowadays quality is a competitive advantage. Excellent organizations deliver excellent products with highest quality.
Quality Definitions
Quality is the cross-cutting principle that serves as a foundation for System Test. It is an essential and integral part of development.
Quality Management (QM) is the overarching framework for managing all quality related activities.
Quality Management Systems (QMS) are systems to manage quality. They typically include quality policies, objectives, planning, control, assurance, and improvement. QMS may fulfill standards like ISO 9001.
The basis of quality management is the PDCA cycle:
Products have both inherent and assigned characteristics:
Only inherent characteristics are quality indicators.
Standards and norms stimulate global trade, guarantee a specific quality level and facilitate clear communication between partners.
The ISO 9000 is a bundle of international quality management standards with several principles:
Quality can be seen as a mindset or attitude. Every individual has an understanding of quality and within an organization a certain responsibility for it. System Test applies quality for a specific project or function.
It is important to find the right degree of quality efforts. Too high quality requirements can slow the design phase. Too many resources in quality effort can make the test phase slow and expensive. Any imbalance also leads to conflicts with development.
The Test Pyramid distributes test activities in three layers:
From bottom to top: number of tests decrease, cost increases, feedback speed decreases, effort increases, automation difficulty increases.
The V-Model (Verification and Validation Model) is one of the most popular models for System Test of embedded complex systems. It has three layers:
PDCA Cycle: Plan-Do-Check-Act (Deming)
DMAIC Method: Define-Measure-Analyze-Improve-Control (foundation for Six Sigma)
The setup is the basis for the System Test with Test Management and other constructs.
If you have a new system to test, one has to start with the setup. It builds and is the basis for the test activities. Test Management is responsible for the setup of System Test — not only done initially, but improved and adjusted constantly.
Test Management handles the strategic planning, organization, monitoring and execution of all test related activities.
The starting point from the idea to test that leads to all that follows. It is the agency of the individual responsible for System Test.
For effective Test Management the structure of the test organization must be clear, including test concepts, test plans, organizing the Test Domains and leading quality management efforts.
A group of individuals working interrelated on the test activities. Can be embedded in a project team or larger organizational structure.
The work executed by the test team follows a defined workflow through test management and test domains to produce test related output.
Physical and digital equipment to enable test work — test benches, test tools, etc.
The subject that is tested. In System Test: components that interact together and features provided by the system. It is important to exactly specify what is the Test Object.
The Test Concept outlines the approach of testing the system. It gives guidance on the landscape of test possibilities and how testing for a specific system is done.
Quality attributes to consider: functionality, usability, reliability, performance, safety, security, privacy, compatibility, portability, maintenance.
Black-Box: Equivalence Partitioning, Boundary Value Analysis, Decision Table Testing, State Transition Testing, Use Case Testing
White-Box: Statement Coverage, Branch Coverage, Path Coverage, Condition Coverage
Experience-Based: Error Guessing, Exploratory Testing, Checklist-Based Testing
Defines the conditions of test execution: analytical, methodical, expert-based, reactive, model-based, process-compliant, change-based, regression-averse strategies.
The Test Plan documents the general information about the system in the test context and the specifics about testing. It is the architecture of system testing for a specific test object.
Master Test Plan: high-level concept, generic test plan for a project, created during early phase, basis for other plans.
Ensures every document from specification to test case and every test activity is linked. The main path is from requirement to test case — bidirectional linkage allows forward and backward traceability.
Complex systems bring inherent risks: interface errors, dependency failures, resource constraints, variant configuration mismatches, environmental disturbances.
Key methods:
Only one direct responsible individual that has influence to the workings of the System Test and the Test Team.
System Test execution is done in different domains. The domains contain the operational work.
Input: reasons to test (Test Trigger) → Doing: convert into needed test activities → Output: Test Execution Plans
Prioritize certain tests. Always test from general/major features to specific/minor features. Corner cases and negative use cases should be tested afterwards.
Definition of Ready: preconditions for test execution beyond test case specifics.
Definition of Done: criteria to end tests — completeness, handling of aborted/failed tests, time-based closure.
Input: requirements → Doing: convert into test cases/test sets → Output: executable test cases
Anything that influences how the product must, should or may behave: requirements, use cases, expert discussions, KPIs, regulations.
Smoketests, bug retests, functional tests, use case tests, performance tests, usability tests, acceptance tests, interoperability tests, regression tests, assertions testing.
Input: project environment → Doing: create up-to-date test infrastructure → Output: usable test environment
Simulates the physical world around the system. Used for autonomous driving tests to simulate vehicle dynamics and environment.
Tests software components in a simulated environment rather than on real hardware. Usually a predecessor to System Test.
Input: test execution plans → Doing: execute tests → Output: test results
Verification checks if the behavior matches the specification. Validation has a broader view and also reviews the specification itself. Holistic System Test incorporates both.
If the system behaves as expected the test passed, if not the test failed. For each failed test case an issue must be raised.
A well-documented issue is the foundation for smooth issue resolution. Essential information includes:
Input: issue reports → Doing: handling issues → Output: issue resolution
Simplest form: New → Progress → Closed. Can expand with: Triage, Pre-Analysis, Rejected, Review, Accepted, Fixed, Resolved, Retest.
Priority groups: Blocker, High, Medium, Low
Severity groups: Critical, Major, Moderate, Minor, Trivial
Screening → Pre-Analysis → Triage → Tracking → Monitoring → Resolution → Retest
Input: test results → Doing: data analysis and visualization → Output: report
Human resources and responsibilities for roles.
The Test Team is a group of individuals responsible for all test activities, working in an interrelated way towards a test related goal.
Administration: setting up System Test, defining test concepts, writing strategic test plans, composing the team, providing leadership.
Operation: operational coordination, test planning, test specification, test infrastructure, test execution, problem management, reporting, monitoring and control.
With increasing project size, roles can split by Domain expertise: Test Infrastructure, Test Planning, System Integration, etc.
Critical Thinking, Curiosity, Proactiveness, Adaptability, Perseverance, Team Orientation, Integrity, Skepticism, Accountability.
Analytical Thinking, Systematic Approach, Attention to Detail, Communication, Time Management, Prioritization, Risk-Based Thinking.
Test Design Skills, Test Automation, Tool Proficiency, Requirement Analysis, Bug Reporting.
Organizes and leads the test team. Acts as the head and takes responsibility for the outcomes of test activities.
Organizing: ensuring the team does things right, resource management, communication with the project.
Leading: ensuring the right things are done, motivation, feedback, vision for the test team.
Leadership is crucial for culture, motivation, performance and commitment. An effective leader is a social architect who understands individuals, groups and organizational behavior.
Key motivational measures: job design, training, participation, choice, transparent rules, connections, meaningful work, safety, proper selection.
Organizational justice types: distributive (fair outcomes), procedural (fair processes), interactional (respectful communication), cultural (equitable treatment across backgrounds).
Foster a culture where quality is in the center: strive towards quality, excellent work, foster quality improvement, take ownership and accountability.
Tooling for System Test operations.
Test Tools help with test activities or even enable them. There are three core test tools for System Test.
Store, organize, and manage test cases in a central system.
Track and manage issues, defects, and bugs throughout the development and testing lifecycle.
Execution establishes workflows into context for test performance.
During a project phase, system test can face different modes of operation.
Happens in the early stage when the product is developed. Things change quickly. Close cooperation between developer and tester is required. Test cases are created along with the requirements.
First-time integration of new components may happen in an integration event, where developer, tester and experts together test and evaluate.
Once all components and features are integrated successfully. Test activities are more structured with proper planning and documentation. The goal is to verify the system towards the product launch.
After the product is launched. Ensures products in the market still work. Important for complex embedded systems that depend on other systems, interfaces or software updates. Problems at this phase have direct customer impact.
Control and monitoring of ongoing test activities is vital:
Keeping test activities on track across all domains: Test Planning, Test Infrastructure, Test Execution, Problem Management, and Reporting.
A constant effort on a day-to-day basis. Gets increasingly important with complexity and agility. Requires overview of the whole product under test and direct control over test team resources.
Handling emergencies, coping with stress, creative problem-solving, managing uncertain scenarios, acquiring new knowledge, interpersonal and cultural adaptability.
The ability to evolve and transform. Feedback loop: Observe → Understand → Adjust → Stabilize.
Resilience emerges when adaptability and flexibility are balanced.
If development is agile, Quality Assurance and System Test have to manage it. Agile development brings challenges in terms of quality — changes are inherent risks. Working in agile projects of complex systems requires versatility.
Scaling factors: variants, test cases, test environment. Includes System-of-Systems Testing (SoS Testing) where multiple independent complex systems interact.
Development: develop for stability, reduce dependencies, clear interfaces, use bugs as feedback loops.
Test: stable test team, good documentation, on-boarding/off-boarding, systems and processes in place.
Test automation for sustainable and effective test execution.
Automating complex systems presents unique challenges due to the interaction of multiple hardware and software components.
Have the right ideology when it comes to automation. Automate everything that makes sense — and what makes sense is sometimes difficult to determine.
A modular approach with well-defined components for automation.
There is a spectrum: on one hand, unit tests that are not possible to do manually. On the other hand, complex systems where their testing cannot be automated.
The scope here is limited. More by dedicated Test Automation Engineering.
How everything interrelates after all components have been explained — the holistic view of the System Test Framework.
System Test Framework relates to established standards and frameworks such as ISO standards and other industry-specific quality frameworks.
System Engineering Principles:
Sustainability and Quality are interconnected domains that can mutually reinforce organizational objectives. Many quality objectives, like durability, are directly linked with sustainability.
Environmental Management Systems (EMS) can be integrated in established Quality Management Systems (QMS). Operationally there are many overlaps as quality tools and methods can be applied to sustainability goals.
Zero Waste Engineering: a concept which theoretically increases productivity to infinity because every product or component could be reused or recycled.
Universe → Existence → Growth → Development → Quality
Just as Product Design seeks to create beauty in products, Product Quality seeks to establish the character of a product.
There is a transcendent approach which views quality as an 'innate excellence' — an absolute and universally recognizable attribute that defies precise definition. Almost like 'beauty'. It is understood through experience and represents striving to the best at any given point in time and space.
Quality seems to be something that humans perceive and then put into a cognitive and rational context. It feels like quality as a perception grows with human progress. Perhaps the ultimate goal is that the concept of quality diminishes when development transcends to some higher form.