A Tool View
The objective of a Test activity is to produce information about a structural or behavioral property of a test item. Figure 1 illustrates associations of several elements of the UML metamodel core infrastructure and the SysML Requirement model element. Test components realize the behavior of a test case and elicit from the SUT a behavior. The context of the SUT’s behavior and the manifestation of its behavior form objective evidence of conformance to a requirement.
Figure 1 – Requirement, Feature, Test Case Triad
Evaluating test information produces an assessment of a quality attribute of a test item. Testing is a necessary activity throughout a product’s development life cycle, as mistakes may occur in all stages of the development life cycle. Mistakes translate into product defects. Defects negatively impact a product’s quality and diminish the value perception of the acquirer.
Test is a sub-process of the ISO/IEC 15288:2008 technical processes Integration, Verification and Validation. A “Test System”, Figure 2, has the responsibility to provide test services to a “System of Interest” during its life cycle. These test services produce the information to assess the quality of the system of interest and the value proposition for the acquirer.
Figure 2 – Enabling Systems
Quality is an attribute that has value to an acquirer. Quality factors are diverse and most require testing to assess, which has diverse methods itself. Figure 3 illustrates a Test Method model. Test represents a substantive component of a product’s cost. The quest for perfection is a noble vision, though seldom is perfection valued at its cost. These realities drive a test system to be inherently effective and efficient and to produce the greatest product quality coverage within given project constraints. Achieving efficiency and effectiveness is driven in large part by test automation and the tools that provide the automation infrastructure elements of a Test System.
Figure 3 – Test Method Concepts
Figure 4 presents the use cases where the System of Interest employs the services provided by the Test System to achieve goals of its technical development processes. The three prime uses cases represent execution of the integration, verification and validation technical processes during the life cycle of the System of Interest. Each one of these use cases has unique properties in its underlying service composition, though all employ core test services.
Figure 4 – Test System Services
Test System Architecture
The architecture of a Test System[i] consists of numerous system elements. Figure 5 – Figure 7 are excerpts from the ISO/IEC 15288 standard describing a system. Figure 7 is an example of an architecture description of a notional Test System in a program context. Our focus is the tool components/elements of a test system architecture.
Figure 5 – System Structure Metamodel
Figure 6 – System Structure Hierarchy
Figure 7 – Test System Architecture in a Notional Context
Tools are “devices for doing work”, according to the Encarta Dictionary. They have an intended use and when used accordingly, they increase the effectiveness and efficiency of an activity. Frequently, they replace mandraulic[i] task elements with automated task elements. The ubiquitous ‘Hammer’ has been relegated to the tool chest by the ‘Air or Battery Powered Nailer’, in the construction industry. As a process execution resource, a powered nailer is clearly far more expensive than the hammer it replaces,
yet it transforms a mandraulic and highly variable task to a repeatable automated one that produces immense value with the scale of its use in the home construction industry. If only one nail needs to be driven, the clear choice is the hammer, but scale the task to 100, 1,000 or 10,000 nails and the return on investment is obvious.
Let us view work as a process activity responsible to transform inputs to outputs; parts to an assembly or possibly, and more abstractly, a problem to a solution. Work should produce value. The output of the process should have a greater value than the sum value of its inputs and expended resources. When inputs are concrete and the problem deterministic, rarely does a recurring process relying on mandraulic tools achieve the return on investment that an automated tool provides. Our focus is on how these tools, as elements of a test system’s infrastructure, add value to a program.
The execution of the ‘Integration’, ‘Verification’ and ‘Validation’ technical processes[i] of a system’s life cycle processes falls largely to the test system, an enabling system, employed by a test organization, as a resource, to accomplish test sub-processes. Figure 8 illustrates ‘Process Concepts’ that are fundamental attributes of all ISO/IEC 15288:2008 defined process models. Appreciate the role a tool has as a resource performing an activity of a process. Either a person or a tool may be assigned a role, both are resources.
Figure 8 – ISO/IEC 15288:2008 Process Concepts
Figure 7 contains a number of tools key to the test system architecture, some (e.g., configuration, build, requirement, change, and defect management tools) provide services to the test system while others are responsible for processes that are performed by the test system (e.g., test management, test execution, reporting). At the system level of a project, the execution of the integration, verification and validation technical processes consumes significant resources. An anecdotal accounting of this resource consumption suggests more resources are consumed planning, managing execution and providing test status than the actual execution of the test procedures. Clearly, our test system architecture requires a competent tool infrastructure capable of off-loading mandraulic tasks from the test organization’s staff. What tasks are well suited to automated tools? The test system supports integration of system/sub-system elements into capabilities. Clearly, a Test Architect will benefit from a test generation tool supporting the development, planning, and management of a functional release plan from the system’s design artifacts. Test engineers define integration procedures where the definition of test cases for the innumerable message combinations and permutations is a formidable book keeping task, tools are particularly adept solving this test generation problem and the category of combinatorial analytics tools are adept at coping with the test case expansion problem created by test generation tools. Perhaps the key problem facing the test system at the system level of integration, verification and acceptance test processes is the sheer number of information elements that must be managed; clearly the test system’s tool infrastructure has this problem as its key objective. This is a problem that computer based tools are particularly adept at and deliver a considerable value benefit.
Test System Services
As illustrated in Figure 9, the test system owns a test service process element, which is treated as a behavioral property of the block. Test service is a sub-process representing capabilities delivered by a collection of process activities. Many of these activities are accomplished by test automation tools directly or through the interaction of an engineer performing a role of the test organization and a test automation tool.
Figure 9 – Test Sub-process Activity Composition
Process Activity Definitions
Test Requirements Analysis
Test requirements analysis activities produce a specification for the Test System’s test architecture to implement. The specification entail test design, test case, test procedure and test data requirements. The specification addresses both structure and behavioral features of test architecture.
Test Data Analysis
Test data analysis activities produce specification of data employed by test cases. Test data may take the form of a test input, test output or test oracle data assertion.
Test planning activities specialize the project processes defined by ISO/IEC 15288 for the test domain. Planning activities address managing resources and assets that realize or support the test system. Tools supporting planning provide insight to resource conflicts, costs, milestones and schedules.
Test tracing activities produce coverage maps of the test architecture specification.
Test generation activities produce concrete test implementations realized from abstract test specifications
Test management activities are closely related to their peer project management process activities. Core tasks are estimation, risk analysis and scheduling of test activities. The latter task is a typical capability of a test management tool. Test management tools frequently possess an extensive portfolio of capabilities (e.g., requirements verification status reporting, status dashboards, defect metrics, test complete/not complete, etc…).
Test execution activities are responsible for implementing the test management plan, test specification (i.e., test script specification, test procedure specification)
Test reporting activities include test reports, test logs, test data analysis reports, etc. Test reporting tools typically output to stakeholder dashboards.
A Reference Architecture
The Test System’s tool infrastructure abstracts tools into tool categories. These categories are: Test Management, Test Execution, Status Dashboards, Test Data Analysis, Test Reporting, Defect Reporting, Test Generation, Requirements Management, Change Management, Configuration Management and Build Management; it is not the intent for this list to be exhaustive. Not all of these tools are contained in the Test system architecture, though the Test System relies on services provided by these tools to automate key process activities.
Figure 10 – Generic System Level Tool Infrastructure
[i] A Test Systems is an Enabling System, as defined by ISO/IEC 15288:2008 Systems and software engineering – System life cycle processes. A test system provides support to the system of interest during its full lifecycle. Specifically, it provides test services in the form of test sub-processes to the lifecycle technical processes of Integration, Verification and Validation. These technical processes apply across the system hierarchy (i.e., component, system element, system) as well as the level of test (i.e., component, integration, system, and acceptance). See IEEE 829-2008 IEEE Standard for Software and System Test Documentation.
[i] ISO/IEC 15288-2008 Systems and software engineering – System life cycle processes Pg 12 Figure 4 Clause 6.4.5, 6.4.6, and 6.4.8
[i] Mandraulic – an informal term used as an adjective meaning ‘labour intensive’ according to en.wiktionary.org