Thought on the development stage of a Test Case

From My perspective of the OMG’s UML Testing Profile

Conceptual Design Stage

The System Architecture description models and its sub-systems architecture description models own the test case conceptual design of each requirement having a SysML «Verify» dependency to a TestCase model element in version 1.4 of the language standard.  A Test Case in the UML Testing Profile (UTP) extends the UML metaclasses behavior and operation.  A Test Case ALWAYS returns a Verdict.


Each requirement subject to verification should have such a dependency

The TestCase is named and it has a dependency relationship to its TestObjectiveSpecification model element per the UTP v1.2

A TestRequirement model element may be used to refine the TestObjectiveSpecification model element and act as a bridge between the System of Interest model and the Test System model


Note: The UTP 2 specification addresses maturing concepts in these elements of a test specification

The TestObjectiveSpecification expresses the conceptual design of the TestCase in a natural language statement.  This statement identifies the most important components of the TestCase from the perspective of the Accepting Stakeholder concerns. The components are: pre and invariant conditions of the system under test (SUT) environment, the SUT and its interacting Actors or Systems, their Input(s) to the SUT, the expected behavior of the SUT in response to its inputs.

This collection of components expresses the ‘acceptance criterion’ of the Accepting Stakeholder.  A TestCase, satisfying its TestObjectiveSpecification, will produce objective evidence verifying, to the satisfaction of the Accepting Stakeholder, that the system requirement has been satisfied by the responsible system feature.

The goal of the Test Objective is to articulate an abstract, yet understandable and unambiguous description of the purpose of the TestCase.  The TestObjectiveSpecification does not state ‘How’ to implement the TestCase. The conceptual design treats the TestCase as a “black box”

The Test System Architecture Imports the conceptual design of a TestCase as the foundation to the development of the Test Design Specification for the TestCase


A conceptual test case design may be realized by a collection of test cases by the Test System


Logical Design Stage

The Test Specification of the System Architecture has the responsibility for the Test Design Specification of each conceptual test case imported from the system architecture

A logical design identifies  the components of a test case’s structure and behavior and their relationships.

The logical design activity is an iterative activity ending when a specification can be realized


Test Architecture and high level Test design is performed

Engineering tasks include:  requirements analysis, evaluation, allocation and component specification

The role each component plays in its test case and its test objective responsibilities is defined and traced

A logical design should not specify concrete values for test data properties.  Specify concepts for test data properties (e.g., inbounds, at Boundary, out of bounds, etc…)


Logical properties can be realized as concrete data values through transformation rules

Allocate a TestCase to it’s owning TestContext

A TestContext owns 1 or more TestCases having a common test configuration and a common composited TestObjectiveSpecification

Test component / SUT connections are detailed by the Test Configuration of the TestContext

Typically an internal block diagram when using SysML

Document the execution sequence/schedule of a TestContext’s TestCases

Specify requirements for: the Test Environment, Test Components, test tools, test data, etc…

The Test Case Specification is fully described and can be realized

Concrete Design Stage

The Test Specification of the System Architecture has the responsibility for the Test Case Specification of each test case specified by the Test Design Specification

Define test data pools, partitions and selectors

Detailed design and specification of test data values

Detailed design and specification of the test environment

The physical design of a test case defines component deployment in the test environment

Identify constraints / limitations of the test environment, data flows, or execution infrastructure

The Test Case Specification is complete when an implementation can be realized

Interface Stimulator Services Enterprise

The Interface Stimulator Services Enterprise (ISSE) is a loosely coupled, modular set of simulation/stimulation service components and a core set of management services. The stimulator service components interact with SUT interfaces that normally exchange information with interfaces external to the SUT.  The ISSE provides test capabilities to the integration, verification, and validation life cycle processes. The ISSE is employable with test automation tools (e.g.: Quality Center, etc) and is planned for employment as a component with the system simulation trainer element.


Overview of the Proposed System
The primary objective of the ISSE is to provide information exchange stimulation capability to all information exchange interfaces external to the system. This is a fundamental enabler to basic system integration and verification operations. The interface stimulation components resident in the ISSE provide a web services managed interface to models, simulations, tools, databases, and hardware used to stimulate system’s interfaces. The stimulator service components have loose coupling and management of the stimulator service components is via core infrastructure services.

ISSE Overview
The ISSE is a fundamental enabler of integration and verification activities of system interfaces. Secondary design objectives are; support integration and verification activities by simulating interfaces, support system integration and verification activities by simulating information exchange interfaces external to the system, and support trainer operations by stimulating all segment information exchange interfaces external to the segment.

The design of the ISSE capability set supports evolution to fulfill the operational needs of system data and control interfaces. This expansion of ISSE functionality is largely dependent on evolving program objectives in the area of interoperability validation.
The final ISSE spiral is the trainer product to support training of operators, maintainers, and supervisors. In this deliverable instantiation the ISSE is a component of the trainer element.
Each stimulator component possesses its own web services management interface and the ISSE provides common services to manage infrastructure and the stimulator components. In addition to stimulation services, data logging functionality with time stamping is in the design plan for all stimulator components to support artifact collection automation.
Users (test personnel, instructors, supervisors) can connect and configure the stimulator components (models, simulations, tools, hardware) with specific data and parameter values to compose a stimulation environment to perform a test or conduct a training exercise. The ISSE is capable of archiving service orchestrations for future reference. Test automation tools like HP’s Quality Center can employ service request transactions directly with the ISSE’ core services interface to invoke archived orchestrations.

A design objective is to provide the capability to interface with virtually any model, simulation, tool, database, or hardware. The initial development spiral will only incorporate models and simulations that match the need to integrate and verify system entities. The models and simulations developed for or incorporated into the ISSE will have varying levels of fidelity:
High Appears to be reality from a system perspective. Dynamic and variable behaviors.
Medium system interactions have a visible but limited effect on behaviors.
Low Correctly formatted static data without dynamic variability of behaviors.
To manage cost and risk; functionality and fidelity of stimulator components will not evolve past the point where the components are suitable for integration and verification activities. Low and medium fidelity should suffice for many of the models and simulations. If additional functionality or greater fidelity is required to meet training and operational support objectives, the infusion of additional funding over and above that required for just system integration and verification will be necessary.

Architecture & Design
Employ SOA design patterns and web services to manage simulation / stimulator components and component strings. Employ open standards from standards bodies with demonstrated domain influence. The Object Management Group is a prime example of one such body. Maintain loose coupling of the simulation / stimulator components.
Maintain a focus on the future evolution spiral of simulation / stimulator components and core services of the ISSE. Keep in mind the evolutionary spiral of the trainer, model use in the tactical applications supporting operations planning, and development of distributed test beds for develop-mental/operational test and evaluation (D/OT&E) of the total enterprise.

Background, Objectives, and Scope
The ISSE engineering effort is responsible for the capture of the capability needs of the element, trainer, segment, operational support models, and system. Translate these needs to requirements and design a capability that is evolvable to provide those needs to the foreseen end state. The systems engineering effort is also responsible to identify models for use in the simulator components development (process model) and operation (parametric model). The implementation is limited to the capability set required to integrate and verify the segment.
The segment of interest engages in system data exchanges with entities external to the system. The possible number of external entities exceeds 10,000 instances. The integration and verification activities require information exchange stimulators to succeed in the testing of these interfaces.
COTS tools exist that may satisfy basic integration and verification of interfaces, at the element level, that exclusively engage in SOA system data exchanges at a single interface. In verification situations where coordination of complex exchanges of information occurs at multiple interfaces, existing COTS tools may prove inadequate. This requirement may emerge where system data exchanges at multiple interfaces require orchestration with a complex operational mission scenario. Coordinated scripts may suffice, but they may be subject to high maintenance overhead as scenarios evolve or change.
Realization of the a distributed test bed concept mandates employment of advanced interface stimulator capabilities to bridge segment interfaces to Distributed Interactive Simulation (DIS), High Level Architecture (HLA), or Test and Training Enabling Architecture (TENA) simulation / experiment environments. The complexity of a virtual system of systems experiment environment is unlikely to be supportable using simple scripted system data exchanges.
The objective of this effort is to define these near-term and far-term capabilities and develop only those essential near-term capabilities for segment integration and verification.
Operational Description

Near-Term Capability Description
The ISSE employs a core web services interface. External service consumers interact with this interface to obtain customized stimulator services. Provision for direct access to the stimulator component web services interface is required. This requirement supports component re-use in other infrastructures.
The ISSE employs composable stimulator components. The components feature a web services interface that serves to encapsulate the model, simulation, database, etc and provide for composition of the stimulator component at the environment build time. Modification of the stimulator application at run time is not required. Control of the stimulator component is required during run time. There is a clear distinction between component composition and component control. Composition implies the creation of service chains, links to external data resources or similar configuration of complex behavior models and simulations; actions that are difficult or impossible to orchestrate in real-time. This is different from the simple exposure of a service or process control at the service interface or to proprietary simulation control interface.
The ISSE interfaces to the user’s GUI through the bus’ core web services interface. Infrastructure services support management of the bus and stimulator components, as well as composition of the bus and stimulator components.
The ISSE provides an environment configuration report service. This service provides information relating to stimulator component composition data, model or simulation version data, data base version data, bus orchestration and component deployment data.
The ISSE provides a simulator component composition service. Simulator component composition ser-vice provides service consumer the capability to control those simulation elements exposed to service consumers. This provides a level of service customization.
The ISSE provides a bus orchestration service. This service coordinates the behaviors of the stimulator components.
The ISSE provides a service consumer notification service. An event notification service provided to service consumers.
The ISSE provides a simulator component deployment service. Supports automated deployment of a set of stimulator components.
The ISSE and stimulator components have configurable data monitoring, acquisition, and storage capabilities.
The ISSE supports third party stimulator service requests through its native web services interface. Third party applications may be COTS Test automation tools capable of interacting with a SOA interface.
The ISSE supports interaction directly with a stimulator component by a third party application.
The ISSE supports real-time stimulation of segment interfaces.
The ISSE provides the capability to stimulate segment interfaces in non-real-time as well as modified epoch time reference point.
The ISSE supports automated operation by software test automation tools such as HP Quality Center via the bus’ core web services interface.
The ISSE provides an automated archive service.
Future Capability Description
System program documents convey the concept of developing a “system in the loop” test capability for the system prior to operational deployment. A “test bed” supports the system level test capability.
The concept of a test bed implies that a system or portion of a system is placed in an environment that exercises the system under test (SUT) as if it were in a real world situation. The test bed provides inter-faces to other systems and/or portions of the same system that stimulate the SUT and accept and respond to outputs from the SUT.
In moving closer toward the real world environment, network components that are geographically distributed will comprise the test bed, removing the collocation requirements needed for earlier testing.
To establish the system test and verification environment one must identify a system test bed where the entire set of entities can be assembled and connected in their near operational state. This can be a virtual environment made up of several Integration Laboratories or it can be a physical single site.
The ISSE fits into the above system test environment as a part of the integration lab. The ISSE may evolve into a component of a test bed as testing evolves, allowing the use of actual external systems rather than simulators.
The system distributed test bed concept extends the integration lab to support program objectives in future development spirals of the Interface Stimulation Service Bus.

Automated Archive Service – Archives are analogous to logs. There is a probability that artifacts will be required for the test evidence or debug. This service automates the collection and organization of the data logged by the different interface stimulator components, whatever they may have collected. Wraps it all up in a neat package and may even submit it to a CM service interface.
Bus Orchestration Service – If there is a need to have behaviors synchronized at various interface stimulator components, this is the service that is responsible for this. This service may be or is very tightly coupled to the timing service. In an HLA it is similar to what the RTI is responsible for.
Component Strings a component string is a concept where 2 or more atomic service components are engaged in a service contract to provide a complex service.
Composition the creation of service chains, links to external data resources or similar configuration of complex behavior models and simulations.
Environment Configuration Report Service – captures the test component environment attributes. Tool version, operating system, serial numbers of hardware. Supports re-execution of a test precisely as it was originally instantiated.
ISSE (Interface Stimulator Services Enterprise) – is a loosely coupled, modular set of simulation/stimulation service components and a core set of management services
Service Consumer Notification Service – This is the service that notifies the service consumer about alerts and could even provide the service output.
Simulator Component Composition Service – A service that employs an intelligent agent to assist a service consumer in composing another service. Service that post internal service process models can be customized to a specific consumer’s needs via this service mechanism.
Simulator Component Deployment Service – A service that might be employed to deploy atomic services to a test bed. Conversely, the infrastructure team may deploy the services and then this service is not required.
Stimulation Components Models, Simulations, tools, hardware that provides the stimulus for the external interfaces.
Stimulator Components see Stimulation Components.