Acceptance Criteria – The criteria a stakeholder employs in the assessment of a product feature to assert that the feature satisfies their need. A feature of a product is specified by a system requirement that requires verification of the feature’s implementation for the product to be accepted by the stakeholder.
The question to the stakeholder – “What objective evidence, produced by the test case, will convince you that the feature specified by the requirement has been satisfied?”
The response could identify a scenario, an instance of a Use Case, and their desired outcome of the Use Case Scenario. It may be a set of input conditions to the product and a measurable outcome, meaning the input and output can be formally defined.
A product requirement expresses a precise attribute set of a product feature. Typically, these attributes are framed as desired behavior. These feature attributes may be expressed in terms functionality, inputs, outputs, actions, sequence, qualities and constraints.
Frequently, requirement statements lack the precision to unambiguously produce evidence through test case execution that the product complies with its specification, a collection of requirements. Requirements are frequently expressed implicitly rather than explicitly. Ambiguous characteristics frequently results in conflict during acceptance testing as a result of emergent requirement statement interpretation.
An approach to clarify the intent of imprecise requirement statements is to author acceptance criteria for these requirements which explicitly define the criteria that support the assertion that the product satisfies its specification. The acceptance criteria of a functional requirement might be expressed using a model-based behavior specification.
Employ explicit acceptance criteria early in the product development period and obtain stakeholder agreement with it. Agreement on acceptance criterion should be established at the requirements review phase of a project. Ideally, requirements should focus on the problems requiring solutions in the stakeholder domain rather than focusing on the system solution. Stakeholders must be held accountable for their requirement statements and agree to an acceptance criterion for each and every requirement prior to the commencement of high level system design. This precludes discord late in the program where the stakeholder’s satisfaction is critically important.
An approach to developing acceptance criteria is to ask questions from the user/stakeholder viewpoint. These criterions will answer the stakeholder question “How do we know the product meets its specification?”
What measured or observed criteria will convince stakeholders that the system implementation satisfies the requirement? Is there a specific operational situation where a system’s features will provide a user a capability which accomplishes a user mission objective? Is there an output provided by a system feature that satisfies a stakeholder criterion that can be measured and subjected to a stakeholder standard?
Answering these questions provides a foundation for test case development. The first answer forms the system context for a test case. What objects in the system’s environment are interacting with it and how are they interacting?
The second answer provides the test oracle that an arbiter uses to assert if the test case has “passed” or “failed”. Acceptance criteria define the successful outcome of a test case. Each requirement should have an acceptance criterion statement. Criterion must be measurable, either qualitatively or quantitatively. Where the measure is qualitative, it is imperative to reach agreement on defining this subjective evaluation.
These answers drive the test strategy which culminates in a demonstration or test that produces the evidence that satisfies the stakeholder’s expectation, thereby establishing the acceptance criterion for the requirement.
Acceptance criteria must be explicitly associated with a requirement and acknowledged formally by the stakeholder as being adequate criterion.
The acquirer’s acceptance criteria for a product should be stated in the acquirer’s product specification at the time of their request for a proposal. In the US DoD MIL-STD 961E section 4 of the system’s specification contains the acceptance criteria in the form a test method and a test specification for all requirements in section 3. If the acquirer of the product has not stated acceptance criterion for all requirements in their product specification, then the proposal must contain scoping acceptance criteria to ensure that the acquirer understands what will be delivered by the proposal both in terms of a product and the evidence that the product satisfies their need as stated in the product’s specification. In any event, the acceptance criterion for every stakeholder requirement must be stated at the product’s requirement review milestone and acknowledged as acceptable by the acquirer of the product. Delaying the establishment of acceptance criteria levies a significant risk of “scope creep”. As the design matures and its capabilities begin to be revealed; the acquirer, or the acquirer’s representatives, may realize that they may have provided a product specification which will not fully satisfy their needs and the acceptance criteria is at risk of becoming far more costly to achieve and verify.
IEEE Std 829TM-2008 IEEE Standard for Software and System Test Documentation calls for the development of a Test Traceability Matrix. This matrix is responsible for establishing the association between each system requirement and the test responsible for producing the evidence that satisfies the requirement’s acceptance criterion. This matrix has a hierarchy which parallels the requirement decomposition hierarchy of system, sub-systems and components.
The need for requirement acceptance criteria does not end at the acquirer’s specification. All engineered requirements need acceptance criteria. The acceptance criterion unambiguously informs the next lower tier in the engineering hierarchy of their stakeholder’s expectations.
The incoming acceptance criteria are a principle driver of the test strategy at that level of the product’s feature realization hierarchy.
Acceptance criteria such as “successful” are qualitative. In this example there is a need to quantify “Success”. We can measure “profitability” by assessing the “return on investment” and stating that if each dollars investment returns a 10% return, the standard used to assert a verdict, then success has been achieved. Success might also be measured by the quantity of process cycles performed in a period of time.
Yes, these quantitative criteria are measures applying to non-functional requirements. The point being is that by measuring the performance of a function, which may be a derived performance measure, there is implicit verification of the functional requirement statement.
The acceptance criterion defines the domain the product is employed in. It describes how the product’s features are measured or assessed. Acceptance criterion may express constraints to be imposed on the product when its features are in use. Acceptance criterion forms the test case development framework. The acceptance criteria statements will explicitly communicate our interpretation of stakeholder requirements.
The current SysML standard, as well as the UML 2 Testing Profile, do not address ‘acceptance criteria modeling’ directly or by inference. A description of a Use Case scenario with an accompanying User goal and satisfaction of outcome criteria expressed in a classifier seems required. The UTP specification of a ‘Test Context’ which is both a structuredClassifier and a behavioredClassifier seems fit for purpose. However, the SysML does not include the ‘Test Context’ in its profile; rather it only includes the ‘Test Case’ which is an ‘Operation’ or ‘Behavior’ meta-class.