Requirement Elicitation

The process of discovering the requirements for a system by communication with stakeholders: customers, system users and others who have a stake in the system development.

Requirements Gathering Techniques

• Methodical extraction of concrete requirements from high level goals
• Requirements quality metrics:

Ambiguity

The requirement contains terms or statements that can be interpreted in different ways.

Inconsistency

The requirement item is not compatible with other requirement.

Forward referencing

Requirement items make use of a domain feature that is not yet defined.

Opacity

A requirement item where rationale or dependencies are hidden.

Noise

A requirement that yields no information on problem world features.

Completeness

The needs of a prescribed system are fully covered by requirement items without any undesirable outcome.

Problem World and Machine Solution

The problem to be solved is rooted in a complex organizational, technical, or physical world.

• The aim of a software project is to improve the world by building some machine expected to solve the problem.
• Problem world and machine solution each have their own phenomena while sharing others.
• The shared phenomena defines the interface through which the machine interacts with the world.

Requirements engineering is concerned with the machine's effect on the surrounding world and the assumption we make about that world.

Descriptive vs. Prescriptive

Descriptive statements

State properties about the system that holds regardless of how the system behaves. E.g. if train doors are open, they are not closed.

Prescriptive statements

State desirable properties about the system that may hold or not depending on how the system behaves. Need to be enforced by system components. E.g. train doors shall always remain closed when the train is moving

Formulation of Requirements Statements

Statement scope: Phenomenon of train physically moving is owned by environment. It cannot be directly observed by software phenomenon. The phenomenon of train measured speed being non-null is shared by software and environment. It is measured by a speedometer in the environment and observed by the software.

Formulation of System Requirement

A prescriptive statement by the software-to-be.

• Possibly in cooperation with other system components
• Formulated in terms of environment phenomena

Example: All train doors shall always remain closed while the train is moving.

In addition to the software-to-be, we also require the cooperation of other components:

• Train controller being responsible for the safe control of doors.
• The passenger refraining from opening doors unsafely
• Door actuators working properly

Formulation of Software Requirement

A prescriptive requirement enforced solely by the software-to-be. Formulated in terms of phenomena shared between the software and environment.

The software "understand" or "sense" the environment through input data.

Example: The doorState output variable shall always have the value 'closed' when the measuredSpeed input variable has a non-null value.

Domain Properties

A domain property:

• Is a descriptive statement about the problem world
• Should hold invariably regardless of how the system behaves
• Usually corresponds to some physical laws

Example: A train is moving if and only if its physical speed is non-null.

Example case: We shall make software for a toll road gate. The system's transmitter sends a signal to an approaching car which returns the car-owner's id. The payment is invoiced to the car-owner.

• Assumption: "The car returns the car-owner's id". This is the responsibility of a single agent in the environment of the software-to-be
• Requirement: "The system's transmitter sends a signal to an approaching car". This is the responsibility of a single agent in the software-to-be

Background

Modeling Goals

Generally modelled by intrinsic features (type, attributes etc.) and their links to other goals.

Specifying Goals

Must be specified precisely. Semi-formal specifications generally declare goals in terms of their type, attributes, and links. May be provided using textual or graphical syntax.

Often includes keyword verbs with predefined semantics, example from KAOS:

Achieve

Corresponding target condition should hold some time in the future

Maintain

Corresponding target condition should always hold in the future unless some other condition holds

Avoid

Corresponding target condition should never hold in the future

This basic set can be extended with qualative verbs, such as improve, increase, reduce, make etc.

Reasoning About goals

Goal refinement

A mechanism for structuring complex specifications at different levels of concern.

A goal can be refined in a set of sub-goals that jointly contribute to it.

Each sub-goal is refined into finer-grained goals until we reach a requirement on the software and expectation (assumption) on the environment.

NB: Requirements on software are associated with a single agent and they are testable.

GNL

• Free text requirement elicitation with the assistance of prescribed words from a dictionary. Gives us consistent requirements, reducing misunderstandings.
• No formal constraints
• Requires minimal expertise

Aim:

• Bridge the gap between unconstrained expression and quality checking when representing requirements as free text. Quality measures: Correctness, consistency, completeness and un-ambiguity.
• Provide the basis for semantic processing and checking of requirements.
• Dictionary -- simple taxonomy or formal ontology

Ontology = Thesaurus + inference rules

• Thesaurus -- Domain concepts: entities, terms and events
• Inference Rules - Relations, attributes and axioms
• Causality, similarity, reflexivity, transitiveness, symmetric, disjoint (contradiction)

Motivation for Use of Templates

Text = unconstrained expression. However, there is a need for a common

• Understanding of concepts used to express the requirements and relations between them
• Format of presentation

Lack of common understanding, make free text requirements prone to ambiguous representations and incosistencies.

Templates introduces some limitations, but reduces possibility of ambiguities and incosistencies.

Boilerplates

A set of structures that can be used to write requirements. They use high-level concept classification and attributes. A boilerplate consists of fixed terms and attributes. It may, or may not, contain one or more modes. They

• Provide an initial basis for requirements checking
• Are easy to understand for stakeholders compared to more formal representations

RE process using boilerplates:

1. Select a boilerplate or a sequence of boilerplates, based on attributes that need to be included and how they are organized -- fixed terms.
2. If needed, identify and include mode boilerplates
3. Instantiate all attributes

Goals

• Validation and verification
  • Finding and removing conflicts between requirements
  • Completeness of requirements
    • Derived requirements cover higher level requirements
    • Each requirement is covered by part of the product
• System inspection
  • Identify alternatives and compromises
• Cerfification/Audits
  • Proof of being compliant to standards

Challenges

• Traces have to be identified and recorded among numerous, hetrogeneous entity instances (document, models, code, etc.). It is challenging to create meaningful relationships in such a complex context.
• Traces are in a constant state of flux, since they may change whenever requirements or other development artefacts change.
• A variety of tool support
  • Based on a traceability matrix, hyperlink, tags and identifiers
  • Still manual with little automation
• Incomplete trace information is a reality due to complex trace acquisition and maintenance.
• Trust is a big issue: Lack of quality attribute
  • There is no use of the information that 70% of trace links are accurate without knowing which of the links forms the 70%

Different stakeholders have different viewpoints:

• QA management
  • How close are we to our requirements", "What can we do better" to improve quality
• Change management
  • Tracking down the effect of each change to each involved component that might require adaptations to the change, recertification or just retesting to proof functionality
• Reuse
  • Pointing out those aspects of a reused component that need to be adapted to the new system requirements
  • Even the requirements themselves can be targeted for reuse
• Validation and verification
  • Traceability can be used as a pointer to the quality of the requirements
    • Completeness, ambiguity, correctness/noise, incosistency, forward referencing, opacity
  • Ensures that every requirement has been targeted by at least one part of the product
• Certification/Audit
• Testing
• Maintenance

Meta-models

Meta-models for traceability are often used as the basis for the traceability methodologies and frameworks:

• Define what type of artifacts should be traced.
• Define what type of relations could be established between these artifacts.

E.g.:

Approaches

The critical task of traceability is to establish links between requirements and other requirements, as well as between requirements and artifacts. It's time consuming and error prone to that manually, so we focus on semi-automatic link generation.

• Scenario driven by observing runtime behavior of test scenarios, but has problems:
  • Semi-automated, but requires a large amount of time from engineers to iteratively identify a subset of test scenarios and how they're related to requirement artifacts
  • Requirements that are not related due to no not matching execution paths may be related due to e.g. calling, data dependencies implementation pattern similarity.
• Trace by tagging: Easy to understand and implement, but depends heavily on human intervention
  • How
    • Each requirement is given a tag, manually or by tool
    • Each document, code etc. are marked with a tag which tells which requirements it belongs to
  • Challenges
    • Quality depends on if we remember to tag artifacts
    • Possible to check if everything is tagged, but not if the tagging is correct

Requirements Testability

The capability of the software product to enable modified software to be validated.

In order to be testable, a requirement needs to be stated in a precise way. Sometimes this is in place right from the start: When the ACC system is turned on, the "Active" light on the dashboard shall be turned on.

Sometimes the requirement has to be changed: The system shall be easy to use.

Ways to check if our goals have been achieved:

• Executing a test. Give input, observe and check output.
  • Black box
  • White box
  • Grey box
• Run experiments
• Inspect the code and other artifacts

Concerns

Need to be considered together:

• How easy is it to test the implementation?
• How test-friendly is the requirement?

When to use what?

T

Tests. Input/output. Involves the computer system and peripherals.

E

Experiments. Input/output but involves the user as well.

I

Inspections. Evaluation based on documents.

Challenges

• Volume of tests needed, e.g. response time or storage capacity
• Type of event to be tested, e.g. error handling or safety mechanisms
• The required state of the system before testing, e.g. a rare failure state or a certain transaction history

Requirements for testability

The customer needs to know what and why he wants something. The why-part is unfortunetaly not often stated as part of a requirement.

Each requirement needs to be

• Correct, i.e. without errors
• Complete, i.e. has all possible situations been covered?
• Consistent, i.e. not in disagreement with other requirements
• Clear, i.e. stated in a way that is easy to read and understand - e.g. using a commonly known notation
• Relevant, i.e. pertinent to the system's purpose and at the right level of restrictiveness
• Feasible, i.e. possible to realize (difficult to implement = difficult to test)
• Traceable, i.e. it must me possible to relate it to one or more
  • Software components
  • Process steps

Key principles for agile requirements:

• Active user involvement is imperative
• Agile teams must be empowered to make decisions
• Requirements emerge and evolve as software is developed
• Agile requirements are 'barely sufficient'
• Requirements are developed in small pieces
• Enough's enough -- apply the 80/20 rule
• Cooperation, collaboration and communication between all team members is essential

Written vs. Verbal Requirements

Written:

• Pros:
  • Can be well thought through, reviewed and edited
  • Provide a permanent record
  • Are easily shared with groups of people
• Cons:
  • Time consuming to produce
  • May be less relevant or superseded over time
  • Can be easily misinterpreted

Verbal:

• Instantaneous feedback and clarification
• Information-packed exchange
• Easier to clarify and gain common understanding
• Easily adapted to any new information at the time
• Can spark ideas about problems and opportunities

User Stories

Seek to combine the strengths of written and verbal communication, where possible supported by a picture.

• They are concise, written descriptions of a piece of functionality that will be valuable to a user (or owner) of the software.
• They are
  • User's needs
  • Product descriptions
  • Planning items
  • Tokens for a conversation
  • Mechanisms for deferring conversation

Should be detailed enough

• For the team to start working from
• To establish further details and clarifications at the time of development

Prioritized in a backlog.

Agile Challenges

• Active user involvement can be demanding on the user representative's time and require commitment for the duration of the project.
• Iterations can be a substantial overhead if the deployment costs are large.
• Agile requirements are barely sufficient:
  • This can mean less information available to new startes in the team about features and how they should work.
• Usually not suitable for projects with high developer turnover or a long-term maintenance contract.

What

• Aims to identify possible misuse scenarios of the system.
• Describes the steps of performing a malicious act against a system.
• Mostly used to capture security and safety requirements.

Example:

Textual representation:

Why

Used in three ways:

• Identify threats -- e.g. "system fails to set alarm"
• Identify new requirements -- e.g. "system shall have two independent alarms
• Identify new tests -- e.g.
  • Disable one of the alarms
  • Create an alarm condition
  • Check if the other alarm is set

Pros and Cons

Helps us

• Focus on possible problems
• Identify defenses and mitigations

but can get large and complex, especially the misuse case diagrams.

UCM paths

Architectural entities that describe causal relationships between responsibilities, which are bound to underlying organizational structures of abstract components. They are intended to bridge the gap between requirements (use cases) and detailed design.

UNSW Experiments

Three experiments run in the same way: 120 student were given code with seeded defects.

Introduce two terms:

Nominal group (NG)

Number of defects identified by the group members before the inspection meeting

Real group (RG)

Number of defects identified by the group members after the inspection meeting

=> RG - NG = the effect of the inspection meeting

The experiment found that there was a clear correlation between how many people discovered the defect individually and how likely it was that it would be reported after the group meeting:

• If nobody found the defect during individual inspection, there was a 10% probability that the group would find it.
• If one person found the defect, there was a 50% probability that it would be reported after the group meeting.
• If more than two persons found the defect, the probability of reporting rose to between 80% and 95%.

This happened not only when the group voted, but also happened due to group pressure.

Numbers tell that there is a 53% probability of process loss by using inspection meetings as opposed to only a 30% probability of process gain.

How can this be improved?

Other numbers showed that while RG performed significantly better than NG when it came to finding new defects, they also performed significantly worse when it came to removing defects. Conclusively the experiments showed that when we experience process

• Loss, few new defects are found (12%), but many old ones are removed (44%)
• Gain, a lot of new defects are found (42%), and few of the old ones are rejected (16%)
• Stability, some new defects are found (26%) but appx. as many are removed (28%)

This leads to the question why are are already identified defects removed, and why are so few new ones found? Answer: Voting process or group pressure.

NTNU Experiments

Experiment 1 Concerned with group size and use of checklists. Run with 20 students, two phases: Individual inspection and group inspection. Half of the group used a tailor-made checklist, half used an ad-hoc approach. The code was 130 lines of Java, with 13 seeded defects.

Individual results:

• Checklist group: 8.4 defects on average
• Ad-hoc group: 6.7 defects on average

Group meetings were done in different sizes, 2, 3 and 5 participants. By comparing the groups with 2 and 5, together with whether they used checklists or not, it was found that the groupsize effect was the only influential effect on success rate. Both checklist effect and checklist-group size interaction effect were negligable.

Careful on conclusion, as few participants.

Experiment 2 Focuses on the influence of experience and the effect on three common types of defects - wrong code, missing code and extra code. Only individual inspections, with a supplied checklist. As observed in experiment 1, this has a beneficial effect on small groups. 21 persons with high experience (PhD students), 21 persons with low experience (third and fourth year students).

Conclusion:

• Missing code: High experience better
• Extra code: Low experience better
• Wrong code: Equal

Surprise: Those with low experience found in total more defects (5.5 vs 5.1). The difference, however, is not significant.

Closer look shows that low experience group had more recent hands-on experience in writing and debugging Java code. High experience group focused more on the overall functionality, and did not really read all the code. Thus, they missed simple, but low level defects like missing keywords.

Defects that had no relations to the checklist, or were described late in the checklist, were harder to find (fatigue effect).

Coverage Categories

Program based

Concerned with coverage related to code

Specification based

Concerned with coverage related to specification or requirements

If a test criterion has finite applicability, it can be satisfied by a finite test set. Generally, test criteria are not finitely applicable. Main reason = possibility of "dead code". By relating test coverage criteria only to feasible code, we make them finitely applicable.

Thus henceforth, all branches = all feasible branches, and all code = all feasible code.

Why

• Learn the product
• Connect testing to documented requirements
• Expose failures to deliver desired benefits
• Explore expert use of the program
• Make a bug report more motivating
• Bring requirement-related issues to the surface
  • Reopening old requirements discussions (with new data)
  • Surfacing not-yet-identified requirements

Type 1

Scenarios used as to define input/output sequences. Similar to requirements elicitation.

Type 2

Scenarios used as a script for a sequence of real actions in a real or simulated environment

How

Rules:

• List possible users - analyze their interest and objectives
• List system events. How does the system handle them?
• List special events. What accomodations does the system make for these?
• List benefits and create end-to-end tasks to achieve them
• Work alongside users to see how they work and what they do
• Read about what systems like this is supposed to do
• Create a mock business. Treat it as real and process its data

'Soap operas'

Build a scenario based on a client/customer experience, but exaggerate each aspect of it:

• For each variable, substitute a more extreme value
• If a scenario can include a repeating element, repeat it lots of times
• Make the environment less hospitable to the case (increase or decrease memory, printer resolution, etc.)

Create a real-life story that combines all of the elements into a test case narrative.

Users

The prospective users - those who will later use the system.

For each identified user, identify his interests. A user will value the system if it furthers his interests.

Focus on one interest at a time. Identify the user's objectives.

System Events

Any occurrence that the system is supposed to respond to.

For each event, we need to understand

• Its purpose
• What the system is supposed to do with it
• Rules related to the event

Special Events

Events that are predictable but unusual, and require special handling.

Triggered under special circumstances.

Benefits

What are the benefits the system is supposed to provide to the users?

Ask the stakeholders, but watch out for

• Mistunderstandings
• Conflicts

Risks

• Not good for testing new code, as we have to postpone the rest of a failed test until after the bug is fixed.
• Acceptance testing mainly
• Functionality testing, not code coverage
• Discover design errors, not coding errors

Using v(G)

v(G) = the minimum number of paths through the code.

v(G) < number of paths < 2**|{predicates}|

Decision Table

Technique to achieve full path coverage, but will often lead to over-testing.

1. Make a table of all predicates
2. Insert all combinations of true/false for each predicate
3. Construct a test for each combination

Only practical for

• Situations where we have binary decisions
• Small chunks of code -- e.g. class methods and small components

Loops

Make tests for going through each loop

• 0 times
• 1 time
• 5 times, and
• 20 times

Error Messages

1. Identify all error conditions
2. Provoke each identified error condition
3. Check if the error is treated in a satisfactory manner (error message is clear, to the point and useful)

Static vs. Dynamic

• Static =>
  • Code inspection
  • Code walkthrough
• Dynamic =>
  • Statement coverage
  • Path coverage
  • Decision coverage
  • All define -- use coverage

Binder's State Control Faults

A list of common state-related problems in software systems, which may be used as an input to

• State based testing
• State machine or code inspection

List:

• Missing or incorrect
  • Transitions -- new state is legal but incorrect
  • Events -- valid message ignored
• Extra, missing or corrupt state -- unpredictable behaviour
• Sneak path -- message accepted when it shouldn't be
• Illegal message failure -- unexpected message causes a failure
• Trap door -- system accepts undefined message

State Test Criteria

Choose one or more of the following test selection criteria:

• All states -- testing passes through all the states
• All events -- testing forces all events to occur at least once
• All actions -- testing forces all actions to be produced at least once

State test strategies

All round-trip paths

• All transition sequences beginning and ending in the same state
• All simple paths from initial to final state

This strategy will help you find

• All invalid or missing states
• Some extra states
• All event and action faults

Type 1

1. Write a chunk of code
2. Write a set of tests
3. Test and correct until the test suite runs without errors
4. Change a random part of the code -- e.g. a "+" to a "-". This is called a code mutant, we only consider mutants that compiles without error messages
5. Run the test suite again
6. If the test suite
   • Runs without errors => Extend the test suite until we discover the defect
   • Diagnoses the defect => Go back to step 4 and create a new mutant

Test is done when all of X new mutants are discovered by the current test suite.

Type 2

Also called fuzzing, many ideas in common with random testing. Main differences are:

• Random testing requires us to generate random tests from scratch
• Type 2 mutation testing starts with an input that works OK and then change part of it in a random way

Basic Scenario Pattern (BSP)

Identify a precondition and a postcondition for each test, then:

1. Generate input to make precondition
   1. False => nothing happens
   2. True => input activation time
2. Check system response
   1. Timeout = true => fails
   2. Timeout = false and postcondition OK => OK
   3. Timeout = false and postcondition not OK => fails

Key-Event Service Pattern (KSP)

Identify a precondition, a key-event and a postcondition for each test, then:

1. Generate key event
   1. Not right event => wait
   2. Right event => set activation time
2. Check precondition
   1. Not OK => wait
   2. OK => check post condition
3. Check postcondition
   1. Timeout = true => fails
   2. Timeout = false and postcondition OK => OK
   3. Timeout = false and postcondition not OK => fails

Timed Key-Event Service Pattern (TKSP)

Identify a precondition, a key-event, a duration and a postcondition for each test, then:

1. Generate key event
   1. Not right event => wait for new event or duration expired
   2. Right event => set activation time
2. Check precondition
   1. Not OK => wait
   2. OK => check postcondition
3. Check postcondition
   1. Timeout = true => fails
   2. Timeout = false and postcondition OK => OK
   3. Timeout = false and postcondition not OK => fails

BSP Example

1. Illegal command
   1. Generate something
   2. Nothing happens => test OK postcondition = "doors unlocked" => test fails
2. Legal command
   1. Generate "alarm disabled"
   2. Activation time = T
   3. Timeout = True => test fails Timeout = False postcondition = "doors unlocked" => test OK