What is software engineering?

• Software are computer programs and their associated documentation. They may be developed for a specific customer (customized products) or the general market (generic products).
• Software engineering is an engineering discipline concerned with all aspects of software production, from system specification to maintenance. This is opposed to computer science (which is more concerned with theories and fundamentals). It usually consists of four activities:
  • Software specification: What will it do?
  • Software development: Making it do it.
  • Software validation: Does it do it?
  • Software evolution: Modify the software after release to reflect new requirements.
• System engineering is more concerned with the whole. This includes aspects such as hardware and process engineering. Software engineering is a part of this process.

Why is it called engineering?

Software engineering is an engineering practice because it, like all other engineering disciplines, applies theories selectively to discover new solutions to problems within given financial constraints (hopefully). Good software engineering takes extra effort, but saves a lot of money because the software is more maintainable.

The process is concerned not only with the technical aspects, but also the surrounding factors, such as project management, choosing a good team and the right development methods.

Roughly 60% of software development costs are related to development and 40% go to testing.

The general attributes of software

Good software should strive to meet these requirements. It is not possible to meet all requirements to their fullest (e.g: higher security might mean less efficiency).

What affects software engineering?

A lot of software is bad (and not in the Michael Jackson way). This is due to a lot of factors, such as:

• Increasing demand: The world wants software, but sometimes faster than what anyone could possibly deliver.
• Low expectations: Developers aren't used to using good methods and develop poor software. Users are used to mediocre results.
• Bad development methods: Factors such as customer contracts (or simply incompetence or fear of novelty) prevents teams from choosing the appropriate development methods. Certain development methods, such as the waterfall method, is old; it has a lot of important uses, but agile methods might in many cases be better.

Software engineering ethics

You should always follow these ethical guidelines.

Software process models

The book looks at three models.

Process activities

Coping with change

Change is inevitable. And it is expensive, because of rework (replacing already finished, but now irrelevant, parts). There are two approaches to reducing these costs:

1. Change avoidance: The development includes activities that can anticipate possible changes before significant rework is required. Prototyping is an example of this.
2. Change tolerance: The process is designed to accommodate changes at relatively low cost – normally involving incremental development.

The Rational Unified process

A modern process model that has been derived from the work on UML and the Unified Software Development process. A hybrid model.

The RUP models from the business perspective rather than the technical. There are four phases:

1. Inception: Establish a business case for the system. Identify all external entities (people and systems) that will interact with the system and define these interactions. Use this information to assess the contribution to that the system makes to the business. If this is minor, the project might be cancelled after this phase.
2. Elaboration: Develop an understanding of the problem domain, establish an architectural framework for the system, develop the project plan, and identify key project risks. This should result in a requirements model for the system (perhaps UML cases), an architectural description, and a development plan for the software.
3. Construction: System design, programming, and testing. On completion of the phase, one should have a working software system and associated documentation that is ready for delivery to users.
4. Transition: Moving the system from the development community to the user community and making it work in its real environment. On completion, one should have a fully working software system with complete documentation.

It also has some static workflows. These are activities that are separated from the phases – you have to do them when you see fit. See the book for a list.

Six fundamental best practices are recommended:

1. Develop software iteratively.
2. Manage requirements: State them explicitly.
3. Use component-based architectures.
4. Visually model new software: Using UML models.
5. Verify software quality.
6. Control changes to software.

Agile methods

Agile methods were a response to huge overheads involved in plan-driven development. The basic principles are outlined in the Agile manifesto (very short; required reading).

The principles of agile methods:

Agile methods have been very succesful for some types of system development:

• Product development where a software company is developing a small or medium-sized product for sale.
• Custom system development within an organization. This makes the customer commit to the development process.

Agile methods are not well suited for critical and complex systems. It also requires all members to have a relatively high skill level.

Things that may come in the way of agile development: - Customers are not able or willing to participate. - Team members aren't comfortable for intense involvement. - Prioritizing changes can be difficult. - Maintaining simplicity requires extra work. - It is hard to gain a contract for agile development, because customers are skeptical; they like (or are required by superiors) to know what they buy up front.

Extreme Programming

Extreme Programming (XP) got its name because it pushed known agile practices to its extreme.

In XP, requirements are expressed as scenarios, called user stories, which are written on story cards. You can read more about this in agile planning.

During the planning game, questions that require further exploration are uncovered. This may require a spike, an increment in which no programming is done. Usually, this involves prototyping, documentation or designing the system architecture.

In a normal increment, the scenarios are divided into tasks and are taken on by programmers, who work in pairs. Tests are always developed before code, and code must execute all tests successfully to be implemented into the system.

Some advantages of pair programming:

• People feel collective ownership of and responsibility for the code.
• Code is reviewed and refactored while it written.
• For inexperienced programmers, it has been shown to be just as effective as programming alone. (Productivity drops a little bit when experienced programmers program in pairs).

Even if productivity seems to drop, the benefits of sharing the knowledge between programmers might be worth a few lines of code less per hour.

Agile project management

Scrum isn't as concerned with programming practices as XP and focuses more on management – the two are often combined.

Scrum has three phases:

• Outline planning: Establish general objectives for the project and design the software architecture.
• Sprint cycles: Each cycle develops an increment of the system.
  • A sprint has a fixed length (2 to 4 weeks).
  • The starting point is the product backlog, a list of work to be done. During the assesment phase, the product backlog is reviewed and priorities and risks are assigned to each task.
  • The selection phase lets the whole team select the features and functionality that will be developed during the current sprint.
  • When they have reached agreement, the team organizes themselves to develop the software. They hold short daily meetings to review progress and reprioritize.
  • Ath the end of each sprint, the work is completed, reviewed, and presented to stakeholders. A new cycle begins afterwards.
• Project closure: Completing the required documentation and assessing the lessons learned from the project.

There is no project manager in Scrum, but there is a Scrum master who arranges the daily meetings, tracks the backlog, records decisions, measures progress, and communicates with customers and management.

Scaling agile methods

Some effort has been made to adapt agile methods to larger systems. However, this has not been very succesful. More complex systems require more analysis before the project starts. However, once some design has been made, it is possible to divide the work to be done into smaller parts that can be developed by smaller teams.

Functional and non-functional requirements.

Customers often specify non-functional requirements in vague terms such as "The system should be easy to use". This should be replaced by certain metrics, such as "A user should be able to use the system's functionality within 4 hours of training". Speed requirements can be assessed through metrics such as bytes processed per seconds. (This can be a problem for customers, because they don't understand what computing terms mean.)

The software requirements document

This document is an official statement of what the system developers should implement. It should include both user requirements and detailed system requirements.

The level of detail depends on how critical the system is (more critical $\rightarrow$ more detail).

Requirements specification

The process of writing down the user and system requirements document. They should be clear, unambigiuous, easy to understand, complete and consistent. Albeit these goals seem simple in theory, in practice it is hard to achieve them all at the same time in.

Some ways to specify:

Requirements engineering processes

Read the book …

Requirements elicitation and analysis

This is done after a feasibility system: You work with customers and end-users to find out how the software should be constructed.

Stages:

1. Discovery: Interacting with stakeholders (the people who the system will affect) to discover their requirements.
2. Classification and organization: Groups requirements into coherent clusters.
3. Prioritization and negotiation: Prioritizing requirements and finding and resolving conflicts through negotiation with the customer.
4. Specification: Requirements are documented and input into the next round of the spiral. (Documents might be produced).

This process might be difficult because:

• Stakeholders don't know what they want or can't formulate it.
• They speak from their own perspective and take certain things for granted. (Even though breathing is important for you as a human, you rarely describe this as a part of your day, do you?)
• Different stakeholders have conflicting requirements.
• Political factors influence stakeholders. A manager might want a system that increases his influence in the workplace.
• The environment of the organization might change.

Some activities you might do:

Use case diagram (CC-BY-SA 3.0 Kishorekumar 62)

Requirements validation

The process of checking that the requirements actually define the system that the customer wants.

• Validity checks: A user might think that a system needs some specific functions. Further thought might bring to light that this is insufficient or can be replaced.
• Consistency checks: Checking that requirements aren't in conflict.
• Completeness checks: The document should contain all functions and constraints.
• Realism checks: Is this possible with current technology, budget and schedule?
• Verifiability: Are the requirements quantifiable (measurable)?

Techniques you might use to test this:

• Requirements review: The requirements document is analyzed systematically by a team of reviewers.
• Prototyping: An executable model is demonstrated to end users and customers.
• Test-case generation: Making tests for requirements often reveals problems.

Requirements management

Requirements for large systems are always changing. The problems you solve are often wicked problems, which are problems that cannot be completely defined. Changes might appear because business and technology changes or users are unsatisfied. Thus you need to keep track of requirements and how they change.

This requires:

• Requirements identification: Every requirement must have a unique identifier, so that it can be traced and cross-referenced.
• A change management process: Set of activites that assess the impact and cost of changes.
• Traceability policies: The relationships among requirements and between requirements and system design.
• Tool support: What tools to use to keep track of the requirements. They should be in a secure, managed, data store that is accessible to everyone involved in requirements engineering.

When a change is proposed, this process is followed:

1. Problem analysis and change specification: You start with a problem or change proposal. This stage checks whether it is valid (i.e. worth caring about and possible to implement). If so, proceed.
2. Change analysis and cost: The effect of the proposed change is analazed and the cost is analyzed. If worth it, proceed.
3. Change implementation: Necessary parts of requirements and the system design are modified.

Context models

Early in the process of specification, you should decide the boundaries of the system: What should it do? Is this covered by an existing solution? Can functionality be left to manual processes or other software? A context model shows the relation between the system and other related systems.

An activity diagram will be used to show the flow of a system process, quite like a flowchart.

Interaction models

These models model the interactions between users and the system. This might be either use case models or sequence diagrams. You can read more about them in the introduction.

Structural models

Structural models display the organization of the system in terms of the components that make up the system, as well as their relationships. Static models show the system design; dynamic models show the organization of the system while it is executing (its different threads). These two kinds of diagrams might differ greatly for a system.

Class diagrams is a structural model that is used when developing an object oriented system. A box represents an object, and lines with different arrowheads show how they're associated. Class diagrams also helps the process of generalization: Classes that share common attributes, might have these attributes combined in a superclass that they all inherit from. When an object is composed of (i.e. contains) objects of another class, it is called aggregation.

Behavioral models

Behavioral models model the dynamic behavior of the system as it is executing. They show what happens (or, to be frank, what is supposed to happen) when a system respons to a stimulus from its environment. There are two types of stimuli:

• Data: Some data that the system should process arrives. The data from the tax processing has arrived.
• Events: Some event triggers the system. Events might have associated data. Your nuclear reactor has a meltdown. Along with it, you get a nice count of the radiation level measured in becquerels.

Model-driven engineering

This is an approach to software engineering where models rather than programs are the principal outputs of the development process. The programs are generated automatically from the models.

• Pros:
  • Allows engineers to think about the system without concern for low-level details. This reduces the likelihood of errors and speeds up the development process.
  • Implementations for different platforms can be generated automatically.
• Cons:
  • Abstractions you make for the model aren't always the right abstractions when implementing.
  • The case for platform-independence isn't very relevant, because the main problem in developing a new system that might be ported to another platform largely lies in other parts of the implementation: Requirements engineering, security, and dependability.

You should make at least these three kinds of models in model-driven engineering:

• Computation independent model: Models the important domain abstractions used in the system. You might create more than one, each modeling its own part of the system.
• Platform independent model: Models the operation of the system without reference to its implementation.
• Platform specific models: Transformations of the platform independent model with a separate model for each platform. There might be several level, such as one level that is middleware-specific, but database independent. When a certain database system is chosen, you go deeper.

It is possible to make such models by adding information to UML diagrams to use them in model-driven engineering. This is done by an extension to UML 2, which is called xUML; this makes the semantics clear and contains a very high-level programming language to specify functions.

Architectural design

Architectural design is a creative process where you design a system organization that will satisfy the functional and non-functional requirements of a system.

Questions you should ask yourself:

1. Is there a generic application architecture that can act as a template for the system?
2. How will the system be distributed across a number of cores or processors? (When designing a system for PCs or embedded systems, you usually use only one core.)
3. What architectural patterns or styles might be used? (Client-server, layered, etc.) Should be based on:
   1. Performance: If performance is critical, localize critical operations within a small number of components. Consider run-time system organizations that allow the system to be replicated and executed on different processors.
   2. Security: A layered structure is best if security is essential.
   3. Safety: If this is critical, all safety related operations should be located in a single or small number of components, thus reducing costs.
   4. Availability: If the system needs to always be available, there should be redundant components, so that parts can be replaced without the system shutting down.
   5. Maintainability: If this is critical, the architecture should have fine-grained, self-contained components that may readily be changed.
4. What will be the fundamental approach used to structure the system?
5. How will the structural components in the system be decomposed into subcomponents?
6. What strategy will be used to control the operation of the components in the system?
7. What architectural organization is best for delivering the non-functional requirements of the system?
8. How will the architectural design be evaluated?
9. How should the architecture of the system be documented?

Architectural views

Krutchen suggests that in addition to the block diagram, you make four additional views – this is called the 4+1 model. His four suggestions:

1. Logical view: Shows the key abstractions in the system as objects or classes. It should be possible to relate the system requirements to entities in this view.
2. Process view: Shows how the system is composed of interacting processes at runtime.
3. Development view: Shows how the software is broken down into components that can be implemented by a single developer or development team.
4. Physical view: SHows the system hardware and how software components are distributed across the processors in the system. (Useful for systems engineers.)

This is rarely done in agile methods. The book's author comments that for plan-driven development, the problem is that they're often made, but rarely used …

Architectural patterns

An architectural pattern is a stylized, abstract description of good practice, such as the Model-View-Controller pattern (MVC) (Wikipedia).

Object-oriented design using the UML

Object-oriented design involves designing object classes and the relationships between them. The advantage of an object oriented design is that it is very easy to change.

Prerequisites:

• Understand and define the context of and the external interactions with the system.
• Design the system architecture.
• Identify the principal objects in the system.
• Develop design models.
• Specify interfaces.

These things are done interleaved.

Design patterns

A design pattern is a pattern for solving a problem, made available for reuse. Useful for object-oriented design.

The essential elements: - A meaningful name. - A description of the problem area that explains when the pattern can be applied. - A solution description of the parts of the design solution, their relationships, and their responsibilities. Just a template – not a concrete description. Very likely to be graphical. - A statement of the consequences (results and trade-offs).

A good example is the Observer pattern, which solves the problem of telling several objects that the state of some object has changed.

Implementation issues

Some things you will probably need to think about …

Open source development

Dilbert 2007-08-03, Scott Adams.

Open source means publishing the code with an invitation for anyone to contribute. The benefits are low cost and high reliability, but this requires that there is an active community of people interested in contributing the code (mostly because they're using it themselves). However, if you are a small company, the customers might trust you more if you make open source software – if you go out of business, they can still ask others to develop the code.

Development testing

This process includes all testing activities that are carried out by the team developing the system. Usually done by the developer, but sometimes each developer gets an associated tester.

There are three levels to development testing:

1. Unit testing: The individual program units or classes are tested. Concerned with objects and their methods.