Principles of Testing

• There is no silver bullet
• Think strategically, not tactically
• The SDLC is King
• Test early and often
• Understand the scope of security
• Develop the right mindset
• Understand the Subject
• Use the right tools
• The Devil is in the Details
• Use Source Code When Available
• Develop metrics
• Document the test results

Testing Teqhniques Explained

Deriving Security Test Requirements

You'll need testing objectives to have successful testing. Testing objectives are specified in security requirements.

Deriving functional and non-functional test requirements

Security Tests Integrated in Development and Testing Workflows

During development the developers should validate and analyze source code, write unit tests and use tools to help find potential vulernabilites.

There should also be security tests as a part of integration tests. These should include white box, black box and grey box tests.

Security Test Data Analysis and Reporting

The metrics used should be well thought out before one starts the testing phase. One also has to decide what level the application should be at when it comes to security. This could mean setting an upper bound for security vulnerabilities, or how it should be compared to other applications. There should be a clear goal with the testing phase, e.g. reduce the number of vulnerabilities to some number.

Data that should be included in a test report:

• The categorization of each vulnerability by type
• The security threat that the issue is exposed to
• The root cause of security issues (e.g., security bugs, security flaw)
• The testing technique used to find the issue
• The remediation of the vulnerability (e.g., the countermeasure)
• The severity rating of the vulnerability (High, Medium, Low and/ or CVSS score)

You can present business cases to show that the security test metrics provides value to the organization's security test data stakeholders, e.g. managers, developers, information security offices, auditors etc.

Phase 1: Before Development Begins

Phase 1.1 - Define a SDLC: You must define the SDLC, where security must be a part of each stage.

Phase 1.2 - Review Policies and Standards: Then you must review standards and policies, as you must know what they are to be able to follow them. Without guidelines, you cannot base yourself on the work done by other experts.

Phase 1.3 - Develop Measurements and metrics Criteria and Ensure Traceability: The process may need modification to be able to capture the data you want, so it's essential that this is done before the complete process starts.

Phase 2: During Definition and Design

Phase 2.1 - Review Security Requirements: First review security requirements, which define how the application should work from a security perspective. The requirements has to be tested. When looking for gaps in the requirements, it's smart to look at security mechanisms such as:

• User Management
• Authentication
• Authorization
• Data Confidentiality
• Integrity
• Accountability
• Session Management
• Transport Security
• Tiered System Segregation
• Legislative and standards compliance (including Privacy, Government and Industry standards)

Phase 2.2 - Review Design and Architecture: Then move on to look at the architecture and the design, and review the documents associated with that. This includes models, textual documents and similar artifacts. To find security flaws in the design/architecture is the most cost-effective, and it's also the place where changes are most effective. E.g. if authorization has to be made more than one place in the system, consider creating a authorization component.

Phase 2.3 - Create and Review UML Models: Create and review UML models, so that the system designers have an exact understanding of how the application works.

Phase 2.4 - Create and Review Threat Models: Move on to create and review threat models, based on the design and archutectural reviews, and the UML diagrams. Design realistic threat scenarios, and analyze the design and architeture to see that these threats have been mitigated, accepted by the business or assigned to a third party (e.g. a insurance firm).

Revisit design if a threat doesn't have a mitigation strategy.

Phase 3: During Development

In theory there shouldn't be any need for phases or strategies in this development, as the development is the implementation of the design. If the design is well thought out, all possible threats have already been mitigated. In practice, that's not the case. Implementation can drift away from design. You can have human errors in implementation, or the design can be faulty. The developer will face many decisions, that can and will affect the security of the application.

Phase 3.1 - Code Walk Through: The security team should do a code walk through with the developers, and in some cases the system architect. This is a high-level walk though. The developers explain the logic and flow of the implementation. This gives the code review team a general understanding of the code. The purpose is not code review, but understand the high level structure of the application.

Phase 3.2 - Code Review: When there's a good understanding of the flow and structure of the application.

These code reviews validate the code against a static checklist, that includes the following:

• Business requirements for availability, confidentiality, and integrity.
• OWASP Guide or Top 10 Checklists for technical exposures (depending on the depth of the review).
• Specific issues relating to the language or framework in use, such as the Scarlet paper for PHP or Microsoft Secure Coding checklists for ASP.NET.
• Any industry specific requirements, such as Sarbanes-Oxley 404, COPPA, ISO/IEC 27002, APRA, HIPAA, Visa Merchant guidelines, or other regulatory regimes.

The return on investment (time invested), these code reviews has extremely high quality return compared to alternatives, and rely the least on the skill of the reviewer.

Phase 4: During Deployment

Phase 4.1 - Application Penetration Testing: A last check to see that nothing has been missed in design or implementation.

Phase 4.2 - Configuration Management testing: Even though the application itself can be safe, the infrastructure and configuration of the deployment should also be tested.

Phase 5: Maintenance and Operations

Phase 5.1 - Conduct Operational Management Reviews: You need to document how the application and infrastructure is managed, and have a process surrounding it.

Phase 5.2 - Conduct Periodic Health Checks: Either monthly or quarterly health checks should be performed on the application as well as the infrastructure to ensure that no new threats has somehow been introduced.

Phase 5.3 - Ensure Change Verification: Changes has to be tested by QA in a QA environment, but also tested in the deployment environment to ensure that the level of security is not affected by the change.

Testing for Information Gathering

• Conduct search engine discovery/reconnaissance for information leakage
  • Search with site:yourwebsite.comin different search engines
  • Look at the Google Hacking database to help build searches
  • The goal is the see if there is any leaked sensitive information
• Fingerprint Web Server
  • Find out what type of web server it is allows the penetration tester to exploit known vulnerabilites in the software
  • Can be done with Netcat or simply curl the URL and look at the headers. Note that there are many ways to obfusate or modifiy the server banner string.
  • One can also look at the ordering of the headers returned, e.g. Apache and IIS has different orderings. The test guide provides different examples
  • Another way to detect the server software is to send a malformed request, e.g. request non-existent pages. The responses will be different for the different server softwares. The test guide provides examples. These things can also vary with the version of the server. Also note that the differences can be shown by sending invalid HTTP methods/ requests.
  • Automated testing, e.g. with Httprint. or an online service like the OWASP Unmaskme Project.
• Review Webserver Metafiles of Information Leakage
  • Involves testing the robots.txt file for information leakage. The file describes accepted behavior of Web Spiders, Robots and Crawlers. E.g. allowed user-agents, and disallowed directories. A crawler/spider/robot can intentionally ignore a robots.txt-file.
  • A robots.txt-file can be analysed using Google Webmaster Tools
• Enumerate Applications on Webserver
  • There are usually more than one application running on a server today, or more than one application associated with a public IP.
  • Is tested with black box tests
  • Issues/factors when testing:
  • Apps can be placed under different subdirectories, e.g. example.com/app1 and example.com/app2. These can be referenced by other web pages and found through those. You can also use scanners. The web server may be misconfigured and let you browse the directories, and you may suddenly find some of the applications. You can also probe URLs that are likely to hold an application, e.g. mail.example.com, example.com/webmail etc.
  • Most webapps lie on port 80 or 443 (HTTP and HTTPS), but apps can run on other ports. Can be easily solved with a port scanner, e.g. nmap, for instance nmap –PN –sT –sV –p0-65535 192.168.1.100, which will run a TCP connect scan and look up all ports on the IP 192.168.1.100, and try to determine which services are running on those ports.
  • Multiple virtual machines can run under a domain. You'll need to find out which DNS names are associated with an IP-address.
    • Determining the DNS name server, and query them for zone transfers. This may return additional DNS
    • DNS inverse queries: Set record type to PTR and issue an IP request
    • Web-based DNS searches
    • Reverse IP-services
    • Googling
  • Review webpage comments and metadata for information leakage: Developers use comments to save information, and some info may not have been removed before putting the application into production. Look at HTML and JS code (black box testing) by opening the code view from your browser.
    • Meta tags can help profile the webapp
    • Meta tags can also be used by the developer to alter HTTP response headers. See if these can be used to conduct an attack, e.g. CLRF
  • Identify application entry points

...

Executive summary

The exec summary sums up overall findings of the research. This is meant for the business managers and system owners, and should give a high level view of the vulnerabilities that has been discovered. The executives won't have time to read the whole thing, so they want answers to the questions "What's wrong?" and "How do we fix it?". These questions should be answered in one page.

Test Parameters

Intro to this chapter should outline parameters of the security testing.

The OWASP Testing guide gives the following suggested headings:

1. Project objective: Objective and expected outcome
2. Project scope
3. Project schedule
4. Targets: Lists the applications and systems that are targeted.
5. Limitations: Outlines all limitations that were encountered, e.g. in regards to methods or certain types of tests.
6. Summary of the findings: Outline vulnerabilites
7. Remediation Summary: Outline an action plan for fixing the vulnerabilities.

Findings

Gives detailed technical information about the vulnerabilites that were found, and the actions needed to solve them. The section is aimed at technical people, so include everything. The section should include:

• Screenshot and commands that indicate what tasks was done
• Affected item
• Technical description of the issue and what it affects
• Section on how to resolve the issue
• A severity rating

1.1: Introduction

The conventional view is that while software engineering is about ensuring that certain things happen (‘John can read this file’), security is about ensuring that they don’t (‘The Chinese government can’t read this file’).

1.2: A Framework

Good security engineering requires four things:

• Policy: What you are to achieve
• Mechanism: Ciphers, access control, hardware, controls, machinery ..
• Assurance: Reliance on a particular mechanism
• Incentive: Motive that guardians of a system do their job properly, and the motive that the attackers have to defeat your policy

1.3 to 1.6: Examples of systems

Read them if you want.

1.7: Definitions

Most of these definitions are from the book.

Definitions when talking about a system. In practice, the term system can denote:

1. a product or component, such as a cryptographic protocol, a smartcard or the hardware of a PC;
2. a collection of the above plus an operating system, communications and other things that go to make up an organization’s infrastructure;
3. the above plus one or more applications (media player, browser, word processor, accounts / payroll package, and so on);
4. any or all of the above plus IT staff;
5. any or all of the above plus internal users and management;
6. any or all of the above plus customers and other external users.

Important terms in relation to security:

Other words and terms:

There are more mentioned, e.g. the difference between trust and trustworthy is explained with an example with the NSA.

1.8 Summary

There's some confusion about the terms in use in the security engineering community. Computer security can be a lot of things, and many things, e.g. social interactions, are related. A robust security design requires protection goals to be explicit.

2.1 Introduction

As technology has advanced, social/psychological attacks to computer systems are the greatest threat to online security, see for example phishing.

2.2 Attacks Based on Psychology

2.3 Insights from Psychology Research

2.4 Passwords

There are basically three options for authentication users to a system:

• Retain physical control of the device ('Something you have')
• Present something you know, e.g. a password ('Something you know')
• Use something like a fingerprint or iris scanner ('Something you are')

Most systems use the second option due to cost.

There are other things than passwords, that aren't obviously passwords, that can be used for the same purpose as the password. E.g. your mother's maiden name or a social security number. The first one is an example of a 'security question', and aren't that hard to guess if you know the name of the person you are trying to guess it for, so that you can research him or her. This has given the rise for the industry identity theft.

There are three broad concerns when it comes to passwords:

1. Will the user enter the password correctly with a high enough probability?
2. Will the user remember the password, or will they have to either write it down or choose one that’s easy for the attacker to guess?
3. Will the user break the system security by disclosing the password to a third party, whether accidentally, on purpose, or as a result of deception?

The following sections describes different problems and challenges with passwords and users.

2.5 System Issues

The fastest growing concern when it comes to passwords is phishing, and it's done most research on the pshychology related to passwords. That being said, there are some technical challenges too.

3.1: Introduction

What protocols are, and how they can fail.

3.2: Password Eavesdropping Risks

An example about how car unlockers in the 90's where too simple, as they used the serial number as the key for locking and unlocking.

3.3: Who Goes There? - Simple Authentication

Notation for protocols is shown with an example. The example is a car garage security mechanism, where the cars get's a key in the car when they enter. To open the bar when they leave, they have to beep the key. It then sends its serial number and then sends an authentication block consisting of the same serial number, followed by a random number, all encrypted using a key which is unique to the device.

Notation: $\{X\}_K $ is message $X$ under key $K$ The example mentioned above:

$$ T \rightarrow G: T, \{T, N\}_{KT}$$

• The in-car token sends its name followed by the encrypted vlaue of $T$ concatenated with $N$, where $N$ is a number used once (aka nonce). It can mean anything that guarantees the freshness of a message.
• Everything within the braces are encrypted.
• The $N$ value's purpose is to assure the recipient that the message is fresh

Verification is simple:

• The parking garage server reds $T$, gets the corresponding key $KT$, deciphers the rest of the message and checks that the nonce $N$ has not been seen before.
• Finally checks that the plaintext contains $T$

One reason many people get confused is that to the left of the colon, T identifies one of the principals (the token which represents the subscriber) whereas to the right it means the name (that is, the serial number) of the token. Another is that once we start discussing attacks on protocols, we can suddenly start finding that the token T’s message intended for the parking garage G was actually intercepted by the freeloader F and played back at some later time. [..] Professionals often think of the $ T \rightarrow G$ to th