Expanded interaction between cloud-hosted and on-premise components has contributed to the increased complexity of companies’ tech stacks. As this and other cyber technologies evolve, so do the associated cybersecurity concerns. Traditional penetration testing, focused on a single component like a web application, has difficulty aligning to this new dynamic. If an identified vulnerability lies between two components, such as a web application and an SSO provider, which team is responsible for fixing it? How critical is it, and how likely is it to be abused by an attacker? These are difficult questions to answer accurately without a holistic security assessment of the entire environment.

Isolated Penetration Testing

Typically, a penetration test focuses on testing the security of a specific product line, feature, or infrastructure component. For example, a company may have recently built a new mobile application and want a comprehensive evaluation of its security posture. Alternatively, a company may have recently hardened its external perimeter and may wish to conduct a red team assessment to see if the hardening was effective. Traditional penetration tests are excellent at looking at a specific item and producing actionable results, as a particular team will own that product and can create a roadmap on how to fix the identified issues.

Traditional penetration tests also are easy to digest from a business perspective. Because they focus on a single product line, a specific area of the company can budget and assume responsibility for addressing any vulnerabilities uncovered. In the case of product assessments, the penetration test can also be worked into the deployment cycle, allowing developers and product owners to plan for remediations before a release.

Downsides of an Isolated Penetration Test

The major downside of an isolated penetration test is that products do not exist in isolation. Imagine a small company with a web application that allows users to buy products. This web application also integrates with an external payment processor. Figure 1 shows the entire hypothetical payment process:

Figure 1: Example web application. Only the purple HTTP requests and responses would be in-scope in an isolated penetration test.

In this hypothetical example, the payment processing integration may require the web application and user to send information formatted in a specific way to prevent an information leak. In an isolated penetration test of the web application, the tester may note that this interaction could be an area of concern, but testing the payment processing integration would be out of scope (as in Figure 2). As a result, they would not assign the interaction an appropriate risk rating, and the company would be less likely to remediate it.

Figure 2: Potential out of scope information disclosure (highlighted in red) in the payment processor integration. 

When attackers target a company, they do not distinguish between the web application, database, or payment processing integration. Instead, attackers understand that all the products interact and have interdependencies with other products. In contrast, identifying and tracking all the interactions between different components is difficult for a company. Complex interactions often are buried in developer documentation or become tribal knowledge. Predicting the security implications is hard because of this interconnected dynamic between multiple exposed components.

This isolation factor ties into the more significant issue with isolated penetration testing: scope. For legal reasons, testers must abide by a specific scope when performing a penetration test. Attackers, for obvious reasons, do not care about scope. Penetration tests may therefore miss realistic threat vectors that were out of scope. In addition, isolated penetration tests do not comprehensively view a company’s people, processes, and technology. Important aspects, such as code quality, DevOps, and package management, are invisible in an isolated penetration test. Therefore, recommendations cannot be as tailored and effective as they could have been under a broader scope.

A Different Model: End-to-End, Holistic Testing

In contrast to isolated penetration tests, end-to-end–or holistic–testing focuses on assessing the entire workflow of several integrated components. Going back to the previous example, a holistic test would focus on testing the whole ordering process instead of just the web application, as in Figure 3.

Figure 3: A holistic approach would cover all the respective HTTP requests and responses made during the payment process.

The value of holistic testing is that the scope can be as broad as desired to mimic a realistic attack vector across multiple services. Examples of workflows that would benefit from the end-to-end approach include:

• Web applications that use external identity providers
• Hardware that is managed by a web application, hosted in a cloud environment (such as POS systems with an administrator web application)
• Mobile applications that use a cloud-hosted API and share it with a traditional web application
• Microservice-based architecture
• Products that ingest and export data streams from and to a variety of external sources

An isolated penetration test would underserve the examples above as they span multiple integrations

While holistic testing offers tangible benefits over isolated testing, companies must account for additional organizational challenges in performing an end-to-end assessment. Due to the increased scope, the testing process must involve multiple teams. Several product owners, directors, and product managers need to schedule time for penetration testing activities. Internal teams must create test environments with all the functionality enabled and working. A central owner of the testing initiative needs to coordinate the testing and the remediation effort. Finally, the necessary budget may cut into several teams’ working budgets.

The value delivered by a holistic, end-to-end test is well worth these drawbacks. With full access to a realistic environment, penetration testers can deliver accurate, actionable recommendations that they have tailored to their client, emphasizing actual high-risk items.

So What?

Holistic testing is most effective when incorporated alongside traditional penetration testing. After all, isolated penetration testing of single components still delivers flexible and valuable results. Combining multiple strategies can bridge any gaps in results. Such combinations might include:

• Isolated testing on all new components with a holistic test once everything is incorporated.
• Isolated testing with “checkpoint” holistic testing
• Yearly holistic testing with isolated testing in-between for high-risk new development

The strategy a company selects will depend on its maturity, budgets, timeline, and a variety of other factors. Regardless of the path forward, companies should be aware of all dependencies between components and have a clear plan for how to secure them. Since attackers will use any opening to cause harm, companies need to secure every avenue.