Keeping deployed web applications up-to-date is imperative to prevent data breaches. It’s an existential challenge for organizations that have previously considered custom web software to built at one time and then shift operations to another team which does not retain staff developers. Organizations are woefully unprepared to make code changes necessary for security, test those changes to ensure intended functionality continues to work, and then deploy to production within hours.
To understand the ramifications, recall the 2017 Equifax breach, where an attacker obtained the database contents through a known vulnerability in the open source Apache Struts framework that Equifax failed to patch in a timely manner[i]. Now ask yourself, how confident are you that your production software can be updated and deployed right now!? If you are afraid it would break production, you’re not alone. The slow testing process is a reason companies are slow to deploy updates.
In 2018 (around the time of the Equifax breach), TCell conducted a survey and found that it took an average of 34 days to patch a critical vulnerability[ii].
| Severity Level | Average Days to Patch |
| All (Regardless of Severity) | 38 |
| High | 34 |
| Medium | 39 |
| Low | 54 |
| Oldest unpatched CVE | 340 |
The report also concluded the most common threats observed, all of which are covered in detail by the OWASP Top 10[iii] both then and now, were:
- Cross-Site Scripting
- SQL Injection
- Automated Threats
- File Path Traversal
- Command Injection
In short, the time to patch is dangerously long. 34 days is plenty of time to get hacked! In a presentation at the Georgia Institute of Technology, Mr. Dmitri Alperovitch, the founding CTO of Crowdstrike, verbally stated that certain foreign state actors are able to pivot within 15 minutes of gaining first access to a system. In the time to patch, the adversary has had time for 3,264 pivots (96 * 34)!
The Common Vulnerabilities and Exposure (CVE) database is a list of publicly known vulnerabilities in a software and components. When a CVE documents a vulnerability is related to a mainstream commercial or open source library, the publication is often timed to correspond with the release of a patch that fixes the problem. When a CVE is released for a software component for your system, it like the sound of the starting whistle for a race between your organization and outside attackers who now have a blueprint on how to compromise your system. Ultimately to win this race, you need to get your organization to a point where a computer should deploy to production without human intervention after all of the automated tests pass!
But how can this be possible? How can anyone just deploy to production without a protracted quality assurance process conducted manually by professionals? The ultimate answer is to have extensive automated test coverage that lives in the same revision control (e.g., a Git repository) as your code. These tests should be able to build and fully run on every developer’s computer using technologies like Docker without the need for IT-approved cloud resources. The automatic tests should be run by your continuous integration (CI) system before any changes are merged into the main branch of your repository. By definition, this means that any code in the main branch is extensively tested and ready for production since all the tests passed at the time the changes were committed.
With automated testing, you build a run book of test scenarios and have a integration test robot typing at ludicrous speed. It fills text into every field and follows the processes your customers follow when using your web application as designed! The robot also rapidly executes unit tests and re-tests the programming-language level methods within your code. These tests are ideally built by your developers as they build your software with input from your professional quality assurance (QA) team. The benefit of thorough automatic testing is that if your tests pass yesterday, a change is made, and the tests still pass, then you can have great confidence that your software will continue to work today. And with that confidence, you can deploy to production with a mind at ease.
What does this sort of testing look like? It starts with “it is irresponsible for a developer to ship a line of code that [he or she] has not executed any unit test for, and one of the best ways to make sure you have not shipped a line of code without testing is to practice [test driven development]”[iv]. So yes, developers should be writing tests as they work. And the body of tests should reduce the need for your QA team to manually run test scenarios and thus frees these highly trained professionals to discover and document novel edge cases where the system fails and to write additional testing to grow the automated test suite even more. When you have existing legacy code without tests, your team can follow Michael Feather’s legacy code change process[v]:
- Identify change points.
- Find test points.
- Break dependencies.
- Write tests.
- Make changes and refactor
In summary, to protect your in-production web application from compromise via a CVE in your supply chain, you need patch within hours, not days, and certainly not over a month. The best way to be able to have the confidence to deploy your update to production is to have extensive automated tests. The best way to have this sort of extensive testing is to have your team practice TDD.
[i] Whittaker, Zack. Equifax confirms Apache Struts security flaw it failed to patch is to blame for hack. https://www.zdnet.com/article/equifax-confirms-apache-struts-flaw-it-failed-to-patch-was-to-blame-for-data-breach/. Retrieved 3/12/2023
[ii] Security Report for In-Production Web Applications: Q2 2018 (2018). Tcell. File name is tCell_wp-stateofsecurity-2018-web.pdf, but link no longer available post Rapid7 acquisition.
[iii] OWASP TOP 10 (2021). Retrieved from https://owasp.org/Top10/.
[iv] Robert Martin in his 2012 debate on TDD with Jim Coplien, video at https://www.youtube.com/watch?v=KtHQGs3zFAM&t=14%3A42
[v] Feathers, Michael. Working Effectively with Legacy Code (Robert C. Martin Series) (Kindle Locations 624-628). Pearson Education. Kindle Edition.
