by lapcat 12 hours ago

> Large scale infrastructure changes are often by nature completely untestable.

You're changing the subject here and shifting focus from the specific to the vague. The two postmortems after the recent major Cloudflare outages both listed straightforward errors in source code that could have been tested and detected.

Theoretical outages could theoretically have other causes, but these two specific outages had specific causes that we know.

> which is why robust and fast rollback procedures are usually desirable and implemented.

Yes, nobody is arguing against that. It's a red herring with regard to my point about source code testing.

JohnMakin 11 hours ago | [-3 more]

I am not changing any subject. These are glue logic scripts connecting massive pieces of infra together, spanning what is likely several teams and orgs over the course of many years. It is impossible to blurt something out like "well, source code testing" for something like this, when the source code inputs are not possibly testable outside the scale of the larger system. They're often completely unknowable as well.

With all due respect, it sounds like you have not worked on these types of systems, but out of curiosity - what type of test do you think would have prevented this?

lapcat 10 hours ago | [-2 more]

With all due respect, it sounds like you have never heard of unit tests.

Cloudflare states that the compiler would prevent the bug in certain programming languages. So it seems ridiculous to suggest that the bug can't be detected outside the scale of a larger system.

JohnMakin 5 hours ago | [-1 more]

Please explain how unit tests stop a problem from propagating across a system that fields 70 million requests a second and I’ll take you more seriously, otherwise I’m done with this particular subthread.

lapcat 4 hours ago | [-0 more]

You're completely missing the point. From the blog post:

  if rule_result.action == "execute" then
    rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
  end
"This code expects that, if the ruleset has action=”execute”, the “rule_result.execute” object will exist. However, because the rule had been skipped, the rule_result.execute object did not exist, and Lua returned an error due to attempting to look up a value in a nil value.

This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur."

The unit tests are for the source code. In this respect, the number of requests a second fielded by the system is irrelevant. Unit tests don't happen in production; that's the point of them.

It's a classic coding mistake, failing to check for nil, and none of your handwaving about "scale" changes that fact.