by jakub_g 12 hours ago
The interesting part:
After rolling out a bad ruleset update, they tried a killswitch (rolled out immediately to 100%) which was a code path never executed before:
> However, we have never before applied a killswitch to a rule with an action of “execute”. When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn’t evaluate the sub-ruleset pointed to by it. However, an error was then encountered while processing the overall results of evaluating the ruleset
> a straightforward error in the code, which had existed undetected for many years
> have never before applied a killswitch to a rule with an action of “execute”
One might think a company on the scale of Cloudflare would have a suite of comprehensive tests to cover various scenarios.
I kinda think most companies out there are like that. Moving fast is the motto I heard the most.
They are probably OK with occasional breaks as long as customers don't mind.