by drysart 12 hours ago
"Fail open" state would have been improper here, as the system being impacted was a security-critical system: firewall rules.
It is absolutely the wrong approach to "fail open" when you can't run security-critical operations.
Cloudflare is supposed to protect me from occasional ddos, not take my business offline entirely.
This can be architected in such a way that if one rules engine crashes, other systems are not impacted and other rules, cached rules, heuristics, global policies, etc. continue to function and provide shielding.
You can't ask for Cloudflare to turn on a dime and implement this in this manner. Their infra is probably very sensibly architected by great engineers. But there are always holes, especially when moving fast, migrating systems, etc. And there's probably room for more resiliency.