Pointing fingers, or The Blame Game

When handling incidents, I advocate for blameless management practices. Solve the issue, learn from it, and set up safeguards for the future.

When you’re relying on black boxes and something goes wrong, sometimes you have to point fingers. That might be the right call!

You might debug an issue or incident and discover it originates from a library or service you use. In some cases you can work around the issue; in others you need to inform the provider. You’ll want to provide as much information as you can. You can empathize with how it feels to receive a bug report and how you would want as much detail as possible.

It’s also another opportunity to be prepared. Add layers of protection between your application and black boxes so if they experience issues, it doesn’t bring everything crashing down. Set up monitoring to make sure you know when they know something is going wrong. It’s better to be able to tell customers that a vendor is having an outage rather than not knowing why your application is malfunctioning.

An example is when I was at Mobify, we had our own CDN subdomain that would point to a CDN provider. We were able to switch traffic out to another provider in an emergency. This also had the side benefit of preferring a particular provider for a particular region of the world if they performed better.