Trying to solve a frustrating problem: why is debugging AI systems still so unclear?

I’ve been thinking about something that keeps coming up when working with AI systems. When a model gives a wrong or weird output, it’s surprisingly hard to figure out what actually went wrong. Most of the time we’re digging through logs or guessing where things broke. As part of the AWS AI Ideas hackathon, I started building a concept called AutopsyAI. The idea is to look at the full pipeline from input to output and try to explain where things might have failed, instead of just showing the final result. It’s still early and not a full product yet, more like an attempt to explore whether this problem is worth solving in a structured way. Curious how others here deal with this. If you’ve worked with AI systems in production, how do you debug failures today? Does this feel like a real problem, or is it already solved better than I think? Please do checkout my article and support it! https://builder.aws.com/content/3AeXXMtLdDuPwRL4xcEjJEBoQkB/aideas-autopsyai-the-missing-debugger-for-ai-s