Originally published on OpenInfra Superuser.
A note on context: I chaired the Interop Working Group while writing this. The article was, in part, an attempt to think through a decision we were facing: whether to change the interoperability process, maintain it as it was, or wind it down entirely. There wasn't meaningful interest in changing things, and maintaining a framework that no longer served its purpose didn't make sense. Shortly after, we decided to end the working group. This article is the thinking that led there.
What interoperability means
Interoperability can be defined as the ability of computer systems or programs to exchange information; more specifically, the ability of applications, equipment, and systems from different vendors to seamlessly communicate and process data without requiring any involvement from end-users.
For OpenStack, the goal of interoperability is to help users make informed decisions and adopt the OpenStack products that best meet their business needs. A guideline-based testing framework managed by the Interop Working Group (formerly Defcore, established in 2014) maintains test repositories using Tempest tests to verify cloud compatibility. Pass those tests, and a cloud deployment is considered interoperable. It's a binary model: you either meet the bar or you don't.
A paradox in the data
Tempest contributions peaked between 2014 and 2017, then declined significantly. OpenStack development activity grew through the Newton release in 2016, then trended downward from Ocata onward. New code contributions and contributor numbers followed the same curve. From a pure community activity standpoint, things have been quieter.
But here's what the activity charts don't show: deployments went the other way entirely. According to the 2022 OpenStack User Survey, OpenStack deployments hit a new milestone in 2022, reaching 40 million compute cores in production, a 60% increase compared to 2021 and an impressive 166% increase since 2020. The community writing the code may have shrunk, but the community running it has never been larger.
That's a paradox worth sitting with. The software is more widely deployed than ever. The people testing and extending it are fewer. And the diversity of configurations running in production is growing, because more organisations, with more different requirements, are now running OpenStack in ways its earlier community never anticipated.
The problem this creates for interoperability testing
The current interoperability process was designed for a world where feature development drove the testing roadmap. New capabilities landed in OpenStack, the Interop Working Group added tests for them, and cloud vendors verified compliance. It was a reasonable model when contributions were flowing.
With declining development activity, that pipeline has slowed. Fewer new features means fewer new tests means a framework that increasingly reflects yesterday's OpenStack rather than what's actually running in production today. Meanwhile, the growing diversity of deployments means the binary pass/fail model is running into its limits. A test that assumes a standard configuration can produce misleading results for a cloud that's legitimately interoperable in the ways that matter to its users, but non-standard in ways that don't.
A different way to think about it
What I'd argue we need is something closer to fuzzy logic: placing interoperability on a scale rather than treating it as a binary condition. Instead of asking "does this cloud pass or fail?", the more useful question is "how interoperable is this cloud, for which use cases, at what level?"
This would let users make much more informed decisions. A deployment that passes 90% of interoperability tests in the areas that matter for a particular workload is genuinely useful to know about, even if it doesn't satisfy every requirement of a blanket certification. The current model makes that nuance invisible.
It also makes the process more honest about the reality of diverse deployments. Requiring every cloud to behave identically made sense when OpenStack was a young ecosystem trying to establish credibility. It's a harder argument to make when 40 million compute cores are running in production across wildly different environments, and the community that writes the compliance tests has a fraction of the capacity it once had.
Why this matters
The OpenStack ecosystem faces a genuine sustainability question: how do you maintain the value of a technical standard when the community that built it is smaller, but the community that depends on it is larger than ever? Interoperability testing is one part of that answer, but only if the framework keeps pace with reality.
A graduated approach to interoperability won't solve everything. But it would make the standard more useful to the users who actually rely on it, more achievable for the vendors who implement it, and more maintainable for a working group that no longer has unlimited capacity. That seems like a better fit for where OpenStack is now.