Beyond the Golden Cages: Modernising Testing Workflows for Agile Network Innovation

Change is difficult, but often necessary. The testing discipline sits stubbornly at the edge of that precipice. Many practitioners still need to understand that bringing out the next generation of testing technology is going to involve breaking some of the processes, tools and technologies that they’re most enamored of. 

This is best expressed in the notion of a “gold” test environment. These brittle arrangements are closely guarded and precisely calibrated to test products over and over again under the same conditions: One stray data point could throw the entire arduous process off. They’re also becoming golden cages, obstructing the next evolution of testing. If we want to get there, we’re going to have to melt some gold. 

How a gold build becomes a golden cage

As I've seen time and time again through my work with APNT and Spirent, test labs generally don’t react well to change. They generally rely on the “gold build” - the highly refined and delicately calibrated set of conditions and settings that test teams have to regress against. 

Once a device or system is slated for release, the configuration files and test artifacts are locked in as “the gold build,” setting a baseline for future tests. This creates a brittle, delicate and static environment which is painstaking to set up and recreate. 

A single gold build has to be consistent every time for the multiple cycles of testing that a given release will undergo. This is a particularly delicate set of conditions within physical testbeds. Routers have to be physically connected to the test tools, and fibre optic cables have to be carefully placed because damage to these can cause significant attenuation changes. Given the careful set up required, test teams are rightly sensitive about other teams rearranging or upsetting their work when they need to test another product. 

If an organisation has one testbed and multiple teams testing multiple products, that gold build has to be undone and the testbed has to be restructured so that other teams can use that equipment. Suddenly the risk of manual human error gets introduced into the process again - and that gold build will have to be painstakingly rebuilt. From there, teams become ever more protective over their particular gold builds, jealously guarding their delicate configurations. 

This is happening at exactly the wrong time: Demand for new technology is skyrocketing, innovation cycles are getting shorter, and pressure is mounting to release. The gold build is increasingly an artifact from a previous age, slowing testing practices when they need to accelerate and scale.

The race to release

First, let's examine how the demands around testing are changing. We’re currently living in an age where companies are competing over a series of potentially transformative technologies such as 5G, AI, IoT and more.

In these races, time to market has a huge effect on market capture. Indeed, the race to release products and services around these technologies is so competitive, that the first mover attains huge advantage and later entrants have a disproportionately diminished advantage.

One report by Harvard Business Review found that those that release first, generally capture about 30 to 40 percent of market share, while competitors don’t even come close compared to their faster colleagues. This is especially pronounced within 5G, which is now seeing both a heated race to release and with whom many operators, struggling to automate testing capabilities. It has - thus far - soaked up huge amounts of investment and operators now want to recoup their investments quickly. Getting their offerings to market quickly will be crucially important to recoup those investments.

Mounting complexity

The technologies that are now being tested are more complex too. Whereas previous generations of technology may have required a handful of test cases to properly evaluate, technologies like 5G and AI and the products and services that stem from them require hundreds, if not thousands of tests across myriad use cases and conditions to be properly tested. That’s partly about the potentially endless series of use cases that these technologies will be used for such as medical imaging or autonomous vehicles and it's partly about the bold new capabilities - such as network slicing - those technologies will come with. Ultimately, complexity of these new technologies require new levels of testing which cannot be achieved manually. 

Take 5G OpenRAN, for example. This revolutionary innovation blends the hardware-based systems of classical telecoms technology, with the software based infrastructure of 5G, thus creating an “open” virtualised system. Within the myriad components and processes OpenRAN will involve, testing this technology will require the expertise of highly-qualified and hard-to-hire personnel and thousands of test cases across the galaxy of components and technologies that OpenRAN involves.

Automation will bring agility

Testing needs to move towards agile operations. These will require dynamic, automated testing environments that can evolve alongside product development. Embracing continuous integration and adaptive testing frameworks will enable teams to test frequently, more efficiently, and at a scale that matches the speed of today’s innovation. By modernising testing workflows, the industry can break free from the limitations of gold builds and usher in an era of smarter, more agile product development.

The manual holdover of the gold build has to give way to an automatable, scalable and shareable environment, virtualised and driven by software. Scaling up physical infrastructure - as would have been the solution in previous generations - is simply no longer feasible. However, by consolidating test beds and automating into a shareable - and crucially, reproducible - environment. This will allow teams to create and store an exact blueprint - that including all the precise resources, configurations and connectivity settings - that can be shared across teams. From there, that gold build blueprint can be restructured instantly, robustly and without the risk of human error. 

This will ultimately allow testing teams to schedule equipment when they need it and share testing resources as required. But most importantly, it will allow test teams to scale their testing hugely while maintaining the precise testing conditions they require. This not only provides a path forward to accommodate modern testing demands but will likely result in significant cost savings as the costs of physical infrastructure are diminished and expensive, energy-hungry lab equipment can be decommissioned and replaced with more efficient solutions. 

Melting the gold build

The essential problem here is that with the rapidly increasing demand, exponentially accelerating innovation cycles, market pressures and the sheer speed of technical growth, gold builds are far too brittle a tool. 

In shackling themselves to gold builds, organisations are inviting inefficiencies and delays. For many enterprises, that should ultimately be a question of long-term survival.

Testing workflows need to be modernised to accommodate the scale and speed of modern innovation. That can only be achieved through modernisation and a dynamic, automated test infrastructure that can embrace continuous integration and adapt to shifting demands. From there, testing technology will be able to scale at a speed that matches modern appetites for agility and innovation.