Bloomberg Finance launched a container test platform using Kubernetes to stress test deployment and operational stability. The move is a unique twist on using Kubernetes, which has become an icon for organizations looking to add management capabilities to their container deployments.
The PowerfulSeal platform was developed by Bloomberg’s Data Technologies team. That team consists of around 5,000 employees including Mikolaj Pawlikowski, a London-based software engineer that worked on development of PowerfulSeal.
“We started on this about 18 months ago when we were looking at building a microservices platform,” Pawlikowski explained. “We wanted to automate the things that are in general annoying to software engineers. Things like monitoring the lifecycle of apps, healing of apps, and deployment models.”
PowerfulSeal is designed to test Kubernetes clusters. It does this by knowing what’s running within each container and being able to recognize what it can disable for testing purposes. Once deployed, it can run in an autonomous mode.
The testing software also includes an interactive mode that allows user to see how it reacts on their specific clusters and use that learning to build their own testing policies.
“You can just point it at a cluster and ask it to delete things, take things up and down, and execute arbitrary commands,” Pawlikowski explained. “It lets you get a good idea of how resilient the application is, and then, with that experience, you can write policies in YAML and deploy them.”
The Bloomberg tech team looked into using various open source options, including Mesosphere and Kubernetes. Pawlikowski explained that Kubernetes “was very young at that time, but showed a lot of promise.”
“We liked the abstraction it was providing,” Pawlikowski said. “There was not any one company trying to force it onto others, and instead it was beginning to form an ecosystem around it.”
Bloomberg is using Kubernetes internally with some production services, though Pawlikowski said the orchestrator was not yet ready to run everything. He explained that testing so far has shown a robust environment, with more issues coming from within Docker containers.
“We are still kicking the tires on Kubernetes,” Pawlikowski said. “PowerfulSeal was really to increase our confidence in the platform.”
Open Source Help
The Bloomberg platform is currently limited to testing Kubernetes clusters running on OpenStack. The company did release the platform into the open source community via GitHub in an attempt to foster further development. Pawlikowski said the move spoke to a broader focus on open source at Bloomberg.
“Over the last five years we have started to adopt more open source in our technology stack and also have increasingly participated in and contributed into a wide variety of open source projects,” Pawlikowski said, citing the Apache Solr, Project Jupyter, and Kubernetes.
“At the end of the day it’s all about scale,” Pawlikowski said of the move. “We may have 5,000 engineers, but still have limited resources. Taking advantage of the open source community helps us solve issues we may not be able to solve, and also contribute back.”