How we accelerated development and reduced infrastructure costs with a large OpenStack distributor

Mirantis OpenStack to replace virtualization for Agile IT, lower costs, and host millions of players in 20 global data centers.

The challenge

• Improve game development time-to-market for game development clients.
• Meet fluctuating player demand across titles & regions.
• Lower IT infrastructure CapEx & OpEx.

The solution

• Mirantis OpenStack Cloud & IaaS.
• Custom Fuel plug-ins to existing middleware.
• Murano for provisioning of VMs.

Benefits

• VM provisioning from 1-3 days to less than 1 hour.
• Substantially higher server usage and lower CapEx.
• Improved developer and IT admin efficiency.

The business

The online multiplayer gaming market is big business! Analysts estimated 2015 worldwide revenue of $65 billion and predict a 12 percent annual growth through 2019 from diverse geographies and demographics. Movie theaters, in comparison, drew only $38 billion in 2015 worldwide revenue.

Many of the largest online multiplayer games, such as Wargaming’s World of Tanks, run on infrastructure pioneered by Gcore. Gcore has designed this gaming IT infrastructure to meet the most demanding player expectations around the world. Its 11 locations across Russia and the Commonwealth of Independent States, each with 40 Gbps of bandwidth, host 100 million players per week and ensure industry-leading latency, availability, and cost metrics. This allows Wargaming and other valued customers to focus on game development instead of infrastructure issues.

With 4,000 employees and 150 million users, Wargaming leads the market with award-winning titles and innovative technology. Free-to-play games and continuous feature improvements have propelled its growth since 2011.

Every minute of every day, millions of players from different countries connect to Wargaming titles on Gcore’ infrastructure to face others in battle – to interact, have fun, and win. This deep collaboration drives the two companies’ passion and pursuit of excellence.

Challenges

Like any hosting company facing tremendous growth, Gcore needed to reevaluate its IT process and infrastructure. The fast-paced gaming market required clients to continuously improve games and quickly launch new features, and Gcore needed to make sure they were in a position to do that.

Industry leading customer, Wargaming, had led the market over the past five years, adding successful titles such as World of Warships, and extending World of Tanks access to consoles and mobile devices. Yet, sustaining this innovation required improved infrastructure and increased commitment to Agile IT.

Gcore’ primary goal was to provide this capability to Wargaming, but it struggled to cost effectively maintain infrastructure performance and scale. Its game hosting spanned thousands of servers in 20 data centers and had even set a world record of 1,140,000 peak concurrent users. As platform expenses rose, however, Gcore needed improved resource utilization and cost margins.

Complicating matters, to shorten release cycles, gaming developers required self-service access to resources. Provisioning VM and bare metal servers, however, often involved release managers opening multiple tickets that took many days and administrators to resolve. The activity was complex, labor intensive, and subject to human error. To help make Wargaming successful, Gcore had to help simplify the process of maintaining continuous integration and deployment of high quality games.

To lower IT capital and operating expenses, Gcore needed to decrease virtualization system costs and improve infrastructure utilization. Server virtualization rates were below 50 percent and most workloads ran on bare metal. Meanwhile, customers’ successful games and free-to-play trends were driving massive online player growth. Gcore simply couldn’t afford the virtualization software fees, capacity utilization, and strain on administrators.

To optimize costs and meet fluctuating player demand across titles, regions, and time intervals without impacting performance or scale, Gcore needed to increase platform flexibility. Some games start slow and later explode with demand while others begin strong and unexpectedly even out. This meant simplifying and standardizing complex servers, storage, and networking designs to more easily monitor and scale resources.

Clients such as Wargaming fostered the DevOps culture and were comfortable with open-source software through the use of CentOS Linux, Puppet, and Fabric. But few had embraced the private cloud. However, with OpenStack’s platform gaining adoption by large finance and entertainment firms with high load and streaming services, Gcore was ready to evaluate cloud architectures for Wargaming’s needs.

Gcore and Wargaming’s game development and IT operations teams could likely benefit from OpenStack’s flexibility, cost, and time-to-market gains. But they couldn’t afford any impact on performance or scale.

The solution

In mid-2015, Gcore teamed with Wargaming to conduct an evaluation of cloud platforms. A move to private cloud infrastructure-as-a-service (IaaS) would mean a major transformation of existing virtualization systems and bare metal infrastructure. Thus, the companies’ engineers approached the idea carefully.

Wargaming development teams are divided into two groups: those focused on gaming engines and those focused on non-gaming workloads such as e-commerce, account management, and discussion forums. Furthermore, all work streams are split into four stages – development, load testing, beta testing, and production, the last two being customer facing. This matrixed framework would allow for a phased cloud transition.

Using DevOps methodologies, Wargaming developers had already grouped applications that ran together for specific projects into OpenVZ containers that were deployed using YAML-based files. This meant Gcore’ transition to OpenStack required custom integrations into this existing framework.

Gcore and Wargaming first turned to Mirantis because of its status as the top contributor to the OpenStack community. Upon meeting the team, however, the gaming partners were quickly impressed by Mirantis’ high degree of experience and professionalism in custom design and development.

The teams also valued Mirantis’ local and global presence.

The initial transition to the cloud prudently began with two internal facing work stream stages. The conversion was also limited to a single game, and only for non-gaming workloads. This allowed Gcore, Wargaming, and Mirantis to proceed with lower risks.

Gcore hired Mirantis to conduct a three-day workshop and create an Architectural Design Assessment document that specified the requirements for a minimum viable product (MVP) of the private cloud. Gcore and Wargaming also selected Mirantis to build Fuel plug-in custom integrations with an Active Directory for authentication, NFS for VM data storage, and Puppet master for a CMDB solution.

Mirantis also wrote a plug-in to enable multiple IP addresses, and thus multiple containers, per VM. This allowed Gcore and Wargaming to benefit from state-of-the-art container technology.

Thereafter, Gcore, Wargaming, and Mirantis built the first two clouds based on the Mirantis OpenStack reference architecture and the four custom Fuel plug-ins. The installation includes core OpenStack services including Cinder, Glance, Keystone, Neutron, and Nova. Additionally, Murano is used for the provisioning of virtual machines.

After a few requested adjustments, the teams deployed two additional clouds in just 10 days for the customer facing work streams – beta testing and production.

Next, Mirantis conducted extensive training on OpenStack and Fuel, which helped Gcore and Wargaming gain strong OpenStack platform management skills. As a result, the two companies were ready to transition the first workloads to the new infrastructure.

Results

In early 2016, Gcore transitioned two non-gaming workloads for Wargaming to two OpenStack-based IaaS clouds. The two companies are now in the process of doing the same for the gaming engine workloads. Furthermore, all Wargaming projects recently received access to development workstream clouds.

The companies’ developer efficiency and creativity have also increased as engineers not only spend less time submitting tickets and waiting for resources, but they also benefit from new test and deployment tools. Now, more time and effort can be focused on innovation and adapting to new market needs.

Furthermore, expensive virtualization system licenses are no longer needed as new cloud servers leverage KVM and Ubuntu. And within the IaaS OpenStack deployment, server utilization is significantly higher and capital expenditures have declined as expected.

Operating expenses have also decreased as labor consuming infrastructure administration has been replaced by automated processes, many facilitated by the deployment repeatability from custom Fuel plug-ins.

Gcore and Wargaming’s next goal is the mass migration of all games and work streams to the OpenStack cloud. This means a huge transition of virtualization systems and bare metal servers to a highly scalable and available Mirantis OpenStack solution that can handle the high load of award-winning games.

In addition to the mass migration, Gcore and Wargaming plan to add the following OpenStack-based services: billing and chargebacks, database, DNS, Hadoop, load balancing, and platform-as-a-service. And Gcore plans to add OpenStack solutions to its broader suite of managed services including hosting, CDN, peer networking, and DDoS protection.

Finally, Gcore and Wargaming have been very pleased with Mirantis services to-date and have recently contracted the OpenStack leader to provide 24/7 Enterprise Support.