Gcore’s failsafe transcoding system

We’ve implemented a new technology in our Streaming Platform. It is a system providing failsafe transcoding and delivering live streams in the HLS format. The new technology will improve transcoding stability and enhance the delivery of video content to the audiences that exceed 1,000,000 viewers.

In this article we will tell you how this system is going to help our clients improve streaming quality.

What the new system is like

The failsafe transcoding and stream delivery system is a cluster composed of transcoding servers managed by a load balancer. The cluster receives the original video stream and returns transcoded adaptive files available via HTTP, i.e., in the HLS or, alternatively, in the MPEG-DASH format.

Major system elements:

1. Three (or more) equally powerful transcoding servers.
2. A load balancer that manages and distributes the load between the servers.
3. A CDN for delivering files to the users.

The key feature of such a transcoding system is that there is no centralized repository where adaptive files are stored. We distribute the elements while streaming video content, which allows us to:

• guarantee uninterrupted delivery of your video content to the end viewers;
• enhance the stability of live stream broadcasts;
• reduce the internal network load;
• speed up content delivery;
• simplify the connection between different elements;
• exclude an additional fault point.

The absence of a centralized repository can create some additional difficulties (for example, the data may become inconsistent in case the transcoding server changes). But our developers have covered all the details and solved all the problems.

How failsafe transcoding and stream delivery works

1. Stream reception for transcoding

It starts with the load balancer generating the necessary parameters, selecting the most appropriate server from the cluster based on these parameters, and sending a command to the server using a Redis-based messaging server.

The server receives the command and the original stream in the RTMP format (or in any other suitable video transmission format).

2. Transcoding

The server transcodes the video to the HLS format. The stream involves two file types:

1. Chunks—short video pieces that are 2–5 seconds long.
2. Chunk lists—playlists, i.e., lists with the names of the generated chunks that constantly get updated. The player regularly requests these files using a permanent URL that is known in advance. This allows the player to always have an up-to-date chunk list to play.

Each chunk has its own unique name connected with the transcoding time. This is necessary in case the transcoding session gets restarted on different servers (for example, this can happen if the load gets redistributed or if a server fails). Unique names make it impossible for different files to overlap, which solves the problem of data consistency.

But the name of the chunk list file must be the same on all servers. Chunk lists are stored separately from the chunks in order to avoid creating several irrelevant files. The folder where they are stored is constantly synchronized with every server, so that each server always has an up-to-date chunk list.

Chunk lists are small files, this is why constant server synchronization does not create any significant network load and doesn’t result in increased delays.

Since all servers have an up-to-date chunk list, there is no need to re-code ready chunks in case the transcoding server is changed. All transcoded pieces will remain in the chunk list and they will be loaded from the previous server.

3. Stream delivery via CDN

When transcoded, the stream is transferred to the content delivery network and sent to the users.

All transcoding cluster servers are specified as a content source for the CDN.

1. The network requests a chunk list from a random server in the cluster (this is possible because chunk lists are the same on all servers).
2. When requesting the necessary chunk, the CDN addresses all servers and finds the desired fragment by its unique name.
3. Video pieces are delivered to the users in the correct order.

4. Fault tolerance provision

As mentioned above, if the transcoding server is changed, there is no need to re-code the chunks because the necessary fragments are downloaded from the previous server. But what if the previous server goes out of order, and all the transcoded fragments become unavailable? In this case, the viewer won’t be able to rewind the video back.

We can solve this problem in two ways.

Way #1: caching ready-made chunks on an intermediate server between the CDN and the transcoding system. In Gcore CDN, it’s the origin shielding that plays the role of such a server. All fragments will invariably be stored in the shielding cache and remain available in case of a failure.

Way #2: creating a pair of servers within a cluster that exchange ready-made chunks. This slightly increases the server load. But if only two servers are involved, the load isn’t going to increase much, while fault tolerance will be achieved.

It’s up to our clients to decide which way to choose. Note that origin shielding is a paid option. But if you broadcast videos to a large audience, opting for the paid variant makes sense because shielding also protects the system against extreme loads.

Technology advantages

• It is suitable for low latency streaming. The system can update chunk lists and deliver ready-made chunks as quickly as possible. The streams are delivered using the HTTP protocol. This means that the new technology allows our clients to deliver their video content with the delay not exceeding 4 seconds.
• Fault tolerance provided. The load balancer distributes the load in the cluster evenly, and the transcoded chunks are duplicated on an intermediate server (shielding) or on the second transcoding server. Your viewers will be able to watch your video content even if one of the servers fails.
• It reduces the network load. As there is no centralized storage, the network load gets reduced. Stable content delivery requires less bandwidth.

How to use failsafe transcoding for streaming purposes

This technology is now available for all clients of our Streaming Platform by default.

Our platform uses advanced video content delivery technologies. It allows you to easily broadcast your videos in up to 8K quality on all device types with minimal delays and without buffering, even if your audience exceeds 1,000,000 viewers.

Summary

1. We have implemented a new technology into our Streaming Platform. It is a system providing failsafe transcoding and delivering live streams in the HLS format.
2. This system is a cluster consisting of three or more transcoding servers managed by a load balancer. Transcoded streams are delivered using Gcore CDN.
3. The load balancer sends video transcoding commands to the servers. The servers convert the stream into chunks and chunk lists. The chunks are sent to the users via CDN with minimal delays.
4. The failsafe transcoding system is suitable for low-latency streaming (with the delays not exceeding 4 seconds), guarantees fault tolerance, and helps reduce network load.
5. The new technology has already been launched on our Streaming Platform and is available to all our clients.

Make your video content available to your users in high quality on any device using our Streaming Platform.

More about Streaming Platform