Generate and translate AI captions for a new VOD

You can upload videos and generate captions automatically during the process of uploading new videos to Gcire Video Streaming at https://api.gcore.com/docs/streaming#tag/Videos/operation/post_api_videos

When uploading a video, add one new attribute: "auto_transcribe_audio_language." The system will automatically create a task for transcribing the video, and the finished result will be added to the video as a subtitle.

      Copy
      
    POST https://api.gcore.com/streaming/videos
{
    "name": "Spritefright Blender",
    "description": "Video copied from an external S3 Storage",
    "origin_url": "https://demo-files.gvideo.io/apidocs/spritefright-blender-cut30sec.mp4",
    "auto_transcribe_audio_language": "auto"
}

Generate and translate AI captions for an existing VOD

To create AI subtitles for an existing video, run the add subtitle method, but instead of specifying the subtitle text, specify the "auto_transcribe_audio_language" parameter. Subtitles will be generated and attached to the video: https://api.gcore.com/docs/streaming#tag/Subtitles/operation/post_api_videos_video_id_subtitles

      Copy
      
    POST hhttps://api.gcore.com/streaming/videos/{video_id}/subtitles
{
    "auto_transcribe_audio_language": "auto"
}

Extended use of AI API for any video inside or outside the Video Streaming platform

AI ​​system can process any video stored in an MP4 container and available via a direct download link. Please note that links to YouTube, Vimeo, etc., are not links to a video file, so they cannot be used for processing.

Step 1. Obtain an MP4 link

If the video is from an external source, get a link to it. The link must be in HTTPS format and the file must be available for downloading without authorization.

Let's look at an example of getting a link from our video storage.

To obtain an MP4 link to your video with 240p, 360p, or 468p quality. That quality is sufficient because it contains an 64Kbps audio track suitable for AI automatic speech recognition and transcription. To get your MP4 link, execute the following API request https://api.gcore.com/streaming#tag/Videos/operation/get_api_videos_id :

      Copy
      
    GET https://api.gcore.com/streaming/videos/{id}

From the response, copy the value of the mp4_url field.

      Copy
      
    GET https://api.gcore.com/streaming/videos/4428031

      Copy
      
    {
    "id": 4428031,
    "name": "Sprite Fright, Blender",
    ...
    "hls_url": "https://demo-public.gvideo.io/videos/2675_G8EuYJFTsvIyWmKt/master.m3u8",
    "iframe_url": "https://player.gvideo.co/videos/2675_G8EuYJFTsvIyW",
    "iframe_embed_code": "<iframe src="https://player.gvideo.co/videos/2675_G8EuYJFTsvIyWmKt" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>",
    "converted_videos": [
        {
            "name": "vod240n",
            "mp4_url": "https://demo-public.gvideo.io/videos/2675_G8EuYJFTsvIyWmKt/qid96v1_200_240.mp4"
        },
       ...
    ]
}

Step 2. Assign a transcription task to AI

To assign a transcription task to Gcore AI ASR, execute the following API request:

      Copy
      
    curl -L 'https://api.gcore.com/streaming/ai/transcribe' \
-H 'Content-Type: application/json' \
-H 'Authorization: APIKey {your_api_key}' \
-d '{
    "url": "https://demo-files.gvideo.io/{your_video_url/{quality}.mp4" 
}'

Ensure you replace the example values {your_api_key} and {your_video_url}/{quality}.mp4 with your actual API token and MP4 video URL.

In the response, you will get a task ID. It'll come in handy in the next step. For example: "task_id": "abc12345-6def…"

      Copy
      
    curl -L 'https://api.gcore.com/streaming/ai/tasks' \
-H 'Content-Type: application/json' \
-H 'Authorization: APIKey 1234$abcd...' \
-d '{
    "url": "https://demo-files.gvideo.io/apidocs/spritefright-blender-cut30sec.mp4",
    "task_name": "transcription"
}'

      Copy
      
    {
    "task_id": "34d0ba5a-23f6-4462-85ff-fe343b0cdb2d"
}

If you need translation

As well as original-language transcription, we support subtitle translation in 100+ languages. The full language list is available in the API documentation. Specify the language explicitly in the audio_language attribute in the body of the API request, and Gcore AI ASR will use this info to create subtitles. If this attribute is not set, AI will run auto language identification, and the subtitles will be in the original language.

By default, the transcription language is the same as the audio language. In the example above, the video was in English and the desired subtitle language was also English, so there was no need to set the subtitle language separately.

However, if the language of the video differs from the desired subtitle language, you need to add to the request body the "subtitles_language" parameter.

So, this is how the request body will look like if the original language is not English, and subtitles are desired in English:

      Copy
      
    curl -L 'https://api.gcore.com/streaming/ai/tasks' \
-H 'Content-Type: application/json' \
-H 'Authorization: APIKey {your_api_key}' \
-d '{
    "url": "https://demo-files.gvideo.io/{your_video_url.mp4}",
    "task_name": "transcription",
    "subtitles_language": "ger"
}'

You can skip the "audio_language" field, and Gcore AI ASR will detect it automatically.

For the "code-switching" feature mentioned above, audio_language must be skipped.

Step 3. Get the task result

Wait a few minutes for the AI to complete the transcription and translation. To get the task result, execute the following API request:

      Copy
      
    GET https://api.gcore.com/streaming/ai/tasks/{your_task_id}

You will receive a response containing the transcribed and translated (if set) text.

The following response elements may be useful for further video processing:

• "subtitles" array contains transcription divided into individual timecoded sentences. It includes "start_time", "end_time" and "text" fields.
• "concatenated_text" field contains the full audio text. It can be used for robot scanning, Google indexing, or summarization using a service like ChatGPT.
• "vttContent" field contains marked-up text that can be saved as a .vtt file and added to any system as ready-to-use subtitles.

      Copy
      
    GET https://api.gcore.com/streaming/ai/tasks/34d0ba5a-23f6-4462-85ff-fe343b0cdb2d

      Copy
      
    {
    "task_data": {
        "url": "https://demo-public.gvideo.io/videos/2675_G8EuYJFTsvIyWmKt/qid96v1_200_240.mp4"
    },
    "task_id": "34d0ba5a-23f6-4462-85ff-fe343b0cdb2d",
    "processing_time": {
        "completed_at": "2024-03-13T08:15:19Z",
        "started_at": "2024-03-13T08:13:27Z",
        "total_time_sec": 112
    },
    "progress": 100,
    "status": "SUCCESS",
    "result": {
        "subtitles": [
            {
                "start_time": "00:00:17.648",
                "end_time": "00:00:18.868",
                "text": "Hello, Mr. Snail."
            },
            ...
            {
                "start_time": "00:10:00.551",
                "end_time": "00:10:00.891",
                "text": "Did it, kid."
            }
        ],
        "vttContent": "WEBVTT\n\n1\n00:00:17.648 --> 00:00:18.868\nHello, Mr. Snail...00:10:00.551 --> 00:10:00.891\nDid it, kid.\n\n",
        "concatenated_text": "Hello, Mr. Snail. Oh, you cute little corny wasp-assum. Latin name, bonus points...You made it, kids. Did it, kid. "
   }
}

Step 4. Add AI ASR subtitles to videos

1. Copy the value of the "vttContent" field from the previous step. This contains the subtitles for your video.

2. Add the subtitles to the video by executing the following API request:

      Copy
      
    curl -L 'https://api.gcore.com/streaming/videos/{video_id}/subtitles' \
-H 'Content-Type: application/json' \
-H 'Authorization: APIKey 1234$abcd...' \
-d '{
     "name": "English (AI-translate)",
     "language": "eng",
     "vtt": "{your_vtt_field_content}"
}'

That's it! You can now generate subtitles using AI, add translations (if necessary), and upload subtitles to your video.