How do I let students upload large video reviews, process them into multiple resolutions for smooth playback, and make them available to users quickly without overloading my backend server?

The solution:

An event-driven, serverless video processing pipeline powered by AWS S3, ECS, Lambda, and ffmpeg.

The Problem

If you try to handle video uploads and processing inside your main backend server, you’ll quickly run into trouble:

• Large uploads will consume your backend bandwidth and slow down other requests.

• Processing videos in real time will block your app and lead to timeouts.

• Serving raw video files isn’t streaming and browser-friendly, so users will face buffering issues.

And the above methods are not scalable at all.

I needed a way to:

• Offload uploads from backend.

• Process videos asynchronously after upload.

• Output streaming ready formats with multiple quality operations.

The Solution

Lets understand the architecture.

Direct upload to S3 bucket with signed URLs

Instead of uploading videos to my backend, the frontend:

1. Requests a signed URL from the backend.

2. Uploads the raw video directly to a temporary S3 bucket.

Why?

No backend bandwidth bottleneck.

Faster uploads for users.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1755379384202/2c8877f5-8ed3-4ea5-9243-c8964d4a5626.png align="middle")

S3 Event → SQS Queue

When a new video is uploaded, S3 sends an event notification to SQS.

Why SQS is great here:

1. Reliability: If processing is delayed, messages will wait in the queue.

2. Scalability: Can handle spikes in uploads without overwhelming the video processing service.

3. Retry support: Failed processing attempts can be retried automatically.

SQS → Lambda → ECS

After receiving a message from the S3 bucket on upload, the SQS queue triggers a Lambda function. The Lambda function reads the message from SQS (a message like the object key). After reading the message, the Lambda function starts the ECS with the proper configurations and environment variables.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1755382636457/3730c8e6-1b41-4bc3-a25b-f2a5d5e89c6c.png align="middle")

Video Processing Inside ECS

Inside ECS:

Download the raw video from the temporary bucket using the object key.

Use FFMPEG to transcode it into multiple resolutions (1080p, 720p, 480p).

Split it into chunks for adaptive bitrate streaming (HLS).

Upload the processed files to the final S3 bucket with the following structure:

You can find a sample FFMPEG usage here.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1755384794462/f6e13236-1e76-4c85-8ff6-c73d18a808e2.png align="middle")

Final Bucket → DB Update

The final S3 bucket is configured with another event notification. This triggers a Lambda function that updates the database with appropriate URLs so that the frontend can display the processed videos.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1755432236440/1884b9d0-e33b-414e-9da2-2e3e05416e86.png align="middle")

Secure and Fast Video Streaming

After video processing, it's crucial to ensure that the videos can be viewed from your platform. Simply hosting videos in Amazon S3 and serving them directly can leave them vulnerable to unauthorized sharing. Additionally, serving videos from S3 is not efficient from a performance perspective.

To tackle these problems, we will use a Content Delivery Network (CDN) which serves videos from edge servers spread around the world. AWS CloudFront provides a CDN service that allows smooth and secure video streaming.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1755432091622/8d3af940-2e85-4ee7-9fa1-63ff94b8dc1e.jpeg align="middle")

Videos Serving Architecture

In AWS CloudFront, we create a distribution that points to the S3 bucket from which we want to serve the content. For security, we must block all public access to the bucket and remove all CORS configurations so that only CloudFront can access resources from the S3 bucket.

Before understanding video streaming, let's understand signed URLs and signed cookies.

Signed URLs: A signed URL is a normal CloudFront URL with a signature, expiration time, and policy information embedded in it. The user must use that exact URL to access the resources. It is suitable when we want to serve a single file.

Signed Cookies: A signed cookie provides authentication details (signature, policy, expiration) in cookies rather than in signed URLs. Once set, all requests from the browser automatically include the cookies. When using signed cookies, the URL structure of the resource remains clean and unchanged. Any request that matches the cookie policy is allowed.

In our use case, where we serve multiple files (a playlist of video chunks), signed URLs would be inefficient because signing every URL is not feasible, so we will use signed cookies.

Generating signed cookies:

Refer to the official AWS documentation to generate signed cookies using AWS SDK.

Note: to generate public and private keys to generate signed cookies, ensure that your node js version is <20.

When accessing assets through cloudfront, then signed cookies are being sent in request headers on every file request. If cookies are not present in header or policy is not matching, then content will not be served.

When cloudfront server receives a request, then it checks its cache, if cache is available then cloudfront serves the content else it fetches the content from S3 bucket once.

With this approach, we can serve content securely as well as fast because there are hundreds of CDN servers scattered around the world.

Conclusion

By combining AWS's event-driven services with ffmpeg, I have created a video pipeline that is scalable, reliable, and streaming-friendly.

• Uploads are offloaded directly to S3, avoiding bottlenecks.

• Processing is handled asynchronously in ECS with ffmpeg, producing multiple resolutions for adaptive streaming.

• Delivery is secured and accelerated using CloudFront with signed cookies, ensuring smooth playback across devices and locations.

This approach keeps the system modular and cost-efficient: each component (upload, processing, delivery) can scale independently based on demand.

If you are building a video-based platform, this architecture gives you a production-ready blueprint to handle video at scale.

Teleportation: The Ultimate Data Transfer

In teleportation, think of our body as a complex JavaScript object. It has properties like name, age, hobbies, etc. But to teleport this object from one place to another, we can’t just send it as it is. To do so, we need to convert this object or body into a transferable format. This is where Serialization comes in.

Serialization: Breaking it Down

Serialization is like dismantling process in teleportation. It takes your complex JavaScript object and converts it into a simpler, more portable format, usually a string like object. In JavaScript, the most common format for serialization is JSON (JavaScript Object Notation). JSON is like a universal language of teleportation, it is easy to understand and works on different platforms.

Here’s how it works:

//Your body as JavaScript object
const person = {
    name: "Aqib Ansari",
    age: 20,
    hobbies: ['coding', 'roaming', 'sleeping'],
    contact: {    
        email: "aqibansari72a@gmail.com",
        phone: 8591738255,
    }
}

// Serialization: Dismantling the person
const jsonString = JSON.stringify(person)
console.log(jsonString)
// Output: {"name":"Aqib Ansari","age":20,"hobbies":["coding","roaming","sleeping"],"contact":{"email":"aqibansari72a@gmail.com","phone":8591738255}}

In above example JSON.stringify() methods acts as a teleportation chamber, breaking down the person object into a JSON string. This string is like a stream of particles that can be transmitted across the network easily or store in the database for future transmission.

Deserialization: Rebuilding at the Destination

Once the serialized data reaches its destination, it needs to be reassembled into its original form. This is where deserialization comes in. It is like the assembly process in teleportation where each stream of particles is reconstructed into the body.

In JavaScript, Deserialization is done using JSON.parse() method. It takes the JSON string and converts it back into a JavaScript object which is ready to be used in our application.

// Deserialization: Rebuilding the object
const personObject = JSON.parse(jsonString)
console.log(personObject)
/* Output:
{
  name: 'Aqib Ansari',
  age: 20,
  hobbies: [ 'coding', 'roaming', 'sleeping' ],
  contact: { email: 'aqibansari72a@gmail.com', phone: 8591738255 }
}
*/

The deserialization process ensures that the data arrives intact and ready to use. Our body or object is now fully functional at its destination.

The Challenges in Teleportation (Serialization)

Since teleportation is a very complex process, it comes with its own challenges. What if during transmission, some data loss occur and at the receiving end, any part of our body goes missing? Similarly, serialization in JavaScript has its limitation. Lets explore some of these limitations.

Functions: The Lost Abilities

In teleportation, our abilities might not be transmitted because they are not the part of the physical structure. Similarly, JSON serialization doesn’t support functions. If our object has methods, they’ll be lost during serialization.

// Our body as JavaScript object
const person = {
    name: "Aqib Ansari",
    age: 20,
    isSmart: function(){
         return true
    }
}

const jsonString = JSON.stringify(person)
console.log(jsonString)
// Output: {"name":"Aqib Ansari","age":20}

To handle this, we need to manually reattach the methods after deserialization.

Data Corruption: A Misplaced Finger

Imagine reassembling the body with an extra toe or a misplaced finger. In JavaScript this can happen when serialized data is altered or misinterpreted during transmission. For example, Dates in JavaScript are objects but when serialized, they become strings. If not handled properly they might not get convert back to Date object correctly.

const person = {
    name: "Aqib Ansari",
    birthDate: new Date()
}

const jsonString = JSON.stringify(person)
console.log(jsonString)
// Output: {"name":"Aqib Ansari","birthDate":"2025-02-12T23:08:25.439Z"}

const personObject = JSON.parse(jsonString)
console.log(typeof(personObject.birthDate)) // Output: string

To fix this, we need to create a custom function during deserialization to convert the string back to Date object.

const personObject = JSON.parse(jsonString, (key, value) => {
    if (key === 'birthDate'){
        return new Date(value) // Convert string to date
    }
    return value
})
console.log(typeof(personObject.birthDate)) // Output: string

Conclusion: Mastering The Art Of Data Teleportation

Serialization and deserialization are the backbone of data exchange in JavaScript, much like teleportation might be the backbone of futuristic travel. By understanding these processes, you can ensure that the data is transmitted and reconstructed accurately without any alteration.

So next time you are working with JSON or debugging with serialization and deserialization issue, imagine yourself as a teleportation scientist, properly disassembling and reassembling data to make it travel seamlessly across the digital universe.

If you find this article meaningful and enjoyable, engage with it.

Thank you for reading!

Well, after reading this article, you'll get the answer and realize that all these behind-the-scenes mechanisms are inspired by real life.

First, lets understand why does internet matters in today’s world.

Why does internet matters?

• Global Communication: Enables instant messaging, emails and video calls.

• Information Sharing: Allows access to vast amount of knowledge, research and news

• Entertainment: Streaming services, gaming and social media

• Education and innovation: Enables online courses, remote work and technological advancements.

In above examples, you will find that in one way or another, the data is being shared from one point to another point. The whole agenda of internet is based on this ‘data sharing’ only. So to understand the internet, we must understand how data is being shared. So now, lets move on to technical terms and understand the internet.

The Packet’s Path: Navigating the Digital Highway

The data which travels from one point to another is broken down into small chunks, these chunks are called packets. Now I will take you to the entire journey of these packets.

Planning The Trip:

When we search for a website (like www.google.com) in a browser, the browser needs the IP address of that website. To get it, the browser sends the website name to the DNS server, which helps to find the IP address of the following website.

Analogy: If we have to go to someone’s house, we require their exact address rather that the house name.

Now we got the IP address of www.google.com which is 142.250.183.100

Preparing The Journey:

Now that we have the address to send the data to, our browser constructs HTTP/HTTPS requests to facilitate this transfer. These requests include:

• Method (GET, POST, DELETE, etc.)

• Headers (User Agent, Host, etc.)

• Request Body (the actual data)

Analogy: This process is similar to packing our luggage before visiting a house.

Starting The Journey:

When our HTTP request is ready, it is sent over the internet using specific protocols that dictate how how data transfer should occur. The two primary protocols are:

• TCP (Transmission Control Protocol): Ensures reliable, ordered and error-checked delivery of data. Used for web pages, emails and file transfer.

• UDP (User Datagram Protocol): A faster but less reliable protocol used for real-time applications like video streaming, online gaming and video calls.

Analogy: TCP is like taking a planned route with checkpoints ensuring you arrive safely while UDP is like taking the fastest route without worrying about checkpoints.

The Role of Routers and Switches:

Data packets don’t travel from sender to receiver directly. Instead they pass through multiple routers and switches to reach the final destination efficiently.

• Routers: These act like highway interchanges, directing packets along the best possible path to reach the destination.

• Switches: These operate withing the local network (e.g. home or office) directing packets to the correct device in the network.

Analogy: They act as traffic cops or local guides, ensuring the data reaches its intended destination efficiently.

Reaching the Destination:

Now our data has reached its final destination. The data is reassembled into its original sequence using defined protocols. The receiver processes the data according to the request and sends a response back if needed.

The entire process in fraction of a second, allowing us to load the web pages almost instantly.

Additional Concepts That Keep Our Internet Running

• Caching

  Web browsers and servers use caching to store copies of frequently accessed content, reducing load times.

  For example, when you visit www.google.com frequently, your browser stores the IP address in memory instead of searching for it every time.

• Firewalls and Security Measures

  Firewalls and encryption protocols such as HTTPS ensure data privacy and protection from cyber threats

  Analogy: A firewall is like a security checkpoint that filters harmful entities before the enter a restricted area.

Conclusion:

From typing URL in browser to receiving a web-page, many such processes occur behind the scenes, ensuring seamless connectivity. These mechanisms ranging from DNS resolution, HTTP requests, TCP/UDP transmission and router navigation are the backbone of the internet.

All these process are complex yet well-structured system inspired by our day-to-day life.

Understand how the internet works through this article? Then engage with it by liking and sharing. Follow me on X and LinkedIn for more insightful tech content.