PRODUCTS
Chunking is a method of splitting large files into smaller, manageable pieces, or "chunks," before uploading them. These chunks are uploaded sequentially or in parallel, and once all pieces reach the server, they are reassembled into the original file. This technique optimizes file uploads, making them faster, more reliable, and less prone to failure.
Chunking is widely used in modern applications like cloud storage platforms (Google Drive, Dropbox), video streaming services, and even APIs like Filestack and AWS S3 for efficient file handling. It’s particularly valuable in environments with slow or unstable networks, ensuring users can resume interrupted uploads without frustration.
Imagine uploading a 5GB video for a client presentation using a spotty mobile network. Halfway through, the network drops for just a few seconds. Without chunking, the upload fails, forcing you to restart the entire process from scratch.
Consider an online video editor where users upload raw footage. A creator working on a deadline uploads a single 20GB file. The process takes hours, and a server timeout occurs just before completion. Without chunking, the user misses the deadline, and your application earns a poor reputation.
During live sports events, fans upload large video clips to share on social platforms. Servers receive thousands of large file uploads simultaneously. Without chunking, server overloads lead to failures and long wait times, frustrating users and damaging the platform's credibility.
In cloud backup solutions, users try to upload a folder containing large video files. A single large file fails to upload due to a network interruption, leaving the backup incomplete. Without resumability, the user’s trust in the service diminishes.
Gamers often upload gameplay videos to streaming platforms. Without chunking, any interruption during a large file upload caused by power cuts or system crashes results in wasted time and resources. Chunking, in contrast, could save hours by resuming from the last successful point.
Chunking greatly enhances the performance of large file uploads by addressing key challenges like interruptions, speed, and server strain.
Traditional uploads are highly vulnerable to network interruptions. If a connection drops, the entire upload must restart. With chunking, only the failed part needs to be re-uploaded, ensuring the process is far more resilient and reliable.
Chunking allows multiple parts of a file to upload simultaneously, maximizing network efficiency. Unlike single-threaded uploads, which can be slow and time-consuming, chunking accelerates the process without compromising stability.
Uploading large files in one go places heavy loads on servers, especially during peak usage. By breaking files into smaller chunks, chunking reduces server stress, enabling smoother operations and better scalability for platforms.
In these scenarios, chunking resolves the headaches by splitting files into smaller parts, enabling resumable uploads, minimizing server strain, and improving user experience. It’s not just a technical enhancement—it’s a lifesaver in the real world.
The graph compares how Filestack, Hivo, and Google Drive/Dropbox benefited from chunking technology across three areas:
Failure rate reduction (Blue bars)
Upload speed improvement (Green bars)
User satisfaction improvement (Red bars)
Chunking clearly improves reliability, speed, and user experience across all these platforms.
This SDK allows for efficient uploading of large files directly from the browser by splitting them up into smaller chunks. It also gives the functionality to pause and resume uploads. Although the SDK is developed in TypeScript, we are releasing only the JavaScript output as an npm package for now. Support for TypeScript users will be included in a later version.
Please note that this is an SDK tailored for use with FastPix and is not a general-purpose upload solution.
Chunking: Files are automatically divided into smaller segments, with a configurable default size of 16MB per chunk.
Pause and resume: This is one such feature provided to pause an upload and resume later.
Retry mechanism : Uploading may fail if there are temporary network failures. Using an exponential backoff strategy, each chunk is retried up to five times to automatically recover from such issues.
Lifecycle event listeners : Developers can be notified about different upload lifecycle events and can give instantaneous feedback to the users who are uploading.
Error handling and reporting : The SDK has robust error handling capabilities so that upload failures can be handled gracefully, and the users are informed about the same.
Customizability : The size of the chunk and the number of retries can be preset at developers' discretion based on specific requirements and network conditions.
To install the SDK, you can use npm or your favourite node package manager :
npm i @fastpix/uploader
To get started with SDK, you will need a signed URL.
To make API requests, you'll need a valid Access Token and Secret Key. See the Basic Authentication Guide for details on retrieving these credentials.
Once you have your credentials, use the Upload media from device API to generate a signed URL for uploading media.
import:
1import {Uploader} from "@fastpix/uploader"
Integration:
1const fileUploader = Uploader.init({
2 endpoint: 'https://example.com/signed-url', // Replace with the signed URL.
3 file: mediaFile, // Provide the media file you want to upload.
4 chunkSize: 5120, // Specify the chunk size in kilobytes (KB). Minimum allowed chunk size is 5120KB (5MB).
5
6 // Additional optional parameters can be specified here as needed
7})
Monitor the upload progress through lifecycle events:
1// Track upload progress
2fileUploader.on('progress', event => {
3 console.log("Upload Progress:", event.detail);
4});
5
6// Handle errors during the upload process
7fileUploader.on('error', event => {
8 console.error("Upload Error:", event.detail.message);
9});
10
11// Trigger actions when the upload completes successfully
12fileUploader.on('success', event => {
13 console.log("Upload Completed");
14});
15
16// Track the initiation of each chunk upload
17fileUploader.on('attempt', event => {
18 console.log("Chunk Upload Attempt:", event.detail);
19});
20
21// Track failures of each chunk upload attempt
22fileUploader.on('chunkAttemptFailure', event => {
23 console.log("Chunk Attempt Failure:", event.detail);
24});
25
26// Perform an action when a chunk is successfully uploaded
27fileUploader.on('chunkSuccess', event => {
28 console.log("Chunk Successfully Uploaded:", event.detail);
29});
30
31// Triggers when the connection is back online
32fileUploader.on('online', event => {
33 console.log("Connection Online");
34});
35
36// Triggers when the connection goes offline
37fileUploader.on('offline', event => {
38 console.log("Connection Offline");
39});
You can control the upload lifecycle with the following methods:
Pause an Upload:
1fileUploader.pause(); // Pauses the current upload
Resume an Upload:
1fileUploader.resume(); // Resume the current upload
Abort an Upload:
1fileUploader.abort(); // Abort the current upload
This SDK supports the following parameters:
endpoint (required): The URL endpoint where the file will be uploaded.
file (required): The file object that you want to upload.
chunkSize (optional): Size of each chunk in kilobytes (KB). Default is 16 MB (16384 KB), with a minimum of 5 MB (5120 KB) and a maximum of 500 MB (512000 KB).
maxFileSize (optional): The maximum file size allowed for upload, specified in kilobytes (KB). This helps prevent excessively large uploads.
retryChunkAttempt (optional): Number of retries per chunk in case of failure. Default is 5 retries.
delayRetry (optional): Delay between retry attempts (in milliseconds) after a chunk upload fails. Default is 1000 ms.
After using the Fastpix Upload SDK to upload a video, the system processes the file by splitting it into smaller chunks. These chunks enable efficient streaming and playback. Here's an example of the response you receive after uploading and playing the video through Fastpix:
example:
1#EXTM3U
2
3#EXT-X-VERSION:6
4
5## Generated with https://github.com/shaka-project/shaka-packager version v3.2.0-53b8668-release
6
7#EXT-X-TARGETDURATION:4
8
9#EXT-X-PLAYLIST-TYPE:VOD
10
11#EXT-X-MAP:URI="https://cdn.fastpix.io/media/642239c1-c34b-4f86-bda5-7fe3ab513a3b/video_360/init.mp4?workspaceId=33a4fe1e-14a9-4903-8938-7ce41a4617d4&resolution=640x360&duration=null"
12
13#EXTINF:4.000,
14
15https://cdn.fastpix.io/media/642239c1-c34b-4f86-bda5-7fe3ab513a3b/video_360/1.m4s?workspaceId=33a4fe1e-14a9-4903-8938-7ce41a4617d4&resolution=640x360&duration=4.000
16
17#EXTINF:4.000,
18
19https://cdn.fastpix.io/media/642239c1-c34b-4f86-bda5-7fe3ab513a3b/video_360/2.m4s?workspaceId=33a4fe1e-14a9-4903-8938-7ce41a4617d4&resolution=640x360&duration=4.000
20
21#EXTINF:2.000,
22
23https://cdn.fastpix.io/media/642239c1-c34b-4f86-bda5-7fe3ab513a3b/video_360/3.m4s?workspaceId=33a4fe1e-14a9-4903-8938-7ce41a4617d4&resolution=640x360&duration=2.000
24
25#EXT-X-ENDLIST
Explanation
Automatic chunking:
Smooth playback:
Optimized streaming:
Chunking helps scale video playback for different users and devices by:
With Fastpix automating this process, developers can focus on building their applications without worrying about video optimization.
Chunking is more than a technique to handling large video uploads effectively. Without chunking, developers encounter unreliable uploads, wasted resources, and dissatisfied users. It breaks files into manageable chunks, ensuring smoother uploads, quicker transfers, and a significantly enhanced user experience. With chunking features like resumable uploads, error handling, and customizable chunk size, the Fastpix Upload SDK simplifies implementation for developers. The Fastpix Upload SDK provides a stable and efficient way of managing big files, which is, therefore, a tool required for developers who have the intent to engage in modern applications with lots of media files. With Fastpix, through chunking, comes the potential for high performance, reliability, and user satisfaction in managing the file upload process.
.png)
.png)
