Uploading GBs of Files without Timeout, Up to Resume — Complete Guide to Implementing Chunk Uploads

Note: The code in this article has been conceptually rewritten based on actual work experience. It is not associated with the actual company code.

Chunk Upload + Resume Flow

---

Introduction

File uploading is a core feature in media platforms. Among the VOD-related features I handled at Catenoid, the most challenging was large file uploads.

The problem was simple. Attempting to upload a 2GB video file would freeze the browser, cause a server timeout, or force the user to restart from the beginning if the network became unstable.

This post summarizes the entire process of implementing chunked uploads and resume functionality from scratch.

---

Problem Analysis: Why Single HTTP Uploads Fail

Browser Memory Issues

If you try to load a 2GB file entirely into the browser's memory and attach it to an HTTP request:

// Do NOT do this
const file = input.files[0]; // 2GB file
const formData = new FormData();
formData.append('file', file);
await axios.post('/upload', formData); // Browser memory explodes

JavaScript has to load the entire file into memory. Exceeding the browser tab's memory limit will cause the tab to crash.

Server Timeout Issues

Uploading 2GB over a standard internet connection can take tens of minutes. Most web servers have a default timeout set between 30 seconds and 2 minutes.

The Cost of Retry on Failure

What if the network drops at 50% upload? You have to start all over again. From the user's perspective, it's a nightmare.

---

Design: Chunked Upload

The solution is to split the file into smaller pieces (chunks) and upload them sequentially.

[File: 2GB]
    ↓ Split
[Chunk 1: 5MB] [Chunk 2: 5MB] [Chunk 3: 5MB] ... [Chunk 400: 5MB]
    ↓ Sequential or Parallel Upload
[Server: Receive and Save Chunks Temporarily]
    ↓ All Chunks Received
[Server: Merge Chunks → Final File]

Since each chunk is small (around 5MB), there are no browser memory issues, and they can easily be transmitted within a 30-second timeout.

---

Client Implementation (Vue.js + TypeScript)

Splitting the File

// composables/useChunkUpload.ts
export function useChunkUpload() {
  const CHUNK_SIZE = 5 * 1024 * 1024; // 5MB

  function splitFile(file: File): Blob[] {
    const chunks: Blob[] = [];
    let offset = 0;
    
    while (offset < file.size) {
      chunks.push(file.slice(offset, offset + CHUNK_SIZE));
      offset += CHUNK_SIZE;
    }
    
    return chunks;
  }
  
  // Generate a unique ID for the file (required for consistent resuming)
  async function generateFileId(file: File): Promise<string> {
    const buffer = await file.slice(0, 1024).arrayBuffer(); // Hash the first 1KB
    const hashBuffer = await crypto.subtle.digest('SHA-256', buffer);
    const hashArray = Array.from(new Uint8Array(hashBuffer));
    return hashArray.map(b => b.toString(16).padStart(2, '0')).join('').slice(0, 16)
      + `_${file.size}_${file.name}`;
  }
  
  return { splitFile, generateFileId };
}

Resume: Checking Upload Progress

async function getUploadedChunks(fileId: string): Promise<number[]> {
  const response = await api.get(`/upload/chunks/${fileId}`);
  return response.data.uploadedChunks; // e.g., [0, 1, 2, 5, 6]
}

async function uploadWithResume(file: File) {
  const fileId = await generateFileId(file);
  const chunks = splitFile(file);
  
  // Check already uploaded chunks
  const uploadedChunks = await getUploadedChunks(fileId);
  const uploadedSet = new Set(uploadedChunks);
  
  // Only send unuploaded chunks
  // Also resend the last successful chunk (might be incomplete due to network instability)
  const lastUploaded = uploadedChunks.length > 0 
    ? Math.max(...uploadedChunks) - 1 
    : -1;
  
  for (let i = 0; i < chunks.length; i++) {
    if (i < lastUploaded && uploadedSet.has(i)) continue; // Skip definitely completed chunks
    
    const formData = new FormData();
    formData.append('file', chunks[i]);
    formData.append('fileId', fileId);
    formData.append('chunkIndex', String(i));
    formData.append('totalChunks', String(chunks.length));
    formData.append('fileName', file.name);
    
    await api.post('/upload/chunk', formData, {
      headers: { 'Content-Type': 'multipart/form-data' },
      onUploadProgress: (progressEvent) => {
        const chunkProgress = progressEvent.loaded / (progressEvent.total ?? 1);
        const overallProgress = (i + chunkProgress) / chunks.length;
        uploadProgress.value = Math.round(overallProgress * 100);
      }
    });
  }
  
  // All chunks uploaded, request merge
  await api.post('/upload/complete', { fileId, fileName: file.name });
}

---

Server Implementation (Node.js + Express)

Chunk Reception API

// routes/upload.ts
router.post('/chunk', upload.single('file'), async (req, res) => {
  const { fileId, chunkIndex, totalChunks, fileName } = req.body;
  const chunkFile = req.file;
  
  if (!chunkFile) {
    return res.status(400).json({ error: 'No chunk file provided' });
  }
  
  // Temporary directory for chunks
  const chunkDir = path.join(TEMP_DIR, fileId);
  await fs.mkdir(chunkDir, { recursive: true });
  
  // Save chunk
  const chunkPath = path.join(chunkDir, `chunk_${chunkIndex.padStart(5, '0')}`);
  await fs.rename(chunkFile.path, chunkPath);
  
  res.json({ 
    success: true, 
    chunkIndex: Number(chunkIndex),
    message: `Chunk ${chunkIndex}/${totalChunks} received successfully`
  });
});

// Retrieve list of uploaded chunks (for resume)
router.get('/chunks/:fileId', async (req, res) => {
  const { fileId } = req.params;
  const chunkDir = path.join(TEMP_DIR, fileId);
  
  try {
    const files = await fs.readdir(chunkDir);
    const uploadedChunks = files
      .filter(f => f.startsWith('chunk_'))
      .map(f => parseInt(f.replace('chunk_', ''), 10));
    
    res.json({ uploadedChunks });
  } catch {
    res.json({ uploadedChunks: [] }); // Empty array if directory doesn't exist
  }
});

Merge API

router.post('/complete', async (req, res) => {
  const { fileId, fileName } = req.body;
  const chunkDir = path.join(TEMP_DIR, fileId);
  
  // Sort all chunk files
  const chunkFiles = (await fs.readdir(chunkDir))
    .filter(f => f.startsWith('chunk_'))
    .sort(); // Sorting by name guarantees correct order
  
  // Final file path
  const finalPath = path.join(UPLOAD_DIR, `${fileId}_${fileName}`);
  const writeStream = createWriteStream(finalPath);
  
  // Read and merge chunks in sequence
  for (const chunkFile of chunkFiles) {
    const chunkPath = path.join(chunkDir, chunkFile);
    const chunkData = await fs.readFile(chunkPath);
    await new Promise<void>((resolve, reject) => {
      writeStream.write(chunkData, (err) => err ? reject(err) : resolve());
    });
  }
  
  await new Promise<void>((resolve) => writeStream.end(resolve));
  
  // Delete temporary chunk folder
  await fs.rm(chunkDir, { recursive: true });
  
  res.json({ success: true, filePath: finalPath });
});

---

Results

• Completely resolved large file upload timeouts and memory shortages.
• Upload success rate reached practically 100%, barring issues with the file itself.
• Automatic resume from the exact point of interruption upon reconnection after a network drop.

---

Additional Considerations

In a real production service, there are more things to consider:

1. Chunk Expiration Management: Automatically delete old temporary chunks if an upload has been suspended for too long.
2. Concurrent Upload Limits: Handling scenarios where the same user uploads multiple files simultaneously.
3. S3 Multipart Upload: Uploading chunks directly from the client to S3 without passing through the server (more suitable for large-scale services).
4. Chunk Encryption: Encrypting chunks on the client side before transmission for sensitive content.