Skip to main content

分片上传

分片上传（Chunked Upload）通常使用 multipart/form-data，但实现方式与普通文件上传有所不同。以下是前端实现分片上传的详细步骤和代码示例：

分片上传的基本原理​

  1. 文件分片：将大文件切割成多个小文件块（chunks）。
  2. 并发上传：并行或串行上传这些文件块。
  3. 合并文件：服务器接收所有分片后合并成完整文件。

前端实现步骤​

1. 文件分片​

interface FileChunk extends Blob {
  // 继承自 Blob，添加额外属性（如果有）
}

function createFileChunks(
  file: File,
  chunkSize: number = 5 * 1024 * 1024
): Blob[] {
  // 默认5MB一个分片
  const chunks: Blob[] = [];
  let start = 0;
  while (start < file.size) {
    chunks.push(file.slice(start, start + chunkSize));
    start += chunkSize;
  }
  return chunks;
}

2. 计算文件哈希（用于断点续传）​

// 声明 SparkMD5 类型
declare class SparkMD5 {
  static ArrayBuffer: new () => {
    append(chunk: ArrayBuffer): void;
    end(raw?: boolean): string;
  };
}

async function calculateFileHash(chunks: Blob[]): Promise<string> {
  return new Promise<string>((resolve) => {
    const spark = new SparkMD5.ArrayBuffer();
    let count = 0;

    const loadNext = (index: number): void => {
      const reader = new FileReader();
      reader.readAsArrayBuffer(chunks[index]);
      reader.onload = (e: ProgressEvent<FileReader>) => {
        if (!e.target?.result) {
          throw new Error('Failed to read file chunk');
        }
        spark.append(e.target.result as ArrayBuffer);
        count++;
        if (count === chunks.length) {
          resolve(spark.end());
        } else {
          loadNext(count);
        }
      };
      reader.onerror = () => {
        throw new Error('Error reading file chunk');
      };
    };
    loadNext(0);
  });
}

3. 上传分片​

interface UploadResponse {
  success: boolean;
  message?: string;
  // 可以根据实际响应结构添加更多字段
}

async function uploadChunks(
  chunks: Blob[],
  fileHash: string,
  fileName: string,
  maxConcurrent: number = 3
): Promise<UploadResponse[]> {
  const uploadSingleChunk = async (chunk: Blob, index: number): Promise<Response> => {
    const formData = new FormData();
    formData.append("file", chunk);
    formData.append("hash", `${fileHash}-${index}`);
    formData.append("index", index.toString());
    formData.append("total", chunks.length.toString());
    formData.append("filename", fileName);

    return fetch("/upload/chunk", {
      method: "POST",
      body: formData,
    });
  };

  const requests: Promise<Response>[] = chunks.map((chunk, index) =>
    uploadSingleChunk(chunk, index)
  );

  const results: UploadResponse[] = [];
  for (let i = 0; i < requests.length; i += maxConcurrent) {
    const chunkResults = await Promise.all(
      requests
        .slice(i, i + maxConcurrent)
        .map(req => req.then(res => res.json() as Promise<UploadResponse>))
    );
    results.push(...chunkResults);
  }
  return results;
}

4. 通知服务器合并分片​

interface MergeResponse {
  success: boolean;
  url?: string;
  message?: string;
  // 可以根据实际响应结构添加更多字段
}

async function mergeChunks(
  filename: string,
  fileHash: string,
  totalChunks: number
): Promise<MergeResponse> {
  const response = await fetch("/upload/merge", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      filename,
      fileHash,
      totalChunks,
    }),
  });
  return response.json() as Promise<MergeResponse>;
}

5. 完整的上传流程​

interface CheckChunksResponse {
  uploadedChunks: number[];
  exists: boolean;
}

async function checkChunks(
  fileHash: string,
  totalChunks: number
): Promise<CheckChunksResponse> {
  const response = await fetch(`/check-chunks?hash=${fileHash}&total=${totalChunks}`);
  return response.json() as Promise<CheckChunksResponse>;
}

async function uploadFile(file: File): Promise<void> {
  try {
    // 1. 创建分片
    const chunks = createFileChunks(file);

    // 2. 计算文件哈希（可选，用于断点续传）
    const fileHash = await calculateFileHash(chunks);
    console.log(`File hash: ${fileHash}`);

    // 3. 检查哪些分片已经上传（断点续传）
    const { uploadedChunks } = await checkChunks(fileHash, chunks.length);

    // 4. 只上传未完成的分片
    const chunksToUpload = chunks.filter(
      (_, index) => !uploadedChunks.includes(index)
    );

    if (chunksToUpload.length > 0) {
      console.log(`Uploading ${chunksToUpload.length} chunks...`);
      await uploadChunks(chunksToUpload, fileHash, file.name);
    } else {
      console.log('All chunks already uploaded, skipping to merge...');
    }

    // 5. 通知服务器合并分片
    const mergeResult = await mergeChunks(file.name, fileHash, chunks.length);

    if (mergeResult.success) {
      console.log('Upload completed successfully!');
      if (mergeResult.url) {
        console.log(`File available at: ${mergeResult.url}`);
      }
    } else {
      throw new Error(mergeResult.message || 'Merge failed');
    }
  } catch (error) {
    console.error('Upload failed:', error);
    throw error; // 重新抛出错误，让调用者处理
  }
}

服务器端处理​

服务器端需要实现以下接口：

  1. /upload/chunk - 接收文件分片
  2. /upload/merge - 合并所有分片
  3. /check-chunks - 检查已上传的分片（用于断点续传）

使用 multipart/form-data 的注意事项​

  1. 分片大小：通常设置为 1-10MB，根据网络情况调整。
  2. 并发控制：避免同时上传过多分片导致浏览器卡顿。
  3. 断点续传：通过文件哈希和分片索引实现。
  4. 进度显示：可以通过监听 xhr.upload.onprogress 或 fetch 的 ReadableStream 实现。

其他优化​

  1. 文件秒传：上传前计算文件哈希，如果服务器已存在相同文件，则直接返回成功。
  2. 断点续传：记录已上传的分片，上传失败后可以从中断处继续。
  3. 压缩：对于文本文件，可以在上传前进行压缩。
  4. 加密：对敏感文件进行加密后再上传。

这种分片上传方式特别适合大文件上传，可以有效避免因网络问题导致的上传失败，同时提供更好的用户体验。