AWS S3 Multipart Upload Part Size & Concurrency Sizer

Calculate optimal chunk sizes, concurrency threads, and memory buffers to maximize upload speeds while respecting S3's 10,000 part limit.

Optimal Part Size

32 MB

Yields 8,000 parts (S3 max: 10,000)

Thread Concurrency

13 Workers

Fits safely inside 512 MB RAM budget

Estimated Duration

1h 11m 34s

At 500 Mbps sustained throughput

Upload Workload Parameters

Total File Size (GB):

Upload Bandwidth (Mbps):

Client RAM Buffer (MB):

JavaScript / TypeScript (AWS SDK v3)

// AWS SDK for JavaScript v3 (@aws-sdk/lib-storage)
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import fs from "fs";

const s3 = new S3Client({ region: "us-east-1" });
const fileStream = fs.createReadStream("./large-payload.tar.gz");

const parallelUpload = new Upload({
  client: s3,
  params: {
    Bucket: "my-production-bucket",
    Key: "backups/large-payload.tar.gz",
    Body: fileStream,
  },
  // Tuned for 250 GB upload
  partSize: 32 * 1024 * 1024, // 32 MB
  queueSize: 13, // Parallel threads
  leavePartsOnError: false,
});

parallelUpload.on("httpUploadProgress", (progress) => {
  console.log(`Uploaded ${progress.loaded} of ${progress.total} bytes`);
});

await parallelUpload.done();

Python (Boto3 TransferConfig)

# Python Boto3 S3 TransferConfig
import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client('s3')

# Tuned for 250 GB upload
config = TransferConfig(
    multipart_threshold=64 * 1024 * 1024,       # 64 MB
    max_concurrency=13,                       # Concurrency threads
    multipart_chunksize=32 * 1024 * 1024,   # 32 MB part size
    use_threads=True
)

s3.upload_file(
    './large-payload.tar.gz',
    'my-production-bucket',
    'backups/large-payload.tar.gz',
    Config=config
)

Mastering AWS S3 Multipart Upload Architecture

For files larger than 100 megabytes, AWS recommends using S3 Multipart Uploads. Multipart uploads split large objects into independent byte-range chunks that upload concurrently across parallel worker threads, improving throughput and providing automatic per-part retry resilience against transient network failures.

The Strict 10,000 Part Limit

Amazon Simple Storage Service (S3) enforces a hard maximum of 10,000 parts per multipart upload. If your upload tool uses the default 5MB part size for a 100GB or 1TB database backup, the upload will abruptly crash with an EntityTooLarge or TooManyParts error once part 10,001 is reached.

Tuning Concurrency vs Client Memory Footprint

RAM Allocation Formula: Memory Buffer ≈ Concurrency Threads × Part Size × 1.2. Running 10 concurrent threads with 64MB parts requires ~768MB of client RAM.
AbortIncompleteMultipartUpload Lifecycle Rule: Unfinished multipart uploads accumulate hidden S3 storage billing unless an S3 Lifecycle Rule is configured to clean up aborted parts after 7 days.