Tips for faster uploads

LIBSAFE Advanced offers a huge scaling capability when handling file uploads and downloads. While the downloads scale without limitation and without any particular approach, there are some recommended techniques for uploading content faster when you plan to upload volumes over 10TB.

Three main elements contribute to faster uploads. Combine them all to obtain the best performance:

  • Upload tool in use

  • Parallelization of uploads

  • Upload prefixes and containers.

You can use any S3-compatible tool to upload content to your Data Containers, but to get the best results, we recommend the use of the most recent version of the AWS CLI. Unlike other tools, it has been optimized for parallelization.

Some Linux distributions install older versions of the client by default. Make sure you are always using the latest version.

You can use the AWS CLI with LIBSAFE Advanced guide for examples on how to use this tool.

Parallelization of uploads

If you are planning to transfer a large amount of files, with a relatively small size each, you can benefit a lot from parallelizing multiple upload processes.

The AWS CLI tool already has some built-in parallelization, but to achieve better results, you can launch multiple processes in parallel.

Upload prefixes and containers

Amazon S3 supports a request rate of 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket. The resources for this request rate aren't automatically assigned when a prefix is created. Instead, as the request rate for a prefix increases gradually, Amazon S3 automatically scales to handle the increased request rate.

If there is a fast spike in the request rate for objects in a prefix, Amazon S3 might return 503 Slow Down errors while it scales in the background to handle the increased request rate. To avoid these errors, you can:

​

PreviousAdvanced API File SearchNextConfiguring Azure SAML-based authentication

Last updated