How to Optimize Transfer Between S3 and EC2

2019/03/30

In this article we’ll examine how to achieve consistent transfer speeds (including writing to disk) of around 500 Megabytes per second between S3 and EC2. The methodology described here is being used successfully in production.

We’ll begin by looking at the hardware required, then move on to thinking about software. Throughout this article we’ll also consider some of the trade-offs and potential downsides of this approach. If you’d like to skip the reasoning and just see the final hardware configuration and software recommendations, jump down to the Conclusions section.

Hardware

As a brief refresher, EC2 allows us to provision virtual servers which in AWS parlance are called instances. The main type of storage used for instances is network-attached block storage called EBS. EBS is provisioned in units called volumes.

An EC2 instance’s instance type determines its hardware specifications. The same applies to a volume’s volume type. We’re going to select a combination of instance type and volume type that will maximize performance by avoiding bottlenecks and prioritizing the specs that matter.

Let’s consider how bytes are going to get from S3 onto our disk. They’re going to be requested from S3 via the network, transferred into RAM by way of the CPU, then written to the EBS volume. Let’s work backwards starting from the EBS volume.

We have two main choices to make when it comes to EBS: volume type and size. For volume types there are:

We’re going to eliminate io1 and sc1 off the bat. io1 is very expensive and to take advantage of its performance the volume must be attached to a Nitro-enabled instance. sc1 is too slow for what we want.

That leaves gp2 and st1 to compare. gp2 volumes have more IOPS which is good for small, random IO. st1 volumes have higher max throughput and are better for large, sequential IO. st1 is a better fit for our goal of maximizing transfer speed.

The next question is how big of an st1 volume do we want? EBS ties performance to volume size. The smallest (and therefore cheapest) st1 volume that guarantees the max 500 MB/s throughput at all times is 12.5 TiB (12800 MiB). Therefore, we’re going to use a 12.5 TiB st1 volume.

Caveat emptor: this configuration sacrifices random IO for sequential read and write speed. A gp2 volume must still be used as the instance’s boot drive.

The next thing to consider is CPU, RAM, and network interface, thus moving on to EC2 options. EC2 instances also have a max EBS bandwidth. We’re going to want an instance with at least 500 MB/s (4000 Mb/s) of EBS bandwidth to avoid bottlenecking. That means we need an EBS-optimized instance. We also need at least 500 MB/s of network bandwidth. Many EC2 instances have network performance “up to” a certain value, or may just be classified as “low”, “moderate”, or “high.” In order to maintain consistent transfer speed, we’re going to choose an instance type with a hard value of at least 500 MB/s.

There are quite a few instance types that meet the above requirements. The most cost-effective that I’ve been able to find is the c4.8xlarge with:

If your needs differ, any instance type that meets the above requirements to avoid bottlenecking should be able to achieve the same performance.

That wraps up hardware considerations - let’s move on to software.

Software

Like many other AWS products, S3 is designed to scale horizontally. This means that in order to achieve fast transfers, we’ll need to take advantage of multipart downloads/uploads with concurrent requests. Therefore, our software needs to either be asynchronous (Node.js being an example), or multithreaded (Go being an example).

Any implementation that is able to make concurrent requests to S3 in order to perform a multipart download/upload should work, but I’m going to use Go as an example here because the AWS SDK for Go includes a package called s3manager that provides this functionality with a very simple and easy to use API.

A deep dive into s3manager is outside the scope of this article, but the following settings have worked for me:

Conclusions

In summary, I’ve found a c4.8xlarge with a 12.5 TiB st1 volume to provide a consistent 500 MB/s of transfer from and to S3 when running software that takes advantage of S3’s horizontal scalability through concurrent requests such as the AWS SDK for Go’s s3manager package.

Thanks for reading - I hope this article has been helpful, interesting, or both. Please feel free to contact me with questions, comments, or criticism.