CASA on Amazon Web Services: Costs

Overview of Costs Associated with AWS

Amazon Web Services (AWS) allows researchers to use AWS resources for NRAO data processing. This document presents costs associated with using these resources. Resource types include: Instances, EBS Volumes, EBS Snapshots, S3, and Glacier.

The primary resource utilized is Instances.

Others resources are methods of storing input and output data: EBS, EFS, snapshots, S3, and glacier.

The way to contain costs is to first determine the needs of the code that is to be run. Then AWS resources can be matched to those needs as efficiently as possible given the circumstances.

CASA Hardware Requirements

Running CASA

Knowlege about the job to be run informs choosing suitable hardware. Hardware requirements for running CASA are listed here: https://casa.nrao.edu/casa_hardware-requirements.shtml.

Choosing an Instance

An instance is the best place to start (allocating resources). A list of on-demand instance costs and capabilities are listed here, though be aware it takes a minute to load and nothing will display under "on-demand" while the data load: https://aws.amazon.com/ec2/pricing/. (Spot instances can be utilized to run jobs at much reduced cost. This is covered in another section of this document.) The goal is to select an Instance type that is large enough to do the job but leaves as few resouces idle as possible.

File I/O

Default EBS is generally used. However, options exist to specify different types of EBS, e.g. storage with higher iops, etc., that cost more. This page has pricing details: https://aws.amazon.com/ebs/pricing/. For reference, there is a detailed discussion of storage types here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html.

Instance Store

Some instance types are pre-configured with attached SSD storage called "instance store". If you start such an instance, part of its intrinsic cost is this storage. More details about instance store are here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/add-instance-store-volumes.html#adding-instance-storage-instance.

Selecting an Instance Type for CASA

What to try First

There are over 40 instance types with varying amounts of RAM and CPU. There are, when you look closely, many storage options, but a few recurrent themes emerge. The simple storage system (S3) is great for storing large volumes of data from a few hours to a few months. S3 can be used to stage input and output. You can share the data with other users. EBS storage is the most often used "attached" storage with performance comparable to a very good hard drive attached to your desktop system. However, it's bandwidth goes up with the core count, contrary to ordinary storage. The number of cores and GB of RAM depend entirely on instance type. If you were to want 4 GB RAM per core (as recommended in casa_hardware-requirements) and 10 cores, you'd find the closest instance on the list of instance types, https://aws.amazon.com/ec2/instance-types/. m4.4xlarge is close with 16 "vCPU" and 64 GB. Despite apearances it does not have enough cores. AWS lists "vCPUs" (a.k.a. hyperthread) instead of cores. A vCPU is an internal CPU scheduling trick that is useful for certain classes of programs, but is nearly always detrimental in scientific computing, e.g. CASA. To summarize, 2 Amazon vCPUs equal 1 core. From here on cores are used, 1 core = 2 vCPUs.

Continuing with the example of looking for an instance that meets the contraints of 10 cores, 4 GB RAM per core. It makes sense to look at core count first. The listing of instances with their core count, memory, and cost are here: https://aws.amazon.com/ec2/pricing/. There are no instances with 10 cores. The closest are 8 and 16 cores. So we'll look at >=16 core instances. Also, we'll throw out any instances that do not have: RAM >= (#cores * 4 GB RAM). What is left (without looking at immensely large and therefore expensive instances) are these:

m4.10xlarge with 20 cores and 160 GB of RAM. Cost: $2.394/hour
x1.32xlarge with 64 cores and 1962 GB of RAM. Cost: $13.338/hour
r3.8xlarge with 16 cores and 244 GB of RAM. Cost: $2.66/hour
d2.8xlarge with 18 cores and 244 GB of RAM. Cost: $5.52/hour

Selecting 10 cores produced a results list that contains the most expensive instances. If it's feasible to use a number of cores that is an exponent of 2, a more efficient arrangement may result, e.g. what instances with 2^3 = 8 cores also meet the criterion of 4GB RAM per core? For example:
m4.4xlarge 8 cores, 64 GB RAM. Cost: $0.958/hour
c3.8xlarge 8 cores, 60 GB RAM. Cost: $1.68/hour (instance store)

The c3.8xlarge, although very similar to the m4.4xlarge, costs 75% more per hour. That's becuase c3.8xlarge comes pre-configured with local (instance store) storage. This is charged for even when not used. It is something to watch out for. Instance store can be useful, but it is tricky to make use of. The use of instance store is outside the scope of this document. When considering 8 core instances, m4.4xlarge appears to be the most attractive option in this case.

m4.4xlarge 8 cores 4 GB/core $0.958/hour
r3.8xlarge 16 cores ~15 GB/core $2.66/hour
r3.4xlarge 8 cores ~7.6 GB/core $1.33/hour

r3.4xlarge is not far behind in price. And it has more RAM as well as 320 GB of instance store storage. So zeroing in on the best instance takes some time. However, it is not time well spent to find the most efficient instance until many instances are to be run or an instance is run for a long period of time.

details what might be most useful for CASA. It is quite specific and, while it can be meaningfully translated to AWS, for now, to get started, Dual Intel E5-2640v3 2.6GHz 16 total physical cores (32 AWS cores) 64 GB RAM (4GB per physical core). 20 TB EBS storage optional: S3 storage for results There are a number of detailed considerations when choosing the right combination of RAM, CPU, and storage -- probably too many combinations to outline here, so when you are ready to run many (100's) of instances, you can dial it in to the most efficient combination of AWS resources. It is easy enough to tear one instance down and try another.

What Instance(s) to Consider for Big or Long Jobs

So, to begin, it is probably best to choose EC2 as your primary storage, S3 for cold storage, and an instance with >=4GB RAM per core. A more detailed discussion of these (and other) hardware considerations is outlined in the preceding "hardware-requirements" link, which is essential to consider if running many instances or running instances for a long duration. What is covered here is what is sufficient to get started. Keep in mind that, since AWS has hyperthreading turned on, their "2 cores" means "1 physical core" (2 hyperthreads) to the result of us. For example, an AWS "8 core" instance type is actually only 4 cores. You should find an instance that has "n AWS cores"/2 * 4GB = n * 2 GB of RAM. So if decide you want a system with 4 actual cores, select an AWS "8 core" system with >= 16 GB of RAM. That should be sufficent to get started. As you use AWS more, you'll want to invest more time in optimizing the instance type based on the details of your run and how well it uses various instances for the job you'll run. If you are running only a few instances, such optimizations are not worth much effort, but if you plan to run hundreds of jobs, this can have a very significant impact on total run time and cost. The 4GB per physical core rule should be sufficient to get started. Once the instance is booted, ssh into the instance. Use the Public IP address shown in the console's display of Instances. For example, ssh -i ~/my-key.pem ec2-user@52.40.193.173

AWS Storage for CASA

Root Volume

Starting an Instance with an NRAO AMI and accepting the storage defaults creates a suitable root volume for CASA. (If desired, exhaustive detail on root volumes is availabe at the AWS website: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/RootDeviceStorage.html.)

Additional EBS Volumes

At any time additional EBS volumes can be added. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-creating-volume.html.

EBS snapshots are mentioned. These are suspended EBS volumes. They get transparently stored in S3 storage until they are recovered.