Set Up AWS CLI and Download Your S3 Files From the Command Line

Henry Bley-Vroman, Former Senior UI Developer

Article Categories: #Code, #Front-end Engineering, #Content Management

Posted on

Have an AWS task that's awkward when done in the web interface? AWS CLI sets up easily and has a full command suite

The other day I needed to download the contents of a large S3 folder. That is a tedious task in the browser: log into the AWS console, find the right bucket, find the right folder, open the first file, click download, maybe click download a few more times until something happens, go back, open the next file, over and over. Happily, Amazon provides AWS CLI, a command line tool for interacting with AWS. With AWS CLI, that entire process took less than three seconds:

$ aws s3 sync s3://<bucket>/<path> </local/path>

Getting set up with AWS CLI is simple, but the documentation is a little scattered. Here are the steps, all in one spot:

1. Install the AWS CLI

You can install AWS CLI for any major operating system:

2. Get your access keys

Documentation for the following steps is here.

  1. Log into the IAM Console.
  2. Go to Users. AWS web interface for viewing all users
  3. Click on your user name (the name not the checkbox). AWS web interface for managing a user
  4. Go to the Security credentials tab. AWS web interface for viewing user security credentials
  5. Click Create access key. Don't close that window yet! AWS web interface for creating a new access key
  6. You'll see your Access key ID. Click "Show" to see your Secret access key. AWS web interface for revealing your secret key
  7. Download the key pair for safe keeping, add the keys to your password app of choice, or do whatever you do to keep secrets safe. Remember this is the last time Amazon will show this secret access key.

3. Configure AWS CLI

Run aws configure and answer the prompts.

Each prompt lists the current value in brackets. On the first run of aws configure you will just see [None]. In the future you can change any of these values by running aws cli again. The prompts will look like AWS Access Key ID [****************ABCD], and you will be able to keep the configured value by hitting return.

$ aws configure
AWS Access Key ID [None]: <enter the access key you just created>
AWS Secret Access Key [None]: <enter the secret access key you just created>
Default region name [None]: <enter region - valid options are listed below >
Default output format [None]: <format - valid options are listed below >

  • Valid region names (documented here) are

    RegionName
    ap-northeast-1Asia Pacific (Tokyo)
    ap-northeast-2Asia Pacific (Seoul)
    ap-south-1Asia Pacific (Mumbai)
    ap-southeast-1Asia Pacific (Singapore)
    ap-southeast-2Asia Pacific (Sydney)
    ca-central-1Canada (Central)
    eu-central-1EU Central (Frankfurt)
    eu-west-1EU West (Ireland)
    eu-west-2EU West (London)
    sa-east-1South America (Sao Paulo)
    us-east-1US East (Virginia)
    us-east-2US East (Ohio)
    us-west-1US West (N. California)
    us-west-2US West (Oregon)

  • Valid output formats (documented here) are
    • json
    • table
    • text

4. Use AWS CLI!

In the example above, the s3 command's sync command "recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files" (from s3 sync's documentation). I'm able to download an entire collection of images with a simple

aws s3 sync s3://s3.aws-cli.demo/photos/office ~/Pictures/work

But AWS CLI can do much more. Check out the comprehensive documentation at AWS CLI Command Reference.

Related Articles