How do I use shell script to check if a bucket exists?

Question

I have aws cli installed. I'm just not sure how to do this in shell script.

when I run command aws s3 ls s3://bucket it would give me something like this

A client error (NoSuchBucket) occurred when calling the ListObjects operation: The specified bucket does not exist

That means the bucket doesn't exist. So I want to run that from shell script and check if grep finds it. But my command doesn't work.

if [ $(aws s3 ls "s3://$S3_BUCKET" | grep 'NoSuchBucket' &> /dev/null) == 0 ] 
then
    echo "$S3_BUCKET doesn\'t exist please check again"
    exit
fi

It just gave me this

backup.sh: 20: [: 0: unexpected operator

Updated

I changed the script to be

echo "S3_BUCKET=$S3_BUCKET"
if aws s3 ls "s3://$S3_BUCKET" | grep -q 'AllAccessDisabled'    
then
    echo "$S3_BUCKET doesn\'t exist please check again"
    exit
fi

And this is the output I got

A client error (AllAccessDisabled) occurred when calling the ListObjects operation: All access to this object has been disabled

So the text contains AllAccessDisabled but I still don't the echo the next line.

Does this answer your question? [AWS S3: How to check if a file exists in a bucket using bash](https://stackoverflow.com/questions/41871948/aws-s3-how-to-check-if-a-file-exists-in-a-bucket-using-bash) — Amri, Apr 28 '20 at 12:02

score 51 · Answer 1 · answered Jan 31 '17 at 15:51

51

The s3api head-bucket is more direct and does not incur the expense of listing a bucket with many files.

http://docs.aws.amazon.com/cli/latest/reference/s3api/head-bucket.html

if aws s3api head-bucket --bucket "$S3_BUCKET" 2>/dev/null; then

answered Jan 31 '17 at 15:51

John Milton

511
1
4
2

1

No it shouldn't, because it doesn't work in the case that the bucket is owned by another user: http://pasted.co/7dae7402 – scubbo Nov 01 '17 at 06:23
1

Does this differentiate between a bucket that doesn't exist and a bucket that you don't have access to? – brianmearns Apr 12 '19 at 15:03
1

No, it doesn't. It will fail if the bucket does not exist as well as if it exists, but is owned by another user...just with different error messages. You need to inspect the error message. – Brooks Sep 29 '19 at 00:28

Etan Reisner · Accepted Answer · 2015-06-26T16:23:32.870

20

The code you listed wouldn't have given you that error.

If you had written the script without the space between the leading [ and the $( that would have.

Also grep isn't going to output 0 in that case so that test isn't going to work the way you want.

If you want to test whether grep found anything then you want to use the -q argument to grep like this:

if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'NoSuchBucket'
then

edited Jun 26 '15 at 16:23

answered Jun 26 '15 at 16:13

Etan Reisner

77,877
8
106
148

1

That's error output. That's probably going to standard error and not standard output. You need to redirect for that. See updated code. – Etan Reisner Jun 26 '15 at 16:24

score 17 · Answer 3 · edited Aug 22 '19 at 00:35

I know this is an old question, but I came here for some answers and using some existing answers and some experimentations of my own came up with a script which handles different return values:

bucketstatus=$(aws s3api head-bucket --bucket "${s3_bucket}" 2>&1)
if echo "${bucketstatus}" | grep 'Not Found';
then
  echo "bucket doesn't exist";
elif echo "${bucketstatus}" | grep 'Forbidden';
then
  echo "Bucket exists but not owned"
elif echo "${bucketstatus}" | grep 'Bad Request';
then
  echo "Bucket name specified is less than 3 or greater than 63 characters"
else
  echo "Bucket owned and exists";
fi

upvote for an easy to understand answer that covers all the cases — jcollum, Aug 23 '21 at 23:30

danelips · Answer 4 · 2016-06-08T01:20:58.360

Here's how I did it, the other answer will say there is a bucket if there is an AWS error that doesn't contain 'NoSuchBucket' such as token expiry

echo "Checking S3 bucket exists..."                                                                                                                                                                                                           
BUCKET_EXISTS=true                                                                                                                                                                                                                            
S3_CHECK=$(aws s3 ls "s3://${BUCKET_NAME}" 2>&1)                                                                                                                                                 

#Some sort of error happened with s3 check                                                                                                                                                                                                    
if [ $? != 0 ]                                                                                                                                                                                                                                
then                                                                                                                                                                                                                                          
  NO_BUCKET_CHECK=$(echo $S3_CHECK | grep -c 'NoSuchBucket')                                                                                                                                                                                     
  if [ $NO_BUCKET_CHECK = 1 ]; then                                                                                                                                                                                                              
    echo "Bucket does not exist"                                                                                                                                                                                                              
    BUCKET_EXISTS=false                                                                                                                                                                                                                       
  else                                                                                                                                                                                                                                        
    echo "Error checking S3 Bucket"                                                                                                                                                                                                           
    echo "$S3_CHECK"                                                                                                                                                                                                                          
    exit 1                                                                                                                                                                                                                                    
  fi 
else                                                                                                                                                                                                                                         
  echo "Bucket exists"
fi

score 5 · Answer 5 · answered Feb 13 '19 at 10:13

5

The way recommended by aws is aws s3api head-bucket --bucket $S3_BUCKET See https://docs.aws.amazon.com/cli/latest/reference/s3api/head-bucket.html

answered Feb 13 '19 at 10:13

Vincent

391
3
10

score 3 · Answer 6 · answered Sep 15 '20 at 13:35

I stumbled upon this question, and would like to add the solution using the AWS recommended way using head-bucket.

BUCKET_EXISTS=$(aws s3api head-bucket --bucket <bucket_name> 2>&1 || true)
if [ -z "$BUCKET_EXISTS" ]; then
  echo "Bucket exists"
else
  echo "Bucket does not exist"
fi

score 3 · Answer 7 · answered Oct 02 '20 at 14:33

3

Without havent to handle errors:

if [[ ! -z $(aws s3api list-buckets --query 'Buckets[?Name==`bucket-name`]' --output text) ]]; then
  echo "Bucket Exists"
fi

answered Oct 02 '20 at 14:33

Christian

3,708
3
39
60

Very elegant. Best answer. – Bobby Koteski Apr 11 '21 at 03:40

Yogesh Gupta · Answer 8 · 2021-04-09T07:15:40.663

2

#!/bin/bash

if [[ -z $(aws s3api head-bucket --bucket my-bucket 2>&1) ]]; then
        echo "bucket exists"
else
        echo "bucket does not exist or permission is not there to view it."
fi

edited Apr 09 '21 at 07:15

answered May 30 '19 at 00:37

Yogesh Gupta

1,226
1
12
23

think you have this backwards `[[ ! -z ... ]]` is if it is NOT empty – Christian Oct 22 '20 at 16:05
@Christian No, the output will be empty in case bucket exists, else it will spit out an error on stdout. Hope that helps. – Yogesh Gupta Apr 09 '21 at 07:16

score 0 · Answer 9 · edited Apr 27 '21 at 02:33

0

this might me simplest approach

if aws s3 ls "s3://$S3_BUCKET" 2>&1 | grep -q 'An error occurred'
then
    echo "bucket does not exist or permission error."
else
    echo "bucket exist"
fi

edited Apr 27 '21 at 02:33

choasia

10,404
6
40
61

answered Feb 08 '18 at 06:15

vinay

1,004
1
11
27

score 0 · Answer 10 · answered May 23 '22 at 17:27

A nice one liner that builds off John Milton's anser:

bucket_exists=$(aws s3api head-bucket --bucket $bucket_name 2>/dev/null && echo "yes" || echo "no")

A successful response will return a 0 status from the aws command and run the first part of the ternary.

A failed response will return a 254 status from the aws command and run the latter part of the ternary.

IMHO this is a lot easier to read, but to each their own.

score 0 · Answer 11 · answered Jul 07 '23 at 06:31

0

aws s3 ls "s3://$S3_BUCKET" 2>&1 || aws s3api create-bucket --bucket $S3_BUCKET --region us-east-1 --no-cli-pager

This worked good for me as one line command.

answered Jul 07 '23 at 06:31

Temel Karabacak

1

This does not provide an answer to the question. Once you have sufficient [reputation](https://stackoverflow.com/help/whats-reputation) you will be able to [comment on any post](https://stackoverflow.com/help/privileges/comment); instead, [provide answers that don't require clarification from the asker](https://meta.stackexchange.com/questions/214173/why-do-i-need-50-reputation-to-comment-what-can-i-do-instead). - [From Review](/review/late-answers/34653861) – ryanwebjackson Jul 14 '23 at 02:13

How do I use shell script to check if a bucket exists?

11 Answers11