Background
Recently I was working on a task that needed to upload and download a test file on AWS S3 bucket. For this, I used AWS Java SDK. The functionality seemed to work fine until the code was deployed in production. In this post, I will try to explain why does this exception occur and how we can fix this.
Code that I had used for the test upload was as follows -
try {
BasicAWSCredentials awsCreds = new BasicAWSCredentials("YourAccessKeyId", "YourSecretKey");
AmazonS3 s3client = AmazonS3ClientBuilder.standard()
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.build();
s3client.putObject(storage.getBucketName(), "test.txt", "Done!");
} catch (AmazonServiceException ase) {
logger.error(
"Caught an AmazonServiceException, which means your request made it to Amazon S3, but was rejected with an error response for some reason. storage : {}",
storage, ase);
logger.error("Error Message: {}", ase.getMessage());
logger.error("HTTP Status Code: {}", ase.getStatusCode());
logger.error("AWS Error Code: {}", ase.getErrorCode());
logger.error("Error Type: {}", ase.getErrorType());
logger.error("Request ID: {}", ase.getRequestId());
} catch (AmazonClientException ace) {
logger.error(
"Caught an AmazonClientException, which means the client encountered an internal error while trying to communicate with S3, such as not being able to access the network storage : {}",
storage, ace);
logger.error("Error Message: {}", ace.getMessage());
} catch (Exception ex) {
logger.error("Got exception while testing upload to S3", ex);
}
I had tested this code by deploying it on one of our EC2 instances and it worked fine. I will probably explain this at a later point as to why it worked with EC2 instance but there is a major flaw with the above code that finally ends up throwing an exception - "Unable to find a region via the region provider chain"
Relevant Stacktrace:
com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:352) at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:386) com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
The problem
The problem with the above piece of code is that we are not supplying the AmazonS3ClientBuilder with the region. Even though S3 is a global service, the bucket that you create are region specific. Each region would have its own endpoint and hence AWS S3 SDK need to know which region to know the endpoint it needs to make the API call to.
The Solution
The simplest solution is to pass the region to the AmazonS3ClientBuilder builder explicitly as follows -
AmazonS3 s3client = AmazonS3ClientBuilder.standard() .withCredentials(new AWSStaticCredentialsProvider(awsCreds)) .withRegion(Regions.US_EAST_1) .build();
NOTE: After you build a client with the builder, it's immutable and the region cannot be changed. If you are working with multiple AWS Regions for the same service, you should create multiple clients—one per region.
Understanding the solution
The problem and the solution might appear simple but there are various aspects you need to understand about setting region in your S3 builder.
There are various ways your AWS SDK can infer the region to use instead if you explicitly providing it yourself.
NOTE: You must use client builders to have the SDK automatically detect the region your code is running in. This is not applicable if you are using client constructor and default region from the SDK will be used.
If you don't explicitly set a region using the withRegion methods, the SDK consults the default region provider chain to try and determine the region to use.
There are various ways your AWS SDK can infer the region to use instead if you explicitly providing it yourself.
NOTE: You must use client builders to have the SDK automatically detect the region your code is running in. This is not applicable if you are using client constructor and default region from the SDK will be used.
If you don't explicitly set a region using the withRegion methods, the SDK consults the default region provider chain to try and determine the region to use.
- Any explicit region set by using withRegion or setRegion on the builder itself takes precedence over anything else.
- The AWS_REGION environment variable is checked. If it's set, that region is used to configure the client.
- NOTE: This environment variable is set by the Lambda container.
- The SDK checks the AWS shared configuration file (usually located at ~/.aws/config). If the region property is present, the SDK uses it.
- The AWS_CONFIG_FILE environment variable can be used to customize the location of the shared config file.
- The AWS_PROFILE environment variable or the aws.profile system property can be used to customize the profile that is loaded by the SDK.
- The SDK attempts to use the Amazon EC2 instance metadata service to determine the region of the currently running Amazon EC2 instance.
If the SDK still hasn't found a region by this point, client creation fails with an exception. And we saw what exception it throws :) It is is the reason I am writing this post. Also, you must have realized why it worked for me with the EC2 instance. As per point number 4, SDK attempts to use the Amazon EC2 instance metadata service to determine the region. Hope this helps!
