Bootstrapping on AWS fails with: AuthFailure: AWS was not able to validate the provided access credentials

I tried manually bootstrapping a controller on AWS using instance profiles, I’ve followed the role permissions assignment as described here: Juju AWS Permissions but I still get the error below:

juju bootstrap aws
ERROR 
The provided credentials could not be validated and 
may not be authorized to carry out the request.
Ensure that your account is authorized to use the Amazon EC2 service and 
that you are using the correct access keys. 
These keys are obtained via the "Security Credentials"
page in the AWS console: operation error EC2: DescribeAccountAttributes, https response error StatusCode: 401, RequestID: af2496c5-d990-44b5-98b4-2cbf454ae70c, api error AuthFailure: AWS was not able to validate the provided access credentials

Hi @dparv,

Please refer to docs Using AWS Instance Profiles With Juju 2.9

As discussed on Matrix we don’t support using instance profiles from the Juju client it self at the moment. This is something that we could potentially add.

Instance profiles need to be configured during bootstrap and told to Juju so that when it is bringing up the controller machine in AWS it knows what instance profile to assign to the machine.

So how do I even bootstrap a controller? What credentials should I use? The credential definitions here say I have to use this:

`auth-type` : `access-key` , which requires you to provide your access key and your secret key

I am adding exactly those from an IAM role that I’ve configured as described in the above mentioned discourse post and getting the error above.

Hi @dparv,

Think we need to take a step back and figure out what you are trying to do.

For some context AWS roles do not have secret keys and access keys. To my knowledge it is only users that can get this. However a user can assume a role to be granted the permissions in the role. Much like instance profiles which provide AWS access via instance metadata and allowed the processes on the given machine to then assume the role that is associated with the instance metadata.

Based on what you have provided above you are currently at the stage of bootstrapping a Juju Controller. How this work is the Juju client juju uses the local cloud credentials on the system to create resources in the specific cloud (this case AWS) so that it can stand up a Juju controller.

One the Juju controller is up and running it then gets it’s own cloud credentials to use for administering the models in that cloud.

We don’t currently support instance profiles for the Juju client. That is assuming that you are running the Juju client from within an EC2 machine and not on your local computer.

The first step is to provide the Juju client with a set of credentials for AWS. This can be done with juju add-credentials. See our reference documentation here Juju | Credential

Once we have configured our Juju client we can now bootstrap a controller to the AWS cloud to use instance profiles and by extension roles. To do this we must pass a few extra configuration flags to Juju bootstrap command to either tell it to make a new AWS instance profile or to use an already established one that you have setup. See my documentation around this here: Using AWS Instance Profiles With Juju 2.9

If you are still having trouble with this process it would be good to know what steps from the above have and haven’t been done. I would say our wording for the error message you are receiving is not great. I will take a look from our side to see if I can improve this to make it more helpful.

Just to recap we have to seed your Juju client with credentials for AWS first. This is an important step so the client can create the resources needed for the controller. This is how we overcome the chicken and egg problem when deploying controllers to a cloud.

Well, it doesn’t really matter if I am using IAM role credentials or personal user credentials. Doesn’t matter if I run it on my workstation outside of AWS or inside an EC2 instance. I get the same error.

Ftr, this is how I am getting the IAM role secret key and access key from within a EC2 instance:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME

And IAM roles with full admin access are indistinguishable from real personal user accounts, so this doesn’t make sense.

I’ve opened a bug about this here: Bug #2052410 “Can't bootstrap controller on AWS” : Bugs : Canonical Juju