AWS Data Sources: Users, Credentials and Permissions

Adding an AWS data source into DataBee using API ingest will require API credentials and the associated roles to be created. This article goes through the process of creating a service user, creating API credentials for that user and assigning a role with required permissions.

This setup walks through:

• Creating or using a service user.

• Generating an AWS Access Key and AWS Secret Key tied to the user.

• Adding the required IAM role with the required permissions for the feed.

Setting up AWS Access Key, Secret Key and Permissions

Creating a New Service User

This section outlines the steps to create a new user, API keys and assigning policy permissions to the user.

1. Sign in to the AWS Management Console.

2. Navigate to IAM (Identity & Access Management).
    
   

3. In the left-hand menu, select Users.
    
   

4. Click Create User.
    
   

5. Provide a username and click Next.
    
   

6. If you have the required IAM policy created, search and Attach policies directly.
    
   

7. If the IAM policy doesn’t exist, click Create policy which will open a new browser tab.
    
   

8. Select the respective AWS Service from the drop down under Service. In this example, CloudWatch Logs is selected.
    
   

9. Select all the permissions that are mentioned in the API documentation for the data source (Describe permissions -> List privilege, Get permissions -> Read privilege).
    
   

10. At the bottom, you can limit the permission by adding specific ARNs and then click Next. For example, in the below screenshot we are restricting specific CloudWatch log groups.
     
    

11. Alternatively, the policy can be specified using the JSON policy editor by clicking the JSON button. Ensure all the mentioned permissions in the data source API documentation are covered. Click Next.
     
    

12. Give the policy an appropriate name and review the permissions, and click Next.
     
    

13. Move back to the Create user browser tab, once the policy creation is successful.
     
    

14. In the Create User tab, click Refresh Policies button and search for the policy name in the Search bar. Select it and click Next.
     
    

15. Review the changes and click Create User.
     
    

16. Once the user is created successfully, you can follow the steps mentioned here to create AWS Access Key and AWS Secret Key.

Using an Existing Service User

To use an existing service user, the API credentials must be created.

1. Sign in to the AWS Management Console with Admin privileges.

2. Navigate to IAM (Identity & Access Management).
    
   

3. In the left-hand menu, select Users.
    
   

4. Find and select the existing service user.
    
   

5. Go to the Security credentials tab.
    
   

6. Under the Access keys, click Create access key.
    
   

7. Select Third-party service and click Next.
    
   

8. Set the Description tag value if required, and click Create access key.
    
   

9. Copy and securely store the Access Key ID and Secret Access Key.
    
   

10. Once the AWS Access Key and AWS Secret Key are securely stored, you will need to add the required permissions through an IAM policy. These steps are outlined here.

Adding permissions to an IAM user

This section walks through how to add specific permissions for the API user.

1. Sign in to the AWS Management Console with Admin privileges.

2. Navigate to IAM (Identity & Access Management).
    
   

3. In the left-hand menu, select Users.
    
   

4. Find and select the existing service user.
    
   

5. Under the Permissions tab, Click Add permissions and then select Create inline policy.

    

   
   

6. Select the respective AWS Service from the drop down under Service. For example, CloudWatch Logs is selected.
    
   

7. Select all the permissions that are mentioned in the API documentation for the data source (Describe permissions -> List privilege, Get permissions -> Read privilege).
    
   

8. At the bottom, you can limit the permission by adding specific ARNs and then click Next. For example, in the below screenshot we are restricting specific CloudWatch log groups.
    
   

9. Alternatively, the policy can be specified using the JSON policy editor by clicking the JSON button. Ensure all the mentioned permissions in the data source API documentation are covered. Click Next.
    
   

10. Name the policy, review its permissions and click Create Policy.
     

Sample Policy JSON:

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "logs:DescribeLogGroups",
               "logs:DescribeLogStreams",
               "logs:GetLogEvents",
               "logs:FilterLogEvents"
            ],
           "Resource": " arn:aws:logs:{Region }:{Account}:log- group:{LogGroupName}"
       }
   ]
}

Next Steps

Once authentication and permissions are set up, you can proceed with Data Source documentation for onboarding it into DataBee to retrieve AWS log events using API Ingestion method.