Amazon Virtual Private Cloud (VPC) lets you create a secure, isolated network within AWS. For detailed information refer to the AWS VPC’s official documentation.

Integration Method: S3

Tables: Network Activity (4001)

This integration supports the following events.

Note:

AWS VPC doesn’t follow a traditional versioning system. Instead, it is a continuously updated cloud service.

Prerequisites

• The user should have access to the AWS portal.

• The user should have access to the VPC dashboard and access to create an S3 bucket.

• The user should have access to the DataBee console.

Configuration overview

1. Create S3 Bucket for AWS VPC flow logs.

2. Create an AWS VPC flow log.

3. Add the AWS VPC data feed in the DataBee console with the below parameters.

   DataBee Parameter	AWS VPC Parameter
   S3 Bucket Name	Bucket Name
   AWS Region	Bucket Region

4. Apply the generated Bucket policy from DataBee to S3 Bucket policy.

5. Create an event notification for bucket with the AWS SQS ARN from DataBee.

AWS VPC Configuration

Create S3 Bucket

1. Sign in to the AWS Management Console and open the Amazon S3 console.

2. In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region in which you want to create a bucket.

   

3. Create Bucket

   1. In the left navigation pane, choose General purpose buckets.

   2. Choose Create bucket. The “Create bucket” page opens.

   

4. For Bucket name, enter a name for your bucket. The bucket name must:

   1. Be unique within a partition. A partition is a grouping of regions. AWS currently has three partitions: aws (commercial Regions), aws-cn (China Regions), and aws-us-gov (AWS GovCloud (US) Regions).

   2. It is between 3 and 63 characters long.

   3. Consist only of lowercase letters, numbers, periods (.), and hyphens (-). For best compatibility, we recommend that you avoid using periods (.) in bucket names, except for buckets that are used only for static website hosting.

   4. Begin and end with a letter or number.

   5. For a complete list of bucket-naming rules, see General purpose bucket naming rules.

   

5. Configure the Object Ownership, Block Public Access settings, Bucket Versioning, and Tags according to your preferences.

6. Select the encryption type for Default encryption of object. If you want to encrypt the data with AWS KMS key, then follow Create a KMS key.

   

7. Click Create bucket.

8. Navigate to Properties of the bucket. Copy the ARN value of bucket for future use.

   

9. Navigate to Permissions > Bucket policy of the bucket. Click on the Edit button.

   

10. Add policy for the VPC Flow logs

    1. Navigate to bucket permissions for VPC flow logs. Copy the permissions and paste in the Bucket policy.

    2. Here replace the following values.

       • my-s3-arn – replace with the bucket arn value.

       • account_id – replace with the account_id. Click on the top-right profile menu to get the account id.

    

    • arn:aws:logs:region:account_id – replace with the aws region and account_id.

    • bucket_name – replace with the bucket name.

    

11. Click on Save changes.

Create VPC Flow Logs

1. Sign in to the AWS Management Console and open the Amazon VPC console.

2. Navigate to the Your VPCs tab.

   

3. Select the existing VPC or create VPC.

4. Create VPC flow logs.

5. Navigate to the Flow logs tab.

6. Click on Create flow log.

   

7. Enter the flow log settings details

   1. Enter the Name.

   2. Select All for Filter.

   3. Select Maximum aggregation interval as per your preference.

   

8. Select the Destination as S3 bucket to publish the flow log data.

   1. Paste the bucket ARN value.

   

9. Select the log record format

   1. Select the Custom format.

   2. Click on the Select All button.

   

10. Select Log file format and partition logs by time

    1. Select Text (default) as Log file format.

    2. Select Partition logs by time as per your preference.

    

11. Click Create flow log button.

DataBee Configuration

1. Login to the DataBee UI, navigate to Data > Data Feeds and click the Add New Data Feed button.

   

2. Search for AWS VPC and click on it as shown below.

   

3. Click on the AWS S3 collection method.

   

4. In configuration, enter feed contact information, confirm the following and click Next.

   • S3 Bucket Name: Paste the bucket name

   • AWS Region: Select the AWS Region in which bucket is created

   • Compression: Select gzip

   • Content Type: Select SSV

   • KMS Encryption Key (optional): Paste the KMS Key if setup the KMS key for bucket

   • SNS Topic ARN (optional): Enter the SNS Topic ARN

   

5. Copy the IAM policy and KMS policy to apply to your AWS S3 bucket.

6. Copy the policy statement to clipboard.

   

7. Go to the Permissions tab in the bucket. Navigate to Bucket policy and click the Edit button.

   

8. Add the bucket policy statement copied from DataBee.

   

9. Save the bucket policy.

10. Click Next.

11. Setup AWS S3 bucket notification.

12. Copy the SQS ARN to clipboard.

    

13. Go to the Properties tab of the bucket.

    

14. Navigate to Event notifications and click Create event notification.

    

15. Enter the General configuration details.

    

16. Select the Event types for Object creation.

    

17. Set the Destination

    1. Select SQS queue as Destination.

    2. Select option to Enter SQS queue ARN.

    3. Paste the SQS queue ARN.

18. Click Save Changes.

    

19. Click on Test Connection button to check the connection between AWS VPC and DataBee.

    

20. Click Submit.

Troubleshooting Tips

• If you are facing an error while testing the connection, then make sure the bucket policy is applied properly and check the SQS event notification configuration as described in the steps.