Amazon S3

You will need permission to update your bucket's Access Policy and Event notification settings.

Amazon Web Services (AWS) makes sharing data through their simple storage service (S3) buckets easy. You will be setting up DataBee to read from your S3 Bucket.

Configure an Amazon S3 Ingest Data Feed

Start by clicking on the Data button and select Data Feeds.

Next, click on the +Add New Data Feed button.

Choose your preferred data feed from the list of supported data feeds.

You will now be directed to choose your ingestion method. To fetch data via AWS S3, click on AWS S3.

You will now be redirected to the "Configure Data Feed" page. Follow the flowchart displayed on the right, for a visual guide, outlining the step-by-step configuration process.

Step 1: Configure Data Feed

Enter data in the fields below.

• Data Feed Name: a user-friendly name for the data feed.

• Owner Name: enter a point of contact for the data feed.

• Owner E-mail: email address for the contact.

• S3 Bucket Name: name of the bucket to ingest data from.

• AWS Region: AWS region of the bucket.

• Delete objects in the S3 bucket after read:  if this field is checked, then the file will be deleted after ingesting the data.

• Compression: indicates what type of decompression (if any) should be applied to the objects before reading.

  

• Content Type: indicates how to parse the uncompressed content.

  

• KMS Encryption Key: provide an optional KMS encryption key if the S3 bucket is encrypted.

  Once you have entered the required information, click Next.

  

Step 2: Apply IAM and KMS policy

• Copy the IAM policy and save the policy that you want to apply to your AWS S3 bucket.

• If KMS is enabled, copy the KMS policy and save the policy you want to apply to your KMS key.

• Apply the recommended bucket policy to your source bucket, as described in the AWS documentation link provided at the bottom of the page.

• If KMS is enabled, apply the recommended KMS policy to the KMS key for your bucket.

  After entering the required information, click Next.

Step 3: Setup S3 Notification

• Configure your source bucket to emit object-create notifications to the Databee SQS queue, as described in the AWS Documentation link provided at the bottom of the page.

  Once complete, click Submit to finalize and complete the configuration process.