HTTP Collector

Using HTTP Collector, you can send security logs to DataBee via API over HTTPS, simplifying log ingestion without additional tools or forwarders. DataBee acts as a central platform that manages multiple HTTP Collectors.

Configure an HTTP Collector Ingest Data Feed

Start by clicking on the Data button and select Data Feeds.

Next, click on the +Add New Data Feed button.

Choose your preferred data feed from the list of supported data feeds.

You will now be directed to choose your ingestion method. To fetch data via http collector, click on HTTP Collector.

You will now be redirected to the "Configure Data Feed" page. Follow the flowchart displayed on the right, for a visual guide, outlining the step-by-step configuration process.

Step 1: Provide Basic Information

Enter the data source and owner details in the fields provided.

• Data Feed Name: the name for the data feed, which will be showcased on the data feed list view

• Owner Name: the name of the owner (e.g. John Doe)

• Owner E-mail: the email ID of the owner (e.g. john.doe@gmail.com)

Once you have entered the required information, click Next to proceed to the next step.

Prerequisites

• The DataBee platform should be accessible from the host

• You must have the Endpoint URL, Tenant ID, Data source ID, and API Key

• Your data source must be healthy

Get Endpoint URL

Click on the settings icon at the top right corner of the UI. From the dropdown menu, select System.

From the left sidebar, select HTTP Collector. Click on the Copy to clipboard button to copy the Endpoint URL.

Get Tenant ID

Click on the account icon at the top right corner of DataBee UI.

Click on the My Profile button to view your personal profile information. Navigate to 'Tenant ID' and click on the Copy to clipboard button.

Get Data Source ID

Once the HTTP Data source is created, you can view the data source details where ID refers to the Data source ID.

Get API Key

Refer to: Generate API Key

In case the API Key is lost, a new one can be regenerated by editing the data source.

Using Postman

• Download and import the Postman Collection script.

  HTTP Collector.postman_collection

  1.38 KB

• After import, you will see a collection named DataBee HTTP Collector.

• Open the POST request named Ingest Logs.

• Replace it with the Endpoint URL copied from DataBee UI.

  

  You can send requests to the DataBee platform with either uncompressed or compressed payload.

  With Uncompressed payload

  • Under the Headers section, add the required headers (TenantID, Authorization, and DatasourceID) obtained by following the above-mentioned steps.

  • Navigate to the section Body, and add the logs to be sent to DataBee.

    Note

    The maximum allowed payload size is 1 MB (uncompressed).

    The maximum number of entries in a batch allowed are 10000.

    

    With Compressed payload

    Add the Content-Encoding: gzip header to the request when sending compressed logs.

    

    Compress the file containing logs to be sent to DataBee.

    Select the binary option under Body. Upload the compressed file (with gz extension).

    

    Note

    The maximum allowed payload size is 5 KB (compressed).
    The maximum number of entries in a batch allowed are 10000.

    Click on the Send button. Upon successful execution, you will receive a 202 (Accepted) status code.

    

Using Code Snippet

CURL

Use the following code snippet by

• Replacing the Endpoint URL, Tenant ID, API Key, and Datasource ID with the values obtained from the DataBee UI.

• Replace the value of data field with your log in the following command:

curl --location '<YOUR-ENDPOINT-URL>' \
--header 'TenantID: <YOUR-TENANT-ID>' \
--header 'Authorization: <YOUR-API-KEY>' \
--header 'DatasourceID: <YOUR-DATASOURCE-ID>' \
--header 'Content-Type: application/json' \
--data ' {
   "field1": "value1",
   "field2": "value2"
} 
'

Now, Send the request. Following is the successful response message that will be displayed on the console:

{"message":"Accepted for processing!"}

Other Programming Languages

Postman also provides support for code snippets of multiple programming languages such as Java, Python, etc.

Troubleshooting

Response Codes

Common Issues

Case 1: Unable to make a request due to unreachable network

Error Screenshot:

Solution:

Check your internet connection. In case of any issues, please contact your network administrator.

Case 2: Unable to make a request due to invalid URL

Error Screenshots:

Solution:

• Make sure your endpoint URL is valid. Refer to the following section to get endpoint URL from DataBee UI: Get Endpoint URL

• Make sure the HTTP method is POST.