HTTP Collector

Using HTTP Collector, you can send security logs to DataBee via API over HTTPS, simplifying log ingestion without additional tools or forwarders. DataBee acts as a central platform that manages multiple HTTP Collectors.

Configure an HTTP Collector Ingest Data Feed

Start by clicking on the Data button and select Data Feeds.

Next, click on the +Add New Data Feed button.

Choose your preferred data feed from the list of supported data feeds. In this walkthrough, we will use the BluVector feed.

You will now be directed to choose your ingestion method. To fetch data via http collector, click on HTTP Collector.

You will now be redirected to the "Configure Data Feed" page. Follow the flowchart displayed on the right, for a visual guide, outlining the step-by-step configuration process.

Step 1: Provide Basic Information

Enter the data source and owner details in the fields provided.

• Data Feed Name: the name for the data feed, which will be showcased on the data feed list view.

• Owner Name: the name of the owner (e.g. John Doe).

• Owner E-mail: the email ID of the owner (e.g. john.doe@gmail.com).

Once you have entered the required information, click Next to proceed to the next step.

--header 'TenantID: <YOUR-TENANT-ID>' \
--header 'Authorization: <YOUR-API-KEY>' \
--header 'DatasourceID: <YOUR-DATASOURCE-ID>' \
--header 'Content-Type: application/json' \

Testing Using Postman

• Download and import the Postman Collection script.

  HTTP Collector.postman_collection

  1.38 KB

• After import, you will see a collection named DataBee HTTP Collector.

• Open the POST request named Ingest Logs.

• Replace it with the Endpoint URL copied from DataBee UI.

  

  You can send requests to the DataBee platform with either uncompressed or compressed payload.

  With Uncompressed payload

  • Under the Headers section, add the required headers (TenantID, Authorization, and DatasourceID) obtained by following the above-mentioned steps.

  • Navigate to the section Body, and add the logs to be sent to DataBee.

    Note

    The maximum allowed payload size is 1 MB (uncompressed).

    The maximum number of entries in a batch allowed are 10000.

    

    With Compressed payload

    Add the Content-Encoding: gzip header to the request when sending compressed logs.

    

    Compress the file containing logs to be sent to DataBee.

    Select the binary option under Body. Upload the compressed file (with gz extension).

    

    Note

    The maximum allowed payload size is 5 KB (compressed).
    The maximum number of entries in a batch allowed are 10000.

    Click on the Send button. Upon successful execution, you will receive a 202 (Accepted) status code.

    

Using Code Snippet

CURL

Use the following code snippet by

• Replacing the Endpoint URL, Tenant ID, API Key, and Datasource ID with the values obtained from the DataBee UI.

• Replace the value of data field with your log in the following command:

curl --location '<YOUR-ENDPOINT-URL>' \
--header 'TenantID: <YOUR-TENANT-ID>' \
--header 'Authorization: <YOUR-API-KEY>' \
--header 'DatasourceID: <YOUR-DATASOURCE-ID>' \
--header 'Content-Type: application/json' \
--data ' {
   "field1": "value1",
   "field2": "value2"
} 
'

Now, Send the request. Following is the successful response message that will be displayed on the console:

{"message":"Accepted for processing!"}

Other Programming Languages

Postman also provides support for code snippets of multiple programming languages such as Java, Python, etc.

Troubleshooting

Response Codes

Common Issues

Case 1: Unable to make a request due to unreachable network

Error Screenshot:

Solution:

Check your internet connection. In case of any issues, please contact your network administrator.

Case 2: Unable to make a request due to invalid URL

Error Screenshots:

Solution:

• Make sure your endpoint URL is valid. Refer to the following section to get endpoint URL from DataBee UI: Get Endpoint URL.

• Make sure the HTTP method is POST.