HTTP Collector

Using HTTP Collector, you can send security logs to DataBee via API over HTTPS, simplifying log ingestion without additional tools or forwarders. DataBee acts as a central platform that manages multiple HTTP Collectors.

Configure HTTP Data Source

To configure your HTTP data source in DataBee, follow these steps:

Click on the Data button and select +Add New Data Source in DataBee UI. Choose your preferred data source from the list of available options. You will now be directed to choose your storage account. To fetch data from your HTTP Collector, click on the HTTP Collector button.

Step 1: Configure Data Source

Enter the data source and owner details in the fields provided.

• Data Source Name: the name for the data source, which will be showcased on the data source list view

• Owner Name: the name of the owner (e.g. John Doe)

• Owner E-mail: the email ID of the owner (e.g. john.doe@gmail.com)

• Format: the format of the data that is being sent (e.g. syslog-rfc5424/syslog-rfc3164/XML/JSON/LEEF/CEF/OTHER)

Once you have entered the required information, click Next to proceed to the next step.

Step 2: Generate API Key

Once you fill up the basic details, you must generate an API key which is a mandatory step for data source creation. Click on the Generate New API Key button. To view your API key, click on Show API Key. You can copy it to your clipboard by clicking the Copy to clipboard button.

Note:

Please ensure to securely store the generated API Key as it will only be visible once. If you need to regenerate a new API Key in the future, clicking the Generate New API Key button will automatically invalidate the previous API Key.

Click Submit to finalize and complete the configuration process.

Sending data using POST request

Prerequisites

• The DataBee platform should be accessible from the host

• You must have the Endpoint URL, Tenant ID, Data source ID, and API Key

• Your data source must be healthy

Get Endpoint URL

Click on the settings icon at the top right corner of the UI. From the dropdown menu, select System.

From the left sidebar, select HTTP Collector. Click on the Copy to clipboard button to copy the Endpoint URL.

Get Tenant ID

Click on the account icon at the top right corner of DataBee UI.

Click on the My Profile button to view your personal profile information. Navigate to 'Tenant ID' and click on the Copy to clipboard button.

Get Data Source ID

Once the HTTP Data source is created, you can view the data source details where ID refers to the Data source ID.

Get API Key

Refer to: Generate API Key

In case the API Key is lost, a new one can be regenerated by editing the data source.

Using Postman

1. Download and import the Postman Collection script.

   HTTP Collector

   1.54 KB

2. After import, you will see a collection named HTTP Collector on the left sidebar.

   

3. Open the POST request and replace it with the Endpoint URL copied from DataBee UI.

   

   Note

   You can send requests to the DataBee platform with either uncompressed or compressed payload.

   With Uncompressed payload

   • Under the Headers section, add the required headers (TenantID, Authorization, and Datasource ID) obtained by following the above-mentioned steps.

   • Navigate to the section Body, and select JSON option (This will set the Content-Type header to application/JSON by default). Add the request body using the below template.

     {
        "logs":[
            "<log1>",
            "<log2>",
            …
        ],
        "tags": {
            "env": "dev"
           …
        }
     }

     Note

     The maximum allowed payload size is 10 MB (uncompressed).

   

   With Compressed payload

   • Add the Content-Encoding: gzip header to the request when sending compressed logs.

     

   • Prepare your payload in JSON format as per the template given below and compress the file.

     {
        "logs":[
            "<log1>",
            "<log2>",
            …
        ],
        "tags": {
            "env": "dev"
           …
        }
     }

   • Select the binary option under Body. Upload the compressed file (with gz extension).

     

4. Click on the Send button. Upon successful execution, you will receive a 202 (Accepted) status code.

   

Using Code Snippet

CURL

Use the following code snippet by

• Replacing the Endpoint URL, Tenant ID, API Key, and Datasource ID with the values obtained from the DataBee UI.

• Adding your JSON payload within the data field

curl --location '<YOUR-ENDPOINT-URL>' \
--header 'TenantID: <YOUR-TENANT-ID>' \
--header 'Authorization: <YOUR-API-KEY>' \
--header 'DatasourceID: <YOUR-DATASOURCE-ID>' \
--header 'Content-Type: application/json' \
--data '{
   "logs":[
       "<LOG1>",
       "<LOG2>",
                 …
   ],
   "tags": {
       "<KEY>": "<VALUE>"
       …
   }
}
'

Now, Send the request. Following is the successful response message that will be displayed on the console:

{"message":"Accepted for processing!"}

Other Programming Languages

Postman also provides support for code snippets of multiple programming languages such as Java, Python, etc.

Troubleshooting

Response Codes

Common Issues

Case 1: Unable to make a request due to unreachable network

Error Screenshot:

Solution:

Check your internet connection. In case of any issues, please contact your network administrator.

Case 2: Unable to make a request due to invalid URL

Error Screenshots:

Solution:

• Make sure your endpoint URL is valid. Refer to the following section to get endpoint URL from DataBee UI: Get Endpoint URL

• Make sure the HTTP method is POST.

Case 3: Receiving 202 response code post disabling the data source from UI

This is due to authorizer caching on the server. You can stop the API client or scripts manually for immediate disabling data flow, otherwise, data ingestion will automatically be stopped within 5 minutes from disabling the data source.

Known Issues

• Unable to filter logs using tags provided while ingesting data on the Search page