HTTP Collector
  • 04 Dec 2024
  • 5 Minutes to read
  • Contributors
  • Dark
    Light

HTTP Collector

  • Dark
    Light

Article summary

Using HTTP Collector, you can send security logs to DataBee via API over HTTPS, simplifying log ingestion without additional tools or forwarders. DataBee acts as a central platform that manages multiple HTTP Collectors.

Configure HTTP Data Source

To configure your HTTP data source in DataBee, follow these steps:

Click the Data button and select +Add New Data Source in DataBee UI.

Choose the data source to be configured for HTTP Collector from the list of supported data sources.

To fetch data from your HTTP Collector, click on the HTTP Collector button.

Step 1: Provide Basic Information

Enter the data source and owner details in the fields provided.

  • Data Source Name: the name for the data source, which will be showcased on the data source list view

  • Owner Name: the name of the owner (e.g. John Doe)

  • Owner E-mail: the email ID of the owner (e.g. john.doe@gmail.com)

Once you have entered the required information, click Next to proceed to the next step.

Step 2: Configure Data Source

Fill in the data source details in the fields below.

  • Format: (Required) Choose the format of the logs sent from the source. It will show a list of supported formats by the feed. Example: json/xml/cef/syslog-rfc5424/syslog-rfc3164/other. In case the log attached in the request payload does not match with the format selected on UI, it will be shown under Unprocessed page.

  • Process array objects as individual log?: (Optional) If the format selected is json, enabling this option will ingest each object of JSON array as one log entry. If disabled, the request payload will be stored in the data lake as is.

  • Split Delimiter: (Optional) The delimiter that is used to separate log lines, such as \n.

  • Tags: (Optional) The tag(s) to be applied to the logs from this feed.

Step 3: Generate API Key

Once you fill up the basic details, you must generate an API key which is a mandatory step for data source creation. Click on the Generate New API Key button.

To view your API key, click on Show API Key. You can copy it to your clipboard by clicking the Copy to clipboard button.

Note

Please ensure to securely store the generated API Key as it will only be visible once. If you need to regenerate a new API Key in the future, clicking the Generate New API Key button will automatically invalidate the previous API Key.

Click Submit to finalize and complete the configuration process.

Sending data using POST request

Prerequisites

  • The DataBee platform should be accessible from the host

  • You must have the Endpoint URL, Tenant ID, Data source ID, and API Key

  • Your data source must be healthy

Get Endpoint URL

Click on the settings icon at the top right corner of the UI. From the dropdown menu, select System.

From the left sidebar, select HTTP Collector. Click on the Copy to clipboard button to copy the Endpoint URL.

Get Tenant ID

Click on the account icon at the top right corner of DataBee UI.

Click on the My Profile button to view your personal profile information. Navigate to 'Tenant ID' and click on the Copy to clipboard button.

HTTP_Tenant_ID(1)

Get Data Source ID

Once the HTTP Data source is created, you can view the data source details where ID refers to the Data source ID.

Get API Key

Refer to: Generate API Key

In case the API Key is lost, a new one can be regenerated by editing the data source.

Using Postman

  • Download and import the Postman Collection script.

    HTTP Collector.postman_collection
    1.38 KB
  • After import, you will see a collection named DataBee HTTP Collector.

  • Open the POST request named Ingest Logs.

  • Replace it with the Endpoint URL copied from DataBee UI.

    You can send requests to the DataBee platform with either uncompressed or compressed payload.

    With Uncompressed payload

    • Under the Headers section, add the required headers (TenantID, Authorization, and DatasourceID) obtained by following the above-mentioned steps.

    • Navigate to the section Body, and add the logs to be sent to DataBee.

      Note

      The maximum allowed payload size is 1 MB (uncompressed).

      The maximum number of entries in a batch allowed are 10000.

      With Compressed payload

      Add the Content-Encoding: gzip header to the request when sending compressed logs.

      Compress the file containing logs to be sent to DataBee.

      Select the binary option under Body. Upload the compressed file (with gz extension).

      Note

      The maximum allowed payload size is 5 KB (compressed).
      The maximum number of entries in a batch allowed are 10000.

      Click on the Send button. Upon successful execution, you will receive a 202 (Accepted) status code.

Using Code Snippet

CURL

Use the following code snippet by

  • Replacing the Endpoint URL, Tenant ID, API Key, and Datasource ID with the values obtained from the DataBee UI.

  • Replace the value of data field with your log in the following command:

curl --location '<YOUR-ENDPOINT-URL>' \
--header 'TenantID: <YOUR-TENANT-ID>' \
--header 'Authorization: <YOUR-API-KEY>' \
--header 'DatasourceID: <YOUR-DATASOURCE-ID>' \
--header 'Content-Type: application/json' \
--data ' {
   "field1": "value1",
   "field2": "value2"
} 
'

Now, Send the request. Following is the successful response message that will be displayed on the console:

{"message":"Accepted for processing!"}

Other Programming Languages

Postman also provides support for code snippets of multiple programming languages such as Java, Python, etc.

Troubleshooting

Response Codes

Response Code

Description

200

The request payload is empty.

This is usually used to test connection from the application or client to DataBee.

202

The request has been accepted for processing.

400

This could be an issue in the request payload formatting or an invalid/empty API version.

401

It is likely a missing or invalid header (API Key, DatasourceID or TenantID).

403

Forbidden. Datasource deployment status is not HEALTHY. Please check that the data source is enabled and HEALTHY.

404

Resource Not Found (Invalid Domain) or invalid request method used.

Verify your endpoint URL (Ref: Get Endpoint URL) and ensure that the method is POST.

413

The size of the payload exceeds the maximum allowed payload size or the number of entries in the payload exceeds 10000 or the cumulative size of 10 logs in a batch is greater than 256 KiB.

408

Request Timeout, the request should be retried after some time.

500

The server encountered an unexpected situation that prevented it from fulfilling the request, the request should be retried after some time.

503

The server is not ready to process the request probably because it is overloaded, the request should be retried after some time.

Common Issues

Case 1: Unable to make a request due to unreachable network

Error Screenshot:

Solution:

Check your internet connection. In case of any issues, please contact your network administrator.

Case 2: Unable to make a request due to invalid URL

Error Screenshots:

Solution:

  • Make sure your endpoint URL is valid. Refer to the following section to get endpoint URL from DataBee UI: Get Endpoint URL

  • Make sure the HTTP method is POST.


Was this article helpful?

What's Next
Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.
ESC

Eddy AI, facilitating knowledge discovery through conversational intelligence