Data Collector API Ingest

The data collector collects data from various sources, applies filters, and adds metadata before securely forwarding it to DataBee.

The data collector is deployed on premises as a virtual machine enabling it to interact with data sources in the network. The data collector is able to connect to on-prem systems using APIs.

For details on configuring and installing a data collector in DataBee, refer to Data Collector article.

This article describes the steps needed to configure a feed using data collector API ingest method.

Configure Data Feed with Data Collector API Ingest

• From the console, click on the Data button and select +Add New Data Feed in DataBee UI.

• Choose a data feed that supports API ingestion. In this article, we will use SonarQube as an example data source.  

• Pick Data Collector You will now be directed to choose your ingest method. To fetch data from your on-prem data collector, click on Data Collector.

You will be redirected to the page to select how to accept events using Data Collector. For feeds that supports API ingestion such as ‘SonarQube’, the API Ingest option will be available.

Click on API Ingest option to proceed.

You will now be redirected to the "Configure data feed" page.

Enter the data feed details in the fields provided and choose a pre-configured data collector of your choice.

• Data Feed Name: a user-friendly name for the data feed.

• Owner Name: the name of the point of contact for the data feed.

• Owner E-mail: email address of the owner.

• Collector: select from a list of collectors to use.

• Include Feed in Entity Resolution: Enable this option to include the feed in the entity resolution process. A checkbox for this field is auto-enabled for all new feeds. You can deselect it to exclude the feed from entity resolution.

  • Full Entity Resolution: Automatically creates and updates entities when they are discovered in the data.

  • Associate Only: Links new events to existing entities without modifying their attributes.

    A checkbox is available for each entity type (user, device, application) to select the desired level of resolution (Full Entity Resolution, Associate Only).

Once you have entered the required information, scroll down to the next section.

Enter the input fields shown below and scroll down to the next section.

• Base URL: enter the IP Address or FQDN of the platform.

• Event Types: choose the event type from the drop down options provided. At least 1 event type is required to proceed.

• Rate Limit: Rate limit configuration depends on the ‘Request Count’, ‘Interval’ and the interval time unit.

  • Request Count: enter the total number of requests that are allowed within the specified time.

  • Interval: enter an integer value for the interval duration and select the interval time unit from the dropdown options- seconds, minutes, or hours.

When API versioning is available for the feed, the following option will be available.

• API version: select the suitable API version from the list of supported versions.

Enter the fields listed below and scroll down to the next section.

• Authorization Method: this will be prefilled with the supported auth method.

• Enable TLS: select this if TLS should be used for secure communications with the API.

If Enable TLS is checked, additional fields will need to be filled.

• Skip Server-side Certificate Verification: indicates to skip verification of the server's certificate during the TLS handshake.

• Root CAS File Path: provide the full path to the Root CA Certificate file. This path should include the directory and subdirectories where the certificate is stored. For example, on Windows, it might be C:\Certificates\rootCA.crt, and on Unix/Linux, it might be /etc/ssl/certs/rootCA.pem

• Certificate File Path: enter the file path to the client certificate used for mTLS authentication.  

• Key File Path: enter the file path to the private key associated with the client certificate.

Step 4: Configure Inbound Proxy

• Enable Proxy: a flag that indicate whether to route data through a configured proxy for outbound connectivity

  If enabled, the following additional fields will need to be specified:

  • Proxy URL: an authenticated URL of the proxy server through which API requests should be routed.

  • Proxy Username: username required to authenticate the proxy server.

  • Proxy Password: password required to authenticate the proxy server.

Click Save Draft to save your current changes.

After entering all the field details click on Submit to finalize and complete the configuration process.

Refer to the SonarQube integration for an example of setting up a feed with Data Collector API Ingest.