Data Collector API Ingest

The data collector collects data from various sources, applies filters, and adds metadata before securely forwarding it to DataBee.

The data collector is deployed on premises as a virtual machine enabling it to interact with data sources in the network. The data collector is able to connect to on-prem systems using APIs.

For details on configuring and installing a data collector in DataBee, refer to Data Collector article.

This article describes the steps needed to configure a feed using data collector API ingest method.

Configure Data Feed with Data Collector API Ingest

• From the console, click on the Data button and select +Add New Data Feed in DataBee UI.

• Choose a data feed that supports API ingestion. In this article, we will use SonarQube as an example data source.  

• Pick Data Collector You will now be directed to choose your ingest method. To fetch data from your on-prem data collector, click on Data Collector.

You will be redirected to the page to select how to accept events using Data Collector. For feeds that supports API ingestion such as ‘SonarQube’, the API Ingest option will be available.

Click on API Ingest option to proceed.

You will now be redirected to the "Configure data feed" page. Follow the flowchart displayed on the right, for a visual guide, outlining the step-by-step configuration process.

Step 1: Configuring Data Feed

Enter the data feed details in the fields provided and choose a pre-configured data collector of your choice.

• Data Feed Name: a user-friendly name for the data feed.

• Owner Name: the name of the point of contact for the data feed.

• Owner E-mail: email address of the owner.

• Collector: select from a list of collectors to use.

Once you have entered the required information, click Next to proceed to the next step.

Step 2: Configure Input

Enter the input fields shown below and click Next.

• Base URL: enter the IP Address or FQDN of the platform.

• Event Types: choose the event type from the drop down options provided. At least 1 event type is required to proceed.

• Rate Limit: Rate limit configuration depends on the ‘Request Count’, ‘Interval’ and the interval time unit.

  • Request Count: enter the total number of requests that are allowed within the specified time.

  • Interval: enter an integer value for the interval duration and select the interval time unit from the dropdown options- seconds, minutes, or hours.

When API versioning is available for the feed, the following option will be available.

• API version: select the suitable API version from the list of supported versions.

Step 3: Configure Authentication Details

Enter the fields listed below and click on Next.

• Authorization Method: this will be prefilled with the supported auth method.

• Enable TLS: select this if TLS should be used for secure communications with the API.

If Enable TLS is checked, additional fields will need to be filled.

• Skip Server-side Certificate Verification: indicates to skip verification of the server's certificate during the TLS handshake.

• Root CAS File Path: provide the full path to the Root CA Certificate file. This path should include the directory and subdirectories where the certificate is stored. For example, on Windows, it might be C:\Certificates\rootCA.crt, and on Unix/Linux, it might be /etc/ssl/certs/rootCA.pem

• Certificate File Path: enter the file path to the client certificate used for mTLS authentication.  

• Key File Path: enter the file path to the private key associated with the client certificate.

Step 4: Configure Inbound Proxy

• Enable Proxy: a flag that indicate whether to route data through a configured proxy for outbound connectivity

  If enabled, the following additional fields will need to be specified:

  • Proxy URL: an authenticated URL of the proxy server through which API requests should be routed.

  • Proxy Username: username required to authenticate the proxy server.

  • Proxy Password: password required to authenticate the proxy server.

After entering all the field details click on Submit to finalize and complete the configuration process.

Refer to the SonarQube integration for an example of setting up a feed with Data Collector API Ingest.