Ingest Methods

DataBee flexible ingestion pipeline supports ingestion from more than 250 data feeds including leading security platforms such as Azure Active Directory, CrowdStrike, Azure MFA, Azure Sign-In, Palo Alto Networks, Zscaler and many more.

Data can be ingested via

• API - DataBee makes API calls to pull in events

• HTTP - Platforms push data securely to DataBee via HTTP/S

• Amazon S3 and Azure Blob - DataBee polls a storage bucket for events

• Syslog - Logs can be sent to a DataBee Data Collector

• Files - The DataBee Data Collector is able to read in files mounted into the file system

Managing data feeds

Data feeds refer to the various systems, applications, and platforms where data is generated or stored. DataBee is designed to extract data from these data feeds, transform it as required, and load it into a target system such as Databricks or Snowflake. In this section, we will explore the various data feeds available in DataBee, how to add, delete, and edit data feeds, and how to determine their current states.

Current data feeds

Click on the Data button and then select Data Feeds from the dropdown menu. This takes you to "Your current data feeds" page where you can find all the data feeds currently configured within DataBee. The page displays a clean and organized view of data feeds which includes the data feed name, its state, the size of raw data, and the data quality score with a graphical representation. At the top of the page, you can find the total number of configured data feeds and the total bytes ingested. You can search for a specific feed by entering its name in the Search Data Feeds field. Click the sort dropdown menu and choose the preferred option to sort the feeds by name, vendor, data usage, state, ingest date or the configuration date. From the time dropdown on the top right corner, you can filter the results by choosing the last hour, day or week’s data.

Refer to the Data Quality article for detailed insights of the feed data.

After you have added and configured your data feeds, check the state of each data feed widget on "Your current data feeds" page. The various states of a data source and their corresponding meanings are mentioned below.

Add new data feed

You can click on the + Add New Data Feed button, which takes you to the "Add new data feed" page where you can select the data source you want to add by clicking on the desired data feed button.

Configure data feed

This takes you to the "Configure data feed" page, where you can enter the details required to connect to the data feed. Make sure that you already have an account of the data feed you are trying to connect. You can refer to the link provided within the page, for more instructions on setting up the data feed. The steps for connecting a new log source are displayed on the right side of the page. Follow the steps and enter the details in the fields to configure the data feed.

Delete data feed

Click on the data feed you wish to delete. A window will open to your right where you can locate the trash can icon. Click on the trash can button to delete the data feed. A confirmation dialog box will appear asking if you are sure you want to delete the data feed. Click on Delete Data Feed in the confirmation dialog box to proceed with the deletion, or click Close to abort the deletion.

Edit data feed

To make changes to the data feed details you entered earlier, go to "Your current data feeds" page and click on the data feed you wish to edit. A window will open towards your right where you can locate the edit icon represented by the image of a pencil. Click on the edit button. This will take you to the "Configure data feed" page where you can make the necessary edits.

Data feed configuration history

Navigate to “Your current data feeds” page. Locate the desired data feed for which you want to view the configuration history. On clicking the data feed, a window will open towards your right where you can locate the history icon. Click on the Feed Configuration History icon. The “Feed Configuration History” page pops up which displays all the configuration changes made for the selected data feed. It displays the time at which the configuration change was made and the user or account responsible for the configuration change. Click on the tiny arrow button at the right, which displays the specific field that was modified, the old value before the change, and the new value after the configuration change. This feature helps you track and analyze the modifications made to the data feed configuration over time.

Supported ingest file types

DataBee accepts CSV, TAR, and TAR.GZ as the default supported ingest file types. Files without these extensions are treated as single JSON objects or JSON arrays based on their content structure. If your files don’t match these formats, contact your support engineer for manual configuration adjustments.