Demultiplexing logs for Data Collector

Introduction

The Data Collector is designed to receive data from various syslog sources, each transmitting data to a distinct port. Hence, it is recommended to set up each syslog source to send data to a distinct port as per the data collector’s requirement. However, we recognize that customer preferences may differ based on their environments, e.g., all syslog sources send logs to a single destination port. This guide is crafted to assist customers in aligning their setup with the Data Collector's specifications, which necessitate a separate log source for each port. 

Furthermore, this article will introduce you to the utilization of tools such as rsyslog and logstash. These tools not only help in segregating logs as per the Data Collector's requirements but also offer advanced log filtering options. Implementing these capabilities allows you to streamline your data collection process, minimize extraneous data, and effectively distribute logs to multiple destinations. 

Prerequisites

• Choosing a demultiplexer tool: There are many tools available for demultiplexing. We recommend using rsyslog for Linux machines and NXLog CE for Windows machines. Both tools are popular for their lightweight footprint and extensive adoption. 

• Port configurations: Ensure that the ports you configure for the data collector and tool aiding demultiplexing do not conflict with other applications and are open in your firewall. 

• It is recommended that the demultiplexer tool should be installed on the same machine as the data collector. 

Installation

Linux

Run the following command to install rsyslog to use for demultiplexing: 

apt install rsyslog 
apt install netcat 

Windows

1. Install NXLog CE.

2. Download and install nmap-7.94-setup.exe from here.

Sample Configurations

rsyslogd (Linux)

/etc/rsyslog.d/1-databee.conf

More powerful conditional statements can be created using a broad range of pre-built properties in rsyslog. Please refer to the following link for more details: rsyslog Properties

NXLog (Windows)

Conditional statements can be created using Event records and fields documented here.

Data Collector (Linux/Windows)

You can create different data sources from the Databee Platform which will be reflected in the inputs section of your config file (fluent-bit.yaml).

Kindly note that the below screenshots are provided as an example. No modifications are required in the existing data collector’s configurations.

Sample Data Ingestion Script (Linux/Windows)

You can send the data directly from the syslog server to validate the setup. Alternatively, you can utilize a script to simulate the ingestion of data. Add the following content to a Python script named ingest.py:

Setup

Linux

Assuming that all the above configuration files whose path is not specified explicitly are present at the /root directory.

1. Restart rsyslog service:

   sudo systemctl restart rsyslog

2. Verify that there are no errors after restarting the rsyslog service:

   sudo systemctl status rsyslog

3. Stop the data collector services using the following command:

   /opt/comcast-databee-collector/collector.sh stop

4. Send the data using sample data ingestion script:

   python3 ingest.py

5. Run the following command to listen to the data on the specified port and validate that the received data is the same as that sent by the log source.

   nc -l -p <port>

6. Start the data collector services after verification of the data received using the following command:

   /opt/comcast-databee-collector/collector.sh start

Windows

1. After updating nxlog configurations, start the Services snap-in.

   1. Press Windows+X on the desktop, and click Run.

   2. Type services.msc. Click OK.

   3. Restart the nxlog service.

2. Verify that there are no errors after restarting the nxlog service in the C:\Program Files\nxlog\data\nxlog.log file.

3. Stop the data collector services using the following command on the Powershell (Administrator) terminal:

   C:\Program Files\Comcast Databee Collector\collector.ps1 stop

4. Send the data using sample data ingestion script:

    python3 ingest.py

5. Run the following command to listen to the data on the specified port and validate that the received data is the same as that sent by the log source.

   ncat -l -p <port>

6. Start the data collector services after verification of the data received using following command on the Powershell (Administrator) terminal:

   C:\Program Files\Comcast Databee Collector\collector.ps1 start