Troubleshooting

View Data Collector logs locally

Linux

Note:

Until configuration adapter service fetches the latest configurations and reloads the fluent bit service, logs of fluent bit service can be viewed using following command:

journalctl -u fluent-bit -e

Windows

Share Debugging Logs for Support

Reach out to DataBee Support after enabling debug mode in the data collector for further troubleshooting.

1. Run the command to stop the collector services:

   Ubuntu:

   /opt/comcast-databee-collector/collector.sh stop

   Windows:

   C:\Program Files\Comcast Databee Collector\collector.ps1 stop

2. Open the collector.yaml file from the below mentioned location:

   Ubuntu:

   /opt/comcast-databee-collector/conf/collector.yaml

   Windows:

   C:\Program Files\Comcast Databee Collector\conf\collector.yaml

3. Set log-level to DEBUG and save the changes.

4. Start the collector services:

   Ubuntu:

   /opt/comcast-databee-collector/collector.sh start

   Windows:

   C:\Program Files\Comcast Databee Collector\collector.ps1 start

5. Compress all the files under the log directory and create a tar.gz (Ubuntu) / zip (Windows). Attach it when reaching out to support.

Common installation issues

Case 1: Unable to install the dependencies on the RHEL system

Issue:

Solution:

The issue occurs when the system is not subscribed to the Red Hat portal. Run the command below on your terminal.

subscription-manager register --username <username> --password <password> --auto-attach

Case 2: Accidental/Unintended misconfiguration during installation

Solution:

1. You need not perform re-installation, in this case.

2. Rectify the details in the collector.yaml file, located under:

   • /opt/comcast-databee-collector/conf (for Linux)

   • C:\Program Files\Comcast Databee Collector\conf (for Windows)

3. Restart all the services:

   • Linux:

     /opt/comcast-databee-collector/collector.sh stop /opt/comcast-databee-collector/collector.sh start

   • Windows:

     C:\Program Files\Comcast Databee Collector\collector.ps1 stop C:\Program Files\Comcast Databee Collector\collector.ps1 start

Case 3: Data Collector keeps trying to open the chunk files but they do not exist

Issue:

Solution:

When fluent-bit is uninstalled, sometimes the system service keeps running even though the unit file is missing. Stop the fluent-bit service manually.

• Linux:

  systemctl stop fluent-bit

• Windows:

  C:\Program Files\Comcast Databee Collector\winsw\fluent-bit stop

Case 4: Incorrect proxy details provided for the data collector

Issue:

The API requests of the configuration adapter and system monitor will fail when incorrect proxy details are provided for the data collector.

Solution:

Manually update the collector.yaml file to remove the incorrect proxy details. After that, check collector status and then manually stop and start collector.sh as described under Management of services.

Case 5: Unable to delete the comcast directory

Issue:

Solution:

Manually delete the folder when the directory is in use. Note that services are already stopped and removed.

Note:

After collector uninstallation, ensure that all components including fluent-bit are also removed. Otherwise, it would be hard to notice CPU utilization if multiple fluent-bit services are running.

Case 6: Data Collector Error: “context deadline exceeds”

Issue:

Solution:

Restart the collector using the script below.

• Linux:

  /opt/comcast-databee-collector/collector.sh stop /opt/comcast-databee-collector/collector.sh start

• Windows:

  C:\Program Files\Comcast Databee Collector\collector.ps1 stop C:\Program Files\Comcast Databee Collector\collector.ps1 start

Case 7: Getting TLS handshake/unexpected EOF error when TLS is enabled for a TCP Data source

Issue:

The data source is configured with TLS enabled and the data collector is unable to send data to the platform. The fluent-bit-collector.log file has the error logs mentioned below.

TLS handshake error

Unexpected EOF error

Solution:

1. Cross-verify whether the certificates generated are valid and not expired.

2. When the DN parameters are the same for both CA and Server Certificates, the certificate will not generate properly. Make sure while giving the DN parameters, users must give the unique Common Name for both CA and Server Certificate. For eg: if CN for CA certificate is comcast.com, then CN for server certificate must not be comcast.com.

3. Verify host network configurations.

Case 8: Fluent-bit is unable to hot-reload due to some reason

Issue:

Sometimes fluent-bit does not hard reload for TCP data source when config changes are received and written to fluent-bit yaml. When the data collector with a syslog data source, which is upgraded to the latest version, changes the configuration to the TCP data source, the fluent-bit is sometimes unable to hard-reload. Check the config adapter logs and fluent-bit yaml to know if the fluent-bit is reloaded or not. If not, we need to hard reload the data-collector. The issue is not observed frequently.

Solution:

To resolve this, you should manually stop and start the services using the commands mentioned below.

• Linux:

  /opt/comcast-databee-collector/collector.sh stop
  /opt/comcast-databee-collector/collector.sh start

• Windows:

  C:\Program Files\Comcast Databee Collector\collector.ps1 stop
  C:\Program Files\Comcast Databee Collector\collector.ps1 start

Case 9: Syslog server sending data of multiple feeds to a single destination port

Issue:

Syslog server sends logs of multiple feeds as a single log stream to a single destination port on the machine where the data collector is installed. However, the data collector requires defining a different port for each data source.

Solution:

Please refer to the following article: Demultiplexing logs for Data Collector

Case 10: Failure to subscribe or open some Windows channels

Issue:

Solution:

The Data Collector only supports Administrative and Operational types of Windows channels. The analytical channel or debugging channel, such as Microsoft-Windows-DNSServer/Analytical, are not supported currently.

Case 11: Incorrect Query configured for a Windows Security Event data source

Issue:

Incorrect Query configured in a Windows security data source. In this case, following logs are shown in the fluent-bit-collector.log file.

Solution:

Please provide a valid XML or XPath query on the UI by following the instructions provided under Windows Event Log Source.

Case 12: Fluent Bit service of data collector restarts unexpectedly on Windows OS

Solution:

• Check config-adapter service logs to confirm whether it is not restarting the service due to checksum mismatch.

• To find the root cause for this issue follow the steps below.

  • Install procmon and filter Path with ‘fluent’ to monitor fluent bit service related logs.

  • Highlight Process Start and Process Exit events.

  • Search for Status Code on Microsoft Documentation for Process Exit event.

  • If the Status Code indicates that the issue relates to Windows misconfiguration or corruption, please reach out to Microsoft support team.

Case 13: Incorrect Channel name configured for a Windows Security Event data source

Issue:

Incorrect Channel configured in a Windows security data source. In this case, following logs are shown in the fluent-bit-collector.log file when xyz is set as channel name.

Solution:

Please provide a valid Channel Name provided under Windows Event Log Source. Please follow the below steps to identify the channel name.

• Open Windows Event Viewer and select the channel from the list of channels. Open properties for that channel.

  

• From the Log properties dialog box select and copy the channel name present inside 'Full Name' property as shown below.

  

Case 14: Duplicate data ingested when multiple Channels with Query containing multiple channels are configured in a Windows Security Events data source

Issue:

When multiple channels along with an event_query containing multiple channels in itself are configured inside the same data source, duplicate events are ingested. This issue is caused due to fluent bit’s open issue. (Ref: https://github.com/fluent/fluent-bit/issues/8747)

Solution:

Following are the workarounds:

1. Filter events while creating Source-initiated subscriptions to collect Windows events on the Central Windows Server machine (as per windows event log collection).

2. Create separate data source for each channel, and provide appropriate event_query in each of them.

Case 15: Unable to send data when using a flat file data source

In the case of Ubuntu/RHEL machines, the following log would be printed in the /var/log/comcast-databee-collector/services/fluent-bit-collector.log file.

In the case of Windows, the error log is not printed by the fluent bit service. Hence, refer to the below section to figure out the root cause.

Please check the following to identify the root cause and remediate the issue accordingly:

1. Check whether Source Files and Exclusion Files are not identical while configuring the data source on UI.

2. Check whether the Exclusion File path is not the parent of the Source Files path while configuring the data source on UI.

3. Check whether a proper path is given in Source Files with the absolute file path or a valid wildcard pattern in case you want to scan the entire folder, i.e. var/log/*.log, var/log/syslog.log

Case 16: collector.yaml gets wiped out

Issue:

In some cases such as toggling proxy, collector.yaml file after installing the data collector successfully gets wiped out randomly.

Solution:

In such cases, uninstall the existing collector and reinstall the latest collector.

Resiliency

Collector

In case of failures or system reboots, collector services will be restarted automatically.

Platform

Platform services run as Kubernetes pods. Kubernetes offers a thorough system for overseeing the lifecycle of these pods, incorporating a restart mechanism to guarantee the availability and reliability of applications. Through restart policies, probes, and supplementary functionalities such as Pod Disruption Budgets, Kubernetes presents a resilient framework for managing restarts in a dynamic and scalable containerized application environment.

Limitations & Known Issues

• HTTPS proxy is not supported.

• Self-signed TLS certificates are not supported for the Syslog ingestion mechanism.

• As of now, there is no tag to distinguish the data metrics as per the host, when the same collector configuration is used for installing the collector on a different host.

• Unusual proxy behavior.

• Query applied for multiple channel is not working properly (Duplicate data ingested)

• collector.yaml wiped out randomly on the data collector when the proxy is toggled multiple times from the UI

• Toggling state of data source triggers unintended historical Windows events collection