The Update Trap: How Cybersecurity Vendors Risk Your Operations ?
Recently, CrowdStrike experienced a significant issue causing widespread Blue Screen of Death (BSOD) incidents on Windows computers running its software. The company acknowledged numerous reports of crashes and is still trying to address the problem. A defect in a recent update for Windows led to the BSODs. This incident underscores the risks associated with frequent kernel driver updates required by traditional cybersecurity solutions to combat emerging threats. Traditional cybersecurity solutions that rely on detection often need to update their kernel drivers frequently to stay ahead of new and evolving threats.
Kernel drivers operate at the core of the operating system, managing critical functions and communications with hardware. The primary reason for these updates is to ensure that the security solution can effectively identify and mitigate new threats. Kernel drivers have privileged access to system resources. Any flaws or bugs introduced during updates can lead to system crashes, such as the Blue Screen of Death (BSOD), resulting in downtime and productivity loss. While updating kernel drivers is necessary for traditional cybersecurity solutions to protect against emerging threats, this approach carries significant risks. These include system instability, compatibility issues, and operational disruptions.
Kernel drivers require extensive testing to ensure they do not interfere with the operating system or other software. Inadequate testing can lead to incompatibility and other operational issues updating kernel drivers often requires system reboots, which can be disruptive in a business environment, particularly if updates are frequent.
The recent CrowdStrike incident likely involves a real-world example where an update to a kernel driver caused significant issues, such as widespread BSOD incidents. This serves as a reminder of the inherent risks involved in the process.
Detection and Kernel Trap
Xcitium does not require frequent kernel driver updates to combat emerging threats due to its innovative unified Zero Trust architecture. Unlike traditional cybersecurity solutions that rely heavily on detection and signature-based updates, Xcitium’s approach focuses on proactive containment and verification of all unknown executables. The Zero Trust architecture operates on the principle that no file or process is trusted by default, ensuring that potentially harmful actions are isolated before they can impact the system. This method eliminates the dependency on kernel-level interventions, which are prone to causing system instability and BSOD incidents when regularly updated. By leveraging this architecture, Xcitium provides robust protection against both known and unknown threats without the risks associated with constant kernel driver updates, ensuring a more stable and secure operating environment.
It’s important to thoroughly test kernel drivers before they are pushed publicly. However, legacy cybersecurity vendors are caught between a rock and a hard place because they have to choose between speedy updates in an attempt to protect their customers and the longer testing needed for quality assurance. This creates a persistent problem: rushing updates can introduce new vulnerabilities and instability, while delaying updates leaves customers exposed to emerging threats. It’s a battle that traditional vendors can’t win, but one that Xcitium sidesteps entirely through its Unified Zero Trust approach, which does not depend on detection thus avoids relying on constant updates and eliminates these inherent conflicts and risks.
Kernel Drivers
A kernel-level driver is a type of software that operates at the core of a computer’s operating system, interfacing directly with the hardware. These drivers are critical components that manage and facilitate communication between the operating system and hardware devices such as graphics cards, network adapters, and storage devices. Because they operate in the kernel mode, they have high privileges and direct access to system memory and hardware resources, enabling them to perform tasks that are essential for the overall functioning and performance of the system. However, this high level of access also means that any issues or bugs within kernel-level drivers can lead to severe system instability, crashes, and security vulnerabilities, making their development and testing a complex and critical process.
Kernel drivers, such as file mini-filter drivers, play a crucial role in endpoint protection products by providing a high-level interface to monitor and control file system operations. These mini-filter drivers operate at the kernel level, allowing them to intercept and manipulate file I/O requests before they reach the underlying file system. This capability is essential for implementing various security features, including:
Real-Time File Monitoring: File mini-filter drivers can continuously monitor file system activities, such as file creation, modification, deletion, and access. This real-time monitoring enables endpoint protection products to detect suspicious or malicious behaviors, such as unauthorized data access or attempts to encrypt files, commonly seen in ransomware attacks.
Access Control and Filtering: These drivers can enforce security policies by controlling access to files based on predefined rules. For example, they can block unauthorized applications from accessing sensitive files or prevent the execution of potentially harmful files, thereby mitigating the risk of malware infections.
File Integrity Checking: Endpoint protection products can use mini-filter drivers to perform file integrity checks, ensuring that critical system files and important user data have not been tampered with. This helps in detecting and preventing file-based attacks that rely on modifying or corrupting essential files.
Content Scanning: Mini-filter drivers can intercept file read and write operations, allowing security software to scan the content of files for known malware signatures or suspicious patterns. This scanning can happen transparently in the background without interrupting the user’s activities.
Encryption and Decryption: Some endpoint protection solutions use mini-filter drivers to implement on-the-fly encryption and decryption of files. This ensures that data is protected at rest and only accessible by authorized users and applications.
Logging and Auditing: By intercepting file system operations, mini-filter drivers can generate detailed logs of all file activities. These logs are invaluable for forensic analysis, helping security teams understand the nature and scope of a security incident.
Because file mini-filter drivers operate at the kernel level, they can provide comprehensive and robust security features that are difficult to bypass. However, their integration and deployment must be handled carefully to avoid system instability and ensure compatibility with other kernel-level components.
Xcitium Unified Zero Trust Platform
ZeroDwell Containment is Xcitium’s advanced security technology designed to isolate and neutralize threats before they can cause any damage. Unlike traditional solutions that depend on identifying threats through detection signatures, ZeroDwell Containment proactively protects endpoints by verifying 100% of unknown executables. This approach ensures that even new and sophisticated threats are contained and unable to affect critical system components. This fundamental shift eliminates the need for continual kernel driver updates to confront emerging threats. From their own statements, other cyber security companies like CrowdStrike admit they must continuously update these drivers to protect against the latest adversaries. This constant updating creates more opportunities for catastrophic events, as witnessed in the recent widespread BSOD incidents.
Xcitium ZeroDwell Containment technology represents a significant advancement in endpoint security. By focusing on proactive isolation and eliminating the dependency on detection and frequent kernel driver updates, Xcitium provides robust protection against ransomware and unknown threats while ensuring system stability and minimizing the risk of BSOD incidents.
BSOD Incidents Caused by Endpoint Security Products
The cybersecurity industry has seen numerous instances where insufficiently tested kernel driver updates have caused BSODs, leading to significant downtime and operational disruptions. All of these issues are either caused by a not-well-tested Kernel Level Driver or False Detection of legitimate Windows processes.
CrowdStrike Falcon Sensor
CrowdStrike agent caused a BSOD loop on SQL Nodes. This impacts many MSSQL servers over the world. The BSOD Error we were seeing was: DRIVER_OVERRAN_STACK_BUFFER.
More details: https://www.reddit.com/r/sysadmin/comments/152iyhl/psa_crowdstrike_falcon_update_causing_bsod_loop/
CrowdStrike agent caused BSOD on MS Server 2022 Terminal servers: random BSODs with the error “KERNEL_SECURITY_CHECK_FAILURE (139 due to a stack-based buffer overrun, potentially allowing malicious control.
Sophos Endpoint Protection
The SophosNetFilter component has been a recurring issue, leading to BSODs especially after Windows updates. Disabling the web filter policy and Real-Time Internet scanning resolved the crashes for many users. This problem often involved conflicts with the NETIO.SYS driver.
More details: https://support.sophos.com/support/s/article/KB-000045244?language=en_US
Cisco Secure Endpoint
Formerly known as Cisco AMP for Endpoints, this solution provides advanced threat protection but has faced issues with agent deployment and process-based queries, which could potentially lead to BSODs if misconfigurations or conflicts arise during its operations.
More details: https://heimdalsecurity.com
Symantec Endpoint Protection
An update to Symantec’s intrusion prevention system (IPS) definitions caused widespread BSODs. This issue was linked to the IDSvix86.sys and IDSvix64.sys drivers. Symantec released updated definitions and provided workarounds involving Safe Mode and command-line adjustments to resolve the problem.
More details: https://www.bleepingcomputer.com/news/security/symantec-fixes-bad-ips-definitions-that-cause-a-windows-bsod/
Microsoft Defender for Endpoint
Defender’s mssense.exe process has been reported to cause file locking issues and system instability. Although users applied exclusions, the process continued to access files, requiring support intervention to manage these exclusions properly.
Check Point Endpoint Security
A BSOD problem was observed on a virtual server, suspected to be related to the Check Point Endpoint Security client. Detailed analysis of the dump logs suggested that the issue might be caused by the endpoint security client, although specific fixes were not detailed.
More details: https://community.checkpoint.com/t5/Endpoint/Possible-reason-for-BSOD-Blue-screen-of-death/m-p/151526
Trellix (formerly McAfee)
Similar issues have been reported with Trellix, where specific driver conflicts and signature updates led to BSODs. Trellix has advised users to ensure their systems are regularly updated and to apply specific policies to mitigate these issues.
More details: https://kcm.trellix.com
VMware Carbon Black
A faulty ruleset in certain versions of Carbon Black’s endpoint security solution caused widespread BSODs across multiple organizations. VMware recommended putting sensors into Bypass mode until the faulty ruleset was rolled back.
More details: https://www.vmware.com/products/carbon-black.html