A threat actor has compromised close to 2,000 thousand Citrix NetScaler servers in a massive campaign exploiting the critical-severity remote code execution tracked as CVE-2023-3519.

More than 1,200 servers were backdoored before administrators installed the patch for the vulnerability and continue to be compromised because they have not been checked for signs of successful exploitation, the researchers say.

RCE exploited to hack 6% of all vulnerable servers

Security researchers at cybersecurity company Fox-IT (part of the NCC Group) and the Dutch Institute of Vulnerability Disclosure (DIVD) have discovered a large-scale campaign that planted webshells on Citrix Netscaler servers vulnerable to CVE-2023-3519.

Although the vulnerability received a patch on July 18, hackers started exploiting it in the wild as a zero-day to execute code without authentication.

On July 21, Cybersecurity and Infrastructure Security Agency (CISA) warned that the vulnerability had been leveraged to breach a critical infrastructure organization in the U.S.

Earlier this month, the non-profit organization The Shadowserver Foundation found that hackers had infected more than 640 Citrix NetScaler servers and planted web shells for remote access and persistence.

Over the past two months, Fox-IT responded to multiple incidents related to CVE-2023-3519 exploitation and discovered servers compromised with several web shells.

Using the details about the backdoors, Fox-IT and DIVD were able to scan the internet for devices that had the web shells installed. Administrators can recognize their scans by checking the Citrix HTTP Access logs for the user-agent: DIVD-2023-00033.

Initially, the scans considered only vulnerable systems but later expanded to Citrix instances that received the update to address CVE-2023-3519.

This revealed 1,952 NetScaler servers backdoored with the same web shells Fox-IT found during the incident response engagements, indicating that the adversary used an automated method to exploit the vulnerability at a large scale.

In a larger context, the 1,952 backdoored servers represent more than 6% of the 31,127 Citrix NetScaler instances vulnerable to CVE-2023-3519 at a global level when the campaign was active.

Of the discovered compromised servers, Fox-IT says that 1,828 remained backdoored on August 14 and that 1,247 had been patched after the hackers planted the web shells.

On August 10, Fox-IT and DIVD started to reach out to organizations, either directly or through national CERTs, about compromised NetScaler instances on their network.

Yesterday, the largest number of compromised Citrix NetScaler servers, both patched and unpatched, was in Germany, followed by France and Switzerland.

Fox-IT says that Europe is the most affected, highlighting that of the top 10 affected countries, only two are from a different region of the world.

Another detail the researchers observed is that while Canada, Russia, and the U.S. had thousands of vulnerable NetScaler servers on July 21, they found compromising web shells on almost none of them.

Fox-IT says that the number of affected Citrix NetScaler servers is declining but there are still plenty of compromised instances.

The researchers warn that a patched NetScaler server can still have a backdoor and recommend administrators perform basic triage on their systems.

They provide a Python script that uses the Dissect forensics and incident response toolkit.

Mandiant has also released a scanner that looks for indicators of compromise related to attacks exploiting CVE-2023-3519. The researchers caution, though, that running this bash script twice results in false positives because “certain searches get written into the NetScaler logs whenever the script is run.”