This post is an informational announcement detailing the Citrix NetScaler Vulnerability and what Datashield is doing to help our customers detect and mitigate the issue.

Since the announcement of the vulnerability in Citrix devices (CVE-2019-19781) Datashield has performed extensive analysis and research on exploit attempts, attack patterns, and the latest intelligence. Citrix released security bulletin CTX267027 containing information on a vulnerability in various Citrix Products. This allows an unauthenticated attack that allows remote code execution.

Detection:

It is currently difficult to determine the exact origin of an attack against Citrix devices using the CVE-2019-19781 vulnerability.  Attacks are encrypted and are therefore difficult to detect without packet capture with decryption solutions, making traditional web attack detection methods less useful and reliable.  The best method for detection of this type of attack is to monitor for unexpected connections from Citrix devices to out-of-country IPs.  For those with global operations, whitelisting certain IPs or IP ranges may be necessary.  

C2 traffic typically consists of payload retrieval that contains a shell, script, or other backdoor.  This is often observed in plaintext and is the most readily identifiable indication of an attack.

Mitigation:

No official patch is currently available, although Citrix mitigations are available for the following Citrix ADC, Citrix Gateway, NetScaler Gateway, and Citrix NetScaler ADC devices:

• Citrix ADC and Citrix Gateway version 13.0 all supported builds
• Citrix ADC and NetScaler Gateway version 12.1 all supported builds
• Citrix ADC and NetScaler Gateway version 12.0 all supported builds
• Citrix ADC and NetScaler Gateway version 11.1 all supported builds
• Citrix NetScaler ADC and NetScaler Gateway version 10.5 all supported builds

Some suggestions for mitigation and containment include:

Block inbound and outbound traffic to

-185.178.45[.]221

-95.179.163[.]186

-62.113.112[.]33

-193.187.174[.]104

-217.12.221[.]12.

-104.168.166[.]234

-104.244.74[.]47

-111.206.52[.]101

-111.206.52[.]81

-111.206.59[.]134

-111.206.59[.]142

-159.69.37[.]196

-167.88.7[.]134

-185.178.45[.]221

-185.212.170[.]163

-185.220.101[.]69

-188.166.106[.]153

-192.236.192[.]119

-192.236.192[.]3

-192.3.255[.]144

-193.187.174[.]104

-217.12.221[.]12

-23.129.64[.]157

-27.115.124[.]70

-27.115.124[.]74

-27.115.124[.]9

-31.134.200[.]75

-45.32.45[.]46

-45.83.67[.]200

-47.52.196[.]15

-47.52.196[.]152

-5.101.0[.]209

-51.68.122[.]93

-61.218.225[.]74

-62.113.112[.]33

-81.110.55[.]125

-82.27.64[.]190

-85.248.227[.]164

-94.140.114[.]194

-95.179.163[.]186

• Implement Citrix’s recommendations that were provided in the communication DATASHIELD sent out Wednesday January 15th.  For your reference, the recommended Citrix mitigations are here: https://support.citrix.com/article/CTX267679
• Pull bash history log. The path of the log should be /usr/bin/bash/bash.log.
• Review cron jobs on the Citrix NetScalers, disable/remove suspicious cron jobs.
• Change passwords for all users on the device(s).
• Pull a list of running processes and terminate any confirmed suspicious/malicious processes (specifically running under the user nobody).
• Review any files that have a file modification date on or after January 10, 2020.  While a file modification date can be manipulated to make forensic analysis harder, it’s recommended to pull and review any of these files.
• Pull the Apache access logs
• Review suspicious files within the directories of /netscaler/portal/templates and /var/tmp/netscaler/portal/templates.  If you identify any suspicious files and delete the file(s)/remove the malicious code.
• Review the Apache error and notice logs for any suspicious activity
• The Following commands will locate any successful exploit attempts against the device:
     -grep -iE 'POST.*\.pl HTTP/1\.1\" 200 ' /var/log/httpaccess.log -A 1
     -grep -iE 'GET.*\.xml HTTP/1\.1\" 200' /var/log/httpaccess.log -B 1

Post Attack Remediation:

Remediation is expected to take the form of a complete wipe and rebuild of any infected Citrix devices.

Citrix has stated a wipe tool will be available by the end of day on January 16th and will announce when it is available.  Datashield recommends applying the mitigation immediately if devices are not affected by infection.  

If they have been impacted, we recommend standing up isolated, fresh builds of the required devices with the mitigation applied, deploying the fresh builds, and then stand down the infected builds.

References:

• https://support.citrix.com/article/CTX267679 (Mitigations)
• https://support.citrix.com/article/CTX267027 (Citrix Bulletin)
• https://nvd.nist.gov/vuln/detail/CVE-2019-19781 (NIST Listing)