This is a tough situation, no question, and a great example of why it's important to hire the best quality personnel you can, then make sure you have workarounds if possible. And reliable backups.

To offer you the most useful possible information, I asked a few colleagues for their advice on this situation. Here's what they offered:

---

Bad news. The company should treat this as a hostile incident, and should ideally erase and rebuild the server from scratch. A backup needs to be made of datafiles (mysql databases and html/cgi-bin) plus other user files. Since the victim doesn't know what other goodies the former employee has left on the machine, they need to assume that other system files have been altered. Every user account on the machine should be confirmed as needed.

#0. Take the machine off the network first thing. Minimize the damage that might be going on.

#1. Back up the data files to CD or DVD or an external firewire hard disk. You do have some kind of automated backup system in place already, right?

#2. Reinstall the system from original disks, or from original downloads from publisher web sites. Install all released system updates and patches.

#3.Choose a new root (admin) password. Make it a strong one, uppercase, lowercase, numeric, shifted characters.

#4. Set up the software (Mysql and apache?) Or is it WebStar? The only servers/services that should be enabled are those that you will use. If you don't need Samba file sharing, don't enable it.

#5. The settings files for the web sites that are using MySQL should ideally have a unique username and password, not using the root. For example, on my Linux boxes, I have several web sites using the Wordpress blog system. Each Wordpress installation has a unique database, username and password that can be used. his helps minimize damage if one of your sites/systems is hacked or compromised.

#6. Each user (the new system admin, regular employe, etc...) should have their own system username and password. "Jim SysAdmin" should not know the password for "Joe SysAdmin" Then later when "Jim SysAdmin" quits/is let go, you can deactivate his account easily. The Worksgroup Manager of OS X 10.4Server is very flexible when it comes to setting up users, and more importantly, groups. OS X Server also allows for Access control Lists which allows for very flexible access to files and directories.

#7. It can't hurt to call the police. Good luck. If you can show proof of a significant dollar figure loss, they may be interested. Probably they won't. Keep all logs showing the dates and times and IP addresses of what was accessed.

#8. You should also be proactive and let your clients know what is going on. Especially since their web sites will go down once you start to rebuild the box.

This victim should look at their disaster plan again. What would they do if the building the server is in caught fire? OR someone spilled their coke into the machine? Or a hurricane came through. Etc...

Good luck, I hope this is helpful,
Michael Clarke, Planet Mike.com.

---

Just a note that the recommended method nowadays is using the GRANT command rather than editing the tables directly:

MySQL: GRANT

or the new SET PASSWORD:

MySQL: Set Password

Jeremy Zawodny, Yahoo.com

---

This will sound a bit Draconian, but any sysadmin worth his salt knows there is no 'fix' except complete rebuilding of the machine in question. The former sysadmin probably left a bunch of 'holes' only he knows about. Even if this isn't the case, the greater than 0% possibility warrants an immediate rebuild.

Again, it seems drastic, but I think this is the only right thing to recommend. Yes, I'm aware this machine is in 'production', but the costs of a machine overhaul are most certainly outweighed by the benefits.

Sahil Tandon

---

Root level passwords will eventually require changed no doubt about that. I am surprise to here of such unethical behavior from SA... very surprised.

The most immediate thing to do is to stop the remote service from running be it vnc, ssh, telnet or whatever. this will effectively restrict access to your server at local level. Of course the assumption here is that it is possible... ie, the server is at your office or somewhere with easy access.

Another quick alternative and somewhat more involved is to restrict access at the firewall level... assuming of course you have one.

i am not entirely familiar with how Macs / BSDs set up the services mentioned above, for the moment however, focus on disabling/killing the target service that provides remote access.

kjteoh, 5lm.net

---

Without knowing the details of how the websites are set up, I can't really give step-by-step instructions on how to go about securing the system but the following might be of help.

First of all, let me mention that if the rogue sysadmin set the system up himself, and is intent on breaking in, simply changing passwords may not keep him out. There's a good chance he has left a back door or two open for himself. If at all possible, the server should be rebuilt. This would mean some downtime for whatever the system is running. If another server is available, it might be worth it to move the services to the backup server while the primary gets rebuilt.

Anyway, to the problem at hand. I see the need to lock down the two ways the server is being accessed; the shell, and the database.

The shell: Changing the password for the root user is the first step in securing shell access. This might not be enough as they suspect someone on the inside might be helping the rogue sysadmin. If the server is running in production mode, updates don't need root level access. My suggestion is to lock the root account (NetInfo Manager : Domain -> Security -> Disable Root User), create a new administrator account and use sudo for administration activity as the new user. Disable all old admin accounts.

The mysql db: The primary problem here is that client sites depend on accessing the server using the root account. The root account for the database should only be locally accessible, i.e. host=localhost in user table. In any case, the root password for mysql needs to be changed, this step needs to be followed up by changing the configuration of each of the applications so they use the new password. If the sites use php, there's typically a config.php or similarly named file in the application somewhere. For java apps, search through .properties files for the password.

NOTE: In some cases, depending on the db driver in use, the new password won't work even after changing it in the configuration files for applications that access the database. This is typically caused by the stronger MD5 passwords in use by mysql that the client or the driver can't work with. The quick solution here is to set the password to the original unix crypt style by a command similar to this "set password for '<account>'@'<host>' = old_password('<password>')"

This should address the most immediate problem. However, I'd suggest setting the system up as follows:

1. Install server software from scratch.
2. Install/upgrade the database software (Mac OS X server comes with MySQL installed, the client version doesn't)
3. Create a separate database for each client application. If they share a common database, use a common database user other than the root account. If the websites are being served from the same server thats running the database, make sure remote access is not allowed for any user account in the database (do a 'select user,host from user' query using the root account and mysql database to see who can access the server from where, there should be no '%' entries in the host column).
4. Install the web application software (I'm assuming these are web applications as they need to access the database)
5. Disable root account, set up admin account and give access to only the people that must have it.

Salman Khattak

---

I hope these ideas can help you recover from this situation.Without question, you need to act, and act now to minimize the loss and exposure of your company to this unethical former employee.

And good luck to you.