Brute Force Protection :- If account is currently locked out because a brute force attempt was detected. Please wait 10 minutes and try again. Attempting to login again will only increase this delay. If you frequently experience this problem, we recommend having your username changed to something less generic.

Information Gathering

Since access to the box wasn’t urgent, I did what I recommend that everyone does in situations like this: gather information first. I wanted to know what caused the lock out first. In other words, who or what was brute forcing the box and caused this issue.

It was quite possible that my box was still under attack. If I simply turned off brute force protection to bypass the block, I could have opened up my box to being compromised.

cPHulk stores all of its information in a database called cphulkd. There are two tables of interest: logins and brutes. The logins table stores login authentication failures. The brutes table stores excessive authentication failures indicative of a brute force attack.

Here is what I saw on the server:

[chris@office ~]$ ssh server
Last login: Wed Oct 14 11:02:14 2009 from host
[user@server ~]$ mysql -u user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is id
Server version: version

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input
statement.

mysql> connect cphulkd
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Connection id: id
Current database: cphulkd

mysql> select IP, BRUTETIME from brutes order by BRUTETIME;
Empty set (0.00 sec)

mysql> select IP, LOGINTIME FROM logins order by LOGINTIME;
+———————————+———————+
| IP | LOGINTIME |
+———————————+———————+
| 220.199.6.48 | 2009-10-14 11:23:10 |
| 220.199.6.48 | 2009-10-14 11:23:10 |
| 220.199.6.48 | 2009-10-14 11:23:10 |
| 118.212.186.59 | 2009-10-14 11:23:40 |
| 118.212.186.59 | 2009-10-14 11:23:40 |
| 118.212.186.59 | 2009-10-14 11:23:40 |
| djdeatheater.liquidweb.com | 2009-10-14 11:24:03 |
| 221.7.58.37 | 2009-10-14 11:24:07 |
| 221.7.58.37 | 2009-10-14 11:24:07 |
| 221.7.58.37 | 2009-10-14 11:24:07 |
| djdeatheater.liquidweb.com | 2009-10-14 11:24:09 |
| djdeatheater.liquidweb.com | 2009-10-14 11:24:15 |
| mail.ingener.com | 2009-10-14 11:24:53 |
| mail.ingener.com | 2009-10-14 11:24:57 |
| 123.147.144.45 | 2009-10-14 11:25:16 |
| 123.147.144.45 | 2009-10-14 11:25:16 |
| 123.147.144.45 | 2009-10-14 11:25:16 |
| 119.62.128.42 | 2009-10-14 11:25:41 |
| 119.62.128.42 | 2009-10-14 11:25:41 |
| 119.62.128.42 | 2009-10-14 11:25:41 |
| pomme.sai.msu.ru | 2009-10-14 11:26:13 |
| pomme.sai.msu.ru | 2009-10-14 11:26:13 |
| pomme.sai.msu.ru | 2009-10-14 11:26:13 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 84-74-21-119.dclient.hispeed.ch | 2009-10-14 11:26:48 |
| 114.143.242.51 | 2009-10-14 11:27:23 |
| 114.143.242.51 | 2009-10-14 11:27:23 |
| 114.143.242.51 | 2009-10-14 11:27:23 |
| 222.179.116.53 | 2009-10-14 11:27:47 |
| 222.179.116.53 | 2009-10-14 11:27:47 |
| 222.179.116.53 | 2009-10-14 11:27:47 |
+———————————+———————+
32 rows in set (0.00 sec)
As you can see, a distributed brute force login attempt was launched starting at 11:23am. Fortunately, cPHulk quickly recognized the attack and had completely shut down login access for the user account.

Thus, when we tried to log in, it of course didn’t work.

Regaining Access

Since I now knew the problem, and now felt good about how cPHulk protected our box, I decided to open up access again so we could go about our work. There are two different ways that I could have done this.

The first way would be to disable cPHulk to regain access, log into WHM, clear out the the block by using the “Flush DB” option in the cPHulk settings page, and then re-enable cPHulk. A number of people recommended this method, but I didn’t like it. I certainly don’t want to disable a security measure that successfully protected the box just to be able to regain access. What would happen if a huge wave of brute force authentication attempts hit the box in the time between disabling and re-enabling cPHulk? The answer is that the box wouldn’t protest and would tell the attacking program whether each attempt was successful or not.

If you need to use this method, the two commands you will want to use are: /usr/local/cpanel/bin/cphulk_pam_ctl --disable and /usr/local/cpanel/bin/cphulk_pam_ctl --enable. These two commands will disable and enable cPHulk, respectively.

I decided to use another method. This method didn’t require disabling cPHulk, and thus, didn’t require reducing protection to regain access. Essentially, I cleared the tables manually, so that I could log in once again. Since this opens up the login process again, a brute force could still proceed but would be quickly shut down again.

While still connected to the database through the MySQL monitor, I ran a couple more queries.

Note: Before I ran these queries. I stored the details of these attacks in a file on the server in case I needed to refer to the information later. Never delete attack data without keeping a record of it somewhere. You never know when you may need it.

mysql> delete from brutes;
Query OK, 0 rows affected (0.00 sec)

mysql> delete from logins;
Query OK, 32 rows affected (0.00 sec)

Now, we can log back into the box.

Finishing Up

I’m sure that many people encountering this situation would just pat themselves on the back and go about their day, but I wasn’t satisfied.

I wanted to ensure that the office wouldn’t be blocked in the event that another brute force attack happens, so I added our office to the whitelist. The whitelist supports both IP numbers and hostnames. So, if you have a dynamic IP, you can use a DDNS (Dynamic DNS) service to get your own hostname to use. Both DynDNS and No-IP offer reliable free solutions that provide you with a hostname to use. In addition, EveryDNS is a free DNS service provider that also offers the ability to create subdomains of your personal domain that can link to dynamic IPs. Many current routers support either DynDNS, No-IP, or both so that the router can handle updating the hostname automatically. Some router firmwares, such as Tomy support a much larger variety of DDNS services.

Another nice feature about using the DDNS services and whitelisting a hostname is that you can update that DNS pointer remotely. This means that if you get locked out of WHM when you are away from the location that is whitelisted, you can update the IP on the DDNS system, wait a few minutes for the IP to update, and then log into WHM via the whitelist.