Even ISP's Get Attacked

I am a member of one of the oldest computer user groups. In Fall 2005, the ISP our members used imploded - complete server crash. Hey, these things happen ... no big deal ... assuming you have backups. We had been assured that backups were made every night.

That's right, Milkern.com had not made backups for over a year.

(As of Sept 2006, Milkern was substantially out of the web service business ... and they never refunded over $200 that they owed ME. I understand that they did not refund money to many other people. This is a real shame ... we received great service from them for more than 10 years, and now they are gone.)

As a result, we hired a new ISP - AlphaOne-Tech. Things went pretty good until about the last week of August 2006. (I would recommend avoiding these people at all costs.)

Loss of Service at AlphaOne-Tech

Sometime near the end of August 2006, the cpcug.org account hosted by AlphaOne-Tech came under attack.

There appeared to be 3 separate attacks at about the same time

One web search engine at one site was trying to read a file that did not exist. This occurred about 5 times a second. I contacted the owner of the site and they fixed the problem.
That same missing file was being accessed about 5 times a minute by various sites all over the world. We just ignored those.
Our site was caught sending some type of phishing email and severely overloaded the server.

Before I got involved, the ISP tried to disable the phishing by first deleting an email account that they thought might be the problem and then by disabling all web based (browser based) email. I was told both

The outgoing emails were stopped when webmail was disabled
Disabling webmail had only a slight effect

No one on the cpcug support team ever saw the phishing email ... so we were never able to try and find the source.

After the webmail was disabled, there was one log file that indicated that the attack was still originating from the cpcug.org web site ... but that was all it said. Apparently, cpanel (the website management software provided by AlphaOne-Tech) has many known design problems ... several allow anonymous attacks and then do not log ANY USEFUL INFORMATION. (I am not providing a link supporting this statement because the data can not be read without a password.)

As a result of the supposed phishing attack, AlphaOne-Tech claims that they had to pay to get their site off of several blacklists. The charges were then passed to us without any evidence (either to support the phishing claim or that they paid off the blacklisters). When we questioned this charge, we received threats that they would turn the charge over to a collection agency if we refused to pay ... but no supporting documentation. (Needless to say, we now have a new ISP.)

No one was ever able to determine the source of the attack.

The primary problems were that

cpanel (the website management software) does NOT produce any logs that can be used to solve problems.
AlphaOne-Tech never provided copies of the outgoing emails
AlphaOne-Tech did not place a software limit on outgoing emails ... even after the problem had occurred for over a week

At some point, the cpanel DNS management functions icon disappeared. The ISP said they knew nothing about it because cpanel updates itself on a regular basis and that they (AlphaOne-Tech) were not responsible for any changes. However, it is possible that that was a part of the attack.

At any rate, the extremely high server load did not stop until after all cpcug.org access to the server was terminated.

Basically, being hacked was OUR fault. AlphaOne-Tech refused to provide the tools necessary to diagnose and fix the problem (in fact, they never provided evidence that cpcug.org was responsible for the problem). But, eventually, AlphaOne-Tech gave up, kicked us off their server, sent us a hefty bill for damages, and threatened to turn the bill over to a collection agency if we did not pay it.

Summary

In summary,

I would avoid AlphaOne-Tech specifically because of their inability to solve real problems
I would avoid cpanel (over 1.4 million sites in 2004) because
- They keep critical information secret
- They don't provide sufficient information (adequate logs) to fix problems
- It appears that the hole was in THEIR system (I could never prove that)
- They are at least partly responsible for our email problems
- cPanel Security Hole Exploited in Mass Hack (posted Sept 23, 2006) details a different type of attack. Additional data.
Anyone who believes that Linux systems are immune to attacks is simply wrong

Author: Robert Clemenzi