It was a cold Wednesday morning, about ten ‘til eight. I had been in the office a few minutes when the phone rang.
Hello. “Yes, hi, the server has been hacked!"
How do you know? “The picture. Look at the picture."
What picture? “The one on the server."
What server are we talking about? “The web server!"
Ah, OK, what’s the url?
“It’s www.example.com/register-for-the-conference “
I opened a browser and navigated to the web site. There was a picture of a woman without any clothes. She was sitting on a stool wearing a black gangster style hat and smoking a cigarette. She looked like a model posing for an art class drawing.
OK, I see the picture. That does not look like it should be there.
“No, it’s not supposed to be there and people are trying to register for the conference on the web site."
Alright. I’ll talk with the system administrator and we’ll take care of it.
“She’s in training on the west coast."
Are there any other technical staff in the office?
“No. It’s just me and the receptionist. Can you come over? We’re on the first floor of the Smith Building. Just come in the main entrance and I’ll meet you in the lobby."
Sure, I’ll drive over now. By the way, what’s your name?
“Thanks, my name is John. We’re getting a lot of phone calls about the picture. Please hurry."
I grabbed my backpack and a spare laptop and drove to the Smith Building.
The Smith Building
The Smith Building was about a 15 minute drive across town (next to the post office). The morning commute was over, and most people were away for the holiday, so I made the drive in about ten minutes.
John was in the lobby waiting for me.
Hello John. I’m Brad… the new security guy.
“Yes, hi, it’s nice to meet you."
Where’s this unfortunate web server?
“I don’t know.”
You don’t know where the server is at?
Alright then, can you take me to where the system administrator works?
“Sure, Janet’s office is on the third floor."
The third floor
We took the elevator to the third floor and walked over to Janet’s office.
Her office was large and well organized. I could tell she was a true hacker. Unix and C books were stacked neatly on her desk and she had a new Sun workstation. There was a nice view of the Town Center from the office window.
“I hardly ever come up here. I run the business office on the main floor."
That was why John didn’t know where the server was at. He was just caught in the middle answering phone calls from people trying to register for the conference.
There was a large solid wood door in a metal frame at the back of Janet’s office that was locked. I could hear the faint hum of machines behind it.
Let’s have a look behind that door John.
John took a large set of keys out of his pocket and found one that unlocked the door.
The server room
It wasn’t a proper data center, but it was a nice server room. It was cooled sufficiently, had three full sized racks, which were roughly half-way filled, and was neatly maintained.
Each rack had its own KVM switch, but they weren’t much use yet. We could not log into anything and still weren’t sure which machine ran the web server software.
I booted my laptop and asked John if he had heard of the program called ping. He had not.
I explained that ping was a program that machines could use to check on each other.
It’s sort of like calling out, “Are you home?" to your next door neighbor. If they call back, “Yes, I’m home”, then you know they are home. If they don’t, then you know they are away.
“OK, but, I don’t understand how we can use that to find the web server."
You’ll run ping continuously, from my laptop, asking the web server if it’s home. It should always respond. Like this:
$ ping example.com
PING example.com (126.96.36.199): 56 data bytes
64 bytes from 188.8.131.52: icmp_seq=0 ttl=58 time=40.740 ms
64 bytes from 184.108.40.206: icmp_seq=1 ttl=58 time=38.695 ms
64 bytes from 220.127.116.11: icmp_seq=2 ttl=58 time=38.298 ms
While you watch the ping responses, I’ll go behind the server racks and unplug one network cable from one server at a time. When I unplug the web server cable, the responses will stop.
“Yes, I see now how ping will help us find it."
We found it
The web server stopped responding to pings when I pulled the sixth Ethernet cable. I was glad because I had no idea what the servers were doing and didn’t want to unplug something essential.
“It just stopped!"
Are you sure?
“Yes… Yes, the responses have stopped. They are no longer scrolling."
OK, let me plug it back in. If this is the right machine, the responses should start again.
“I see the responses again now. We’ve really found it!"
Yes, You’re right. We have found it.
A quick fix
John stayed with me while I figured out the KVM. I was glad he was there. I could not have found the server without him.
I had a forensic recovery CD and ran dcfldd to get an image of the hard drive.
After the image had finished, I mounted the file system (read-only) and poked around and found some old PHP code that could be abused to upload files to the web server along with logs indicating that was indeed what had happened.
I left the forensic OS running and wrote a bash script to serve up a static web page asking people to call the main office line to register for the conference.
while : ; do cat conference.txt | nc -l 80; done
The conference.txt file contained these HTTP headers and HTML code:
HTTP/1.1 200 OK
Content-Type: text/html; charset=ISO-8859-1
<p>The registration system is down for maintenance. Please call 1-800-123-4567 to register.</p>
This wasn’t ideal as John and the receptionist would have to answer phone calls, but given the circumstances, we agreed that it was the best thing to do.
We were unsure what other vulnerable code might be on the web server and would not know until I had time to do a more thorough investigation of the image.
John found some help handling the conference registration phone calls. It was the last day to register so the call volume wasn’t as bad as we thought it would be.
We became good friends and I got to meet Janet a few weeks later too. I was right, she was a real hacker and we had a lot of great conversations while we worked together.
But, that was a long time ago.