The Storm

What happens if one pokes a stick into a hill of fire ants? You’re in trouble. The same happened to me when I engaged in some discussions with #infosec on Twitter. What could have been a nice weekend, turned into a lot of work.

It was Friday afternoon when I was lazily monitoring Twitter because I had nothing else to do. A #infosec message caught my attention:

I felt .. a strong .. hmm .. urge .. to contribute and wrote:

I would never hire any #infosec person that doesn’t have a strong background in software engineering. Take those fancy infosec certificates and stuff ’em you know where 🙂

Agreed – my choice of words may have been a tad too aggressive. But come on – we all know that there are actually a number of “certifications” that don’t really mean a lot. And I truly believe that in-depth knowledge of software and IT-environments should be required for any advanced infosec specialist. You can’t really talk about buffer-overflows if you don’t know the technical reasons or problems associated with those pesky bugs. I personally don’t know a lot about Windows because I seldom use it. I would never offer any Infosec advise to Windows users – just the usual: Don’t click everything – use a virus scanner – be careful.

Anyway – a heated discussion ensued. Knowing that emotional Twitter discussions sometimes lead into unexpected territory – especially when you engage folks that have all those “pen-testing” toys at hand – I made a full backup of all servers under my jurisdiction.

I also had a brand new Debian based system fully configured and waiting to to be deployed – though because some of my scripts never had “experienced” PHP 7 – I wanted a little more time to play around with. But it was ready to replace the old CentOS box at a moments notice.

So I went ahead and continued to stuff my foot into my mouth and wrote:

I work for a company that has never had any of it’s client breached or compromised in any way. None of my clients in this and other companies has been breached or compromised, I am doing this since > 20 years. Questions?

Now – while this statement is basically true, it was incomplete (I don’t do Windows or Macs) and it was probably not the smartest thing to say. But – then again – I had the new Debian system ready, additional backups and a free weekend ahead – why not let it play out.

I guess my servers have never been probed this hard. And yes – problems were found and fixed. Though the server on CentOS was old, it (and the PHP core) were patched up to the latest update. And – while the wordpress-installations on the new Debian system were up-to-date, the versions on the CentOS were not. I was just a little lazy – and that is never good. Lesson learned.

The load increased steadily during Friday night and I had to block a few IP addresses that tried some destructive stuff, but all in all folks behaved fairly.

And while people were playing with my servers, I ran the necessary tests to make sure I was able to switch the system over to the new Debian environment. I actually switched a few times between the CentOS and Debian, but nobody noticed.

Because I made sure to keep the old server signatures.

The Debian server has now been running for a while and people are still trying to prove a point. The old server signatures are still being displayed because, you know .. em .. I don’t want to spoil the fun.

Here’s the summary and what I learned:

For whatever reason, the INDEXES keyword was in the old apache config. That allowed people to view the contents of the wordpress directories. Not professional at all.

Lot of people probed for wordpress .php backup files and for good reason. Something like wp-config.php.1 would be displayed instead of executed revealing all the contents of the configuration files. There weren’t any on my servers, but I’ll keep that in mind for sure.

Others probed for wp-login.php. I usually keep those files in place, but decided to move them during the onslaught of requests. This is inconvenient of course, so I might move them back at some point.

I also activated a module that blocks the IP-address of users who request the ‘wrong’ files or get too many error messages within a time limit. Talking about wp-config backup files 😀 Happy to share, let me know.

Now – some people suggested that all this must have been a ploy in order to get some free pentests going on. This wasn’t my intention. Though I let it proceed after it started. I could have shutdown the servers and wait for this to blow over. I didn’t and learned in the process.

Here’s what was cool:

Most people behaved very professionally and I truly appreciate that. No denial-of-service, no excessive probing (for the most part).

Here’s what wasn’t cool:

Personal attacks. I mean .. come on – typos? Political statements? Denigrating statements about my country of origin (Germany) or the city I am living in? Hey typo police: Once your German is as good as my English, I salute you and officially allow you to correct me in public.

Was any server breached?

I don’t know. And it doesn’t really matter as there was nothing really important on the server, the Debian platform had been created before the shit started to flow in and all digital keys and passwords have been changed.

Summary:

I realize that I was somewhat disrespectful to the infosec community and thus I had to spend my weekend in front of the screens instead of enjoying some R&R with the dogs and a few quiet quiet hours with a good book or a movie. Just to be sure: I never invited anybody to probe my systems and it leaves a sour taste that some people are willing to use their trade and tools just to prove a point.

Note to myself: Never brag that nobody has shot me yet.

I will make all logs available for download (here) after I clean the IP-addresses and some preliminary screening.