I have several websites. The way my web host has them set up, like many hosts who use cPanel, is that one site is a “master” and the others essentially exist as directories within that site. My master site is smallaxesolutions.com, which is the “company” under which I sometimes do my web design and network support business.
One of the things I (used to) do as Small Axe Solutions was publish the core code of the engine that powers blog.adamscheinberg.com, Small Axe. Small Axe code was built up as 0.1, then 0.2, then 0.3. At that point, I had added several features to blog.adamscheinberg.com that I had yet to merge upstream into Small Axe. So, I created a build system so I could slowly integrate the changes. In short, it worked like this: I had a directory called “build_source” which contained my current code. Of course, it had all kinds of problems out of the box, like the config files which pointed to nonsensical location like /path/to/your/blog/. It had no valid database connection info. The flatfiles were unwritable. So, in short, the code was (usually) solid, but PHP couldn’t compile it.
Meanwhile, another directory called “demo” was waiting silently.
Lastly, a third directory, outside the web root, called “static” was sitting with pre-built config files, db connection files, and some other stuff.
Then it was just a matter of a simple shell script. The script did the following: it deleted everything in the “demo” directory. Then it copied all of the files in the “build_source” directory into the demo directory. It deleted the config file and overwrote it with a copy from the “static” directory. Same for the db connection and a few other files. It left the demo directory as a live, fully functional build of the current code. Then it zipped everything in the “build_source” directory and put it into my downloads section. It ran this script every 30 minutes for probably 2 years now. I only chose 30 minutes because it made sense from a development standpoint to see the updates quickly. I stopped working on that version some time ago, but never got around to updating or changing the script.
Fast forward to a few weeks ago, I was cleaning out a bunch of old directories. Within 5 minutes, EVERYTHING was gone: my mail, *all* of my sites, my temp files, everything in my home directory that wasn’t a hidden file preceded with a dot. I didn’t realize this for several hours, but I then I restored from a backup and within 45 minutes, everything was gone again! Oh noes!
I immediately begin researching security and disabling all of my upload scripts. Something is wrong, I thought. I searched high and low. But, as you guessed, I didn’t find anything wrong, because there was nothing wrong. In my cleanup, as you may have gussed by now, I decided to delete the “demo” folder. The first line of my shell script is “cd /home/adam/public_html/build_source.” Then second, scary line, is “rm -rf *“. Since there was no “build_source” folder, the first line flat out failed, leaving the script in /home/adam. Then, unfortunately, it ran rm -rf * in the root of my home directory. Killer!
It took my some time to swallow my own stupidity. All I had to do was comment out the cron job to prevent this disaster. But alas, I dropped the ball. We’re back online now, and a little smarter.
