Work

Well on Sunday I got a call from Jay, there is a forest fire going on in the Boundary Waters again. So I drove up here on Sunday night, and we’re finally settled in with the network deployed, and I have some time to get online and catch up.

For those who are unaware, I helped a local company (www.tacticalsatellite.com) build a deployable emercency communications system. This system connects up through a satellite uplink, and uses Military grade Mesh Wireless to build a Mesh Network. We use this mesh network as the backbone of the network, and run 802.11 and wired LAN on top of it. This allows us to provide Internet, Phone, and Fax out through the satellite. The system is more complex than that, however, but that is the basic gist of it.

The fire this year, is the Cavity Lake fire. It started from a lightning strike, and as continued to grow. As of this morning, the fire covers over 15,000 acres, and is more than 5 times as large as the Alpine lake fire that happened last year.

I’m hoping to get out with my digital camera and take a bunch of pics. This afternoon we are going to be shooting the mesh out a mile or so, so that we can light up the heli-base with internet and phone. This will be fun, especially since the national guard has brought in a pair of blackhawk helicopters to help with air attack operations.

Anyways, enough for now, I’ll post more as I have time. All I got to say is, I love my job!

So the other week I was in the datacenter doing a routine check on one of our servers, and I noticed that Chimera (our MySQL server) had a blinking orange light on it, and it was making this *beep beep beep* noise.  Great!   The Raid array has failed, or possibly the hard drive.

Well it doesn’t really matter now, because I’m at the datacenter again preparing to fix this problem.

Thankfully, we use RAID.  RAID stands for a Redundant Array of Inexpensive Disks (some claim the I stands for independent).  In particular, this server runs RAID 1, or mirroring.  That means that it has 2 hard drives, and one mirrors the other in real time, so both drives contain the exact same data.

This is a good thing, because in the case that the hard drive is actually bad, I just call up Dell, they send me a new hard drive, and I re-image the array off the good disk.  But hopefully the case is that the array just broke it’s sync, and I just need to rebuild it.  In any case, it means I have to take our database server offline to do this process, which means thousands of databases for our shared hosting customers are going to be offline for a wee bit.

I feel really bad about this, because it means that they will have an interruption of service, but then again, they are only paying $10/mo to share a server cluster with hundreds of other customers.  This kind of stuff comes with the territory.  Unlike our major dedicated server customers who have their own database servers and redunant load balancers etc.  Then again, they are paying hundreds of dollars per month for the assurance that they will not have an outage.

This is a little off topic, but as I’m sitting in the datacenter here backing up the server before this process (in case we need to do a bare-bones re-install), a centipede like bug just ran across the floor from under the rack.  I think I need to tell the DC techs that there are bugs in the datacenter

Anyways, the main point of this little tirade is that any reputable web host will have redundancy built into their system for cases like this.  Modern hardware and computer equipment is not infallible, rather it is quite fallible and will break or fail at some point.  The key is to know what you are going to do when it happens, and what you are going to do to minimize the situation with pre-planning.

I just got a call from Jay, one of my clients, that we built a rapid deployable emercency communications system for.  It looks like there is a 90% probability of being called out to deployment within the next few days.  If we are deployed, it means I get to pack my bags and head to Wyoming.  So right now I’m scrambling to make sure everything is all set with the system so we can get out there and it works.

I’m excited about this, I had a blast last summer when we deployed the system at the Alpine Lake fire, so I’m looking forward to this now.  The only downfall is that my friend Will is getting married the 24th of June, and I wanted to go to his wedding.  It also means that I’ll be away from Melanie and my son for a while.

On my to-do list.

• Complete the startup page for the system’s intranet
• Make sure all components in the system are functional
• Do laundry, pack a suitcase
• Reconcile my bank statements and get as caught up in my billing as possible
• Quick test my remote access capabilities to the office, so I can still get work done on the road.  (Remote desktop is going to be a pain through the satellite)
• Pick up another TV series on DVD to watch

Well, back to work!