Audiobook update

Wow, the narrator makes all the difference. We picked up a different book for the drive home from Thanksgiving, “Basic Black” by Cathie Black on business. The 6 hour trip was over before we knew it. We’ve still got 3 hours left in the book, but a good audiobook makes the drive go by so much easier.

We’re planning on having a full library on our ipods before all trips from now on. And while I’ve still got 5 months left in my transition out of Webshots, I’ll be listening to a lot more books during my commute up into SF.

Malibu Fires

We spent the weekend in SoCal with family for Thanksgiving and got to see what a California Forest Fire looks like close up. The Malibu fire was about 15 miles from where we were staying, but it looked like it was right over the next ridge. Took a couple pics with the cell phone, I’ll have to get them uploaded later.

It’s a bit of a surreal experience. 1000+ firefighters actively working on it, dozens of aircraft in the air all day, last I heard 50+ homes destroyed. But from what we could see, life was just going on as normal. People were still out and about running errands and visiting friends. Just shows that different areas deal with different types of disasters.

To Partition or Archive?

I’m getting close to having to make a decision with Cloudgrove on what to do about the main content table. Cloudgrove already has 5M posts and is growing rather quickly. Currently the table indices still fit within memory on the DB server, but at some point in the near future that will no longer be the case. When that happens, performance is going to start to suffer dramatically.

The two main ways to handle this are to either partition the table with the possibility of moving some of the partitions off to other DB servers or to setup an archive system to move older content off to another system. The access pattern to this content strongly favors reading content less than 2 weeks old. So having a small primary table with the latest content and a larger archive table with the rest of the content could work. The more standard system though would be to partition the table into say 10 sub tables. If I went this route I could use something like Hibernate Shards for data access and have to change very little code.

An advantage of using partitions is that the backend system is hidden from the application. So if access patterns change in the future there would be minimal impact. With an archive system, if the older content became used it could cause problems.

I’ll have to do some experiments with Hibernate Shards to see if it’s ready for production use and will do what I need it to.

My intro to audiobooks

For Thanksgiving this year we drove down to the LA area to visit family. It’s about a 6 hour drive for us which isn’t too bad. To make it go a bit quicker though we picked up a couple audiobooks from Audible.

The first one that we listened to was Stephen Colbert’s “I am America (and so can you)”. If you’re familiar with the Colbert Report you’ll probably like this. It’s not really a book but an extended (3 hour) audio version of his show. Comedy is definitely great for making a road trip go by faster

The second book we chose was the 3rd book in the Dexter series. I love the TV show and Jaimie had read the first two books so we thought it would be a good choice for the trip. We had time to get about 1 hour into the 6 hour book on the way down. However, I’m not sure that I can recommend it yet. It was a bit of a shock to listen to my first real audiobook. The narrator speaks rather slowly and likes to give dramatic pauses every few sentences. Personally I enjoy reading rather quickly and found the slow pace to be very hard to handle. I keep wanting to hit fast forward. I’ll have to wait and see if this is just an issue with this narrator or if its an issue that I’ll have with all audiobooks.

PG&E is not my friend

I recieved a nice little notice from PG&E a couple of weeks ago that they would be doing some routine maintenance in my neighborhood and that they would need to shut the power off for several hours to complete it. The letter was a nice heads up that the power would probably be shut off between the hours of 10 & 2 on the upcoming saturday. This wasn’t a huge problem, I’ve got a rack of servers in the garage serving several properties that I’m working on, but since none of them are generating revenue yet or are still in an early beta stage, a little down time wouldn’t be the end of the world.

I made a promise that I wouldn’t sink the money for colo space until the properties were generating enough revenue to pay for it. The bandwidth to my garage is sufficient and the power up until now had been very reliable. Also this way I would be pushed to do more with constrained resources and wouldn’t be careless with expenses.

Saturday morning I woke up to the sound of the puppy running around the house and my wife cursing about not being able to get a load of laundry done. It turns out that PG&E had jumped the gun and the power went off at 8am instead of 10am. My plan of carefully shutting down all of the servers went out the window, not much you could do about it now though. So I spent the rest of the afternoon out running errands waiting for the power to come back up.

Later that afternoon the power was returned and I quickly went about the task of bringing everything back online. The first problem to creep up was that my backup server with the large disk shorted out immediately and nothing was going to bring it back. One server down.

The rest of the servers came back online without to much of a hitch. They all booted properly and seemed to be functioning normally. Upon closer inspection though I found that the mysql database had corrupted and that I would have to restore from the nightly backup (thankfully this was also copied to a server other than the backup server). I would lose 12 hours of data, but everything else would be restored. The only problem was that it was 7Gb now and would take awhile to process.

2 hours later, the database was back online and all of the sites were back up. Everything seemed to be fine. I just needed to find some new hardware for the backup server and get those cron jobs running again.

The next morning however, I started to see some issues on my gateway server that handles web, email, cvs, and dns. Disk errors all over the logs and that horrible clicking noise coming from the hard drive. The machine quickly went down and wouldn’t reboot to allow me to copy the files since the backup was also gone. One trip to Frys to pick up a new hard drive and 6 hours of going through the configs again to get all of the services running and I was back in business. It turns out that mounting a bad disk under a rescue boot will allow you to get a lot off of it. I also took this opportunity to get the backup server online with new hardware.

Everything was smooth for a couple of days, but then the DB server went down for no apparent reason. Turns out that one of the 3 fans on the power supply had failed and the server would rather shut down hard than run with a iffy power supply. 20 min to copy the DB to another server and reconfigure everything to point at the new location and everything is back up and running, if at a slightly slower speed.

I put in a call to my CDW rep (who also happens to be my brother) and learn that the part I need to replace is no longer made and while HP might be able to get it to me it’ll probably be a few months and a special order(read costly). So I turn to ebay and quickly find a reseller in OH that is willing to ship me the part today for $14, SOLD.

Last night I installed the new power supply, reconfigured the db back on the db server and hopefully ended the saga of the early power shutoff. I’m working very hard to get some cash flow coming in so that I can move these servers out of my garage and into a proper colo and not have a repeat of all this excitement. It’s always interesting to see what a bad shutdown can do to a server that’s used to running continuously.