Audiobook update

Wow, the narrator makes all the difference. We picked up a different book for the drive home from Thanksgiving, “Basic Black” by Cathie Black on business. The 6 hour trip was over before we knew it. We’ve still got 3 hours left in the book, but a good audiobook makes the drive go by so much easier.

We’re planning on having a full library on our ipods before all trips from now on. And while I’ve still got 5 months left in my transition out of Webshots, I’ll be listening to a lot more books during my commute up into SF.

Malibu Fires

We spent the weekend in SoCal with family for Thanksgiving and got to see what a California Forest Fire looks like close up. The Malibu fire was about 15 miles from where we were staying, but it looked like it was right over the next ridge. Took a couple pics with the cell phone, I’ll have to get them uploaded later.

It’s a bit of a surreal experience. 1000+ firefighters actively working on it, dozens of aircraft in the air all day, last I heard 50+ homes destroyed. But from what we could see, life was just going on as normal. People were still out and about running errands and visiting friends. Just shows that different areas deal with different types of disasters.

To Partition or Archive?

I’m getting close to having to make a decision with Cloudgrove on what to do about the main content table. Cloudgrove already has 5M posts and is growing rather quickly. Currently the table indices still fit within memory on the DB server, but at some point in the near future that will no longer be the case. When that happens, performance is going to start to suffer dramatically.

The two main ways to handle this are to either partition the table with the possibility of moving some of the partitions off to other DB servers or to setup an archive system to move older content off to another system. The access pattern to this content strongly favors reading content less than 2 weeks old. So having a small primary table with the latest content and a larger archive table with the rest of the content could work. The more standard system though would be to partition the table into say 10 sub tables. If I went this route I could use something like Hibernate Shards for data access and have to change very little code.

An advantage of using partitions is that the backend system is hidden from the application. So if access patterns change in the future there would be minimal impact. With an archive system, if the older content became used it could cause problems.

I’ll have to do some experiments with Hibernate Shards to see if it’s ready for production use and will do what I need it to.

My intro to audiobooks

For Thanksgiving this year we drove down to the LA area to visit family. It’s about a 6 hour drive for us which isn’t too bad. To make it go a bit quicker though we picked up a couple audiobooks from Audible.

The first one that we listened to was Stephen Colbert’s “I am America (and so can you)”. If you’re familiar with the Colbert Report you’ll probably like this. It’s not really a book but an extended (3 hour) audio version of his show. Comedy is definitely great for making a road trip go by faster

The second book we chose was the 3rd book in the Dexter series. I love the TV show and Jaimie had read the first two books so we thought it would be a good choice for the trip. We had time to get about 1 hour into the 6 hour book on the way down. However, I’m not sure that I can recommend it yet. It was a bit of a shock to listen to my first real audiobook. The narrator speaks rather slowly and likes to give dramatic pauses every few sentences. Personally I enjoy reading rather quickly and found the slow pace to be very hard to handle. I keep wanting to hit fast forward. I’ll have to wait and see if this is just an issue with this narrator or if its an issue that I’ll have with all audiobooks.

PG&E is not my friend

I recieved a nice little notice from PG&E a couple of weeks ago that they would be doing some routine maintenance in my neighborhood and that they would need to shut the power off for several hours to complete it. The letter was a nice heads up that the power would probably be shut off between the hours of 10 & 2 on the upcoming saturday. This wasn’t a huge problem, I’ve got a rack of servers in the garage serving several properties that I’m working on, but since none of them are generating revenue yet or are still in an early beta stage, a little down time wouldn’t be the end of the world.

I made a promise that I wouldn’t sink the money for colo space until the properties were generating enough revenue to pay for it. The bandwidth to my garage is sufficient and the power up until now had been very reliable. Also this way I would be pushed to do more with constrained resources and wouldn’t be careless with expenses.

Saturday morning I woke up to the sound of the puppy running around the house and my wife cursing about not being able to get a load of laundry done. It turns out that PG&E had jumped the gun and the power went off at 8am instead of 10am. My plan of carefully shutting down all of the servers went out the window, not much you could do about it now though. So I spent the rest of the afternoon out running errands waiting for the power to come back up.

Later that afternoon the power was returned and I quickly went about the task of bringing everything back online. The first problem to creep up was that my backup server with the large disk shorted out immediately and nothing was going to bring it back. One server down.

The rest of the servers came back online without to much of a hitch. They all booted properly and seemed to be functioning normally. Upon closer inspection though I found that the mysql database had corrupted and that I would have to restore from the nightly backup (thankfully this was also copied to a server other than the backup server). I would lose 12 hours of data, but everything else would be restored. The only problem was that it was 7Gb now and would take awhile to process.

2 hours later, the database was back online and all of the sites were back up. Everything seemed to be fine. I just needed to find some new hardware for the backup server and get those cron jobs running again.

The next morning however, I started to see some issues on my gateway server that handles web, email, cvs, and dns. Disk errors all over the logs and that horrible clicking noise coming from the hard drive. The machine quickly went down and wouldn’t reboot to allow me to copy the files since the backup was also gone. One trip to Frys to pick up a new hard drive and 6 hours of going through the configs again to get all of the services running and I was back in business. It turns out that mounting a bad disk under a rescue boot will allow you to get a lot off of it. I also took this opportunity to get the backup server online with new hardware.

Everything was smooth for a couple of days, but then the DB server went down for no apparent reason. Turns out that one of the 3 fans on the power supply had failed and the server would rather shut down hard than run with a iffy power supply. 20 min to copy the DB to another server and reconfigure everything to point at the new location and everything is back up and running, if at a slightly slower speed.

I put in a call to my CDW rep (who also happens to be my brother) and learn that the part I need to replace is no longer made and while HP might be able to get it to me it’ll probably be a few months and a special order(read costly). So I turn to ebay and quickly find a reseller in OH that is willing to ship me the part today for $14, SOLD.

Last night I installed the new power supply, reconfigured the db back on the db server and hopefully ended the saga of the early power shutoff. I’m working very hard to get some cash flow coming in so that I can move these servers out of my garage and into a proper colo and not have a repeat of all this excitement. It’s always interesting to see what a bad shutdown can do to a server that’s used to running continuously.

Webshots is sold again

Well, this makes it twice now, that Webshots has been sold while I was working there. This looks like it will be the last time that I will get to go through a sale of this property since I won’t be making the move out to Cleveland to join the American Greetings Team. I want to wish them the best of luck with Webshots in the future. It’s been quite the learning experience. The two sales were wildly different and the emotions involved were different as well.
Come May of 2008, I’ll be looking for the next opportunity to continue my education.

Syriana

I watched “Syriana (Widescreen Edition)” on one of the movie channels a few nights ago and it brought up an idea for me that I might have understood at a base level, but hadn’t given too much thought. The idea is that while the world is full of possibilities and a lot of times all it takes to change it is for someone to care, there are also areas where the world is the way that it is for very specific reasons. Very powerful interests may be working very hard to maintain the status quo for reasons that you can’t see or don’t understand and in these situations it can be extremely difficult to bring about change. The trick is to understand which type of situation you’re facing.

Another movie with a similar theme , but presented in a bit more accessible way, that I also enjoyed was “Tucker – The Man and His Dream“.