Don’t forget to change your hdfs / mapred config when a drive fails with hadoop

Just ran into a little gotcha when running a huge job against my CDH4 cluster. One of the servers lost a drive at the 50% mark. Each server has 4 1TB drives mounted, so losing one isn’t a huge deal. With the new config “dfs.datanode.failed.volumes.tolerated” set to 2 it was possible for the datanode to keep right on going and not impact the larger job.

To get ready to replace the drive later, I unmounted the drive, leaving only the mount point dir. Then I made the mistake of bouncing the datanode so that I could start collecting ganglia stats, which are great by the way and really easy to set up.

Now the datanode determined that the mount point was back, nevermind that it was on the root device. So a day later, when the root device filled up and the tasks on that server started failing, I realized what I had done wrong.

If you’re going to temporarily take a drive out. Take it out of the config as well or else you’re going to forget about it and get yourself into trouble.

Leap Second Issues

There was so much hype with Y2K, but it turns out that it’s a leap second that takes out portions of the web. I had my Amazon EC2 instances taken out with this bug. This little code snippet brought the java cpu load back to normal.

/etc/init.d/ntpd stop; date; date `date +”%m%d%H%M%C%y.%S”`; date

Then you just need to restart ntpd. Some have reported having to wait awhile to restart ntpd so that the issue doesn’t happen again.

iPad Kindle Update

This is just a quick post for any of the Amazon iPad Kindle Reader developers out there that might be listening. As someone that has purchased and read hundreds of kindle books, the latest update is a step backwards.

Instapaper has better reading options and that’s a done by a single developer. The margins change is really bugging me. I realize that you just cant pick a single setting that is going to make 100% of the people happy, so there should just be a slider for margins. Then everyone can choose what they’re comfortable with. A slider for brightness and a slider for font size. Then you could let people choose their favorite font and you’d have the perfect reader.

I know that some of these changes are difficult. How would you calculate page numbers with infinite variations in the text? But making difficult things look easy is why Amazon is great.

My Representatives

With MyElectedRep, I get a view into the people that are representing me. Whether I agree with the votes that they’re making on my behalf or if I would like them to vote another way.

To help me make decisions on votes, I can look at how some trusted organizations recommend I vote and read the analysis that they’ve provided.

With each Representative, I can vote on upcoming legislation, so that they can determine how the district feels. I can also go through their past votes and either agree or disagree with the votes that they’ve made. If I feel strongly about certain votes, I can contact my Representatives directly to let them know how I feel.

Once I’ve gone through the different bills and voted on the ones that I wanted to, I can look at how each Representative scores. They get a district score as well as a personal score. I can then use this score to determine if I should vote for this person to be my Representative again next time or if someone else would do a better job.

My Representatives are: