PG&E is not my friend

I recieved a nice little notice from PG&E a couple of weeks ago that they would be doing some routine maintenance in my neighborhood and that they would need to shut the power off for several hours to complete it. The letter was a nice heads up that the power would probably be shut off between the hours of 10 & 2 on the upcoming saturday. This wasn’t a huge problem, I’ve got a rack of servers in the garage serving several properties that I’m working on, but since none of them are generating revenue yet or are still in an early beta stage, a little down time wouldn’t be the end of the world.
I made a promise that I wouldn’t sink the money for colo space until the properties were generating enough revenue to pay for it. The bandwidth to my garage is sufficient and the power up until now had been very reliable. Also this way I would be pushed to do more with constrained resources and wouldn’t be careless with expenses.
Saturday morning I woke up to the sound of the puppy running around the house and my wife cursing about not being able to get a load of laundry done. It turns out that PG&E had jumped the gun and the power went off at 8am instead of 10am. My plan of carefully shutting down all of the servers went out the window, not much you could do about it now though. So I spent the rest of the afternoon out running errands waiting for the power to come back up.
Later that afternoon the power was returned and I quickly went about the task of bringing everything back online. The first problem to creep up was that my backup server with the large disk shorted out immediately and nothing was going to bring it back. One server down.
The rest of the servers came back online without to much of a hitch. They all booted properly and seemed to be functioning normally. Upon closer inspection though I found that the mysql database had corrupted and that I would have to restore from the nightly backup (thankfully this was also copied to a server other than the backup server). I would lose 12 hours of data, but everything else would be restored. The only problem was that it was 7Gb now and would take awhile to process.
2 hours later, the database was back online and all of the sites were back up. Everything seemed to be fine. I just needed to find some new hardware for the backup server and get those cron jobs running again.
The next morning however, I started to see some issues on my gateway server that handles web, email, cvs, and dns. Disk errors all over the logs and that horrible clicking noise coming from the hard drive. The machine quickly went down and wouldn’t reboot to allow me to copy the files since the backup was also gone. One trip to Frys to pick up a new hard drive and 6 hours of going through the configs again to get all of the services running and I was back in business. It turns out that mounting a bad disk under a rescue boot will allow you to get a lot off of it. I also took this opportunity to get the backup server online with new hardware.
Everything was smooth for a couple of days, but then the DB server went down for no apparent reason. Turns out that one of the 3 fans on the power supply had failed and the server would rather shut down hard than run with a iffy power supply. 20 min to copy the DB to another server and reconfigure everything to point at the new location and everything is back up and running, if at a slightly slower speed.
I put in a call to my CDW rep (who also happens to be my brother) and learn that the part I need to replace is no longer made and while HP might be able to get it to me it’ll probably be a few months and a special order(read costly). So I turn to ebay and quickly find a reseller in OH that is willing to ship me the part today for $14, SOLD.
Last night I installed the new power supply, reconfigured the db back on the db server and hopefully ended the saga of the early power shutoff. I’m working very hard to get some cash flow coming in so that I can move these servers out of my garage and into a proper colo and not have a repeat of all this excitement. It’s always interesting to see what a bad shutdown can do to a server that’s used to running continuously.

Webshots is sold again

Well, this makes it twice now, that Webshots has been sold while I was working there. This looks like it will be the last time that I will get to go through a sale of this property since I won’t be making the move out to Cleveland to join the American Greetings Team. I want to wish them the best of luck with Webshots in the future. It’s been quite the learning experience. The two sales were wildly different and the emotions involved were different as well.
Come May of 2008, I’ll be looking for the next opportunity to continue my education.

Why the plug-in hybrid is better

There seems to be some reluctance on the part of the automakers to push plug-in hybrids. There has been some research out there showing that by using plug-ins there really isn’t a energy benefit. All that you are doing is shifting the source of the energy from one place to another. While this may not seem to be a huge benefit, in actuality it really is.
With an oil based economy there are a few sources of oil and very defined paths for how energy gets from these places to the consumer. However, by using the electric grid it very much becomes a free for all. The grid acts as an open market / network allowing energy to come from many different sources. This allows for innovation in energy technology to be plugged into the grid as it becomes viable. Also different techniques can be used in different areas. There is no longer a need to come up with a solution for everyone.
Local generation of electricity will take off in the near future. 50% of energy is lost in transmission, so by generating it locally already there is a huge savings. As micro-turbines and solar begin to reach similar prices to coal and natural gas you will see communities begin to install these.
By allowing vehicles to take advantage of the incremental upgrades in energy technology you are creating a more flexible vehicle and economy. There is no longer a need for major infrastructure changes. That same money can be used for infrastructure enhancements that have wider unforeseeable benefits.

Flickr posts some numbers

Over at flickr, they had a little downtime and decided to post some numbers about how impressive they are while they’re down.

  • 12,000 photos a second at peak times
  • 2,070,075 photos in 24 hours!
  • 8.5 million registered members
  • 10 million unique tags

Some pretty impressive numbers, but nothing too crazy compared to the several other large photo sharing sites. Was discussing what sets apart photo sharing sites with a few people the other day and being able to communicate with photos is definitely a draw. Flickr stumbled across an easier way to accomplish this with tags very early on. Other sites have experimented with ideas, but it’s a difficult question on how to get people to interact with photos without trying to replicate all of the features of a pure social networking site. To go down the social networking path dilutes your strength as a photo site, but tags allows for the interaction and expression while keeping the focus on the photo.
Also just goes to show that even large sites have downtime sometimes. As much money as you spend on hardware there are always going to be single points of failure that are not hot-swappable. When you’re running a small startup, focus on the product first, scalability second, fault tolerance third. When you’re building things up from nothing you just can’t devote that much energy to making things bulletproof. There just isn’t the payoff in the end. If the products not there it doesn’t matter that you have 5 9s of uptime.

Suggestion for Google Video

I love using Google Video for their Google Eng Edu videos. This series of ~1hour long video of talks given on all sorts of techie topics is great. I love being able to get an inside access to these presentations. It’s just like being at school again and having the weekly presentation from the visiting professor.
The only issue with the format is that while it’s great to have the long duration video, instead of the normal 5-15 minute videos being pushed by other sites, it’s a pain to deal with the files. The optimum way for me to watch the videos is to batch them up and then watch them when I have some free time, say on a flight across the country. For this to work though you need offline access and while Google allows you to download these files, it just doesn’t work well. You essentially have to pretend that you’re watching the video on pause and let the player download continue in the background. Sometimes this works, sometimes it doesn’t and the worst part is that you can only do one video at a time.
The system is almost there, if Google would allow you to download multiple videos in the background so that you could watch them at your convenience it would be optimal. Of course I’m sure there are revenue issues, whole lot of bandwidth being used with no ads being served. However, I would sit through a 30 sec ad at the beginning in order to get this feature.

Getting it wrong on Net Neutrality

There was an interesting piece by Cody Willard from the thestreet.com posted the other day about the cracks forming in Google. I feel the article managed to get Google’s stance on Net Neutrality completely wrong. I do agree though that there are some cracks forming in Google’s reputation. The general public is starting to realize how much data that Google collects. As long as you trust them then that’s ok, but what if they lose that trust, it’s hard to take the data back at that point. Google has shown that they’re not infallible with a couple of devastating security holes in the last couple weeks. The holes were quickly fixed, but these little chips could be just the start. The fact that even the mighty Google can screw up sometimes is going to bring people to reexamine the trust they put into that company.
The final point of the article from thestreet.com though claimed that Google was evil for supporting Net Neutrality. I found it rather difficult to fathom how someone could make this claim. You can either claim that the telecoms are evil for trying to subvert the networks, where evil here is used to mean that they are substituting the publics long term gain for their own short term gain. However, to claim that Google is evil for trying to stop the telecoms makes the assumption that the telecoms are doing good. I have yet to see anyone argue that a tiered internet is good for anyone other than the telecoms.
My reasoning on net neutrality:
Well to start at the beginning, the core of the problem is that the internet was designed to be a robust way to move data from point A to point B, but never made any guarantees about when it would arrive, the End-to-End principle. All sorts of fail-safes and checks were put in to make sure that if a router went down that traffic would automatically find other routes to get to its destination. This allowed the internet to grow in the face of different companies creating the backbone with different hardware and also various hardware failures and back-hoes causing interruptions. For most people everything worked great, sometimes pages would load a bit slowly or your email would take 30s instead of 5s but it would all get there.
The problem became noticeable when technology started to come along that required that the data get to the destination within a certain time frame, ie. voice and video. When you have enough excess bandwidth capacity it’s not a problem, but when network links get up around 80% full, you start to see a lot of collisions where packets run over each other and need to be resent. This is horrible for these time sensitive applications because it creates a delay.
The next issue that contributed to this problem is that with network links sold to consumers the telecoms modeled usage off of past behavior where consumers were checking a little email and browsing a few web pages. With these models in mind, they’re able to oversell their networks and maximize their profits. If you’ve ever used a cable modem in an area that’s been oversold in the evening after work it can be incredibly painful. Also, if you read the fine print in most of these contracts, just because you bought a 6Mb/s connection doesn’t mean that you can use it and if you do use it you’re going to get kicked off of the network as an abuser. With the rise of Vonage, Youtube, and other wildly popular sites the average usage of the consumer was skyrocketing and people were beginning to notice that they weren’t getting what they were sold.
So at this point there are two solutions for the telecoms, increase capacity and give the consumer what you told them they were supposed to get or tweak the network so that for certain types of traffic you could make the guarantee that it would get there on time. One is expensive while the other gives you a way to charge extra. It’s completely obvious why the telecoms chose the way they did as a public company. The problem is what are the implications for users of the internet as a whole?
If the telecoms have a special value added network where they are making all of their profit what is their motivation to make any expenditures to improve the standard network? And why with their past history of hyping improvements and then not following through would we believe them if they told us they would maintain the old network? Wouldn’t this just lead to a balkanization of the internet? Would the democratizing force of the internet be destroyed as the old internet rusts away and content is controlled on the new?
I would argue that the answer to all of those questions is yes. The internet has become a critical piece of infrastructure that needs to be kept open for the long term good of businesses and consumers and we have seen that amazing things can come out of an open internet. Things that no one at this point in time can even imagine.
“Google says it’s acting in the best interest of consumers and end users. Why the use of force then? A truly “non-evil” company would have no interest in using governmental force to stop attempts at innovation.”
Here you are claiming that Google is evil for lobbying, yet Verizon is just innovating, even though Verizon’s lobbying effort is much more extensive and Verizon’s innovation is really just attempts to sidestep building out their network.
“Google’s evil here stems from the fact that it knows it has won this version of the Internet and wants the government to make sure it stays on top.”
This just doesn’t even make sense. Google and Verizon aren’t in the same business and don’t compete. Google is a customer of the telecoms. They pay huge monthly bandwidth bills to the telecoms for the right to use their network. Google is not getting anything for free. If Verizon doesn’t feel that they’re being fairly compensated for services that they’re supplying then why don’t they raise their rates? The issue is again that Bandwidth is Bandwidth and there is no value add there.
I would argue that Verizon may be jealous of Google’s ability to charge for what it provides, but jealousy doesn’t mean that you’re going to get your way. The reality that Verizon has to deal with, is that no one cares which network they use as long as they can get to the end points that they care about, hence Google has pricing power while Verizon does not.

Reducing Spam

Sunday night I got a nice little heads up while cruising through my Digg feed. Apparently in Mac Mail there is a nice little button that you can put in the toolbar, using customize, that allows you to send back user unknown bounce messages to people that send you spam. Mail already does a decent job of determining what is spam and putting it in it’s own little spam folder. Now I can just go into that folder, select all, and then hit the bounce button.
A decent percentage of spam is completely forged so you’ll get back more bounce messages that the people you’re bouncing to don’t exist either. So one simple rule later, to automatically dump the double bounces into the trash and we’re off. Up to this point I was getting 50-100 spam messages a day. So it’ll be interesting to see if this makes a noticeable impact on that number, but it does feel good to be able to send them back.

No HD Ads?

Since my wife and I picked up a HD tv at the beginning of the year and started watching most of our primetime shows in HD I’ve found it interesting that the uptake in ads in HD is incredibly low. HD, when done right, has an incredible wow factor and since just getting someones attention is one of the primary hurdles of advertising why are they passing it up? With all of the stories of people watching sporting events that they didn’t even care about because the picture was so amazing it seems odd that advertisers aren’t jumping all over it.

It's all about Execution

There have been a whole load of posts around the internet from all sorts of respected people within the startup ecosystem, but it just can’t be said enough.
IT’S ALL ABOUT EXECUTION
Anyone can have an idea in their shower. They could envision the coolest thing ever, but if they don’t turn it into a product then it never happened. Ideas have zero value if you don’t have the execution to make them happen.
There are all sorts of examples of ideas that many people had, YouTube wasn’t the first to put video on the internet, but it was how they went going about doing it that made them successful.
If you’re in a small company, start each day going over what customer pain point it is that you’re trying to solve. Then ask what it is that you’re doing to make your answer to that pain point, ie. your product, better and more effective. If you don’t stay focused on that, you’re going to find yourself working on things with nothing to do with anything and out of money before too long.

Writing about startups

While I’ve titled this blog “Redwood City Startup” which very accurately describes my focus, I’ve actually found it rather difficult to write a whole lot about about that topic. My time and focus is definitely invested in running a startup in Redwood City, but with not being a founder of the startup, just lead engineer, I have to be very careful about what I say. It’s actually quite limiting.
I’m hoping to someday be able to go over my experiences, I’ve even started writing privately with the hope of one day releasing these posts to the public. In the meantime I’ll just have to keep things abstract.