In this thread, entitled The past 2 days: what went wrong, and what’s being worked on:
RevolvingDork says:
Hi all,
On Saturday night, Haim, Mike, and I headed to the datacenter to perform some heavy maintenance on the servers. We planned to install a new, larger switch, several new image servers, and a reorganization of the network architecture. This required system downtime.
Despite a well-documented plan (with charts and everything!), the work took much longer than anticipated. It ended up seeping overnight into Sunday afternoon, as almost everything that could have gone wrong did go wrong. We eventually got the site online, but there were still problems.
[Edited for brevity: RD wrote quite a long explanation - please see the original thread for the full post]
We recognize these issues as a failure on our part, and we’re deeply sorry for the disruption to the tens of thousands of people who tried to access the site during these times. Those with showcase dates during the effected days (Sunday and Monday) will be issued a make-up day next month.
Needless to say, we will be doing our best to prevent this kind of thing from happening again. We’re working now on getting fully operational and facilitating future growth. Keeping the site up and functioning quickly is our #1 priority, and we’re working day and night toward this goal.
If you are having any issues that are not described in the list above, please let us know here, and we’ll look into them as soon as possible.
Again, we are deeply sorry for the weekend’s problems, and we appreciate truly appreciate your patience.
Posted at 3:57 pm, April 23 2007 EST
April 23rd, 2007 at 4:15 pm
This communication is really impressive. They’ve come a long way, baby.
April 23rd, 2007 at 4:27 pm
“It was sorted out, and now the site is just about at the point it was before we began the maintenance.”
so this means the work they went there to do Saturday didn’t actually get done? We’re just back to where we were before Saturday? They are going to need another downtime to install the image servers they didn’t get a chance to install.
April 23rd, 2007 at 5:19 pm
Yep, it’s excellent communication. I’m really pleased RD took the time to write it out after what has no doubt been a very long and unpleasant few days.
April 24th, 2007 at 2:03 am
I agree Elizabeth - RD’s post is excellent. I hope he continues in this vein.
April 24th, 2007 at 2:04 am
I also appreciate the info itself. =)
April 24th, 2007 at 3:03 pm
yes.
yes. yes and more yes.
this was a super professional post / response to what was probably a frustrating weekend to most people involved, and i think they handled it super well.
April 25th, 2007 at 9:06 pm
Etsy: Please pay attention to the retail calendar and do not attempt a major site update at least 6 weeks prior to a major retail holiday. Mother’s Day is coming up, the site is not fuctioning for its 4th consecutive day, and you have eaten fatally into any sales that might be realized by Etsy Sellers. This mistake will do more damage to the reputation of the site to all prospective buyers for this holiday and for future holidays. Some customers will stay away forever, but you have also damaged the trust of the selling community. Some of that will never be repaired. It often takes 5 times longer to undo the bad vibe you create because you ignored the retail selling needs of the sellers. It could also have long lasting effects on the bottom line.