inicio mail me! sindicaci;ón

Archive for Down Time

Etsy migration to new data center may cause access issues

According to this Etsy thread, users of some ISP’s have been reporting connectivity issues to Etsy which are caused by Etsy moving their servers to a new data center. (These are the physical location of the server machines)
alternate link to Etsy thread if you cannot access Etsy

Rokali says:
Hi everyone,

As people here have seen, this is a DNS issue. Anytime any changes to DNS are made, those changes have to propagate to servers around the world. This can take a bit of time.

We’re in the process of migrating from one data center to another (larger) data center. As part of this migration, we moved our DNS servers. This means they got a new IP address. (There are always at least two DNS servers, for redundancy, so it’s multiple IP addresses, and we set Primary, Secondary, Tertiary etc. status for each of them).

It’s these new IP addresses that needs to propagate. Being well aware of this, we kept our old DNS servers up & running, so as the new IP addresses propagated, even if people still hit the old IP addresses, it would work.

This did work, but we recently discovered an edge case, which causes problems. This edge case is a good example of how intertwined the Web is.

In order for Etsy not to be loading for you, the following three ingredients need to be in the recipe of how you connect to the site:

1. Your internet service provider (ISP) has been slow to update its own DNS servers (this happens daily for almost all ISPs), and hasn’t recognized the new IP addresses of our DNS servers

2. You’re hitting the secondary DNS server at our old data center

3. You’re getting a specific type of error message from this DNS server, due to how your request has been routed (what “hops” it’s taking)

*Then* you have problems accessing Etsy.

We have made special changes to how the secondary DNS server handles these requests, and we expect this to resolve the issue. It might still take a bit of time, because the fix needs to propagate out to the rest of the world.

In an ideal world, our changes would propagate out to all the DNS servers in the amount of time they’re supposed to (1-2 days). Alas, some ISPs take their time, and for that ~1% it can take more like 5 days.

As some people here have noted, you can manually change what DNS servers you use on your computer, but in general this isn’t something we’d recommend. It can slow down accessing other sites on the Web.

Of course please keep letting us know here if any of the issues persist.

Footnote: What is an IP address? It’s a numerical locator, that gets mapped to a domain name. For example, www.etsy.com is also 72.37.157.20 — you can copy that string into your Web browser and use it to visit this site. (Your browser might try and force you to use the domain name though.)

Back when the Web first started, there weren’t domain names, there were only IP addresses. But these were really hard to remember, hence the need for domain names.

A DNS server is what connects the domain name (www.etsy.com) to the IP address (72.37.157.20), approximately speaking. There are master lists on DNS servers around the world, and anytime a domain name points to a different IP, these servers need to update what domain points to what IP address.

Disclaimer: This explanation is oversimplified, and I’m not a system administrator. This is what I’ve learned from reading up on the issue, and getting some input from Etsy’s own sys admins, who have been amazingly busy keep the site up & fast. They’re doing a great job.

If you *really* want to learn more, there’s always Wikipedia:
http://en.wikipedia.org/wiki/Domain_Name_System
http://en.wikipedia.org/wiki/IP_address
Posted at 11:19 pm, June 28 2008 EST -

earlier in the same thread, Revolving Dork said:

RevolvingDork says:
Hi all,

We’ve gotten some scattered reports of users being unable to access Etsy, generally receiving “cannot connect” errors from their browsers.

We’ve run all of our diagnostic tools, and everything appears to be running properly within our network. Our global monitoring tools also have not shown any activity out of the ordinary.

We have been in touch with various ISPs, and we believe we have found the root of the issue. With their cooperation, the issue should be resolved quickly.

We’ll share more information with you as we receive it. In the meantime, if you or anyone you’re in contact with is having Etsy connectivity problems, please post their location here and the nature of their problems.

Thank you!
Posted at 5:24 pm, June 28 2008 EST -

There was an earlier thread that was locked, in which RD said the problem was not on Etsy’s end, and the failure was most likely due to some ISP.

RevolvingDork says:
Hi all,

We’ve gotten some scattered reports of users being unable to access Etsy, generally receiving “cannot connect” errors from their browsers.

We’ve run all of our diagnostic tools, and everything appears to be running properly within our network. Our global monitoring tools also have not shown any activity out of the ordinary.

The evidence we have so far is pointing towards a failure in one or more ISPs ( internet service providers ) around the world, causing routing problems and cutting some users off from Etsy’s servers. This issue is unfortunately out of our hands, as we only have control over our internal network.

We are in contact with our ISP to see if they have any more information about potential issues. They are also reporting no problems currently, but we’re working with them to figure out the root cause of the issue.

We’ll share more information with you as we receive it. In the meantime, if you or anyone you’re in contact with is having Etsy connectivity problems, please post their location here and the nature of their problems.

In Case of Etsy Emergency: Off-site Blog for Updates

As reported in this Storque article, Etsy will now be using their old blog for emergency updates, if the main Etsy site goes down. The Storque is hosted on the same servers at the ETsy site so when the site is down, so is the Storque. The old Etsy blog is hosted on a different server, so it will still be up and can be used for emergency updates.

Etsy says:

Since we’ll only use this blog in the event that Etsy itself is totally unreachable for a significant amount of time (which means, I hope we never use it), you might want to make a note of the url now and bookmark the link. Typing the address in directly will be the only way to get there.

We have decided to use the off-site blog in lieu of emailing everyone for several reasons: it takes over 24 hours to send all Etsians an email; when our servers are down or unreachable we most likely can’t send any emails; and we’d be emailing thousands of people who don’t want to get an email.

UPDATE May 22, 2008
The site for updates in an Etsy emergency has been changed to fix.etsy.com.
Bookmark this siite and look there for updates when Etsy.com and the Storque blog are down.

Etsy is down, 1:30 AM, EDT

Etsy is currently down, and some tips report it has been down a few hours (at least). I cannot confirm the exact duration of the downtime. If anyone was using Etsy at the exact time it went down, please post your info.

Edit: The site has been down since at least 12:15am EDT.

Edit: Site is back up approximately 2:30am EST.

A little piece of info from admin:

MikeH says:
The site was unfortunately down for a couple of hours due to a network issue. There will be a Storque article with more details coming up as soon as the guys get back from the datacenter.

More info about the downtime from Admin

haim says:
Hi everyone,

Around 11:45PM EST our internet connecting fiber lines were accidentally cut. I’ll have a Storque article up later today with complete details including the stuff we have planned to make sure stuff like this doesn’t take us down in the future.

Etsy goes splat

Etsy is experiencing unexpected downtime as of approximately 2:45pm EST. It appears all parts of the site are affected, including the Storque.

Update: As of 3:40pm EST, Etsy is back online.

Update: 3:58pm EST, Etsy is offline again.

Update: Back up and here’s the explanation!

Update by JB: More explanation of the downtime in this Storque article

UPDATE :: The cause of the issue appears to have stemmed from a mechanical failure on one of our firewalls. Our backup firewall kicked in, but was not fully configured for the sudden shift in traffic. Haim and Dusten are now getting both firewall machines back up to speed to get us back to 100%.

Looks like Etsy is down 1:40 am EDT

Etsy is down, for about the last 5 minutes.
started around 1:38 am, (eastern daylight savings time zone)
I don’t have any other info yet, but this is for all the people wondering “is it just my computer”?
No, it’s not!

update 1:45 am, it’s back up!

Site Outage and Reset Item Views

In this thread, Etsy admin RevolvingDork explains what happened:

Today at roughly 2:15pm EST, there was a failure on one of our load balancing machines that rendered the site inaccessible. We recognized the failure immediately, but were unable to restore full connectivity until 3:30pm EST.

Unfortunately, tracking down the issue required all of our servers to be reset, which in turn erased all view tally data from listings. It is unlikely that we will be able to retrieve this data to bring them back, but we will try.

We’re now investigating the causes of the issue, and tightening everything back up. Those that had a showcase scheduled for today will be refunded.

We apologize for any inconvenience this outage has caused, and we’re doing our best to ensure that it won’t happen again.

Posted at 3:40 pm, February 23 2008 EST

Edit:
RD has stated that all showcases for today will be refunded.

Edit (Feb. 27/08):
RD has started a new thread about the site outage and view reset.

Christmas Morning Downtime

In this forum thread, RevolvingDork writes:

We’ll be taking the site down very early on Christmas day to perform an upgrade of the Master Database. This upgrade will help us facilitate future growth and make it easier for us to develop new features.

The estimated downtime will be:

Dec 25th
1:00am - 6:00am

Not living in NY, NY? Here’s a handy time zone converting tool: World Time Server. Just choose your location out of the second box.