« y.ah.oo! | Main | continued hiccups »
December 14, 2005
power failure
Our datacenter has had full power outage, and we are now recovering from the aftermath. About half of the machines are still down and we are working on bringing them back as soon as possible. To keep things moving, I've turned off RSS and some other features so that posting will still work and so that you will still have access to your bookmarks.
5pm EST Update: Everything is back up. We apologize for the inconvenience. It looks like a number of slave servers are corrupted. We're rebuilding them now and taken the site down so this goes as quickly as possible.
Update: Some servers are coming up. Should be going much faster from here on in.
PS -- we are obviously not yet on the Yahoo! infrastructure, and this is exactly the kind of stuff they can do a lot better for us (and you.)
Update: Most stuff is up, finally. Tag intersections are currently not working, and search is not updating -- a few servers would not come up. We'll bring them up tomorrow when we return to the east coast.
Posted by joshua on December 14, 2005 at 10:31 AM in service alert | Permalink | Bookmark This
Comments
Lets say: yahoo! ;-) BTW, sending the verification email doesn't work at the moment.Posted by: New User | Dec 14, 2005 10:40:15 AM
clueless RSS user question: Is there a way for me to subscribe to a feed for "service alert"? I have a special folder in my aggregator set aside for these types of announcements (my ISP, mail provider, etc.). The main del blog feed wouldn't aptly go there.Posted by: shekay | Dec 14, 2005 11:29:52 AM
Posting works, but: - the pagecount doesn't increase. - the tagcount doesn't increase, nor does the adding of newly defined tags. - the verification email sending doesn't work. Hope you get everything back up real soon. Best of luck! B!Posted by: Bramus! | Dec 14, 2005 1:44:37 PM
Please, let us know what datacenter it is so we never host anything important there.Posted by: teo | Dec 14, 2005 2:27:37 PM
I always hated yahoo :) Why not google? ;) With google you'l never have power failure.Posted by: Yasha | Dec 14, 2005 2:45:48 PM
they're probably not in the yahoo server system a few days after the announcement ;)Posted by: Judson | Dec 14, 2005 2:51:03 PM
Is this why I can't rename tags?Posted by: Brock Tice | Dec 14, 2005 3:41:02 PM
Why are there mysql errors all the time?Posted by: Ivan | Dec 14, 2005 4:10:34 PM
Seen anyone in a Google t-shirt with a pair of wirecutters recently?Posted by: Steve | Dec 14, 2005 4:18:31 PM
Don't make the mistake of thinking that just because Google is big they do not have power difficulties: http://hardware.silicon.com/servers/0,39024647,39154982,00.htm Again, the problem is scale. You can solve the hardware scaling problem, but you're still dependent on California's power grid.Posted by: Wyclif | Dec 14, 2005 5:03:51 PM
...I'm starting to realize that del.icio.us is really important to me.Posted by: ZEO | Dec 14, 2005 5:33:19 PM
glad to hear all the servers are back up. however, still getting lots of mySQL errors, and the tagging problems others have listed above are not resolved. any outlook on when things will be functioning normally?Posted by: hillary | Dec 14, 2005 5:38:26 PM
These things happen. but could you do a little bit better than "del.icio.us is down for maintenance. we'll be back in one hour." Without a timestamp, that's meaningless.Posted by: Judi Sohn | Dec 14, 2005 10:30:53 PM
Ack, I want my del.icio.us! Anyway, I'm sorry to hear about your woes and wish you the best of luck in getting everything back up and running. I know how awful this stuff can be sometimes. As one tech guy to another, JaredPosted by: Jared White | Dec 14, 2005 10:48:42 PM
A link to the blog post about the downtime would be handy on the downpage, the next time there's an outage. Don't spend any time on it this time, though - outages suck :(Posted by: Richard Soderberg | Dec 14, 2005 10:57:03 PM
del.icio.us: Where are you? I miss you.Posted by: Michael Kebbekus | Dec 15, 2005 2:00:09 AM
Is there an updated ETA on this? It's been a long "one hour."Posted by: nowak | Dec 15, 2005 3:05:45 AM
Yahoo is copying all the links to MyWeb2.0 site! In the mean time, you are fooled to believe it's a power failure! Good that i made a offline backup yesterday!Posted by: Jackie | Dec 15, 2005 3:10:24 AM
I wanted to know what's going on. I think that it is necessary to separate to read BLOG at any time.Posted by: m.mashima | Dec 15, 2005 4:50:16 AM
So long and thanks for all the fish!Posted by: Jerome | Dec 15, 2005 5:16:55 AM
(LOL at Steve's comment) Hey guys, besides what they said about linking the 'we got trouble' page to this blog/status page, will ya remember to put the time of the update when you edit the post to say 'Update:' ? Just a simple ("7pm EST Update: keep your pants on") type thing. Thanks.Posted by: Tollie | Dec 15, 2005 5:20:08 AM
No co-lo?!Posted by: Lloyd D Budd | Dec 15, 2005 6:02:07 AM
del.icio.us seems to be recovered completely for me. Thanks your best efforts!Posted by: yukkie(JA) | Dec 15, 2005 6:24:51 AM
Congratulation to del.icio.us! It seems to up again here too! Btw, is there a good way to manage an offline backup from del.icio.us? (Please send tag for:toydi to me, if you have a good solution, thanks in advance!)Posted by: toydi | Dec 15, 2005 6:32:52 AM
I was really scared back there... no del.icio.us? I shudder at that thought! :DPosted by: Kevin | Dec 15, 2005 8:48:41 AM
I picked the wrong day to see what this whole del.icio.us thing was about. Still haven't received my email to set up an account.Posted by: 706GL | Dec 15, 2005 9:06:52 AM
I thought it was being transferred to Yahoo and the transition went bad. We have had one of our data centers down because of power failure recently. Energy is actually becoming a bigger problem for web hosting, just as Google warned the other day.Posted by: Son Nguyen | Dec 15, 2005 11:31:39 AM
My tag counts are wrong.Posted by: Dave | Dec 15, 2005 2:09:36 PM
My tag counts are wrong too and new tags don't show up in the list.Posted by: Linda | Dec 15, 2005 3:05:10 PM
Intersection pages are getting cleared for me every time I add a tag to a link that is included in that intersection. Is there an ETA on when those issues are to be resolved?Posted by: Nick | Dec 15, 2005 5:42:49 PM
You know, people, it's nice outside: you could leave the internet alone for maybe a couple hours, and then when you came back to it, if it wasn't all better you might be at least have calmed down and got some perspective.Posted by: RichS | Dec 17, 2005 8:49:10 PM
I can't loggin to my account! (SyncMaster)... please help me!Posted by: Francisco | Dec 18, 2005 6:19:40 AM
on my page: "All your items (11) ...[list_of_my_tags]... No items". ??Posted by: george kronberg | Dec 18, 2005 6:29:59 AM
I can loggin now. Thanks for fixed the problem.Posted by: Francisco | Dec 18, 2005 8:19:23 AM
down againPosted by: Jean-François | Dec 18, 2005 5:02:19 PM
I'm getting down for maintenance on Sunday 12-18. What's the story?Posted by: Will | Dec 18, 2005 5:16:07 PM
I'm getting down for maintenance too. Good luck getting everything back up and running asap. -Jon http://philoneist.comPosted by: Jonathan | Dec 18, 2005 6:21:25 PM
Any info would be great. Did godzilla attack the data center?Posted by: keith | Dec 18, 2005 6:25:16 PM
Still down for me too (Sunday, 12/18/05 6:32 PM PST). I use an RSS feed to show my podcast's archived shows (so folks can surf for the audiobooks easier). I noticed during the last outage you guys went back up but didn't enable RSS? Is that goign to happen this time too? I'm gearing to send some emails out about my podcast but will hold off if the RSS feeds won't be showing.Posted by: eileen | Dec 18, 2005 9:28:01 PM
Perhaps it was King Kong who attacked the data center... Seriously, guys... ANY info about the cause of this 'emergency maintenance' and outtage would be great. ~RPosted by: Robert | Dec 18, 2005 10:54:22 PM
So bad. Soooooo bad. How much was this company bought for?Posted by: Paul | Dec 18, 2005 11:43:08 PM
Looking for a Bookmark Manager to store your private Bookmarks? No social bookmarks, no tags, no endless clicking and searching. Just your own bookmarks - accessible all over the World, in any internet browser. Try Web Favorites. http://www.webfavs.de/?lang=en Greetings.Posted by: Gerd Müller | Dec 27, 2005 6:47:07 AM
Congratulation to del.icio.us! It seems to up again here too! Btw, is there a good way to manage an offline backup from del.icio.us? (Please send tag for:toydi to me, if you have a good solution, thanks in advance!)Posted by: Nina | Jan 23, 2006 5:31:38 PM
Thanks for fixed the problem.Posted by: alaa | Mar 29, 2006 9:02:09 PM
