We all had a list of suspects. Is the database only utilizing half its memory? Are aggressive spiders taking down the site? Is there a problem with the way Rails talks to FastCGI? But more than any other area of suspicion, we all knew there was something wrong going on with subscriptions.
So as a test, we turned off subscriptions around noon today. Guess what? The site perked up considerably. Errors dried up, the site stayed up, and “Oops” messages dropped down to a murmur (15 in the last 2 hours out of 105,000 page views).
We are going to monkey around with subscriptions a bit more, but it seems we may have found the greatest offender to site stability. If the site dies in the next few hours, it is probably us testing turning subscriptions back on. For now though, it looks like we got lucky right out of the gate.
What else is to come with Lickety Split? Fixing what ails the problems tool in 43 Places. Reworking the way we get rid of spammers and miscreants on the site. Getting subscriptions to be fast and painless. Lots of caching, some new hardware, some new site monitoring. We will keep going until the site is humming along reliably.
8 responses so far ↓
Blake // October 30, 2006 at 3:03 pm |
Still borked.
http://www.43places.com/city/wanderlust/106707
oops for me right now.
Daniel // October 30, 2006 at 3:10 pm |
Hi Blake — that’s a separate issue that Buster just fixed (we’ll be pushing it out soon). It’s only those city links on the 43T homepage. That link will break every time until the fix goes out — but thanks for pointing it out. The good news is the site is still up and in great shape post-subscriptions.
NYCinephile // October 30, 2006 at 3:13 pm |
Interesting that subscriptions might be a culprit, Josh.
I’ve reported elsewhere that the subscribed links on the New York Locals page are often slow to update.
Would you like me to delete the ones I’ve added?
gypsydog38 // October 31, 2006 at 1:34 am |
What’s happened? 43things was running at an astonishing rate this morning but now when I try to log in it’s just saying “this page cannot be displayed” I used to complain about “ooops…” but with that at least I was connected.
gypsydog38 // October 31, 2006 at 1:37 am |
Ooops I take back my last comment its working fine now!
EricFoster // October 31, 2006 at 7:29 am |
Help…
Yesterday I was on 43 things and it logged me out… and now I cant log back in…
It looks like my account is gone?
Help…
morrigirl // October 31, 2006 at 1:28 pm |
Ah, so that’s what’s been going on! I’ve been trying to post a particular album review for the last 12 hours, and every time I hit that “Save” button it’s all “Oops… yadda yadda yadda.” Good to know it’s not just me.
Daniel // October 31, 2006 at 1:49 pm |
Morrigirl — you still having trouble? I just posted to All Consuming and it worked fine. Let me know if it’s till Ooops’ing — if so can you email the album you’re reviewing to robots at robotcoop dot com? Thanks!