October 31, 2006
We had a couple quick victories (we think) the last 2 days. We think we sorted a problem in subscriptions that was taking down the site about 2-3 times per hour. Nasty. We also tweaked some database configurations that seems to have sped up the site a noticeable amount. Yea for 43 Things!
Today we turned people’s feeds and subscriptions all back on, finished our work on images (which should all be loading now), and made a bit of headway on the 43Places problems tool. Hopefully we’ll add a new webserver this week as well. So much progress on Lickety Split.
Please go ahead and let us know about any problems you are seeing on the site. The more specific you can be the better. And of course, we love all the compliments and kind words. It is not as much fun to work on the backend of the site, but it makes it more gratifying when we hear how happy it makes you!
36 Comments | 
Announcements | 
Permalink
Posted by joshp100
October 30, 2006
It seems I’ve fixed all the image problems for 43 Things, and 43 Places and 43 People will be fixed up tomorrow. Give us a heads-up if you see any broken pages or broken image links on 43 Things.
UPDATE: 43 Places and 43 People now seem to be up to date. If you see anything amiss, please leave a URL for the broken image and the page you found it on. Some may show up after a reload or two as our image servers may be busy rebuilding images for you.
15 Comments | Site Status | Permalink
Posted by drbrain
October 30, 2006
We all had a list of suspects. Is the database only utilizing half its memory? Are aggressive spiders taking down the site? Is there a problem with the way Rails talks to FastCGI? But more than any other area of suspicion, we all knew there was something wrong going on with subscriptions.
So as a test, we turned off subscriptions around noon today. Guess what? The site perked up considerably. Errors dried up, the site stayed up, and “Oops” messages dropped down to a murmur (15 in the last 2 hours out of 105,000 page views).
We are going to monkey around with subscriptions a bit more, but it seems we may have found the greatest offender to site stability. If the site dies in the next few hours, it is probably us testing turning subscriptions back on. For now though, it looks like we got lucky right out of the gate.
What else is to come with Lickety Split? Fixing what ails the problems tool in 43 Places. Reworking the way we get rid of spammers and miscreants on the site. Getting subscriptions to be fast and painless. Lots of caching, some new hardware, some new site monitoring. We will keep going until the site is humming along reliably.
8 Comments | Announcements | Permalink
Posted by joshp100
October 30, 2006
About two months ago we started work on cupcake, our latest release of 43 Things. But in the middle of that release, about three weeks ago, it became clear we had a problem. The site was slowing down. Too many people were getting the dreaded “Oops!” page.
For the last few weeks, we’ve put aside any new development projects to focus on the site’s performance. We added new hardware. We’ve found some new staff to help with system administration. We’ve fixed lots of bugs. But unfortunately, we have not yet turned the corner. The site is still too slow and too many people are getting that “Oops!” message. So we decided to make it official, we are creating a new release called “Lickety Split” focused around ![]()
nothing other than making 43 Things fast and reliable. The whole co-op is now working on “Lickety Split” and we hope to soon have all the sites living up that name.
We’ll keep you appraised of the progress we are making. This week we’ll be adding a new server to help with the load of traffic we are getting. But we are also reworking old parts of the site to make them faster and more reliable. In the process, we may need to change a few things – but we think the result will be worth it.
We’d appreciate it if you keep letting us know about your experience with the site. It is always nice to hear some of the good along with the bad. Thanks for your support!
23 Comments | Announcements | Permalink
Posted by joshp100
October 19, 2006
The new drives are installed and image upload is now working on all the sites. There’s now plenty of room for all your photos so bring it on. Let us know if you encounter any errors, but we’ve been uploading photos successfully all morning.
P.S. I am experiencing an upload failure only in Safari about 50% of the time—so we’ll work on that one but I suspect most people are using Explorer and Firefox.
14 Comments | Site Status | Permalink
Posted by Daniel Spils
October 18, 2006
A couple days ago we began running out of disk space for storing all the photos you’ve been uploading to the sites. This has resulted in plenty of “Ooops” pages and folks not being able to upload pictures to the sites. We’re in the process of getting new, large drives installed on our servers to handle the volume of photos that we’re all uploading. We expect the drives to be online within 24 hours. We’ll keep you updated on the progress here.
I’ve personally got a bunch of pictures from my recent Massachusetts trip that I’m dying to upload to 43 Places—hold tight and we’ll have plenty of image space for all our photos.
19 Comments | Site Status | Permalink
Posted by Daniel Spils
October 12, 2006
Josh won today. Additionally we had a secret lunch guest named Russell Dicker. It was great to see the Dickertator (as we affectionately call him) and chaw the chew.
I’m looking forward to Olympic Sculpture Park opening in 2007.
P.S. we didn’t talk about mulch at unch
No Comments » | Lunch | Permalink
Posted by Daniel Spils
October 11, 2006
The briefest lunch today resulted in drbrain winning1 at CCR2 (he wins the least) and him giving me the gift of blog. Here goes.
There was talk of stale object errors, one of a few complicated issues which have been impacting the sites of late. In our journey to make it up and make it happen we have arrived at an inflection point in our travels. It’s time for us to temporarily lay down our bindle stick3 and pause by the tracks to contemplate stale object errors, memory leaks, bugs and other puzzles of our own creation with the hope of improving site stability and performance. We know it’s been frustrating at times but we’re happy to have you along with us on our collective journey. We’ve made some great progress the last two days – thanks for all your help reporting issues to us. We couldn’t do it without you, our fellow 43T hobos, leaving us easy marks4 along the way.
So, that was lunch talk. I split early to attend a memorial service in Gig Harbor which reminded me that there is sad comfort in celebrating a friend’s life. But when the love is gone we are truly lost. Tell someone you love them and find your way again.
1 In CCR “winning” is actually losing (or loosing5). So, if someone wins at CCR that means they actually lost and had to pay the bill.
2 CCR = credit card roulette. CCR is a game whereby everyone at lunch hands in their credit card to one person called the “shuffler”. The shuffler then shuffles the credit cards under the table and the “picker” picks the winning credit card. The “winner” wins and pays the entire bill and is granted the privledge of assigning the act of blogging to any of the “loosers”. CCR was brought to us by Jason at 37 Signals.
3 Bindle stick is a collection of belongings wrapped in cloth and tied around a stick.
4 Easy mark is a hobo sign or mark that identifies a person or place where one can get food and a place to stay overnight.
5 “loosing” is a reference to the most popular mispelled goal on 43 Things, loose weight (it’s actually supposed to be lose weight). Sadly, these same people are known to pull over near Reno, NV for “lose slots”. Thankfully there are about 2,000+ people on 43T that want to loose weight and 14,000+ people that want to lose weight.
2 Comments | Lunch | Permalink
Posted by Daniel Spils
October 10, 2006
We’ve been experiencing gnarly site performance issues the last few days. The sites have been slow or timing out altogether (sorry about that!). The good news in all of this is that we seem to be back on track. Let us know if your’e experiencing any site performance issues.
The culprit? A backend issue where folks were trying to merge duplicate Manhattans using the Report A Problem tool. As it turns out Manhattan is a big place and merging it brought the sites to their knees. Leave it to Manhattan to take the mighty 43 Things down! On that note, I think we all deserve a delicious Manhattan … it’s on the Robots.
31 Comments | Site Status | Permalink
Posted by Daniel Spils
October 10, 2006
The time has come for us to hire a dedicated system administrator. We’re looking for a great person who is passionate about the sys admin role and is willing to make it up and make it happen. Here’s the official job description …
The Robot Co-op is looking for a passionate sys admin w/ experience managing open-source clusters & Rails. Our software includes: FreeBSD, MySQL, Apache, Rails, MemCache and MogileFS. We run our 5 sites on a 6-machine cluster. You must be comfortable setting up new servers, optimize existing servers and be able to perform server surveillance, monitoring and maintenance. We need you to keep us clear of system issues, performance crunches, & daily hiccups.
To apply email [email protected]
No Comments » | Job | Permalink
Posted by Daniel Spils
