The Robot Co-op

The last few days email was backed up (notifications, account verification, etc.). Apparently some spammer(s) had a few bogus email addresses that clogged up the pipes. The pipes are now clear and email is catching up. I think we’ll be back on track by the end of the day.

In other news, it’s a magical day at the Co-op. We installed a printer today! After 2+ years of no printer or fax machine we now have more than that: a printer/scanner/fax/copier. We don’t have a phone so faxing is still a thing of the future. If you’re curious, we’re using an HP Photosmart 3180. Things are looking up!

update: Josh has informed me that it’s actually not a fax machine. It’s simply a scanner/copier/printer. Faxing remains as elusive as ever.

We’re tweaking the database and the site has to be down for a few minutes. We’re hoping we’ll be up shortly. Thanks for your patience!

UPDATE: we’re back at 11:10 PST! Down about 10-15 minutes. Thanks folks.

The site will be down tonight (Dec. 12, 2006, 9pm Seattle time) for approximately 60 minutes while we do some scheduled database maintenance. Ivan and Chuq will be working together on this together—if anything goes wrong we’ll update you here. This should be the last time we pull the sites down for a long while. Thanks everyone!

UPDATE: site went down at 9:36PM PST and came back up at 10:55PM PST. That’s 84 minutes of downtime. We should be good for a while. Thank you Chuq and Ivan.

CAN’T GET THE SITE? A few folks have reported not being able to get 43T or just their profile page to load. In these cases clearing their Firefox browser cache has solved the issue. Try this this sequence of clicks:

tools > options > privacy > cache > clear cache now

Thanks to Gary and Laurie for helping troubleshoot the issue.

We’re going to activate the new database we’ve been talking about in recent weeks. This will allow the site to be up 24/7. We’ll no longer bring the sites down daily to backup the database. Exciting times!

The best part is that it’s an avid 43T user who is doing the behind-the-scenes work to make this happen. Chuq will be bringing the db online tomorrow. Go Chuq!

Chuq is pretty much a legend around the office as he was the first person to push the limit of things done. Currently Chuq has done 3,441 things. No joke.

We plan to be down for less than 60 minutes. Could be less; could be more if we run into any unforeseen issues. We’ll post updates here if things get crunchy.

UPDATE (Nov. 29 10pm PST): Aborting mission tonight. Chuq is diligently working with Eric & Ivan. It’s taking longer than expected so we decided to do a little more prep work and take another swing at the piñata in a day or so. Stay tuned kids.

“Box” in this case means webserver. Prior to last week there were two webservers dedicated to 43 Things. Now there are three. One could argue (as I will) that this makes the site more stable. Not necessarily faster, but healthier in general. We’ve managed to stablize quite a few performance issues and errors over the past few weeks and we’re still working on delivering more Woot! and less Oops!

It’s true. The sites go down daily for about 30 minutes at 1am PST (Seattle time) while we backup our single database. That’s 9am London time, 4am New York time, 8am Sydney time. During these 30-ish minutes the sites simply don’t operate. However, we do have a plan to replicate the database.

A new, second database is already in the fleet of boxes that run the site and we hope to have it up and running … soon. Soon means that we’re learning as we go along here. Ivan and Eric are working on this together. We’ll let you know when the second database is up and running. At that point we’ll be able to backup the database daily without shutting the sites down and a celebration will be in order!

I should add that we originally chose 1am PST as it was mostly folks on the West Coast of the US using the sites. It began as us and our friends and grew out from there. Over time things have shifted and we’re now an international group of users. New York has displaced Seattle as our largest user base. London is a close third and Sydney and Shanghai are big players. That’s all a long-winded way of saying our night may be your day and we’re going to make the sites work 24-7 for all of us.

It seems I’ve fixed all the image problems for 43 Things, and 43 Places and 43 People will be fixed up tomorrow. Give us a heads-up if you see any broken pages or broken image links on 43 Things.

UPDATE: 43 Places and 43 People now seem to be up to date. If you see anything amiss, please leave a URL for the broken image and the page you found it on. Some may show up after a reload or two as our image servers may be busy rebuilding images for you.

The new drives are installed and image upload is now working on all the sites. There’s now plenty of room for all your photos so bring it on. Let us know if you encounter any errors, but we’ve been uploading photos successfully all morning.

P.S. I am experiencing an upload failure only in Safari about 50% of the time—so we’ll work on that one but I suspect most people are using Explorer and Firefox.

A couple days ago we began running out of disk space for storing all the photos you’ve been uploading to the sites. This has resulted in plenty of “Ooops” pages and folks not being able to upload pictures to the sites. We’re in the process of getting new, large drives installed on our servers to handle the volume of photos that we’re all uploading. We expect the drives to be online within 24 hours. We’ll keep you updated on the progress here.

I’ve personally got a bunch of pictures from my recent Massachusetts trip that I’m dying to upload to 43 Places—hold tight and we’ll have plenty of image space for all our photos.

We’ve been experiencing gnarly site performance issues the last few days. The sites have been slow or timing out altogether (sorry about that!). The good news in all of this is that we seem to be back on track. Let us know if your’e experiencing any site performance issues.

The culprit? A backend issue where folks were trying to merge duplicate Manhattans using the Report A Problem tool. As it turns out Manhattan is a big place and merging it brought the sites to their knees. Leave it to Manhattan to take the mighty 43 Things down! On that note, I think we all deserve a delicious Manhattan … it’s on the Robots.