The Robot Co-op

We’re transferring the DNS settings for All Consuming this morning and that means it’ll be down for a little bit. The back story on this one is that allconsuming.net was registered by Buster long before the Robot Co-op was around. He and I spent many hours over the last 3 years trying to transfer the domain from Gandi to Joker (where we have 43T, 43P, 43Peeps, LOB, etc. registered). This was not easy. In fact, it was a total pain in the ass (damn you Gandi!).

But the good news is the transfer has occurred and we’re yanking the band-aid off the DNS settings today. Soon (hopefully later today) All Consuming will be humming again.

 UPDATE (Aug 8): All Consuming is back up as of a couple days ago.

You may have noticed two missing sites the last few days:

• robotcoop.com was down as we swapped the blog over to Wordpress — it should be a stable experience now. No more multiple posts and comments or application errors. Phew.
• All Consuming was down for about 16 hours — this stemmed from trying to renew the domain and incorrectly having the credit card denied. The credit card has now been accepted and we’re back in business!

Hope you enjoy the new blog.

7/26 UPDATE: we’ve added more RAM to the slave database and are adding 4GB more RAM to the master database on Monday.  Are people feeling things are zippier?

UPDATE: sites are back up—and things are better than before. This is game of inches on the performance front but I think we gained about 2 inches with the last buffer move. Okay, back to goaling!

UPDATE: we’re taking the sites down for 5 minutes to increase some mysql buffers—that’s a fancy way of saying hold on for just a sec and things should get much better.

Are you getting a lot of Oops pages in the last few days? You are not alone. I could make up an elaborate explanation, or simply say: we’re working on it. More specifically, we should have two new boxes (aka webservers) up by the end of the week. That may help things hum along. In addition we’re looking into other database efficiencies to deliver us from Oopsland to the promise land.

In other news I was lucky enough to make it this weekend to the Bay Area 43T Meetup in San Francisco and met some great folks. It was rewarding as a user of 43T to meet up with a bunch of smart, insightful 43T’ers. I highly recommend attending and/or organizing a 43T meetup of your own. If you do, let us know how it goes.

P.S. If you’d like to lodge your frustration about Oops-ness, feel free to do so here in the comments. You’ll feel better if you get it off your chest.

Thanks for all the email reports regarding performance issues with Lists of Bests in the last couple days. We’re making some tweaks now that should stabilize performance—and we’ll continue to work on getting the site healthy again.

Sorry for the hassle but thanks for your detailed feedback. It’s helpful to us.

We’re working on 43 People, All Consuming & Lists of Bests right now—they may be down for a few minutes to an hour as we work on some database issues. Hope to have things up before lunch!

The sites may be down periodically on Sunday April 15th from 12am – 1am-ish (Central time). Here’s the official word from our hosting company, Rackspace:

The window for this maintenance will be from 12:01 AM to 1:00 AM Central time on Sunday morning, April 15, 2007.

Rackspace will be performing upgrades and routine maintenance in the data center networking infrastructure. Due to this maintenance, there may be several brief interruptions of service while traffic fails over from one device to another. No maintenance will be performed on your specific devices but applications utilizing persistent TCP connections like SSH and RDP may lose connectivity to the internet during the traffic migration.

Hopefully it’ll go without many hiccups. Thanks for your patience.

The last few days email was backed up (notifications, account verification, etc.). Apparently some spammer(s) had a few bogus email addresses that clogged up the pipes. The pipes are now clear and email is catching up. I think we’ll be back on track by the end of the day.

In other news, it’s a magical day at the Co-op. We installed a printer today! After 2+ years of no printer or fax machine we now have more than that: a printer/scanner/fax/copier. We don’t have a phone so faxing is still a thing of the future. If you’re curious, we’re using an HP Photosmart 3180. Things are looking up!

update: Josh has informed me that it’s actually not a fax machine. It’s simply a scanner/copier/printer. Faxing remains as elusive as ever.

We’re tweaking the database and the site has to be down for a few minutes. We’re hoping we’ll be up shortly. Thanks for your patience!

UPDATE: we’re back at 11:10 PST! Down about 10-15 minutes. Thanks folks.

The site will be down tonight (Dec. 12, 2006, 9pm Seattle time) for approximately 60 minutes while we do some scheduled database maintenance. Ivan and Chuq will be working together on this together—if anything goes wrong we’ll update you here. This should be the last time we pull the sites down for a long while. Thanks everyone!

UPDATE: site went down at 9:36PM PST and came back up at 10:55PM PST. That’s 84 minutes of downtime. We should be good for a while. Thank you Chuq and Ivan.

CAN’T GET THE SITE? A few folks have reported not being able to get 43T or just their profile page to load. In these cases clearing their Firefox browser cache has solved the issue. Try this this sequence of clicks:

tools > options > privacy > cache > clear cache now

Thanks to Gary and Laurie for helping troubleshoot the issue.

We’re going to activate the new database we’ve been talking about in recent weeks. This will allow the site to be up 24/7. We’ll no longer bring the sites down daily to backup the database. Exciting times!

The best part is that it’s an avid 43T user who is doing the behind-the-scenes work to make this happen. Chuq will be bringing the db online tomorrow. Go Chuq!

Chuq is pretty much a legend around the office as he was the first person to push the limit of things done. Currently Chuq has done 3,441 things. No joke.

We plan to be down for less than 60 minutes. Could be less; could be more if we run into any unforeseen issues. We’ll post updates here if things get crunchy.

UPDATE (Nov. 29 10pm PST): Aborting mission tonight. Chuq is diligently working with Eric & Ivan. It’s taking longer than expected so we decided to do a little more prep work and take another swing at the piñata in a day or so. Stay tuned kids.