The Robot Co-op

Entries from October 2006

The software and hardware that runs our sites

October 10, 2006 · 4 Comments

Robot Co-op Hardware

quantity memory cpu disks purpose
4ea Dual 3GHz Xeon 6GB 70GB RAID 1 Apache, FastCGI, MogileFS storage node, memcached, image serving
1ea Dual 3GHz Xeon 2GB 70GB RAID 1 Staging, mail, backend jobs
1ea Dual Opteron 246 12GB 5×73GB in RAID 5 MySQL

Eric has more to say on this topic

Robot Co-op Software

All our online machines run FreeBSD 6.0-RELEASE. We use Amanda for backups which gets rsynced off-site. Subversion is used for revision control of all our configuration data.

We have our own CVSup mirror and package build machine which builds packages for all our boxes to keep things in-sync and to decrease the load on FreeBSD’s CVSup mirrors. We also NFS-mount the src and ports directories to reduce wasted disk space. portaudit is used for vulnerability monitoring and portupgrade for performing package upgrades.

Critical processes are watched by dwatch. Clocks are synchronized with ntpd.

Webservers (x4)

The webservers all run PAE kernels with 6GB of ram. 2GB is allocated to memcached and the rest gets chewed up by Rails processes. Each of our sites gets its own set of processes and we have 25 total Rails processes running per host.

For pages our webservers are all Apache 1.3 with mod_fastcgi. Apache logs are rotated by cronolog and processed by AWStats (but I might switch to Visitors).

We’re using Rails (of course) to generate our pages. We use a library very similar to CachedModel to fetch ActiveRecord objects out of memcached. We don’t use any page or fragment caching.

For images our webservers are WEBrick using a Ruby MogileFS library I wrote (but have not yet released). Image resizing is performed by RMagick, occasionally on-the-fly.

We use the lang/ruby18-nopthreads port of Ruby because we experienced an incredible load increase with the default pthread version.

Each machine runs memcached with 2GB of cache (so be sure to set kern.maxdsiz and kern.maxssiz in /boot/loader.conf appropriately). We store sessions, ActiveRecords and random other expensive-to-compute data in memcache.

Each machine is a MogileFS tracker node and file store. All our images get stored in MogileFS. We use NFS mode due to problems with components of Perlbal on FreeBSD. (With the way we’ve implemented image serving and FreeBSD’s NFS implementation it works out fine.)

Our webservers all run sendmail set up as a smart host pointing to a machine running postfix. This keeps outbound mail sending fast and reduces the amount of maintenance.


We have one database server, it runs MySQL 4.1.x and isn’t all that special. Bob tuned it based on Wikipedia’s MySQL configuration.


We have one machine that does miscellaneous jobs. It runs postfix for inbound and outbound mail, runs crons that update memcached, analyzes log files and a small handful of other unimportant things.

Read more about our Software on Eric’s blog.

Categories: Job

New day, new bug fix

October 9, 2006 · 2 Comments

We’ve had a problem for the last few days that made it difficult to delete or edit a goal. It should be working now. Thanks for letting us know about the problem (and let us know if it is still happening).

Categories: Announcements

Closing accounts

October 6, 2006 · 12 Comments

We’ve got a little bug that is keeping users from closing their accounts. We’ve found the error and should have a fix out in a few hours. If you are running into this problem, try back this afternoon and you should be able to bury your 43 Things account with all due ceremony.


Categories: Site Status

Meaty Eggplant - Snowy Donuts

October 4, 2006 · 3 Comments

Berikster McLenson won at credit card roulette today and entrusted me with the words we shared at lunch. We’re all about habits on Mutual Improvement this month and it’s no wonder that the topic found its way into our lunchtime chatter. Much like our half meat/half eggplant pizza, we defined two separate but similar sides of the same pie: habits and routines.

During my years in Anchorage it once snowed 4 feet in a 24 hour period. The building my parents owned and ran the family business out of was just down the road from our house. It had a flat roof. My dad woke me up around 5am and we went over to shovel off the roof before it had a chance to cave in. That was somewhat of a life altering event for me. I mean who wants to wake up at 5am in the middle of the winter to go shovel? Not me. But we had to. It took a lot of personal will … but there also was company in our misery (we had each other). We finished up around 8am and headed straight downstairs to open up the business.

Now for the life of me I can’t remember how at lunch we separated habits out from routines! I feel like shoveling the flat roof became a habit and that opening the doors for business every day was a routine. I see habits as little programs we create that run like clockwork. Routines are clockwork that simply organizes our lives. Habits are so ingrained that we may not know we’re running the “chew my nails” program until people give us that look. Conversely, it’s much simpler to develop a new morning routine to fit, say, a new job or fall semester.

Thinking back to the old “time to make the donuts” Dunkin Donuts ad campaign, I wonder where habits and routine intersect. That seems like a case where someone’s routine (his job was to make the donuts) became his habit (if he were fired, he’d likely still show up to make the donuts).

Can somebody help me out?

Categories: Uncategorized