mlcastle

The Five Boro Bike Tour this morning was fine (though apparently not if you weren’t in it),...

Sun, 06 May 2012 21:00:00 -0400

The Five Boro Bike Tour this morning was fine (though apparently not if you weren’t in it), but the best moment was the fleeting one after the ride, when hundreds of cyclists at a time got off the ferry and momentarily outnumbered cars in lower Manhattan, not interfering with but becoming the traffic of Nieuw-Amsterdam. This is what I want New York to look like every day; motorists, as far as I’m concerned, can keep the BQE.

I gave a brief talk on how Android uses images, and how to deal...

Mon, 23 Apr 2012 19:41:00 -0400

I gave a brief talk on how Android uses images, and how to deal with the variety of device sizes and resolutions in the Android ecosystem. I made sure to give the talk as boring a title as possible.

The target audience was a group of experienced web developers just starting to learn about mobile development, mostly focusing (except for this “guest lecture”) on iOS. Have a look at my slides if you like.

Why gender parity in computing matters

Tue, 03 Apr 2012 09:26:02 -0400

Why gender parity in computing matters:

Dr. Klawe and others say the underrepresentation of women in the field is detrimental in a larger sense. Computer science, they say, is as vital to propelling society forward in the digital era as mechanical engineering was in the industrial age.

Ride Report: Manhattan to Staten Island via George Washington and Bayonne Bridges

Thu, 29 Mar 2012 22:13:00 -0400

I rode this weekend from my home in lower Manhattan to Staten Island. The only bike-accessible route for doing this involves riding up Manhattan (the cherry blossoms were in bloom!), across the George Washington Bridge, down through New Jersey along the west bank of the Hudson River, and across the Bayonne Bridge to Staten Island. (Conversely, if you’re actually interested for some reason in going to Staten Island, it’s much quicker just to take the ferry.)

A friend then asked how the ride was; in case my advice is useful to anyone else, here it is.

The ride is totally doable, but

Much of the route is on this weird walkway along the Hudson which is mostly bricked rather than asphalted. I think bikes are allowed there (and anyhow no one tried to stop me) but am not totally certain. It seems like it is also still being constructed in parts. Anyway you shouldn’t expect to go particularly fast on it, if that’s a concern.
The walkway disappears and reappears, and unless you have a map, you might not know from the road when it is reappearing, as you often have to ride through a parking lot or something to get there.
When you have to be on a road, the road is full of New Jersey drivers.
This is apparently New Jersey’s idea of a designated signed but not striped “bike route.”
As far as I could tell, the easiest way to get around the Hoboken train station is to get off the bike and just walk it through the station.
There are stairs on the NJ side of the Bayonne Bridge, and technically you’re supposed to walk your bike across, but seriously I saw one pedestrian the whole way across and it wasn’t a huge problem to avoid hitting him while riding.

Good news:

Except for the approach to and from the GWB, it’s pretty much flat
There’s a very nice segragated cycleway for four whole blocks of Hoboken
The southern part of Jersey City and all of Bayonne and S.I. are on normal roads with cars on them, but reasonably narrow roads which felt reasonably safe.
Free ferry back to Manhattan.

The Tumblr of the moment to follow is geraldoinahoodie. Have a...

Fri, 23 Mar 2012 22:18:13 -0400

The Tumblr of the moment to follow is geraldoinahoodie. Have a look if you like to look at hypocritical Fox News talking heads.

makingmeetup: We noticed some interesting contrasts in the...

Fri, 16 Mar 2012 18:43:43 -0400

makingmeetup:

We noticed some interesting contrasts in the weekday usage of Meetup on different platforms.

Saturday is the biggest day for Meetups to take place. Since people are away from their desks, they use our mobile apps and mobile web site to find their way — but visits to Meetup.com fall off.

On Monday, however, people return to their desks and start planning their week on the website, while usage of the mobile platforms is lower.

(Data is from the past nine weeks, normalized so that the average day for each type of access is equal to 1.0.)

"When you transfer Subscriber Content to Tumblr through the Services, you give Tumblr a..."

Sat, 03 Mar 2012 16:56:29 -0500

“

When you transfer Subscriber Content to Tumblr through the Services, you give Tumblr a non-exclusive, worldwide, royalty-free, sublicensable, transferable right and license to use, host, store, cache, reproduce, publish, display (publicly or otherwise), perform (publicly or otherwise), distribute, transmit, modify, adapt (including, without limitation, in order to conform it to the requirements of any networks, devices, services, or media through which the Services are available), and create derivative works of (including, without limitation, by Reblogging, as defined below), such Subscriber Content.

Note also that this license to your Subscriber Content continues even if you stop using the Services…

”

- Tumblr Terms of Service

Android Photo Spy

Wed, 29 Feb 2012 08:58:05 -0500

The New York Times published an article noting that iOS applications can access your photo library if they request permission to know your device’s location. They noted that “Google declined to comment on how its Android operating system for mobile devices handles this issue.”

Of course Google declined to comment: Android apps can access your photos with no special permissions whatsoever. This is, apparently, by design.

So, I wrote a quick demonstration app which gets a list of your photos (and their locations, if available) and lets you look at the photos if you like:

Just to say it again: the app requires no special permissions whatsoever. In fact, because it doesn’t have INTERNET permission, it can’t send your photos anywhere and so is not a true “spy,” but of course almost all real apps have INTERNET permission, and so this is not a limitation in actual practice.

You can download the app and look at the source code on GitHub.

For further Android privacy leaks, see packetlss’s android-locdump project, which reads your phone’s location cache and which I used to make a map of my travels. (This map was also published by Spiegel Online.)

HTML-like Tags in Android String Resources

Thu, 16 Feb 2012 00:54:46 -0500

It’s not a secret that you can put some HTML-like tags in Android string resources files. The documentation lists only three such supported tags: , , and .

However, reading the source code reveals that some other tags are also supported! According to the Android source code at android.content.res.StringBlock#applyStyles, the complete list of supported tags is:

 creates boldface text
 creates italicized text
 creates an underline
<tt> creates text in a monospaced font
<big> and  increase and decrease the font size
 and <super> create superscripts and subscripts
<strike> creates a strikethrough
<li> creates a paragraph with a bullet next to it
<marquee> creates a span with TruncateAt.MARQUEE set.
The  tag supports a few attributes:
- height accepts an integer and sets the line height accordingly
- size accepts an integer, setting the size of the enclosed text in dip.
- fgcolor and bgcolor set the foreground and background color. Note that these attributes accept only integer arguments, so you need to say “#ff00ff00”, not “green”.
The <a> tag accepts the href attribute and creates a link.
The <annotation> tag accepts arbitrary attributes, creating arbitrary Annotation spans.

Don’t be afraid to use the source or to experiment in order to get further details.

makingmeetup: We’re refactoring some of the user interface in...

Fri, 03 Feb 2012 14:41:48 -0500

makingmeetup:

We’re refactoring some of the user interface in our Android app. As you can see, we want event comments to feel more like a conversation, so we’re making them feel like the native Android Messaging app.

As we do this, we’re trying to take some of the new Android ICS design guidance into account; you can see in particular the action bar pattern here. We’re using Action Bar Sherlock so that we remain compatible with Android 2.x devices.

And, finally, the reason we have two screenshots above is that we’re taking care to use UI elements that will be familiar to our users, depending on the version of the Android platform that they’re using. In particular, the share icon, add-a-comment text field, and send button are set to match the usual platform style: Gingerbread on the left and Ice Cream Sandwich on the right.

Tumblrs, how do they work?

Fri, 03 Feb 2012 02:10:33 -0500

Tumblrs, how do they work?

Purchasing 4′33″ Outside the UK

Mon, 13 Dec 2010 14:25:00 -0500

The Cage Against the Machine campaign aims to make John Cage’s 4′33″ the number one single in the UK this Christmas. This is a campaign which I support wholeheartedly, and I wished to participate.

However, UK digital music vendors limit sales to computers with IP addresses actually in the UK. Here is a summary of the steps I took to purchase the album anyway. (Note that the operative word here is “summary”; some degree of technical skill may be needed to follow this procedure.)

WARNING: Even though this post explains how to purchase music, not how to pirate it, it is entirely possible—maybe even likely—that following the steps described here violates some terms of service agreement or is otherwise illegal.

Procedure

Here are the steps I followed. You may need to vary them somewhat to suit your taste.

Download and install Tor if you haven’t done so already. Set it up in such a way that you can browse the web anonymously. (Note that steps taken later in this document will completely break your anonymity.)
Use Vidalia, Tor Status, or some other means to gather the fingerprints of a few reasonably fast exit nodes in the UK.
Edit your torrc file to contain the following lines, and restart tor to make your changes take effect.
```
   1  ExitNodes $fingerprint1, $fingerprint2, ...
   2  StrictExitNodes 1
```
Have your browser use tor, and verify that your HTTP connections are indeed coming from the UK.
Visit 7digital’s Cage Against the Machine page and follow the steps there to purchase the single or the 8-track collection of remixes.
You can sign up for an account using your usual contact details. On the payment page, make sure to use a credit card. It’s apparently okay if you leave the country on “United Kingdom”; just enter your US ZIP code (or other foreign postal code) in the postcode box. Do not attempt to pay with PayPal—logging in to PayPal through tor will apparently cause your account to be locked.
After you have paid, you will be taken to a page with links to download your purchase. After you have arrived at this page, but before you click on the download link, disconnect your browser from tor.
Now you can download the files without tor, conserving the bandwidth of the tor network and allowing your download to complete in a reasonable amount of time. (You’ve already completely destroyed your anonymity by providing your credit card number.)
Remove the lines you added to your torrc and restart tor again, so that you will use any nodes in the network next time you use tor.

Cooking Roasted Vermicelli

Mon, 26 Jul 2010 15:28:00 -0400

When I began freelancing and formed a company, I called it “Roasted Vermicelli” more-or-less at random: I had previously purchased the domain name vermicel.li, and sticking the word “roasted” in front was required to make the business name unique within New York State.

After some years of this—plus a blog tagline which promises musings on pasta but has yet to deliver any—when I cooked dinner the other day, it was time to cook using some roasted vermicelli.

Roasting

The actual process of roasting the vermicelli is quite simple: heat a little bit of fat (oil or ghee) in a pan, put the uncooked vermicelli in, and toast for a few minutes, stirring constantly, then remove from the heat. The result looks something like this:

If this is too much work, it’s probably just as well to skip making dinner altogether, but my local Indian market does stock pre-roasted vermicelli.

Then What?

There are a few different recipes one can then make using the roasted vermicelli. Semia payasam is a dessert with condensed milk, sugar, and nuts. I, however, made vermicelli upma, which is a savory dish with lentils, nuts, and vegetables. Apparently it’s normally served for breakfast, but I ate it for dinner and didn’t get struck down by any gods of propriety.

The recipe I followed is from a site called “Show Me the Curry”, and the result looked like this:

It tasted good, and I was able to take some photographs for my next batch of business cards. A successful project!

Queueing async_observer tasks from the command line

Thu, 22 Apr 2010 18:27:00 -0400

For one of our projects, we’re using beanstalkd in combination with the async_observer plugin for Rails to run slow tasks in the background.

This is a nice setup. Although we’re not using all of its more advanced features, beanstalkd is a very powerful and efficient queueing system, and async_observer provides a clean integration with Rails. (On other projects, I’ve happily used delayed_job for similar purposes, but it was not suitable for this project due to its heavy dependency on a ActiveRecord and a SQL database. By contrast, async_observer’s support for ActiveRecord was quite easy to rip out.)

However, we would like to be able to queue async_observer jobs from cron or the command line. Why launch a new process with a copy of the Rails stack when there’s already a long-lived one running? So, I wrote a ruby script with minimal dependencies which can stick tasks in beanstalkd for our async_observer worker to run.

The Script

Here’s the script I wrote. Stick it in RAILS_ROOT/script/beanstalk-queue.

   1  #!/usr/bin/env ruby
   2  
   3  #
   4  # == Synopsis
   5  #
   6  # This is a replacement for script/runner which avoids loading up the
   7  # rails environment and instead queues things for async_observer to
   8  # do. Note that, unlike script/runner, a filename is not accepted; you
   9  # must provide the code to run on the command-line.
  10  #
  11  # == Usage
  12  #
  13  # beanstalk-queue [OPTION] ... 'CODE'
  14  #
  15  # -d DELAY:
  16  #   amount of delay before the job is run (default 0 seconds)
  17  #
  18  # -e ENVIRONMENT:
  19  #   set the rails environment (falls back to ENV['RAILS_ENV'] or
  20  #   'development' if unset)
  21  #
  22  # -p PRIORITY:
  23  #   set the job priority (default 65536)
  24  #
  25  # -t TTR:
  26  #   set the job's time to run (default 120 seconds)
  27  #
  28  # -v, --verbose:
  29  #   be verbose
  30  #
  31  # -h, --help:
  32  #   display this help
  33  #
  34  # CODE: the ruby code to be called asynchronously
  35  #
  36  
  37  require 'rubygems'
  38  
  39  require 'beanstalk-client'
  40  require 'getoptlong'
  41  require 'rdoc/usage'
  42  require 'yaml'
  43  
  44  RAILS_ROOT = File.expand_path(File.join(File.dirname(__FILE__), ".."))
  45  
  46  opts = GetoptLong.new(
  47    ['--help', '-h', GetoptLong::NO_ARGUMENT],
  48    ['-d', GetoptLong::REQUIRED_ARGUMENT],
  49    ['-e', GetoptLong::REQUIRED_ARGUMENT],
  50    ['-p', GetoptLong::REQUIRED_ARGUMENT],
  51    ['-t', GetoptLong::REQUIRED_ARGUMENT],
  52    ['--verbose', '-v', GetoptLong::NO_ARGUMENT]
  53  )
  54  
  55  pool = Beanstalk::Pool.new(%w(localhost:11300))
  56  
  57  tube = 'default'
  58  delay = 0
  59  environment = ENV['RAILS_ENV'] || 'development'
  60  priority = 65536
  61  ttr = 120
  62  verbose = false
  63  
  64  opts.each do |opt,arg|
  65    case opt
  66    when '--help'
  67      RDoc::usage
  68    when '-d'
  69      delay = arg.to_i
  70    when '-e'
  71      environment = arg
  72    when '-p'
  73      priority = arg.to_i
  74    when '-t'
  75      ttr = arg.to_i
  76    when '--verbose'
  77      verbose = true
  78    end
  79  end
  80  
  81  if ARGV.length != 1
  82    $stderr.puts "missing code to run (try #{File.basename(__FILE__)} --help)\n"
  83    exit 1
  84  end
  85  
  86  code = ARGV[0]
  87  
  88  case environment
  89  when 'production'
  90    if File.exist?(File.join(RAILS_ROOT, "REVISION"))
  91      tube = File.read(File.join(RAILS_ROOT, "REVISION")).strip
  92    end
  93  when 'development'
  94    tube = 'ourproj-development'
  95  end
  96  
  97  puts "using tube #{tube}" if verbose
  98  
  99  pool.connect
 100  pool.use(tube)
 101  job = YAML.dump({:type => :rails, :code => code})
 102  
 103  puts "job:", job if verbose
 104  
 105  job_id = pool.put(job, priority, delay, ttr)
 106  
 107  puts "job #{job_id} sent to #{pool.last_server}" if verbose

Download this code.

Usage

Because this code does not load Rails or your application, it is unfortunately unable to determine how you have configured your application. Therefore, you must configure the script itself.

First, if you’re not using the default beanstalkd configuration of one server running on localhost:11300, edit line 55 appropriately. Next, you’re probably not using the same beanstalkd tube naming convention as we are, so edit lines 88–95 to properly map rails environment names to beanstalkd tube names.

Full documentation for running the script is at the top of the file, or run script/beanstalk-queue --help, but basically it works like script/runner with some extra optional parameters for controlling the queueing:

$ ./script/beanstalk-queue -e production 'MyModel.some_long_task'

cron and whenever integration

If you use whenever for dealing with programatically your crontab file, here’s a bit of code to define a new beanstalk job type, analogous to the runner job type:

   1  module ::Whenever
   2    module Job
   3      class Beanstalk < Whenever::Job::Default
   4        def output
   5          path_required
   6          %Q(#{File.join(@path, 'script', 'beanstalk-queue')} -e #{@environment} #{task.inspect})
   7        end
   8      end
   9    end
  10  
  11    class JobList
  12      def beanstalk(task, options = {})
  13        options.reverse_merge!(:environment => @environment, :path => @path)
  14        options[:class] = Whenever::Job::Beanstalk
  15        command(task, options)
  16      end
  17    end
  18  end

Download this code.

I just stuck this code near the top of my config/schedule.rb file, though a possibly cleaner approach would be to put it in its own file and then require that file from schedule.rb.

Converting a Rails Database From MySQL to PostgreSQL

Thu, 04 Feb 2010 00:55:00 -0500

For a number of reasons, we want to convert one of ours Rails applications so that it is backed by PostgreSQL rather than MySQL. Although Rails and ActiveRecord abstract many of the differences between the two database engines, there are still some issues to be aware of and no one-size-fits-all solution to the problem. Here are some of the challenges we encountered in making the switch.

Converting the schema

The first thing in the conversion process is converting the schema. Fortunately, if you have a fairly simple schema, then Rails makes this relatively easy.

Look in config/environment.rb and make sure that config.active_record.schema_format is unset or set to :ruby. This is the default, but some projects change it to :sql for various reasons.

Then do rake db:schema:dump to generate db/schema.rb with your current database schema. Edit config/database.yml to use a PostgreSQL database instead of a MySQL one and do rake db:schema:load to load the schema in.

If your database migrations are simple, you might instead be able to do rake db:migrate to construct the Postgres database from scratch, but in my experience most Rails projects eventually end up with some kind of database-specific SQL in at least one migration.

If, on the other hand, your database schema is so complex that ActiveRecord’s schema dumps can’t handle it, you might consider looking at SQLFairy to help you manage the transition.

Converting the code

This is likely to be the trickiest part of the conversion process. You must go through your code and replace anything MySQL-specific with something PostgreSQL-specific or (better) database-agnostic.

Good test coverage of your application is definitely a huge help here: if you can run tests and fix bugs as they appear, you dramatically increase the likelihood of catching everything.

Here are some of the issues that we encountered in our codebase; of course, different applications and coding styles may reveal other issues.

Differences in quoting

MySQL allows you to quote table and column names with backquotes; PostgreSQL requires double-quotes. One of our tables has a column in it called when, which must be quoted whenever we use it.

Rails will of course handle the quoting for you if you do something like OurModel.find_by_when(Time.now). But if you are constructing your own SQL or conditions, you need to handle the quoting yourself. Given MySQL-compatible code like this:

OurModel.find(:conditions => ["`when` < ?", @yesterday])

You need to either use Postgres-specific code like this:

OurModel.find(:conditions => ["\"when\" < ?", @yesterday])

Or, better, convert it to something database-agnostic:

OurModel.find(:conditions => ["#{OurModel.connection.quote_column_name(when)} < ?", @yesterday])

In addition to quote_column_name, ActiveRecord also provides a quote_table_name function for quoting table names and, crucially, a quote function for quoting the data you actually store in your database. These functions are your friends in database-agnostic programming—be sure to use them!

Booleans

MySQL lacks a native BOOLEAN type, so if you create a boolean column in Rails, you will end up with a TINYINT(1) column which has values of 0 and 1 for false and true respectively. There’s no particular need to be aware of this fact—except that most Rails developers are aware of this fact and so often use 0 and false somewhat interchangeably. PostgreSQL has a native BOOLEAN type, so these shenanigans will no longer fly. If important is a boolean column, then code like this will work in MySQL-backed rails but not under PostgreSQL:

m = OurModel.new(:all, :important => 1)
m.save

Instead, be sure to use true and false where you mean to be using booleans:

m = OurModel.new(:all, :important => true)
m.save

Similarly, if you are using booleans in your conditions, this will work in MySQL:

OurModel.find(:all, :conditions => "important = 1")

Instead, take advantage of ActiveRecord’s helpful quoting capabilities and replace your code with the database-agnostic equivalent:

OurModel.find(:all, :conditions => ["important = ?", true])

You may also be tripped up by this problem in your fixtures; replace 0 and 1 with false and true in your YAML files: they’ll then work in both MySQL and PostgreSQL.

Other differences

There are other differences which we encountered. In one part of our code, we need to select rows from the database at random. In MySQL, this is expressed as ORDER BY RAND(), but in PostgreSQL, the function is called RANDOM(). Since there is only one place in the code where this occurs, we did not bother to implement any kind of sophisticated database-agnostic function. Instead, we took advantage of the connection.adapter_name function to be able to write this:

OurModel.find(:all, :conditions => ["value < ?", @max_wanted_value],
              :order => OurModel.connection.adapter_name == 'PostgreSQL' ? 'RANDOM()' : 'RAND()',
	      :limit => 50)

Additionally, we had one unexpected issue: MySQL DATETIME columns store their data with integer second precision, but PostgreSQL timestamp without time zone columns are based on floating point numbers. We had to modify our code in a couple of places to round times to the nearest second before storing them in the database in order to get certain equality comparisons to work properly.

Copying the data

Once we had the application more-or-less working as we wanted, it was time to copy the data from the MySQL database into Postgres. Naïve approaches actually do not work well here; due to the difference in quoting rules, the differing storage for booleans mentioned above, and a couple of issues, calling mysqldump and then attempting to read the output into Postgres fails.

Instead, we did the data transfer taking advantage of ActiveRecord’s knowledge of databases. We based the method on this one provided by Rama McIntosh but we had to make a couple of changes. We modified the code to manually generate INSERT statements rather than constructing ActiveRecord objects. This has the chief advantage of working with has_and_belongs_to_many-style join tables without id columns and the secondary advantage of being somewhat faster. We also modified the code to automatically update the PostgreSQL sequence objects so as not to break future inserts into the table.

The resultant code has been checked into github. Feel free to use it and modify it further if you like.

If you are working on a similar conversion project, good luck!

timemachine-fu, a Timemachine plugin for Rails

Thu, 12 Nov 2009 18:08:00 -0500

Tobis Timemachine is a very useful tool for bringing websites back to “the amateur web of 1996.”

However, there is is some overhead with the Timemachine: it requires your users to have already installed it in order to be useful, and those visitors need to be using a compatible version of Firefox. What if you want to make a 1996-era website for all your site’s visitors?

Thus, timemachine-fu, the Timemachine plugin for Rails.

Demonstration

PBCore Database before timemachine-fu:

and after:

Note that this static demonstration does not really show the full power of timemachine-fu: in actual installation, many of the images will be animated for further effect, and there are three stylesheets, one of which will be randomly selected for each page load. In addition, positioning of some of the images will be randomly determined anew on each page load, and one of the themes includes a MIDI file for your listening pleasure.

Use it in your own projects

timemachine-fu is provided as an easy-to-install Rails plugin. It is compatible with either Prototype, as provided by default with Rails, or with JQuery, if you’re using jrails.

You can get the code, see installation instructions, and do whatever else you want on GitHub.

A Lightweight “Migrations” System for CouchDB

Tue, 10 Nov 2009 20:10:00 -0500

As I’ve mentioned before, I’m working on a Rails project where we’ve dropped ActiveRecord in favor of a CouchDB database.

One thing I miss from ActiveRecord, though, is the ability to do migrations. While SQL migrations in their most conventional use—to modify the schema of our database—are of course not needed in schemaless CouchDB databases, there is still sometimes a need to have a block of code which runs exactly once on every installation of our codebase. For example, we might have a key in our documents which becomes so important that we want to break its values off into new documents, or vice versa, we might have a bunch of documents which should all get combined into attributes of one master document.

What we aren’t doing

The correct way to solve this problem, some people argue, is to simply put some smarts into our model and support versioning there:

   1  class MyClass < CouchRestRails::Document
   2    attribute :version
   3  
   4    def stuff
   5      if self.version && self.version >= 2
   6        Stuff.by_parent_id(:key => this.id)
   7      else
   8        self["stuff"]
   9      end
  10    end
  11  end

No thanks

But that seems like a lot of work to maintain. If our system had millions of models in it and converting them was expensive, we might have no choice but to do it, but I’d rather just convert all existing models to our new schema (or, rather, our new but still schemaless way of being) and write all our code to use the updated way of thinking.

So, I designed a simple, lightweight “migrations” system. It lives entirely in Rake and is only a couple dozen line of code.

Here it is, in its entirety:

   1  desc "Perform unperformed CouchDB migrations"
   2  task :migrate => :environment do
   3    schema = begin
   4               CouchRestRails::Document.get("migration-list")
   5             rescue RestClient::ResourceNotFound
   6               CouchRestRails::Document.new(:_id => "migration-list")
   7             end
   8  
   9    migrations = Rake::Task.tasks.select{ |t| t.name =~ /^migrations:/ }
  10  
  11    begin
  12      migrations.sort{|a, b| a.name <=> b.name}.each do |migration|
  13        unless schema[migration.to_s]
  14          puts
  15          puts "== running #{migration.to_s}"
  16  
  17          migration.invoke
  18  
  19          schema[migration.to_s] = true
  20        end
  21      end
  22    ensure
  23      schema.save
  24    end
  25  end

Download the code and stick it in lib/tasks/migrator.rake. Then, create a new file, lib/tasks/migrations.rake, and create some tasks in the :migrations namespace. Here’s an example file you can modify to suit your needs. If you end up with a lot of migrations, you can of course split them among several files.

Any un-run migrations will be run in lexical order, so you can prefix your tasks with consecutive integers as shown in the example, with timestamps, or with whatever else makes you happy, as long as you are consistent within any one project.

Then, doing rake migrate will run any migrations which have not yet been run against your database.

This system has its limitations of course: among them is the need to stick everything in rake, possibly causing slowness if you end up with too many migrations. But as a lightweight, judiciously used replacement for ActiveRecord migrations, I find that it is sufficient for our needs.

Dipping a Toe Into the Cloud, and Other Mixed Metaphors

Fri, 30 Oct 2009 16:50:00 -0400

Cloud computing is supposedly all the rage these days: the idea is that instead of running your own infrastructure, you just outsource everything.

There are some cases where this is almost self-evidently a good idea: renting a VPS is much cheaper and more reliable than sticking a server in your home and then dealing with the noise, electricity, and bandwidth that such a thing would consume. This blog, for instance, is hosted on a VPS at Panix. On the other hand, an Amazon EC2 instance at the seemingly low price of $0.10 per hour will cost $72 if you leave it on for a whole month—more expensive than many low-end dedicated physical machines.

But the cloud, we are promised, can do more for us than just save us money: it can run our services for us. Programmers no longer need know how to do anything but program the things specific to their application: everything else will be taken care of by “the cloud.”

I’ve traditionally run all my services on machines (usually VPSes) that I administer. But it would be nice to get out of the system administration business.

The Experiment

So, I tried moving one of my small websites, the PBCore Validator, completely away from servers I control and into “the cloud.” This is such a tiny site that I was able to make the move completely for free, which makes for a nice experiment.

The most important component of the setup is, of course, the hosting of the application itself. I selected Heroku for three reasons: they have a limited plan available for free; they support the development of Sinatra, the framework I’m using; and their service is incredibly slick.

Setting up the hosting was quite easy:

$ sudo gem install heroku
$ heroku create validator
$ git push heroku

This is certainly easier than creating a new nginx configuration, setting up Thin, and all the rest. However, there was an extra step here: I had to modify my application to work with Heroku’s restrictions. Previously, my application statically precompiled its homepage and stored it in the filesystem. However, Heroku provides you with a read-only filesystem, so I had to modify the application to dynamically generate the file and then to set some caching headers so it won’t actually be dynamically generated on every request.

The changes here were trivial, but a larger application might have required more changes and thus more work. This highlights a potential pitfall of cloudlike services: you have to adapt your application to your providers’ conventions; if you need to do something which can not be done under the your providers’ constraints, you need new providers.

Another pitfall was highlighted when I made a small change to the application, then tried to push a new version to Heroku, and got this:

Faced with such a vague, unhelpful error message, I could do nothing but try again after a few minutes, and when that failed, contact Heroku support. They fixed the problem and got back to me within a couple of hours, which is certainly not bad for a response to a nonpaying customer outside of business hours, but if this had happened to a more critical application under more critical circumstances, I might have been rather upset.

Other Services

Having successfully migrated my application away from my servers, what about the other services on which it depends?

DNS was easy. Heroku comes with an integration with Zerigo, so I could set that up with a few commands:

$ heroku addons:add custom_domains:basic
$ heroku addons:add zerigo_dns:basic
$ heroku domains:add pbcorevalidator.org
$ heroku domains:add www.pbcorevalidator.org

After a quick trip to my registrar to switch the domain to use Zerigo’s servers, I was all set. I expect that my application will never get so much traffic that their free level of service would be insufficient.

The final component of the move was of the source code hosting. I use git, so there is not really any need for a centralized server, but since the Validator is a Free Software project, I want to provide access to the browse or download the code and to have an option for others to be able to commit to the project if they desire. It’s easy enough for me to host my own repositories, but if I want to move into the “cloud,” then the obvious choice is GitHub, so now the PBCore Validator code is on GitHub. As the Validator is a small Free Software project, the GitHub free plan is more than sufficient here as well.

And of course, the morning after I made these changes, GitHub went down for about an hour and a half. As they point out, this isn’t the end of the world with a distributed system like git, but it’s still downtime that I had given up any ability to do anything about.

Conclusions

I was able to relatively quickly and easily move a small service completely off of servers that I control for a total cost of $0. Of course, I now rely on the largess of three companies—if any of them goes out of business or decides that providing free limited service is unprofitable, I will need a new option. If lots and lots of people start to want to validate lots and lots PBCore documents, it’s possible I will have to pay to upgrade my service. For comparison, my Panix VPS isn’t free, but I can easily host a few smallish websites on it for one fairly low monthly cost.

Reliability is definitely a concern: I hit two snafus less than twenty-four hours after my move. On the other hand, the uptime of machines that I control seems to be somewhere under 100% as well.

I can’t say for sure if I would take this approach with larger projects, but I will definitely consider it.

An Adventure in Old Internet Memes

Tue, 27 Oct 2009 01:35:00 -0400

Presented without (much) comment.

An Old Internet Meme

An Analysis

Bolded text indicates cognates in all three columns; italicized text indicates cognates which were missed in the false translation. Not all such cognates are marked; notably, Moskau / Moscow, hey, ha, and ho are unmarked because they appear so often.

A Letter to the NYC Department of Transportation

Fri, 23 Oct 2009 18:36:00 -0400

I wrote a letter to the Commissioner of the NYC Department of Transportation, urging the marking of a safe route between the new bike lanes on Allen and Pike Streets and the East River Greenway. Here’s a copy.

The Letter

Dear Commissioner Sadik-Khan and Borough Commissioner Forgione:

It has been a nice surprise over the past couple of months to see the new bike lanes and other improvements along the center of Allen and Pike Streets in lower Manhattan. These cover a route that I use frequently, and I feel much safer in these new, clearly-striped lanes.

However, there is one part of the project which I feel has been overlooked: at the southern end of Pike Street, I often wish to turn onto the East River Greenway, and when returning, I often wish to turn from the Greenway north onto Pike Street. However, there are no clear markings or signals for how to make this turn. There is not even a curb cut to get on and off of the Greenway. In addition, crossing the busy car traffic on South Street always feels unsafe, especially when coming off of the Greenway onto Pike Street, when there are no visible traffic signals and cars do not generally expect my presence.

I hope that the project can be extended to encompass a safer, easier connection between two of Manhattan’s best bike routes, and I await hearing your response on this issue.

And if you’d like to see it exactly as sent, you can view the letter as a PDF.

Responses

Update November 5, 2009: I received a response from the DOT.

Update March 15, 2010: I received another response, this time more detailed and by email:

Dear Mr. Castleman,

This is in response to your October 23, 2009 email regarding the bicycle lane on Pike Street. We apologize for the delay in response.

As you may be aware, DOT recently concluded an unprecedented three-year plan to expand New York City’s on-street bicycle network by 200 miles and improve safety for cyclists on the streets; 204.5 miles was the final total. These bicycle lanes have encouraged a low-cost, healthful mode of transportation and provided measurable safety improvements to all street users.

Thank you for reporting your observations of a missing ramp and unclear markings at the intersection of Pike Street and South Street. Currently, cyclists should use the pedestrian ramps. In the near future the malls and the East River Esplanade will be reconstructed; when those two projects are implemented the crossing will occur in the middle along with curb ramps.

Thank you for your interest in bicycling in New York City.

Sincerely,

Bicycle Program
NYC Dept. of Transportation