I did some work for ecolect recently, integrating search across their main site, a Beast forum and a WordPress blog. It was pretty straightforward once I had it figured out, but I couldn’t find a walkthrough on the net.

So, I decided to write one.

The Search Engine

Ecolect was already using acts_as_ferret for their main site search, so it was a no-brainer to keep using it. For a fun and thorough introduction to acts_as_ferret, see the Rails Envy Tutorial

Searching the Beast Forum

(Note: the beast Forum search isn’t live at the moment as the Forums are not fully functioning yet.)

Beast setup

In Beast, each Forum has many Topics (saved in the topics table), and each Topic has many Posts (saved in the posts table).

The Beast forum was set up by Shanti Braford as described in option #3 of this great article on integrating a beast forum into a Rails app.

In this setup, the Beast tables are added to the main site’s database. So, the posts and topics tables that we want to search are already in the main site’s database. This made things pretty easy: you can search the Beast forum just like you would any database table.

Searching a non-integrated Beast forum

If you don’t have the Beast tables integrated in to your main site’s database, you can still search them. You just need to point the Topic and Post models to the correct database. This is a two step process.

First, set up a database entry for your Beast forum in config/database.yml. Something like this:

1
2
3
4
5
6

beast:
  adapter: mysql
  database: beast_forum
  username: app
  password: your_password
  host: localhost

Then, in the bottom of config/environment.rb, add the following lines:

1
2

Post.establish_connection "beast_forum"
Topic.establish_connection "beast_forum"

I haven’t actually tried this, so let me know if you get it working or if you needed to make any changes to what I’ve written here.

The Topic Model

Even with the integrated setup there were a few wrinkles. First, although the Beast tables are in the database, there are no models associated with them. I wanted to search post body and topic titles, so I created Topic and Post models.

Here’s the Topic model. It’s only here so that the Post model can search a posts’s titles, so there’s not much to it.

1
2
3
4
5

class Topic < ActiveRecord::Base
  
  has_many :posts
    
end

The Post Model

The Post model is a bit more complicated.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

class Post < ActiveRecord::Base
  
  belongs_to :topic
  
  SEARCH_FIELDS = [:scrubbed_body, :topic_title]
  
  acts_as_ferret :fields => { :scrubbed_body => {:boost => 0, :store => :yes}, 
                              :topic_title => {:boost => 3, :store => :yes}}  
  
  extend FullTextSearch
  
  # remove the html tags from body
  def scrubbed_body
    body_html.gsub(/<\/?[^>]*>/, "")
  end
  
  def topic_title
    topic.title
  end
      
  # Construct the url to the post    
  def url
    File.join("http://forums.ecolect.net", "forums", topic.forum_id.to_s, 
              "topics", topic.id.to_s + "#post_#{id}")
  end
  
end

acts_as_ferret

The acts_as_ferret declaration makes the Post model searchable. Notice that the actual fields being searched are not taken directly from the database; they are both manipulated in some way. acts_as_ferret doesn’t really care if the stuff it is indexing is coming directly from the database or from methods you have added to your model.

scrubbing the html tags

The post body is stored with HTML tags in them, so I wanted to search and show the posts with tags scrubbed out of them. This is done using the Post#scrubbed_body method, which is just an ugly regexp that takes out anything between < and > signs.

the url method

I also wanted to link to the posts, so I created a Post#url method which is used in the view.

Finally, the actual search is done using the FullTextSearch mixin, which adds a class method Post::full_text_search to Post. The FullTextSearch mixin is described in more detail below.

Searching the WordPress Blog

The only table from the WordPress Blog that you really care about is the wp_post table. To get access to it in your Rails app, make a WpPost model and point it at your WordPress db.

First, create a database entry in config/database.yml that looks like this:

1
2
3
4
5
6

wordpress:
  adapter: mysql
  database: wordpress
  username: app
  password: your_password
  host: localhost

WpPost.establish_connection "wordpress"

Here’s the model:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class WpPost < ActiveRecord::Base
  
  primary_key = "ID"
  
  SEARCH_FIELDS = [:scrubbed_title, :scrubbed_content]
                   
  acts_as_ferret :fields => { :scrubbed_title => {:boost => 3, :store => :yes},
                              :scrubbed_content => {:boost => 0, :store => :yes}}
  
  extend FullTextSearch
  
  def id
    read_attribute(:ID)
  end
  
  # title and content need to have the html tags removed from them, and should
  # only be searchable if the post has been assigned a url (the guid) and the post 
  # is actually a post and not an asset.
  def scrubbed_title
   post_title.gsub(/<\/?[^>]*>/, "") if post_type == "post" and !guid.empty?
  end
  
  def scrubbed_content
    post_content.gsub(/<\/?[^>]*>/, "") if post_type == "post" and !guid.empty?
  end
end

There are a few things to note here:

WordPress uses ID, rather than id, as its primary key. The line primary_key = "ID" lets Rails know about that. You also need to add an id method that returns ID to get ferret indexing things properly.

You will need to scrub the html tags from the content and title; that’s what the scrubbed_title and scrubbed_content methods do.

• Real posts will have a post_type of 'post'.
• An unpublished post won’t have its guid set.

This is taken care of by only returning titles or content if post_type == "post" and !guid.empty?.

The FullTextSearch mixin

This is based on code by Roman Mackovcak’s article on full text search in Rails. All I did was extract the method he provides out in to a mixin so I could use it in multiple models.

To use the mixin in a model, the model needs to define SEARCH_FIELDS and have an acts_as_ferret declaration. SEARCH_FIELDS is an array of symbols giving the model fields to be searched.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

module FullTextSearch
  
  ##
  # FERRET SEARCH METHOD
  # This method requires that you set the following in the model:
  # SEARCH_FIELDS: 
  # a list of symbols giving the fields to be searched.
  # E.g., SEARCH_FIELDS = [:post_title, :post_content]
  #
  # The acts_as_ferret declaration.  
  # Use :store => :yes for each field if you want to use highlighting for that field.
  # E.g., 
  # acts_as_ferret :fields => { :post_title => {:boost => 3, :store => :yes},
  #                             :post_content => {:boost => 0, :store => :yes}}
  ##
  DEFAULT_PER_PAGE = 10
  
  def full_text_search(q, options = {})
     return nil if q.nil? || q == ""
     
     default_options = {:limit => FullTextSearch::DEFAULT_PER_PAGE, 
                        :page => 1, 
                        :lazy => self.const_get(:SEARCH_FIELDS)
                       }
     options = default_options.merge options

     # Get the offset based on what page we're on
     options[:offset] = options[:limit] * (options.delete(:page).to_i-1)  
     # Now do the query with our options
     results = self.find_by_contents(q, options)

     return [results.total_hits, results]
  end  
  
  
end

1
2

WpPost.full_text_search('test')
WpPost.full_text_search('test', :limit => 30, :page => 2)

Re-indexing the non-local models

As Ruben pointed out in the comments, I forgot to mention how I deal with re-indexing the WordPress and Beast database tables. This is necessary as these tables have data that is modified by another application, so your Rails app doesn’t know that changes have been made to them.

To deal with this, I wrote a simple Rake task that reindexes the WpPost, WpArticle and Post models, and then added a line to the crontab to run it hourly. Here’s the rake file, which I put in RAILS_ROOT/lib/tasks/ferret.rake:

1
2
3
4
5
6
7
8
9
10
11

namespace :ferret do
  
  desc "rebuild the non-local Ferret indices, as these need to be done manually."
  task :rebuild_nonlocal_indices => :environment do
    [WpPost, WpArticle, Post].each do |model|
      puts "rebuilding #{model} index"
      model.rebuild_index
    end    
  end
  
end

and, for posterity, here’s the crontab line:

20 * * * *  cd <rails_root> && /usr/local/bin/rake ferret:rebuild_nonlocal_indices >> <log_directory>/ferret_reindex.log  2>&1

Here’s the final refactoring from the Rubyize This workshop. See the first refactoring for an explanation of what’s going on and why this code is so darn ugly! The second refactoring is worth checking out as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/usr/bin/env ruby

def delete_large_files(directory, max_size)
  
  # Make sure directory ends in a slash
  if directory !~ /\/$/
    directory += '/'
  end
  
  # Find all of the files in the directory 
  files = Dir.glob(directory + '*')
  
  # Delete all files with size larger than max_size
  for file in files
    size = File.size(file)
    if size >= max_size
      File.delete(file)
    end
  end
  
end

max_size = 1024 * 50 # 50 kb
directory = './files_to_delete'
delete_large_files(directory, max_size)

This was the final refactoring, and the end of the conference, so we sort of ran out of time.

I didn’t get a change to show off my solution. Here it is.

1
2
3
4
5
6
7
8
9

def delete_large_files(dir, max_size)

  files_to_delete = Dir.glob(File.join(dir, '*')).select do |file|
    File.size(file) > max_size
  end

  File.delete(*files_to_delete)

end

What’s going on with that File.delete call?

First, File.delete takes multiple arguments and deletes all of them.

Second, I used the asterisk operator. From here:

“The asterisk operator may also precede an Array argument in a method call. In this case the Array will be expanded and the values passed in as if they were separated by commas.”

I don’t think I’ve every actually used the asterisk operator in production code, but it sure came in handy here.

Here’s the second refactoring from the Rubyize This workshop. See the first refactoring for an explanation of what’s going on and why this code is so darn ugly! Don’t forget to check out the third and final refactoring

This script loads the file full of random numbers from the first refactoring and makes a beautiful ascii-art histogram from it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#!/usr/bin/env ruby

require 'pp'

BIG_NUMBER = 32768

# Get data from a file, turn it in to a float, and find the max
data = File.readlines('random.txt')
max = -BIG_NUMBER
for n in (0 ... data.length)
  data[n] = data[n].chomp.to_f
  if data[n] > max
    max = data[n].to_i + 1 # max is ceil(max(data[n]))
  end
end

# Create the empty histogram
histogram = []
for n in (0 .. max)
  histogram.push(0)
end

# Fill the histogram
for n in (0 ... data.length)
  histogram[data[n].to_i] += 1
end

# Print the histogram
pp histogram
puts
for n in (0 .. max)
  puts "*" * histogram[n]
end

Here’s a very concise one-liner from the crowd:

0.upto((data=File.readlines('random.txt').collect {|e| e.chomp!.to_f}).max.to_i) {|i| puts i.to_s + " " + data.select{|a| a==i}.size.to_s}

Here’s Owen’s refactoring:

1
2
3
4
5
6
7
8
9
10
11
12

require 'pp' 
histogram = [] 
File.readlines('random.txt').each do |value| 
  i = value.chomp.to_i 
  histogram[i] ||= 0 
  histogram[i] += 1 
end 

# Print the histogram 
pp histogram 
puts 
histogram.each { | v | puts "*" * v }

Here’s Sam Livingstone Gray’s refactoring

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#!/usr/bin/env ruby

require 'pp'

# Get data from a file, turn it in to a float, and find the max

module Enumerable
  def value_counts
    h = Hash.new(0)
    each { |e| h[e] += 1 } 
    h
  end
end

lines = File.readlines('random.txt')
histogram = lines.map { |line| line.chomp.to_i }.value_counts

# Print the histogram
pp histogram
puts
histogram.keys.sort.each { |n| puts '*' * histogram[n] }

I really like this one. Sam created a simple extension to Enumerable that I can see using over and over again.

Here’s what I came up with, with some debugging help from the group

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/usr/bin/env ruby

require 'pp'

data = File.readlines('random.txt').collect {|datum| datum.chomp.to_f}
max = data.max

histogram = data.inject([0] * (max + 1)) do |histogram, datum|
  histogram[datum.to_i] += 1
  histogram
end

pp histogram
histogram.each do |bucket|
  puts "*" * bucket
end

At rubycamp today, I ran a little Rubyize This workshop. Rubyize This is a game invented by François Lamontagne. The idea is that someone puts up a chunk of code that is written in Ruby, but in a not very Rubyish way. Then, everyone in the audience gets to Rubyize it!

We tried to use the most excellent RefactorMyCode.com, but the WiFi was too saturated so we had to resort to posting the code on my blog and people plugging their laptops in to the projector to present their results.

So, this post and the following two will show the original code. Keep in mind that this is intentionally ugly!

We had some great refactorings. If you were at the workshop, please post your refactorings in the comments. I’ll make the pretty and put them in the posting.

The other two refactorings are at Rubyize this #2 and Rubyize this #3

Without further ado, here’s the first refactoring. The code is supposed to create a file filled with random numbers, one number per line.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

#!/usr/bin/env ruby

# create num random numbers between min and max 
# and print them to a file called file_name, one per line
def create_random_numbers(file_name, min, max, num)
  file = File.open(file_name, 'w')
  n = 0
  while n < num
    r = rand(max - min) + min
    file.puts(r)
    n += 1
  end
  file.close   
end

create_random_numbers('random.txt', 0, 10, 1000)

1
2
3
4
5

def create_random_numbers(file_name, min, max, num) 
  File.open(file_name, 'w') do |f| 
    1.upto(num) {|i| f.puts rand(max-min)+min }
  end 
end

Here’s one from Sam Livingston-Gray

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env ruby

# create num random numbers between min and max 
# and print them to a file called file_name, one per line

class Range
  def random_member
    offset = rand(max - min)
    min + offset
  end
end

def create_random_numbers(file_name, range, num)
  File.open(file_name, 'w') do |file|
    num.times { file.puts(range.random_member) }
  end
end

create_random_numbers('random.txt', (0..10), 1000)

Here’s what I came up with:

1
2
3
4
5
6
7

def create_random_numbers(file_name, min, max, num)
  File.open(file_name, 'w') do |file|
    num.times {file.puts(rand(max - min) + min)}
  end
end

create_random_numbers('random.txt', 0, 10, 1000)

format.json  { render :json => @bookmark }

Now, the easiest way to do this would be to add JSON response to the scaffolding. The templates are in RAILS_ROOT/vendor/rails/lib/rails_generator/generators/components/scaffold/. Open up the controller.rb template and add in the JSON response shown above to the index, show, update and destroy methods and you’re done.

Unfortunately, I had already mostly developed the API when I realized that I had to add in the JSON responses (It’s not Gerald’s fault, he told me long ago). So, I decided to write a quick rake task to do the job for me. To use it, create a file called add_json.rake in RAILS_ROOT/lib/tasks and copy the following code in to it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

namespace :rails do
  
  desc "Adds a 'format.json ...' line for each 'format.xml ...' line in every controller"
  task :add_json => :get_controller_list do
    @controllers.each do |controller|
      new_lines = File.open(controller, 'r').readlines.collect do |line|
        if line =~ /^\s+format\.xml/
          [line, line.gsub(/xml/, 'json')]
        else
          line
        end
      end.flatten
        
      file = File.open(controller, 'w') do |file|
        file.puts new_lines
      end
    end
  end
  
  task :get_controller_list => :environment do
    @controllers = Dir.glob(File.join(RAILS_ROOT, 'app', 'controllers', '*_controller.rb'))
  end
end

Call the task from RAILS_ROOT like this: rake rails:add_json

The task opens up every controller, and every time it finds a line that starts with format.xml, it makes a copy of that line just below with every instance of ‘xml’ replaced with ‘json’. It’s pretty very brain-dead: it doesn’t check if the json response is already there, so don’t run it twice on the same project.

I’m working on the sample application for the Intermediate Ruby on Rails Workshop that Gerald Bauer and I are putting on in January. I had to remind myself of some of the foibles of Single Table Inheritance (STI). I thought others might find this useful, so here’s what I found.

The Set-up

I have two types of posts in the application I’m building: posts by people with an empty room looking for a roomie (RoomiePosts), and posts by people looking for a room (RoomPosts). They both share a lot of the same attributes, so it’s not very DRY to create two separate models.

This is where Single Table Inheritance comes in. STI is basically a way of subclassing a single model, creating subclasses that all use the same database table.

What I did is create a Post model and two subclasses: RoomiePost and RoomPost. Here’s how you do it.

Creating the Database Table

Here’s the migration for the posts database table. Note that this uses the new migrations syntax in Rails 2.0.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.integer :rent
      t.date :start_date, :expiry_date
      t.boolean :includes_utilities
      t.string :text, :type
      
      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

Most of the columns are pretty straightforward. text will contain the text of the post. start_date and expiry_date will determine when the post will start and stop being shown on the site. rent and includes_utilities will only be used by the RoomiePost subclass, but still need to be created for all subclasses of the Post model.

The interesting column is type. This allows Rails to determine which subclass the row should be loaded in to. If you set type = "RoomiePost", then it’s a RoomiePost. If you set type = "RoomPost", then it’s a RoomPost.

Creating the Models

You need to create a Post, RoomiePost and RoomPost model. They look like this:

1
2
3

class Post < ActiveRecord::Base 
  
end

1
2
3

class RoomiePost < Post
  
end

1
2
3

class RoomPost < Post
  
end

Note that RoomiePost and RoomPost are both subclasses of the Post model. Here’s where you run in to a “gotcha”.

You have to create the models in separate files. If you don’t, Rails ignores the subclassed models.

For more information, and another workaround, see this blog post. The comment by Chris at the bottom gives the ‘create the models in separate files’ solution.

Okay, so now you have the models all set up. You can do things like

>> p = RoomiePost.create(:rent => 800, :includes_utilities => true, :text => "Great room in a shared house in the Commercial Drive area") 

And it will create a row in the posts table with type = "RoomiePost". If you have 5 RoomiePosts and 2 RoomPosts, then Posts.count will return 7.

1
2
3
4
5
6

>> RoomiePost.count           
=> 5
>> RoomPost.count
=> 2
>> Post.count
=> 7

Adding behaviour to STI models

I want to be able to determine if a post is active by looking at the start and expiry dates, and to get a list of all active posts, roomie_posts and room_posts. I also want to have a different icon for each post type.

The Post.active_post? and Post::active_posts methods are straight-forward:

1
2
3
4
5
6
7
8
9
10
11

class Post < ActiveRecord::Base
  
  def is_active?
    (start_date .. expiry_date) === DateTime.now
  end 
  
  def self.active_posts
    self.find(:all).select {|p| p.is_active? }
  end  
  
end

The only tricky part is that I used self.find(:all) instead of Post.find(:all) in the Post::active_posts method. This makes sure that RoomiePost::active_posts only returns RoomiePosts and RoomPost::active_posts only returns RoomPosts.

Getting the icon to work took a little more figuring out. If I wanted to do this in straight-up Ruby, I’d do something like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/usr/bin/env ruby

class Post
  ICON_PATH = "icons"
  
  def initialize
    @icon_name = 'base.gif'
  end
  
  def icon
    File.join(ICON_PATH, @icon_name)
  end
  
end

class RoomiePost < Post
  
  def initialize
    @icon_name = 'roomie.gif'
  end
  
end

class RoomPost < Post
  
  def initialize
    @icon_name = 'room.gif'
  end
  
end

Then I could do this

1
2
3
4
5
6
7

ruby $>irb -r subclass_test
irb(main):001:0> Post.new.icon
=> "icons/base.gif"
irb(main):002:0> RoomiePost.new.icon
=> "icons/roomie.gif"
irb(main):003:0> RoomPost.new.icon
=> "icons/room.gif"

This doesn’t quite work in the Rails version. Not to worry, you only have to make one simple change:

replace the initialize method with an after_initialize callback.

The final post.rb, roomie_post.rb and room_post.rb files look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class Post < ActiveRecord::Base

  ICON_PATH = "/images/icons"
  
  def after_initialize
    @icon_name = 'base.gif'
  end
  
  def icon
    File.join(ICON_PATH, @icon_name)
  end  
  
  def is_active?
    (start_date .. expiry_date) === DateTime.now
  end 
  
  def self.active_posts
    self.find(:all).select {|p| p.is_active? }
  end  
  
end

1
2
3
4
5
6
7

class RoomiePost < Post
  
  def after_initialize
    @icon_name = 'roomie.gif'
  end
  
end

1
2
3
4
5
6
7

class RoomPost < Post
  
  def after_initialize
    @icon_name = 'room.gif'
  end
  
end

One final note: because of the way the after_initialize callback works, you have to actually define an after_initialize method. You can’t do something like this:

1
2
3
4
5
6
7
8
9
10

class RoomiePost < Post
  after_initialize :set_icon_name

  private

  def set_icon_name
    @icon_name = 'roomie.gif'
  end
  
end

If you want to learn more about STI, and a whole slew of other info on Rails, sign up for the Intermediate Rails Workshop on January 25th, 2008 in Vancouver.

Early Bird rates are available until January 9th!

RubyCamp is a free one-day gathering for Rubyists and Railsers.

When and Where

WorkSpace in downtown Vancouver, B.C., Canada
January 26th, 2008 from 9:00 to 5:00

Who should come

Anyone who’s interested in Ruby and Rails, whether you’re just interested in learning what this Ruby thing is all about or you know Ruby inside out.

The Conference Track

A conference-style track with “classic” talks on Ruby or Rails topics. We’re looking for a few more speakers, so see here if you’re interested in giving a talk.

The Hackathon Track

An informal un-conference track focusing on hacking some Ruby code, showing off a cool feature you just added to your Rails application or demonstrating a new addition to Rails 2.0.

If you’re interested in working on some code or showing off something, get started by promoting your ideas and getting some buzz going .

A weekend of Ruby

There are two more reasons to come to Vancouver for those of you who are out of town.

On the Friday before RubyCamp, Rails Advance is giving a one day intermediate Ruby and Rails workshop. (Rails Advance is Gerald Bauer and I, by the way)

On the Sunday after RubyCamp, Peter Armstrong is giving a one day workshop on Flex and Rails.

For more information, see the RubyCamp web site

See you all at RubyCamp in Vancouver!

The slides from the talk are here. (Yes, they’re hosted on S3).

There are two points in the presentation where I switched to a different window.

At the ‘S3SH DEMO’ slide, I did some live coding showing how you can work with S3 using s3sh. It basically followed the script shown in ‘s3sh demo script’ below, so read that part when you see the ‘S3SH DEMO’ slide.

At the ‘Example: S3Syncer’ slide, I switched over to textmate and showed the code for a simple script to synchronize a single directory to S3. I then demoed the script to show it working. So, when you see the ‘Example; S3Syncer’ slide, read the s3syncer code and s3syncer demo sections below.

s3sh demo script

$> s3sh

Create a bucket. Show that you can create a bucket multiple times if you own it, but trying to create a bucket that somebody else owns raises an error.

>> Bucket.create('spatten_s3demo')
=> true
>> Bucket.create('spatten_s3demo')
=> true
>> Bucket.create('test')
AWS::S3::BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/error.rb:38:in `raise'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:72:in `request'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:83:in `put'
        from /usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/bucket.rb:79:in `create'
        from (irb):3

>> b = Bucket.find('spatten_s3demo')
=> #<AWS::S3::Bucket:0x14ae7b8 @attributes={"prefix"=>nil, "name"=>"spatten_s3demo", "marker"=>nil, "max_keys"=>1000, "is_truncated"=>false, "xmlns"=>"http://s3.amazonaws.com/doc/2006-03-01/"}, @object_cache=[]>

>> S3Object.store('test.txt', 'This is a test', 'spatten_s3demo')
=> #<AWS::S3::S3Object::Response:0x10830590 200 OK>
>> b.objects
=> [#<AWS::S3::S3Object:0x10804170 '/spatten_s3demo/test.txt'>]
>> pp b.objects[0].about
{"last-modified"=>"Wed, 05 Dec 2007 19:56:49 GMT",
 "x-amz-id-2"=>
  "JACm9T+m9CgZhmj4q6q00OSGHgSyBVAbQ1cgRWGydYZLTKdhLc/IUZ+K7b/1snOc",
 "content-type"=>"text/plain",
 "etag"=>"\"ce114e4501d2f4e2dcea3e17b546f339\"",
 "date"=>"Wed, 05 Dec 2007 19:57:03 GMT",
 "x-amz-request-id"=>"CA170D2AA5DEB0C9",
 "server"=>"AmazonS3",
 "content-length"=>"14"}
=> nil
>> b.objects[0].key
=> "test.txt" 
>> b.objects[0].value
=> "This is a test" 

>> S3Object.store('vampire.jpg', File.open('vampire.jpg'), 'spatten_s3demo')
=> #<AWS::S3::S3Object::Response:0x10764700 200 OK>

Show the photo in browser

This doesn’t work, as the file is only readable by me. Make it public readable and do it again.

>> S3Object.store('vampire.jpg', File.open('vampire.jpg'), 'spatten_s3demo', 
     :access => :public_read)
=> #<AWS::S3::S3Object::Response:0x10747950 200 OK>

Show it in a browser again. It works this time.

Look at bucket.objects. We have to reload the bucket to show the new object.

>> b.objects
=> [#<AWS::S3::S3Object:0x10804170 '/spatten_s3demo/test.txt'>]
>> b.objects(:reload)
=> [#<AWS::S3::S3Object:0x10708080 '/spatten_s3demo/test.txt'>, #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>]

>> b['vampire.jpg']
=> #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>
>> vamp = b['vampire.jpg']
=> #<AWS::S3::S3Object:0x10708070 '/spatten_s3demo/vampire.jpg'>

A look at metadata

>> vamp.content_type
=> "image/jpeg" 
>> vamp.size
=> 10817
>> vamp.metadata
=> {}
>> vamp.metadata['subject'] = 'Claire'
=> "Claire" 
>> vamp.metadata['photographer'] = 'Nadine Inkster'
=> "Nadine Inkster" 
>> vamp.store
=> true

>> picdata = vamp.value
=> "\377\330\377\340\000\020JFIF\000\001\002\000.......

Downloading a picture by streaming it to an IO object.

>> File.open('vampire_downloaded.jpg', 'w') {|file| file.write(vamp.value)}
=> 10817
>> exit
s3demo $>ls
flowers.jpg             vampire.jpg
test.txt                vampire_downloaded.jpg
s3demo $>open vampire_downloaded.jpg 
s3demo $>

S3Syncer Code

Please note that this code is really only useful as an example of how to synchronize with S3.

It won’t recurse directories and it dies a horrible death if there are any symlinked files in a directory.

If you are looking for something to synchronize directories, check out s3sync.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

#!/usr/bin/env ruby

require 'digest/md5'
require 'aws/s3'
include AWS::S3

class S3Syncer
  attr_reader :local_files, :files_to_upload
  
  def initialize(directory, bucket_name)
    @directory = directory
    @bucket_name = bucket_name
  end
  
  def S3Syncer.sync(directory, bucket)
    syncer = S3Syncer.new(directory, bucket)
    syncer.get_local_files
    syncer.connect_to_s3
    syncer.get_bucket
    syncer.select_files_to_upload
    syncer.sync
  end
  
  # This does not recurse directories.
  def get_local_files
    @local_files = Dir.entries(@directory)
  end
  
  def connect_to_s3
    Base.establish_connection!(
        :access_key_id     => ENV['AMAZON_ACCESS_KEY_ID'],
        :secret_access_key => ENV['AMAZON_SECRET_ACCESS_KEY']
      )
  
    raise "\nERROR: Connection not made or bad access key " +
          "or bad secret access key.  Exiting" unless AWS::S3::Base.connected? 
  end  
  
  def get_bucket
    Bucket.create(@bucket_name)
    @bucket = Bucket.find(@bucket_name) 
  end
  
  # Files should be uploaded if 
  #   The file doesn't exist in the bucket
  #      OR
  #   The MD5 hashes don't match
  def select_files_to_upload
    @files_to_upload = @local_files.select do |file|                 
      case
      when File.directory?(local_name(file))
         false # Don't upload directories
      when !@bucket[file]
         true  # Upload if file does not exist on S3
      when @bucket[file].etag != Digest::MD5.hexdigest(File.read(local_name(file)))
         true  # Upload if MD5 sums don't match
      else
        false  # the MD5 matches and it exists already, so don't upload it
      end
    end
  end
  
  # This will choke on symlinked files
  def sync
    (puts "Directories are in sync"; return) if @files_to_upload.empty?

    @files_to_upload.each do |file|
      puts "#{file} ===> #{@bucket.name}:#{file}"
      S3Object.store(file, File.open(local_name(file), 'r'), @bucket_name)      
    end
  end
  
  private 
  
  def local_name(file)
    File.join(@directory, file)
  end
  
end

if __FILE__ == $0
  S3Syncer.sync('/Users/Scott/versioned/spattendesign/presentations/s3-on-rails/s3demo', 
                'spatten_syncdemo')
end

S3Syncer demo

Start with spatten_syncdemo bucket empty, and four files in the local directory.

s3demo $>ls
flowers.jpg             vampire.jpg
test.txt                vampire_downloaded.jpg
s3demo $>s3syncer
flowers.jpg ===> spatten_syncdemo:flowers.jpg
test.txt ===> spatten_syncdemo:test.txt
vampire.jpg ===> spatten_syncdemo:vampire.jpg
vampire_downloaded.jpg ===> spatten_syncdemo:vampire_downloaded.jpg

s3demo $>s3syncer
Directories are in sync

s3demo $> vi test.txt
Make some changes using vi
s3demo $>s3syncer
test.txt ===> spatten_syncdemo:test.txt

s3demo $>s3syncer
flowers.jpg ===> spatten_syncdemo:flowers.jpg

So there you go, a quick intro to the wonders of Amazon S3.

I’ve re-written the plugin to use s3sync.rb rather than trying to figure out which files have changed in svn. It’s a much better solution, so hopefully those of you who were having trouble will find it works for you now. Thanks, especially, to Chris in the comments for helping me figure out what was going wrong.

The plug-in also now has a permanent home in my new Projects section, here.

I’m going to be giving a talk on Amazon S3, Ruby and Rails at the next Vancouver Ruby and Rails meetup. It’s this Monday, December 3rd.

Here’s the talk teaser:

Amazon’s Simple Storage Service (Amazon S3) is an online storage system. It can be used for backup, serving assets for your Rails application, storing and streaming large media files or storing customer generated assets.

The AWS/S3 gem, written by Marcel Molina, provides an elegant interface to Amazon S3. I’ll be talking about how S3 works, how to use it in your Ruby scripts and Rails applications and showing off some example code.

Hope to see you there!

Note: This article is out of date. The latest version of this article is on the new permanent page in the projects section.

So you’ve got multiple asset hosts running in your Rails application, and you’re using Amazon’s S3 to host your assets. Now you want to make sure that your assets are kept up to date. This plugin is a Capistrano recipe that keeps the asset hosts synchronized with the public directory in your subversion repository.

Usage

After you get everything setup and do your first deploy, just run cap deploy as normal and all changed files in RAILS_ROOT/public will be uploaded to all of your asset host buckets before the final deploy:symlink task.

The following tasks are also available:

• cap s3_asset_host:get_s3_revision
• cap s3_asset_host:find_changed
• cap s3_asset_host:list_changed
• cap s3_asset_host:find_all
• cap s3_asset_host:upload_changed
• cap s3_asset_host:upload_all
• cap s3_asset_host:upload
• cap s3_asset_host:reset_and_upload
• cap s3_asset_host:setup
• cap s3_asset_host:create_buckets
• cap s3_asset_host:delete_all
• cap s3_asset_host:connect

You can get documentation on these tasks by running cap -T

Requirements

This plug-in is a Capistrano extension. It requires Capistrano 2.0.0 or greater.

You will also require the aws-s3 gem

• assumes that you are using the ‘checkout’ method of deployment.
• only works with svn.

If you are using another version control system, I think all you’ll have to change is the two methods in lib/scm.rb. If you do get something other than svn working, please let me know.

If you want to use more than one asset host, then you have to either install the multiple asset hosts plugin or upgrade to Rails 2.0 (see setting up multiple asset hosts in Rails)

Setup

To set-up, you need to do the following

• Install the plug-in
• Install the AWS-S3 gem.
• Set up your Rails application to use asset hosts.
• Set up your asset hosts.
• Configure Capistrano.

Installing the plug-in

From RAILS_ROOT, run:

script/plugin install svn://svn.spattendesign.com/svn/plugins/synch_s3_asset_host

Installing the AWS-S3 gem

You need to do this on both your local computer and the computer that is defined as the asset_host_syncher (see Capistrano Configuration, below).

$> sudo gem install aws-s3

Setting up your Rails app to use asset hosts

Single asset host

For a single asset host, simply add the following line to RAILS_ROOT/config/environments/production.rb:

config.action_controller.asset_host = "http://assets.example.com"

Multiple asset hosts

Follow the instructions in setting up multiple asset hosts in Rails

Setting up your asset hosts

Set up a CNAME entry for each asset host pointing to s3.amazonaws.com. How you do this depends on your domain host. Here’s what it looks like on easydns

You may need to wait up to 24 hours for the DNS entries for these new hosts to propagate.

Configuring Capistrano

Capistrano installation

This plugin requires Capistrano 2.0.0 or greater.

To upgrade to the latest version (currently 2.1.0):

$> gem install capistrano

Once the plug-in is installed, make sure that the recipes are seen by Capistrano

$> cap -T | grep s3_asset_host

should return a bunch of tasks. If you don’t see anything listed, then you need to update your Capfile by doing the following (this is from Jamis Buck):

$> cd RAILS_ROOT
$> rm Capfile
$> capify .

If you do not want to delete your Capify file, or if you are using Capistrano 2.0.0, add the following line to your Capify file:
Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

Capistrano configuration

Create a new file in RAILS_ROOT/config called synch_s3_asset_host.rb. Add the following lines to it, and edit to suit:

# =============================================================================
# S3 ASSET HOST OPTIONS
# =============================================================================
set :asset_host_name, "assets%d.example.com" 
set :aws_access_key, "your Amazon AWS access key" # You can also set this in your environment as AMAZON_ACCESS_KEY_ID
set :amazon_secret_access_key, "your Amazon AWS secret" # You can also set this in your environment as AMAZON_SECRET_ACCESS_KEY
# set :dry_run, false# Set to true if you want to test the asset_host uploading without doing anything on Amazon S3
before "deploy:symlink", "s3_asset_host:upload_changed" 

You have to do one more thing: in RAILS_ROOT/config/deploy.rb. Specify one of your web hosts as an “asset_host_syncher”, like this:

role :web, webserver1, :asset_host_syncher => true

The first deploy

Commit all changes to your rails application and do the initial bucket setup:

$> cap s3_asset_host:setup
$> svn commit -m "Adding synch_s3_asset_host plugin" 
$> cap deploy

• Create your Amazon S3 AWS buckets
• upload everything in RAILS_ROOT/public (in your svn repository) to each bucket
• Set the revision in each bucket to the latest revision in your repository.

This could take a while if you have lots of images or other big files.

You’re done!

That should do it. Now, every time you run cap deploy, your asset hosts should be updated with any changes to files in RAILS_ROOT/public.

Let me know if you have any problems, suggestions or comments.

One of the nice goodies coming in Rails 2.0 is the ability to use multiple asset hosts. This article explains how you can use this feature now, without waiting for Rails 2.0, and why you would want to use asset hosts.

What’s an asset host, and why would I want to use it?

An asset host is another server, somewhere on the internet, where you store your static files. These can be javascripts scripts, CSS stylesheets, images, static html files and anything else that doesn’t change often. Basically, anything that lives in your public directory. You can then use a link to the asset host whenever you want to include an image, javascript or stylesheet. For example, Plot-O-Matic stores its assets at assets0.plotomatic.com — assets3.plotomatic.com. You can see the logo here: assets0.plotomatic.com/images/o-logo.png.

So, why would you want to use an asset host? It turns out that many browsers limit the number of simultaneous connections to a host. For Internet Explorer, that number is two. If you are serving a lot of small images, or you haven’t bothered to bundle your scripts or stylesheets, this can be a real bottleneck. Asset hosting allows you to increase the number of hosts a web page is loaded from, removing the bottleneck.

Asset hosting in Rails

config.action_controller.asset_host = "http://assets.plotomatic.com"

<%= image_tag 'o-logo.png' %>

<img alt="o-logo" src="http://assets.plotomatic.com/images/o-logo.png?1193172652" />

Asset tag helpers are things like image_tag, stylesheet_link_tag or javascript_include_tag. Links made without asset tag helpers will not automatically use your asset host.

Multiple asset hosts

Now, having a single asset host is nice, but using multiple asset hosts can really speed up those page load times. Here’s how you do it:

Get Rails ready for multiple asset hosts

There are two ways you can do this.

Upgrade to Rails 2.0

For instructions, scroll to the bottom of this post.

Install the multiple_asset_hosts plugin

script/plugin install svn://svn.spattendesign.com/svn/plugins/multiple_asset_hosts

All this plugin does is monkeypatch in the multiple asset host functionality from Rails Edge.

Pick a name for your asset hosts

The change from single to multiple asset hosts is simple. You simply put a %d somewhere in your asset host name in production.rb. For Plot-O-Matic, the setting is

config.action_controller.asset_host = "http://assets%d.plotomatic.com"

Rails will replace the %d with either 0, 1, 2 or 3.

Set up your asset hosts

You need to provide four asset hosts, so I set up four asset hosts named assets0.plotomatic.com, assets1.plotomatic.com, assets2.plotomatic.com and assets3.plotomatic.com.

I used Amazon S3 for my asset hosts, which was simple:

1. set up four buckets called assets0.plotomatic.com, assets1.plotomatic.com, assets2.plotomatic.com and assets3.plotomatic.com
2. Set up a CNAME entry for each asset host pointing to s3.amazonaws.com. How you do this depends on your domain host. Here’s what it looks like on easydns

Deploy it

• Commit your changes to your version control system
• Upload everything in your public directory to each of your asset hosts.
• Deploy and restart your web server

A simple Capistrano 2.0 recipe, installable as a plugin, that runs all of your tests before the final symlink test.

Synopsis

This plug-in is a Capistrano extension. It requires Capistrano 2.0.0 or greater.

Once installed, running cap deploy will run all of your tests before doing the final symlink.

If the tests fail, the symlink will not be created and your deployment will roll back.

to deploy without tests,

cap deploy:without_tests

Installation

Plugin installation

You should be able to install with the following command (from rails root):

script/plugin install run_tests_on_deploy

If that doesn’t work, try

script/plugin install svn://svn.spattendesign.com/svn/plugins/run_tests_on_deploy

If that doesn’t work, send me an e-mail at scott@spattendesign.com

Capistrano configuration

This plugin requires Capistrano 2.0.0 or greater. To upgrade to the latest version (currently 2.1.0)

gem install capistrano

Once the plug-in is installed, make sure that the recipes are seen by Capistrano

cap -T | grep deploy:without_tests

should return

cap deploy:without_tests # deploy without running tests

If capistrano is not seeing the deploy:without_tests task, then you need to update your Capfile.

(The following is from a post by Jamis Buck)

In Capistrano 2.1.0 or above: you can delete your Capify file in rails root, and then, from rails root, run

capify .

If you do not want to delete your Capify file, or if you are using Capistrano 2.0.0, add the following line to your Capify file:

Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

Details

• prepares your test db with rake db:test:prepare
• runs all of your tests with rake test at a nice level of -10

The deploy:run_tests task won’t work if you call it by itself, as it runs from the release_path directory, which won’t exist unless called after deploy:update_code.

to deploy without tests, cap deploy:without_tests

you can also set the run_tests option to 0 from the command line like this:

cap -S run_tests=0 deploy

this allows you to do things like deploying with migrations but without tests:

cap -S run_tests=0 deploy:migrations

Genesis

The original idea for the :run_tests task is from the testing rails blog, which, sadly, seems to be defunct.

Here’s a simple little Capistrano 2.0 recipe to update your rdig search index after every deploy.

Add the following lines to config/deploy.rb

1
2
3
4
5
6
7
8

namespace :deploy do 
  after :deploy, "deploy:update_rdig"

  desc "Update the rdig database.  This has to be done after re-starting, as it looks at the live web-site"
  task :update_rdig do
    run "cd #{current_path} && rdig -c config/rdig_config.rb"
  end
end

This will run automatically after every deploy. If you just want to run it manually, comment out the line that starts with after .... You can then run it manually with cap deploy:update_rdig

Here are a few tips and tricks for unit tests

To find all of the command line options for the test::unit suite, call a unit test with --help:

ruby some_test.rb --help

The output will look something like this

~/tests $>ruby some_test.rb --help
Test::Unit automatic runner.
Usage: test/unit/graph_test.rb [options] [-- untouched arguments]

    -r, --runner=RUNNER              Use the given RUNNER.
                                     (c[onsole], f[ox], g[tk], g[tk]2, t[k])
    -n, --name=NAME                  Runs tests matching NAME.
                                     (patterns may be used).
    -t, --testcase=TESTCASE          Runs tests in TestCases matching TESTCASE.
                                     (patterns may be used).
    -v, --verbose=[LEVEL]            Set the output level (default is verbose).
                                     (s[ilent], p[rogress], n[ormal], v[erbose])        --                           Stop processing options so that the
                                     remaining options will be passed to the
                                     test.
    -h, --help                       Display this help.

Deprecated options:
        --console                    Console runner (use --runner).
        --gtk                        GTK runner (use --runner).
        --fox                        Fox runner (use --runner).

Running only some tests

The -n option is my favourite. You can use it like this to call a single test by giving it the name of a test method:

ruby some_test.rb -n test_something_very_important

Even better, you can use a regular expression and all of the matching tests will be run. For example, if I had a graph model that tested the creation of .png, .eps and .gif files with methods called test_create_png_file, test_create_eps_file and test_create_gif_file, I could run them all with the command

ruby test/unit/graph_test.rb -n /_create_.*_file/

you can also use the -n option multiple times. So, if I just wanted to test eps and png file creation, I could use

ruby test/unit/graph_test.rb -n test_create_eps_file -n test_create_png_file

The same thing applies to selecting test cases using the -t option.

Cranking up the verbosity

Sometimes you want to see what tests are failing without waiting for all of the tests to run. Use the -v or --verbose options to do this.

ruby test/unit/<some_test>_test.rb -v v

or

ruby test/unit/<some_test>_test.rb --verbose=verbose

Then, instead of just the boring old periods appearing as each test passes

~/rails/grapher/trunk/vendor/plugins/ploticus $>ruby test/ploticus_test.rb 
Loaded suite test/ploticus_test
Started
.................
Finished in 0.10768 seconds.

17 tests, 118 assertions, 0 failures, 0 errors

you’ll see this instead:

~/rails/grapher/trunk/vendor/plugins/ploticus $>ruby test/ploticus_test.rb -v v
Loaded suite test/ploticus_test
Started
test_data_from_columns(PloticusTest): .
test_data_from_columns_with_ragged_data(PloticusTest): .
test_data_from_hash(PloticusTest): .
test_data_from_hash_with_nils(PloticusTest): .
test_data_from_hash_with_ragged_data(PloticusTest): .
test_data_from_rows(PloticusTest): .
test_first_proc_by_name(PloticusTest): .
test_has_proc(PloticusTest): .
test_num_procs_by_name(PloticusTest): .
test_pad_to(PloticusTest): .
test_ploticus_proc_has_line(PloticusTest): .
test_ploticus_proc_line(PloticusTest): .
test_png_graph_creation(PloticusTest): .
test_proc_with_colons_in_it(PloticusTest): .
test_procs_are_enumerable(PloticusTest): .
test_procs_by_name(PloticusTest): .
test_svg_graph_creation(PloticusTest): .

Finished in 0.042398 seconds.

17 tests, 118 assertions, 0 failures, 0 errors

Obviously you don’t want to do this all of the time, but it can come in handy.

Running tests with a GUI

If you get bored with the command line, try using the “runner” option. The tk option worked for me without having to install anything:

ruby some_test.rb -r tk

You’ll see something like this:

Other valid options are console, fox, gtk, gtk2, tk.

You can mix the -n and -t options as well if you want

ruby test/unit/graph_test.rb -r t -n /downloadable/