File Transfers
If you happen to develop a platform that will have to support many file transfers there are a few things you could do to speed them up and go easier on your Rails application.
I won’t cover the network part nor I’ll go into underground details - I’ll save it for another time. I’ll just give out some quick notes for all Rails developers that deal with file transfers.
If you’re one of the developers of rapidshare or megaupload (or any other massive file sharing system) you probably already know everything I’ll state here. If you’re not one of these file transfer gurus, make sure to keep reading.
Downloading
Serving files to your website clients is a useful feature in many kinds of services. There’s a problem, though: If you’re using send_file, which you probably are, this useful method will make your Rails process deal with reading the file from the disk and sending it to the user creating an extra, unneeded, overhead.
If you’re using mongrel, you’ll notice that the :stream option doesn’t seem to be doing anything in terms of memory usage and your mongrel process doesn’t free the memory right after it finished sending the file.
There’s a simple trick in order to keep your Rails process from reading the file, making it the webserver’s job - it’s called X-Sendfile. It consists in a HTTP header that instructs your webserver to read and serve the file directly without bothering your Rails server. From mod_xsendfile page we can read:
“If it encounters the presence of such header it will discard all output and send the file specified by that header instead using Apache internals including all optimizations like caching-headers and sendfile or mmap if configured.”
Although Mongrel natively supports this you’ll probably need to tweak your webserver if it’s not this one. If it’s apache have a look at mod_xsendfile stated above. In case your using Lighthttpd you should know that mod_proxy_core supports X-Senfile after a small configuration. Finally, if you’re using nginx you should know that it has it’s own implementation of X-Sendfile, called X-Accel-Redirect. In case you’re using any other webserver you’re probably skilled enough to figure out how to get this working.
Right now, your code might look something like this:
send_file "/path/to/file", :filename => "filename"
In order to use X-Sendfile in the transfer, you’ll just need to add the x_sendfile option, like this:
send_file "/path/to/file", :filename => "filename", :x_sendfile => true
And it’s done! If you’re using nginx, though, you might want to have a look at my X-Accel-Redirect Rails plugin. It supports all the options that send_file does and uses this feature provided by nginx. The downside is that you’ll have to change your send_file calls to something like:
x_accel_redirect "/path/to/file", :filename => "filename"
Do not include the :x_sendfile option on this call or you’ll be mixing headers and weird things may happen - the plugin will do everything else for you.
The nginx plugin is a bit fresh. In a few days I’m planning to add some real documentation. Sit tight and keep an eye on the github page if it interests you.
Update: I’ve updated the plugin documentation at github. You can now completely override Rails send_file with x_accel_redirect, making things even more transparent (see the documentation for details on how to do this).
Uploading
Does your website allow users to upload files? Either photos, documents or large files, there’s a way to go easier on your Rails application - it’s called ModPorter. I won’t describe what it does since it’s better done at their website:
“Porter is designed to make supporting large file uploads as simple as possible. By doing all the heavy lifting in the web server, your application processes are left free to serve user requests.”
It comes with an apache module and a Rails plugin and everything is clearly explained in their webpage - the setup it quite simple since it only requires a few configuration. The only possibly tricky part is that you must refactor your upload handler code. Something like this:
# files_controller.rb
@file = File.new
if @file.uploaded_file params[:file][:file] and @file.save
# file.rb
def uploaded_file incoming_file
self.name = incoming_file.original_filename
self.size = incoming_file.size
File.open("/path/where/its/going/to/be", "wb") do |f|
f.write incoming_file.read
end
end
Would have to be converted to something like this:
# files_controller.rb @file = File.new if @file.uploaded_file params[:file][:file] and @file.save # file.rb def uploaded_file incoming_file self.name = incoming_file.original_filename self.size = incoming_file.size FileUtils.cp(incoming_file.path, "/path/where/its/going/to/be") end
This changes are needed because your application no longer has to deal with the upload itself - ModPorter does it for you!
If you’re using Phusion Passenger, though, you won’t need to worry about any of this since when version 2.0 has a cool feature - it detects large file uploads and stores them in a temporary file. Only after finishing uploading the file, will it be forwarded to the Rails application - just like ModPorter’s behavior.
