Technology Musings
July 05, 2009
Platforms / Migrating to Phusion Passenger
Spent the evening migrating one of my Ruby on Rails servers to Phusion Passenger. It was a nice upgrade, but not the trouble-free upgrade they promise. Here were the gotchas that I ran into:
- Remember to install all of the gems into your new enterprise ruby that were on your regular ruby. This one is pretty obvious, but its easy to miss some
- Passenger preloads your application. This means that if there were models that you never used (and thus never got loaded into Rails before), Passenger will load them for you, and if they are broken, your app will not start. This also affected gem loading, as there was one gem which was loaded by a file that was unused, and therefore I didn't have it installed. But Passenger required that I install the gem.
- Passenger basically works by default as a SetUID program. It looks like whatever user owns the application is the user that Passenger will run it as. This can cause all sorts of problems. It can cause problems accessing log files and session files. When I finally got my gem issues sorted out, I kept getting 500 errors and I found the following in my log file: file /tmp/ruby_sess.97c076be86e7dd90 not readable - This was because it was running as my development user rather than as Apache. I had to use BOTH of the directives "PassengerUserSwitching off" and "PassengerDefaultUser apache" to get it to work.
- Then I had to find and fix all of the badly-created permissions that had accrued while it was running as the wrong user
June 26, 2009
Snippets / What's Taking So Long?
There are a lot of performance analysis tools for Rails logs. However, sometimes you just need something quick and dirty. This one helped me out. 1-line perl script to show all Rails log lines which took more that 100milliseconds to produce:
perl -n -e 'print if(m/\((\d+\.\d+)ms\)/ && $1 > 100);' log/development.log
UPDATE - Here's another handy one:
tail -20000 log/production.log|grep 'Completed in'|cut -d" " -f3,11|sort -n
May 18, 2009
General / FTP Problems with Extended Passive Mode (EPSV)
I just went through many hours of trying to figure out wierd FTP problems. It looks like Peer1 has a weird firewall which gets confused on EPSV FTP connections. The way that the problem presented was that after login, when the FTP client attempted a directory listing (or any other command), after giving the EPSV command it would just hang.
For those who don't know, EPSV is "extended passive mode", and is a newer extension to FTP's historical passive mode (PASV), which is used to make FTP work in firewalled environments. I really don't know much about EPSV beyond that, although I think it was developed for enhanced future compatibility with IPv6.
Anyway, the problem is that most recent FTP clients attempt EPSV first, and then only use the traditional PASV if it fails. Also, most recent FTP servers support EPSV. Therefore, even if the firewall is blocking EPSV, the client will think that the command is successful, because the server is trying to do it, not knowing that it is being blocked.
I use vsftpd. The only way I found to prevent EPSV mode is to use their command whitelisting feature, and whitelist everything except EPSV and EPRT (the extended version of the FTP PORT command). So, here's what I added to my vsftpd.conf file, and it seems to work so far:
cmds_allowed=ABOR,CWD,DELE,LIST,MDTM,MKD,NLST,PASS,PASV,PORT,PWD,QUIT,RETR,RMD,RNFR,RNTO,SITE,SIZE,STOR,TYPE,USER,CDUP,HELP,MODE,NOOP,STAT,STOU,STRU
So now when a client attempts an EPSV command, it will respond with "550 Permission denied." and the client will usually fall back to regular PASV mode. I wish I could have blacklisted the command rather than whitelisting all other commands, but oh well.
If you have access to the client side of the connection, and don't/can't mess with the server side, you can usually turn off extended passive mode there as well. With curl, you need to add --disable-epsv. With regular ftp, you need to issue the command "epsv4" after connecting.
February 10, 2009
Amusing / This Friday is the Geek Apocalypse
See here. On Friday, UNIX time will reach 1234567890. This is, like, the end of the world :)
January 23, 2009
Amusing / Historically Bad Ideas in Programming
The QCon developer's conference this year has an interesting track - historically bad programming ideas. However, I have to say that I'm actually a big fan of the null pointer, despite the issues that it gives some people. The fact is, without null, nil, undefined, or 0 (depending on language), the actual task of programming would be much more difficulty. And, I would argue, would cost more than the bug-savings that we would get from having non-null type systems. In order to get around them you would have to write code that is more confusing and more time-consuming, and you might even wind up with the same set of bugs, just moved around slightly.
January 20, 2009
Features / I Dream of acts_as_tsearch2
Two plugins I would love to have for Ruby on Rails, and would love to build if I had the time:
acts_as_tsearch2
This would be an ActiveRecord layer for integrating with PostgreSQL's tsearch2 full-text indexing. Right now, I am not aware of a Rails plugin that does that.
search_coalesce
This would take the results of multiple paginated queries and combine them together. For instance, right now I'm writing a program using acts_as_solr for searching. However, I also need to refine the results using named_scopes, and some of the searches would most naturally fit into a named_scope. Therefore, I would love to be able to take the results of each query and combine them together on the fly in a paginated way. If this sounds impossible, I've already dreamed up a basic architecture. Here's a hint - just make sure that all queries have the same sorting.
December 18, 2008
Snippets / Docx and Open Packaging Conventions for Ruby and Rails
I just finished the first iteration of my Ruby on Rails plugin for handling Microsoft's new Open Packaging Conventions file format, which is their new container format for XML-based file formats in Microsoft Office. DOCX, Microsoft's new XML-based format for Word is probably the most widely used of these.
Open Packaging Conventions is basically a zipfile containing one or more components, and those components' relationship to each other is defined by other files within the zipfile.
To add rubyopc to your Rails app, do:
script/plugin install http://rubyopc.googlecode.com/svn/trunk/rubyopc
Here is how you would make a DOCX file with it:
OpenPackagingConventions::Package.with_package("test.docx") do |p|
p.add_part("/word/document.xml", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument", "application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml", <<EOF)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:document xmlns:ve="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:m="http://schemas.openxmlformats.org/officeDocument/2006/math" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" xmlns:w10="urn:schemas-microsoft-com:office:word" xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:wne="http://schemas.microsoft.com/office/word/2006/wordml">
<w:body>
<w:p w:rsidR="00EA68DC" w:rsidRPr="00C703AC" w:rsidRDefault="00EA68DC" w:rsidP="00EA68DC">
<w:pPr>
<w:rPr>
<w:lang w:val="es-ES_tradnl"/>
</w:rPr>
</w:pPr>
<w:r>
<w:t>
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Nunc at risus vel erat tempus posuere. Aenean non ante. Suspendisse vehicula dolor sit amet odio. Sed at sem. Nunc fringilla. Etiam ut diam. Nunc diam neque, adipiscing sed, ultrices a, pulvinar vitae, mauris. Suspendisse at elit vitae quam volutpat dapibus. Phasellus consequat magna in tellus. Mauris mauris dolor, dapibus sed, commodo et, pharetra eget, diam.
</w:t>
</w:r>
<w:r w:rsidRPr="00C703AC">
<w:rPr>
<w:lang w:val="es-ES_tradnl"/>
</w:rPr>
<w:t>
Nullam consequat lacus vitae mi. Sed tortor risus, posuere sed, condimentum pellentesque, pharetra eu, nisl.
</w:t>
</w:r>
</w:p>
</w:body>
</w:document>
EOF
p.add_part_to("/word/document.xml", "/word/styles.xml", "http://schemas.openxmlformats.org/officeDocument/2006/relationships/styles", "application/vnd.openxmlformats-officedocument.wordprocessingml.styles+xml", <<EOF)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<w:styles xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">
<w:style w:type="paragraph" w:styleId="Normal">
<w:name w:val="Normal" />
<w:rPr>
<w:b />
</w:rPr>
</w:style>
</w:styles>
EOF
This will create a small word document called "test.docx".
You can find the official documentation for these file formats here.
September 10, 2008
Features / Separating Value from State
Lambda-the-Ultimate had an link to an interesting language called Clojure:
By identity I mean a stable logical entity associated with a series of different values over time. Models need identity for the same reasons humans need identity - to represent the world. How could it work if identities like 'today' or 'America' had to represent a single constant value for all time? Note that by identities I don't mean names (I call my mother Mom, but you wouldn't).
So, for this discussion, an identity is an entity that has a state, which is its value at a point in time. And a value is something that doesn't change. 42 doesn't change. June 29th 2008 doesn't change. Points don't move, dates don't change, no matter what some bad class libraries may cause you to believe. Even aggregates are values. The set of my favorite foods doesn't change, i.e. if I prefer different foods in the future, that will be a different set.
...
In Clojure's model, value calculation is purely functional. Values never change. New values are functions of old, not mutations. But logical identity is well supported, via atomic references to values (Refs and Agents). Changes to references are controlled/coordinated by the system - i.e. cooperation is not optional and not manual. The world moves forward due to the cooperative efforts of its participants and the programming language/system, Clojure, is in charge of world consistency management. The value of a reference (state of an identity) is always observable without coordination, and freely shareable between threads.
It is worth constructing programs this way even when there is only one participant (thread). Programs are easier to understand/test when functional value calculation is independent of identity/value association. And it's easy to add other participants when they are (inevitably) needed.
Sounds incredibly interesting!