Syntax Highlighting

| No Comments
#!/usr/bin/perl
use warnings;
use strict;

print "Hello World\n";

Capistrano and svn prompts

| No Comments
I've been configuring rails 2 apps for automation with Capistrano 2.6 lately and I encountered a pretty irritating bug with svn. When you run the deploy:update command, Capistrano prompts you for your svn password and your machine password - like it should - and then it hangs forever on the actual deploy step. The problem occurs only when svn continually asks if you want to save your password with one of these dialogs:
-----------------------------------------------------------------------
ATTENTION!  Your password for authentication realm:
    Subversion Repository
can only be stored to disk unencrypted!  
...
-----------------------------------------------------------------------
Solution: make svn stop asking you if you want to save your password. To do this, modify your ~/.subversion/config so that it has (at least) this section:
[auth]
store-passwords = no
store-auth-creds = no
If you're interested, read more to view the error messages from Capistrano.

Uploading Files with Drupal 6

| No Comments

Here's a relatively painless example for file uploads in Drupal 6.

This code is in a module file called usertab.module:

#create the form
function usertab_admin($form_state) {
    
  #prepare the file upload form
  $form = array('#attributes' => array('enctype' => 'multipart/form-data'));

  #file selector element
  $form['csv'] = array(
      '#type' => 'file',
      '#title' => 'Upload users',
      '#description' => t('Pick a csv file to upload')
  );

  #submit button
  $form['submit'] = array('#type' => 'submit', '#value' => 'Upload');
   return $form;
}

#handle the form submission
function usertab_admin_submit($form, &$form_state) {

  #this leads us to sites/mysite.example.com/files/
  $dir = file_directory_path();
 
  # unlike form submissions, multipart form submissions are not in 
  # $form_state, but rather in $FILES, which requires more checking
  if(isset($_FILES) && !empty($_FILES) && $_FILES['files']['size']['csv'] != 0)){
    
    #this structure is kind of wacky
    $name = $_FILES['files']['name']['csv'];
    $size = $_FILES['files']['size']['csv'];
    $type = $_FILES['files']['type']['csv'];

    #this is the actual place where we store the file
    $file = file_save_upload('csv', array() , $dir);
    if($file){
      drupal_set_message("You uploaded $name!");
    }
    else{
        drupal_set_message("Something went wrong saving your file.");
    }
  }
  else {
    drupal_set_message("Your file doesn't appear to be here.");
  }
}
Here are some of the strange things going on here:

First, the structure of the $_FILES array in php is kind of strange. There's a good diagram of it here: http://www.php.net/manual/en/features.file-upload.post-method.php#91479. Notice that it's the second one in the example.

Next, Drupal's (6) file_save_upload() method. The API documentation is here: http://api.drupal.org/api/drupal/includes--file.inc/function/file_save_upload/6, but the parameters can be kind of confusing so here are some details:

  1. $source is the name of the file in the $_FILES array (also the name of the file field on form.
  2. $validators are outside the scope of this example.
  3. $dest is relative to the drupal root directory.
  4. $replace is actually self-explanatory

P.S. At some point, I will figure out how to get some syntax highlighting going here.

Debian/Shorewall routing issues

| No Comments

I've recently deployed a new router running Debian Squeeze and Shorewall. The problem started like this:

I have my network configured with five vlans for different areas (servers, workstations, sandboxing, guests, and DMZ) and a /28 subnet assigned for each zone all the fancy network math done.

Everything worked fine until I had to get machines to talk to the outside world.

Shorewall has configurations for 5 zones (corresponding neatly to the vlans) and a few filtering rules for each one. One is behind NAT and the others all have public addresses for which shorewall handles proxyarp.

With and without shorewall running, the router could ping machines both inside and outside the perimeter. In both cases, machines on the inside of the perimeter couldn't ping machines on the outside, but could ping each other.

Something, clearly, was wrong.

I'm pretty sure I spent at least 6 hours trying everything from verifying the upstream switch configuration to checking sysctl variables (mental note: make sure to enable the .forwarding keys!) to checking arp caches everywhere to even swapping out cables and hardware.

As it turns out, the problem was with shorewall. Since shorewall ultimately ends up making a lot of the routing decisions, a badly configured value will ultimately foobar everything else you work so hard to configure. Just remember that even though every other almost value in shorewall.conf is {Yes|No}, IP_FORWARDING takes {On|Off|Keep}.

This was a most extreme case of not RTFMing closely enough.

Passenger on Solaris

| No Comments

So today I set about installing passenger on a Solaris 10 machine. This turned out to be way harder than it seemed.

I went down the "install via gem" road, so it wasn't too painful to being with. When I reached the phase of actually installing passenger to apache, I discovered that the configuration tool is a bit confused by Solaris, especially the apache APU component.

Here's the problem:

* GNU C++ compiler... found at /opt/csw/gcc4/bin/g++
* Ruby development headers... found
* OpenSSL support for Ruby... found
* RubyGems... found
* Rake... found at /opt/csw/bin/rake
* rack... found
* Apache 2... found at /opt/csw/apache/bin/httpd
* Apache 2 development headers... found at /opt/csw/apache/bin/apxs
* Apache Portable Runtime (APR) development headers... not found
* Apache Portable Runtime Utility (APU) development headers... not found

No big deal - we'll just pass it some options, right? Wrong. The CLI only accepts arguments for apxs2 and apr, but not apu.

Running with --apr-config-path /opt/csw/apache2/bin gets us slightly further:


* GNU C++ compiler... found at /opt/csw/gcc4/bin/g++
* Ruby development headers... found
* OpenSSL support for Ruby... found
* RubyGems... found
* Rake... found at /opt/csw/bin/rake
* rack... found
* Apache 2... found at /opt/csw/apache/bin/httpd
* Apache 2 development headers... found at /opt/csw/apache/bin/apxs
* Apache Portable Runtime (APR) development headers... found at /opt/csw/apache2/bin/
* Apache Portable Runtime Utility (APU) development headers... not found

Finally the APU. This is a bit tricky to track down.

This was my process:

passenger-install-apache2-module secretly runs this file:

/opt/csw/lib/ruby/gems/1.8/gems/passenger-2.2.9/bin/passenger-install-apache2-module

Right away we see the following lines:

37 require 'phusion_passenger/dependencies'
52 Dependencies::Apache2,
53 Dependencies::Apache2_DevHeaders

So we follow them:

lib/phusion_passenger/dependencies.rb reveals that it secrely calls PlatformInfo to look for the APU:

304 if PlatformInfo.apu_config.nil?

So we jump into lib/phusion_passenger/platform_info.rb and discover (with some casual grepping):

284 if env_defined?('APU_CONFIG')
285 return ENV['APU_CONFIG']

This means that if we do the following, we can convince the installer that we do, in fact, have the necessary files:

setenv APU_CONFIG /opt/csw/apache2/bin/apu-1-config
passenger-install-apache2-module --apr-config-path /opt/csw/apache2/bin/

Hopefully this will make your life easier.

Drupal GCal Events module and time zones

| No Comments

If you've been using the GCal Events module, you might have noticed a problem where the timezone causes event times to be displayed incorrectly.

In my situation, the time is correct in Drupal, but incorrect in the corresponding GCal link. To force the calendar to interpret the timezone correctly, you can pass the current timezone argument as a GET parameter with the link.

In your GCal Events block, configure the Event Template like this:


<P><A HREF="#URL#&ctz=America/Chicago">#TITLE#</A><br>#LOC##DATE##TIME#

This should force GCal to interpret the times as specific to Central Time.

See the Google API Atom Reference for more information about the ctz option. An example of the link in practice is visible on the Minnesota Population Center (shameless department promotion!) website (http://www.pop.umn.edu) on the right-hand events sidebar.

Monitors and displays

| No Comments

If anyone has an elegant solution to extending gnome panels over multiple monitors, please let me know!

I other news, if you have two monitors and stack them with the panel on the top of the bottommost monitor, you'll discover how gnome handles hiding panels. Set the panel to auto-hide and it'll just slide up to the bottom of the top monitor! That's not hiding my panel anymore, that's just making it really hard to control the screen layout.

More and more I find myself just using a whole pile of screen sessions and putting jobs in the background...this makes for quite a mess when I need to reboot.

Also, if you have one of those convenient USB hubs on your monitor, make sure that you know which one your keyboard and mouse are plugged into before you go shutting off monitors!

Drupal and Shibboleth

| No Comments

With the imminent switch to Shibboleth, I've decided to try out hooking up a Drupal instance...to...Shibboleth. Exactly as the title implies.

I'm using Ubuntu Server 10.04 for my server and rather then authentication against a real IdP, I'm using the public TestShib Two service as an IdP and configuring the SP myself.

Here's how I did it:

0) Register with OpenIdP and get a machine ready for testing.

1) Install the necessary packages:
apache mpm-prefork
php5
libapache2-mod-shib2
mysql (for drupal)!
...and all their friends.

2) Get drupal and install that properly. There are plenty of tutorials out there on how to do this.

3) Configure your apache server so that it looks something like this:



ServerAdmin webmaster@localhost
ServerName x-128-101-79-21.pop.umn.edu
UseCanonicalName On

DocumentRoot /var/www

Options FollowSymLinks
AllowOverride None

Alias /secure /var/www/drupal

Options Indexes FollowSymLinks Multiviews

Order allow,deny
Allow from all

AuthType Shibboleth
ShibRequireSession On
require shibboleth


...and a whole bunch more as needed. You can actually base it off the default-ssl site (...like I did.)

4. Somewhere in your shibboleth2.xml (probably in /etc/shibboleth), you have something that looks like this:









This is the really important part. It's telling you exactly which paths on the server are going to be secured by

Dynamic Layer 2 filtering (and a concert)

| No Comments

Last week I saw Celtic Woman in concert - they put on an excellent performance, though the audience around me left a lot to be desired.

Onto more relevant topics. It turns out that building a MAC address + IP address + authentication application is actually easier than it sounds. The challenge is hovering near the network stack, specifically with the firewalls.

Here's the deal:

Ubuntu - Everyone's favorite Linux is several versions behind on both iptables and shorewall, not to mention that ipset isn't even in the kernel/package tree. If you want to, you can hack at it and compile in support.

FreeBSD - While it has the benefit of pick-your favorite firewall (pf, ipfw, firewall, etc...) it lacks layer2 filtering in a pretty big way. Gleb Kurtsou wrote a kernel patch as part of the Summer of Code '08 but sadly, it's not ready yet (and doesn't seem to work seamlessly with 8.0 Stable).

Debian - Well, according to the debian package repository it has ipset, iptables, and shorewall all at reasonably high version numbers. I'm going down this road now and I'll let you know how it turns out when I'm finished.

The big challenge at this stage is figuring out how to track valid IPs leased to DHCP to permitted MAC addresses. Simply adding the IPs to an allowed list doesn't work since someone could steal an active lease once the original client left the network (window between disconnect and lease return-to-pool). A MAC-IP pair helps track who has the lease at a given time (good for auditing), but a database tracking MAC-IP pairs + username at captive portal/proxy login is the best.

Basically, not until a user logs in with a valid mac address and a valid username and gets logged can they access the network. By default all IP addresses are in a blocked ACL and then moved over to a pass ACL when the login process is finished. Storing these MAC-IP pass-OK ACLs is the challenge. While we could statefully [keep-state] reload the firewall ruleset each time we wanted to add a new pass or remove an expired one, that seems pretty expensive. Instead, I'm going to try using in-memory tables.

Maybe next week I'll post more about the entire system (written mostly in perl).

vhosts with modperl

| No Comments

This post deals with configuring mod_perl environments with vhosts. Our objects for the post are to configure an Apache2 prefork server with mod_perl 2.0 with the following specifications:

1) Each vhost should have its own interpretter pool with its own paths.
2) vhosts do not need to use custom directives during startup.
3) Custom liraries in each vhost should not be shared with the others.
4) Users should be able to execute mod_perl scripts only from docs/bin.

Here's a snip from my apache2ctl status: "Server Version: Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0"

Let's get started!

I have three vhosts {a, b, c} each listening on ports 5050, 8080, and 9090 respectively.

Here's my initial /etc/apache2/sites-available/vhosts.conf file:


<VirtualHost *:5050>
DocumentRoot /var/www/a
#PerlPostConfigHandler MyApache2::Test->post_config
</VirtualHost>

<VirtualHost *:8080>
DocumentRoot /var/www/b
#PerlPostConfigHandler MyApache2::Test->post_config
</VirtualHost>

<VirtualHost *:9090>
DocumentRoot /var/www/c
#PerlPostConfigHandler MyApache2::Test->post_config
</VirtualHost>


Here's my initial /etc/apache2/httpd.conf file:

ServerName localhost


PerlSwitches -I/var/myperl/dep_lib
PerlPostConfigHandler MyApache2::Test->post_config

/var/myperl/dep_lib/MyApache2::Test.pm looks like this:


package MyApache2::Test;
use Apache2::ServerRec;
use Apache2::Const -compile => OK
use strict;
use warnings;

sub post_config {
my ($self, $conf_pool, $log_pool, $temp_pool, $s) = @_;
warn "HELLO " . $s->port . "\n";
}

1;

When we start the server, this is the output we get:

root@larx:~# tail -fn 0 /var/log/apache2/error.log
HELLO 0
HELLO 0
[Tue Mar 09 19:09:00 2010] [notice] Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
[Tue Mar 09 19:09:00 2010] [info] Server built: Nov 13 2009 21:59:28
[Tue Mar 09 19:09:00 2010] [debug] prefork.c(1032): AcceptMutex: sysvsem (default: sysvsem)

This shows that the global configuration (*:0) is saying hello when it calls the PostConfigHandler. After the first call to PerlPostConfigHandler, Apache restarts itself to test its graceful restart ability. [1]

Now what happens when we try to implement a PostConfigHandler in a VirtualHost? According to the documentation, since PerlPostConfigHandler is valid in the SRV scope, we should be able to do this:

/etc/apache2/httpd.conf:


ServerName localhost
PerlSwitches -I/var/myperl/dep_lib
#PerlPostConfigHandler MyApache2::Test->post_config

/etc/apache2/site-available/vhosts.conf:


<VirtualHost *:5050>
DocumentRoot /var/www/a
PerlSwitches -I/var/myperl/dep_lib
PerlPostConfigHandler MyApache2::Test->post_config
</VirtualHost>

<VirtualHost *:8080>
DocumentRoot /var/www/b
</VirtualHost>

<VirtualHost *:9090>
DocumentRoot /var/www/c
</VirtualHost>


and the startup:

root@larx:~# tail -fn 0 /var/log/apache2/error.log
[Tue Mar 09 19:19:23 2010] [notice] Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
[Tue Mar 09 19:19:23 2010] [info] Server built: Nov 13 2009 21:59:28
[Tue Mar 09 19:19:23 2010] [debug] prefork.c(1032): AcceptMutex: sysvsem (default: sysvsem)

Notice that the PerlPostConfigHandler hasn't said anything. This is actually by design, contrary to what the documentation might say. In reality, PerlPostConfigHandler is only valid in the server-wide configuration context, NOT in vhosts. There are two ways to get around this.

1) Patch your mod_perl installation using a patch (possibly outdated) available on the mod-perl mailing lists here [2].
2) Use startup.pl files in a per-host configuration scheme.

Since I have less than no desire to go hacking around inside the monster, I've opted for choice 2.

To achieve this, we need to provide each vhost with a way to load some file at server startup. In /var/www/(a|b|c) we place a file called startup.pl that looks remarkably like this: (just make sure to s/a/(b|c) when and where appropriate)

/var/www/a/startup.pl:


#!/usr/bin/perl

use lib qw( /var/www/a/lib /var/www/a/lib/dep_lib );

use Data::Dumper;

use warnings;
use strict;

warn "Dumping the A startup.pl:\n";
print STDERR Dumper(\@INC);
warn "Finished processing A startup!\n";

1;

Notice that "1;". That's pretty important since the script is effectively being require()'d by mod_perl and therefore must return true.

Next we need to configure the vhost:

/etc/apache2/sites-available/vhosts.conf:


<VirtualHost *:5050>
DocumentRoot /var/www/a
PerlPostConfigRequire /var/www/a/startup.pl
</VirtualHost>

<VirtualHost *:8080>
DocumentRoot /var/www/b
</VirtualHost>

<VirtualHost *:9090>
DocumentRoot /var/www/c
</VirtualHost>


This is the outcome from a startup run:

root@larx:~# tail -fn 0 /var/log/apache2/error.log
[Thu Mar 11 16:54:12 2010] [info] removed PID file /var/run/apache2.pid (pid=4057)
[Thu Mar 11 16:54:12 2010] [notice] caught SIGTERM, shutting down
Dumping the A startup.pl:
$VAR1 = [
'/var/www/a/lib',
'/var/www/a/lib/dep_lib',
'/var/myperl/dep_lib',
'/etc/perl',
'/usr/local/lib/perl/5.10.0',
'/usr/local/share/perl/5.10.0',
'/usr/lib/perl5',
'/usr/share/perl5',
'/usr/lib/perl/5.10',
'/usr/share/perl/5.10',
'/usr/local/lib/site_perl',
'.',
'/etc/apache2'
];
Finished processing A startup!
Dumping the A startup.pl:
$VAR1 = [
'/var/www/a/lib',
'/var/www/a/lib/dep_lib',
'/var/myperl/dep_lib',
'/etc/perl',
'/usr/local/lib/perl/5.10.0',
'/usr/local/share/perl/5.10.0',
'/usr/lib/perl5',
'/usr/share/perl5',
'/usr/lib/perl/5.10',
'/usr/share/perl/5.10',
'/usr/local/lib/site_perl',
'.',
'/etc/apache2'
];
Finished processing A startup!
[Thu Mar 11 16:54:15 2010] [notice] Apache/2.2.11 (Ubuntu) mod_apreq2-20051231/2.6.0 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
[Thu Mar 11 16:54:15 2010] [info] Server built: Mar 9 2010 21:04:15
[Thu Mar 11 16:54:15 2010] [debug] prefork.c(1032): AcceptMutex: sysvsem (default: sysvsem)


Notice that we see the Dumper being called twice. This is, again, because immediately after starting up for the first time Apache starts once, and then restarts to test its graceful restart ability. [1]

Really interesting side note: when you reload (aka "graceful restart," "-k graceful," SIGUSR1) the apache server will start new children with the new configuration settings. On the other hand, restarting (aka "restart now, "-k restart", SIGHUP) stops all children, reloads the parent, and then loads all children with the new configuration. This means that when you send "reload" you get one startup pass from modperl instead of two (since the server is already running it doesn't test it's ability to restart) whereas with restart, the server (for all intents and purposes) stops and starts again meaning that you see the output from modperl twice. [3]

Now that we know what's going on with the modperl startup, let's finish our setup. The remaining adjustments are fairly straightforward:

/etc/apache2/httpd.conf:


ServerName localhost
PerlSwitches -I/var/myperl/dep_lib
#This option allows us to say things like print "Content-Type: text/plain\n\n"
PerlOptions +ParseHeaders
#We still might want to do something server-wide
PerlPostConfigHandler MyApache2::Test->post_config

/etc/apache2/sites-available/vhosts.conf:


#I've actually ignored one of the criteria I set earlier (because I can, and partially to show a point):
#vhost a allows scripts to run under modperl from docs/bin even though the documentroot is under ExecCGI.
#The PerlOptions +Parent gives vhost a it's own perl interpreter pool complete with its own @INC.
<VirtualHost *:5050>
DocumentRoot /var/www/a/docs
Options ExecCGI +Includes FollowSymLinks +MultiViews Indexes
DirectoryIndex index.html

PerlPostConfigRequire /var/www/a/startup.pl
PerlOptions +Parent

<Directory /var/www/a>
AllowOverride All
</Directory>

<Directory /var/www/a/docs/bin>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
</Directory>
</VirtualHost>

#vhost b is configred identically to vhost a.
<VirtualHost *:8080>
DocumentRoot /var/www/b/docs
Options ExecCGI +Includes FollowSymLinks +MultiViews Indexes
DirectoryIndex index.html

PerlPostConfigRequire /var/www/b/startup.pl
PerlOptions +Parent

<Directory /var/www/b>
AllowOverride All
</Directory>

<Directory /var/www/b/docs/bin>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
</Directory>
</VirtualHost>

#vhost c is a little different. This vhost allows scripts to be run under modperl from anywhere it its document root.
<VirtualHost *:9090>
DocumentRoot /var/www/c/docs
Options ExecCGI +Includes FollowSymLinks +MultiViews Indexes
DirectoryIndex index.html

PerlPostConfigRequire /var/www/c/startup.pl
PerlOptions +Parent
SetHandler perl-script
PerlResponseHandler ModPerl::Registry

<Directory /var/www/c>
AllowOverride All
</Directory>

</VirtualHost>

There are two things that should be explained:

PerlOptions +Parent
...creates a separate interpreter pool for each of the vhosts. These pools do not inherit any settings from other pools.

SetHandler perl-script
...tells Apache to parse all files in that Directory (also valid for Locations) as perl-scripts. ("SetHandler modperl" also works just as well here). More on this in a week or two.

PerlResponseHandler ModPerl::Registry
...tells modperl that everytime it needs to process something, use ModPerl::Registry to generate the response by compiling/parsing/executing your code when it needs to.

Finally, to prove that this actually works (in addition to having the error log), here's a script that you can actually run under modperl in each of the vhosts. Notice the lack of a #! line and a 1;. They're not necessary [in this particular instance]!

test.pl:


use Data::Dumper;

print "Content-type: text/plain\n\n";
for(sort keys %ENV) {
print "$_ => $ENV{$_}\n";
}

print Dumper(\@INC);

Finally, my directory structure looks like this:

root@larx:/var/www# tree
.
|-- a
| |-- docs
| | |-- bin
| | | `-- test.pl
| | `-- index.html
| |-- lib
| | `-- dep_lib
| `-- startup.pl
|-- b
| |-- docs
| | |-- bin
| | | `-- test.pl
| | `-- lib
| | `-- dep_lib
| |-- lib
| | `-- dep_lib
| `-- startup.pl
`-- c
|-- docs
| |-- bin
| | `-- test.pl
| |-- lib
| | `-- dep_lib
| `-- test.pl
|-- lib
| `-- dep_lib
`-- startup.pl

19 directories, 8 files

Try visiting http://localhost:8080/bin/test.pl and http://localhost:9090/test.pl . Now you have mod_perl working with isolated vhosts!

Join me next week as I delve further into mod_perl. Possible topics include authentication and authorization, ModPerl::Registry and/or ModPerl::PerlRun, custom environment variables, and further per-vhost configuration options.


References:

[1] http://perl.apache.org/docs/2.0/user/handlers/server.html#Server_Life_Cycle

[2] http://www.gossamer-threads.com/lists/modperl/modperl/60796#60796

[3] http://httpd.apache.org/docs/2.2/en/stopping.html#graceful