Which brings me to my point.

Matt Zumwalt, active code4lib member and Ruby on Rails/institutional repository developer, boldly declared XForms to be dead. I offer this critique:

There are some inaccuracies in this post that I would like to address. First of all, HTML 5 forms do not supplant XForms as an option for collecting user inputted data. HTML5 is much simpler, and thus has broader appeal. XForms enables the creation of much, much more complex models, with far more sophisticated controls and validation. Moreover, if XForms was a dead language in January 2008, with the release of the HTML5 specification, and that IBM had dropped support, then why do you suppose the XForms 1.1 specification was released in October 2009, edited by a representative of IBM?

No, XForms is very much alive. It has a small, but very active community, which is especially visible with the Orbeon development community. XForms is best used as a definition of dynamic forms that are processed server-side, not in the browser (which pushes a lot of processing demand onto the user, which isn’t good). There are some good, open source frameworks out there. Orbeon is the best, and has many users from both industry and academia, including Pfizer, Leap Frog, Wachovia, UCSB, Stanford, and the National Archives. In fact, Orbeon XForms applications form a large part of the enormous workflow of the NARA Electronic Records Archives project, which is a multi-year project contracted to Lockheed Martin and has a financial backing of close to a half a billion dollars (I have heard). XForms, dead?

A lot of the design flaws you describe are in actuality implementation flaws. Development of a Rails-based framework seems to me like an enormous waste of time and money. You can adapt the MODS editor developed by Brown to such a task. It has already been proven that you can interact with metadata delivered through REST from a Fedora repository. And MODS is fairly simple as a a metadata standard. Care to take a stab at TEI or EAD?

When you began your research in 2007, Orbeon was a fairly young application. But the standard and its delivery and processing applications have evolved since then. Only in the last two or three years has XForms grown into a viable solution. Moreover, since it is a W3C standard, you can pick your forms up and migrate them to a new framework fairly easily. Is your Rails application sustainable in the long term? Are today’s jQuery functions going to work in 2015’s browsers? These are things you need to consider when contemplating a web form standard.

Fedora is a Tomcat application. So is Solr. So is adore-djatoka, which UVA/Hydra utilizes for jpeg2000 delivery. And so is Orbeon. ActiveFedora and any Rails-based MODS editor seem to me like the third wheel in the repository relationship. But in all seriousness, the sustainability of a boutique Rails application that is heavily dependent on the javascript functions of 2010 should be a serious concern to repository developers. jQuery is all the rage today, but it could blow away in the wind five years from now. This is the very thing that the XForms working group set out to prevent when they introduced a standard approach to dynamic webforms.

All two-dimensional maps (as opposed to globes) are projected. That is, the curved three-dimensional surface of the Earth is transformed onto a flat two-dimensional surface. This can be done in an infinite variety of ways, many of which have been mathematically characterized and named by cartographers, for whom they are necessary tools. We must note, however, that no such transform can obtain a perfect representation of a section of the Earth. The mapmaker must choose which qualities to preserve and in what measures. Is it more important to provide an accurate depiction of relative areas or of relative lengths? Is the area around Greenland to be kept in the focus of accuracy, or that around New Zealand?

Each map therefore carries with it from its creation certain choices like these, part of the arguments the map makes about the world by its very construction. We chose WMS on which to start building our tools because, amongst other reasons, it allows for the transmission of projection information as part of its operation. This fact allows us to produce imagery from historical maps (themselves in any number of projections) and maintain the original choices the mapmaker made. Google Maps and Open Street Map are not WMS sources. They can be described as tile caches, huge reservoirs of rendered imagery. As such, they offer their own choices about how the world is to be projected. (Google’s choice has become so closely associated with Google that it is known widely as “the Google projection”.)

Now we come to an important technical distinction; WMS services are able (depending on the capabilities of the specific software in use) to reproject their contents. That is, in response to a specific request for imagery, they can produce the imagery in a projection different from the one in which it was stored. GeoServer, the software we are using for Neatline, has a library of thousands of projections to which users can add more as desired. This allows us to take imagery from a WMS source and lay it under a historical map layer while maintaining the original projection for that of the map as a whole. Tile caches, by and large, do not allow for this. (Google Maps offers its one projection, and Open Street Map offers two.) This means that in order to lay historical map imagery over a layer from one of these sources, we would have to reproject the foreground (historical imagery) overriding the choices of the mapmaker and introducing additional choices of our own about what facets of the geographies at stake are to be preserved and which abandoned.

(Neogeographers will remark that georectifying a digital image introduces similar issues. This is true, but unavoidable for our purposes. We would like to avoid compounding the matter in a way that is subtle and hard to detect.)

We are working out means by which we can provide the undeniable utility of popular tilecaching services in a way that is respectful of the historical context and story of map artifacts. Until we do, we will continue to concentrate on the more flexible and sophisticated apparatus provided by WMS.

Most recently, I have turned my attention to expanding the descriptive abilities of Omeka into the realm of collections of artwork. Omeka items are described with Dublin Core, which is capable of describing anything, though not particularly well. I developed VraCoreElementSet, which incorporates VRA Core fields into the Edit Item form. VRA Core is a much more semantically appropriate schema for describing art and artifacts. Since it was conceived as an XML standard (not strictly a flat list of fields), some elements have hierarchical sub-componenets. For example, a work may have several agents involved in its production, and each agent has a name as well as a role, culture, birth date, and, as the case may be, a death date. The VraCoreElementSet plugin creates a table for agents so that a user may enter this data separately. Then in the Edit Item form, the user may select VRA Core agents from a drop down menu restricted by the records in the agents table. Items may also be exported to schema-compliant VRA Core XML. There is still some work remaining on this plugin, but it is well on its way toward completion.

Now that the Scholars’ Lab has contributed EAD Importer and VRA Core Element Set plugins, Omeka may attract new institutional users from the library, archive, and museum fields, who may have otherwise settled for proprietary applications to disseminate their digital collections.

The Scholars’ Lab/NEH Institute for Enabling Geospatial Scholarship was held at the University of Virginia Library May 25-27, 2010 and concluded with a set of two-minute, three-slide lightning talks by Institute attendees on their own spatial humanities projects and works-in-progress.

We’re crowd-sourcing the keynote to the final round of the Scholars’ Lab/NEH 2009-2010 Institute for Enabling Geospatial Scholarship. With all of these fantastic attendees on hand — not to mention the Institute faculty — how could we let the opportunity slip by?

Frontiers in Spatial Humanities:
Lightning Presentations

We are pleased to host 40 rapid-fire, 2-minute demos of boundary-pushing projects in spatial humanities. The scholars presenting their work come from 27 different institutions, and were competitively selected to attend this prestigious program, funded by the National Endowment for the Humanities. Some of our Institute faculty will also offer brief glimpses of their work as part of a whirlwind tour of emerging work in humanities GIS.

While admission to the Institute itself is now closed, “Frontiers in Spatial Humanities” and the reception that follows are open to the public!

I’d like to thank the NEH for its generous funding of our training program, and the University of Virginia Library for supporting the Scholars’ Lab — as well as the “Frontiers” reception, to which you’re all invited!

Thursday, May 27th, 3:30-5:00pm
Harrison-Small Auditorium

For more information about the SLab and our NEH-funded Institute for Advanced Topics in the Digital Humanities, please visit:
http://lib.virginia.edu/scholarslab/geospatial/

Qualification

People often think of the Scholars’ Lab as a Ruby (and Rails) shop, which isn’t quite the case. We code in many different languages. In the past several moths, we have written The Mind is a Metaphor in Ruby (with Rails). For Better For Verse uses Wordpress with TEI and JSP, and some more recent work on a William Faulkner audio archive employs Cocoon with Solr. In addition to those collaborations with UVa faculty, we’ve been writing plugins for Omeka (and dusting off our PHP skills) and have created a discovery interface for our GIS infrastructure in Ruby (with Sinatra). If you analyze the technologies we currently deploy, it turns out we use Cocoon + Solr more than anything else, though we’re starting to move away from that particular stack as our approach for tool development.

The Scholars’ Lab has a lot of experience with all types of languages, and depending on the circumstances, we choose different tools to accomplish any given task. However, after quite a bit of time helping people get started on a programming path, I’ve come to appreciate some of the features Ruby provides in getting new programmers up to speed.

Learning Curve

Every language has a learning curve. However, once you get the hang of some of the basics of computer languages (flow control, data structures, objects, etc.), the biggest differences come from syntax. All web languages make certain programming exercises easy, and once you buy in to the way in which that language handles programming constructs, moving between languages for experienced programmers becomes a simpler exercise in exploring syntax and built-in functionality.

As one of my computer science professors posited, generally the first language you learn governs the way you code until something significantly better comes along. For a lot of folks getting in to programming for the first time, this usually means either taking a class or finding someone to show you the basics. If you’re in higher education, this has typically meant the tool of choice is PHP. However, having seen the look of bewilderment on the faces of enough graduate students and faculty members as I attempt to explain the difference between sprintf and printf (printf returns the length of the formatted String and sprintf returns the formatted String), I’ve come to believe that the syntax of a programming language (and it’s readability) is an exceptionally important part of a language, especially when teaching basics of software construction.

Method Chaining

Without getting into how the PHP and Ruby duck-type primitive data types and data structures, one big difference in syntax between the two is how one combines multiple method calls together. Ruby uses method chaining for objects whereas PHP uses “bolted-on” functions (think of these as order-of-operations from your high school Algebra class). Let’s look at this brief example of addressing and sorting an associative array in PHP and its equivalent in Ruby:

$projects = array("solr" => 4, "php" => 1, "rails" => 2, "jsp" => 3);
$keys = array_keys($projects);
sort($keys);
$sorted = array_slice($keys, 0, 3);

If you’re a little more advanced, you might refactor (rewrite) the code to look more like this (methods anonymously “bolted-on” to one another):

$projects = array("solr" => 4, "php" =>  1, "rails" => 2, "jsp" => 3);
$sorted = array_slice(sort(array_keys($projects)), 0, 3);

Now, the same examples in Ruby syntax:

projects = {"solr" => 4, "php" =>  1, "rails" => 2, "jsp" => 3}
sorted = projects.keys.sort.slice(0,3)

Or, even more concisely:

projects = {"solr" => 4, "php" =>  1, "rails" => 2, "jsp" => 3}
sorted = projects.keys.sort[0..3]

I’ve found that Ruby’s method chaining syntax makes more sense to new programmers than the more mathematical “bolted-on” syntax.

Blocks

Ruby has a neat construct that you use all over the place to create anonymous functions (a technical term for creating specific functionality without defining a new function to define the action). Let’s take a function to sort an array of projects. First, in PHP:

function sort_projects_by_count($a, $b)
{
    if($a -> counts == $b -> counts)
    {
        return 0;
    }
    return($a -> counts > $b -> counts) ? +1 : -1;
}

usort($projects, "sort_projects_by_count");

And the same thing in Ruby:

projects.sort do |a, b|
    a.counts <=> b.counts
end

Ok, so this is a bit of an unfair comparison, but here is an analogous version of the Ruby code in PHP:

usort($projects, create_function($a, $b, 'if($a->counts == $b->counts){
    return 0;}return ($a->counts > $b->counts ? +1 : -1));

No matter how you slice it, Ruby syntax just feels more human. Even if you don’t know exactly what’s going on, looking up one operator in the Ruby syntax as opposed to following the logic flow and determining what “? +1 : -1″ means (it’s shorthand for an if-then statement) makes the act of reading code much easier.

Monkeypatching

If you’re not familiar with the term, “monkeypatching” is what programmers call changing or extending a base class (like an array or string object) to add functionality or change the way it works. Let’s say you really need to be able to test a string to see if it looks like an integer, you could create a monkeypatch along these lines:

class String
    def is_int?
        self =~ /^[-+]?[0-9]*$/
    end
end

This code snip extends the String class and uses a Regular Expression (regex) to test if a given String is an integer (number) by simply calling “is_int?” (notice the question mark at the end of the definition; this is used for methods that return a Boolean value). That’s a little advanced, but it does show off a very useful piece of functionality of the language that allows you to do a better job dealing with a duck-typed language.

Frameworks

Many people when talking about Ruby associate its use in web development with the Rails MVC framework. Just as PHP has Zend, CodeIgniter, CakePHP, symfony, etc., Ruby has Rails, Merb, Sinatra, Camping, and many more. Rails is the 900-pound gorilla of Ruby frameworks, and has a lot of nice features to get new applications off the ground quickly and some really great online guides to setting things up (I frequent RailsGuides). Since we often suggest Rails to our collaborators I’ll focus on this framework, but there are several other frameworks out there to choose from.

Think of Ruby (or PHP for that matter) as a pile of building materials: you can to build anything you want if you know how to put everything together. Rails, on the other hand, is like a prefab house where workers pour a foundation, set the house up, and then leave you to add the drywall, siding, windows, and roof. If you need a new component for your prefab house, a sales representative is standing by to help immediately ship you what you need.

Generators

To extend the previous metaphor, generators are like sales representatives that allow you to place orders for new aspects of your site.  To create all the erb templates (the default templeting language for Rails), controller methods, model, database migrations, routes, and tests for a new project with a single line, you might run something like the following:

script/generate scaffold topic title:string description:text

I moved away from using scaffolding pretty quickly, but it does provide a nice starting point for new programmers to build interactions with data models.

Templates

The default templating engine in Rails is erb which provides a convenient method for generating views of  your models. One of erb’s most important features is the use of “partials,” pieces of code that are used in multiple views by calling the render method. I often replicated this behavior in PHP by calling an include somewhere in a view.

ActiveRecord

The key to database interactions in Rails is ActiveRecord. As an SQL expert, I have to admit this part of the Rails framework drove me a bit batty at first, but then again I’ve been writing SQL for over 10 years (my colleagues often roll their eyes when I start writing it with relational algebra nomenclature), so allowing a framework to abstract this particular piece took some getting used to. If you’re new to programming, though, this means you don’t have to learn SQL but instead can use Ruby-style syntax to interact with your database without necessarily needing to care what your RDBMS back-end happens to be.

Take this example of looking up a book review where you have both a “book” and “review” table. In PHP you would do something like this (this snip will only work with a MySQL connection but has some sub-selection stuff going on):

function query($sql)
{
    global $conn;
    return mysql_query($sql, $conn);
}

function recent_reviews($count)
{
    $query = query(sprintf('SELECT b.book_title, r.review_id, r.created_at, r.id, review_counter.review_total
        FROM reviews r, books b,
            (SELECT count(*) AS review_total FROM reviews) AS review_counter
        WHERE r.book_id = b.book_id
        ORDER BY created_at DESC
        LIMIT %d', $count);

    return $query
}

print_r(recent_reviews(5));

Now, for the ActiveRecord equivalent:

reviews = Review.find(:limit => 5,  rder => "created_at DESC");
puts reviews.inspect

Because of the way in which ActiveRecord sets up its model associations, you’ll have access to the different name scopes to print out the same information, just in far less code. However, if you really want to, you can pass your SQL to get more granular control over the syntax:

reviews = Review.find_by_sql("SELECT b.book_title, r.review_id, r.created_at, r.id, review_counter.review_total
        FROM reviews r, books b,
            (SELECT count(*) AS review_total FROM reviews) AS review_counter
        WHERE r.book_id = b.book_id
        ORDER BY created_at DESC
        LIMIT ?", count);

One of the real beauties of the ActiveRecord methods is that as long as you’re using the generic ActiveRecord syntax, your data persistence layer can be pretty much any RDBMS and be changed with a couple lines in the configuration file. The trade-off however, is that you lose a few things and can make slightly more work for yourself than you might anticipate. One important caveat is that ActiveRecord doesn’t create foreign keys when you set up reference fields. This is actually by design as it’s using an object-oriented idiom (an object should validate the presence of another, without the underlying persistence layer enforcing any type of constraint), but I find myself adding these in to ensure that the RDBMS takes advantage of the pre-calculated indexes to improve overall performance.

I should also mention that I think the ActiveRecord model has some real limitations. As you develop your models, you will most like be tweaking its fields, which in turn requires new migrations, and you may forget which fields are actually in your models. There are plugins that help with this, but you do need to take additional steps to have this information placed somewhere convenient (I use a pre-commit git hook that calls the annotate gem to dynamically annotate my model schemas).

Security

You’ll notice in the last examples I was doing some funny stuff in both the PHP and Ruby examples to protect against SQL injection attacks. If you’re using the ActiveRecord methods of addressing objects, Rails will take care of this for you. If you’re using PHP, you’ll either need to do this yourself (sprintf is commonly used) or rely on a framework to parametrize your statements (you don’t want someone deleting everything in your database).

You also need to protect yourself from Cross-site Scripting Attacks (XSS) by escaping HTML from fields with dynamic content. erb has a helper function html_escape (h is the shorthand) which escapes this data. In Rails 3, this will change slightly and erb will automatically html_escape model output unless you explicitly tell it not to escape the field. One less thing to remember!

Testing

Testing is important, and I try to preach its virtues every chance I get. After finishing a project, the typical programmer won’t touch the code again until the application breaks. Good testing will save you (or the person that inherits the code) a lot of time discovering exactly what broke the application.

Let’s face it: testing is a pain in PHP, and I rarely see it done well. What I’ve really enjoyed about Ruby development is the fact that no matter how you code there is an appropriate testing framework available (I use rspec). There’s a strong emphasis on not only unit testing, but also integration and acceptance testing. There are also libraries that give you an idea of how well your code is tested, something I sorely missed from my Java coding endeavors. One other huge plus is that every gem on the rubygems.org site includes test-coverage metrics to give you an idea of how well the code you want to install is tested.

PHP also has testing frameworks with PHPUnit and PHPSpec being rather popular. I won’t say too much about the PHP testing frameworks other than to say that there are analogous frameworks for writing and running tests in PHP and Ruby. However, I’ve noticed a slightly more concerted effort to think through the inclusion of the tools in Ruby and their integration into the coding workflow than I’ve experienced with PHP. With the latter language, I’ve often fell in to the trap of writing the code, getting it to where I want it, and then, really as an afterthought, writing basic unit tests to get rather skimpy code coverage.

As a case in point, a mantra in Rails development is TATFT.

BryanL on TATFT from Bryan Liles on Vimeo.

Deployment

Two typical objections raised when contemplating Ruby development are that “Ruby doesn’t scale” and that server setup is a real pain. These are valid concerns, but as with many open source projects with a large number of fanatical supporters, the Ruby community has steadily made improvements in these areas. Actually, for the vast majority of our readership, these issues can be filed away in the solved category.

Scaling

I’ve always found objections to scaling a bit troubling. Scaling is one of those over-used terms that means different things to different people, but most of the “Rails doesn’t scale” comes from Twitter’s experience with the framework. They found, as they scaled horizontally (adding more servers) to handle loads of 11,000+ requests per second, that a bottleneck existed at the data persistence level as Rails doesn’t, by default, provide a mechanism to to address multiple databases. Twitter has since moved parts of their code base to Scala but has retained the majority of their code in Rails and has developed some rather ingenious messaging capabilities to talk to the appropriate abstraction layers that one needs in very large enterprise applications.

While Twitter shows that Rails is capable of scaling (with lots of work), quite honestly the likelihood of any of our applications needing this level of engineering is slim. I will say, however, that there are relatively simple methods of scaling with your infrastructure should you start running into performance issues. We have, for example, an application written in pure Ruby on Rails deployed as a Tomcat application (the details of which are completely outside the scope of this article, but the application gets all the benefits of an Enterprise class Java environment with the ease of Rails development).

Server Support

The Ruby language is included in most (if not all) modern Linux package systems and makes installation a snap. The other “major” piece of software you’ll need is RubyGems (a package manager for Ruby libraries), which is also generally available as a managed package.

Note: There is a major change occurring with the development of Rails 3. Rails is moving from a system of system-wide gems to application-level gems with the introduction of GemBundler. This approach is a more stable method of deploying application requirements which not only allows you to ensure that application libraries are properly resolved, but also provide better granular control over which libraries are deployed in specific contexts.

There was a time where deploying Rails applications was a real bear. Then along came Phusion Passenger (aka mod_rails). This allows you to run Rails (actually any Rack-based application) through Apache and Nginx without any other port management, service process monitoring, file cleanup, etc. As long as Apache is running, so is your Rails app!

Community of Support

The Rails community is pretty great in getting folks off the ground. As with any technology there are a fair number of curmudgeons, but leaders in the community as quick to remind people to be nice (see Yahuda Katz’s The Blind Men and the Elephant: A Story of Noobs). There are several corporations backing Rails development (EngineYard is a big one) and when the Google Summer of Code for Rails program wasn’t continued, the Rails community was able to raise $100,000 in three days to support a Ruby Summer of Code.

There are a number of really good podcasts (Ruby5 and The Ruby Show are good), vodcasts (Railscasts), tutorial sites (ACSIIcasts), blogs (RailsDispatch, ThoughtBot, EngineYard), open source projects (lots on github), open source books (Rails Tutorials), and some really good reference books from The Pragmatic Programmers. There’s even some humor…

Summary

As much as it drives language purists crazy when I say it, there’s nothing that you can do in Ruby that you can’t replicate in PHP (and vice-versa). In my experience, getting people set up with a functioning web application is far easier with Ruby than it is with PHP (and less prone to spaghetti code), and the deployment options make Ruby (plus a framework) a great starting point for new programmers to get their feet wet with web application development (check out Heroku). If you’re an experienced developer, does it make sense to drop PHP and rewrite your code base? Absolutely not. However, at some point, you will be faced with the prospect of needing to migrate a legacy application where Ruby may make a lot of sense. As someone who has spent quite a bit of time de-tangling spaghetti code from Perl CGI and mixed HTML and PHP pages, I’m hoping that the people that will eventually be migrating my Ruby code will not need to perform the level of coding archaeology we’ve needed to perform.

Like most things in life, choosing the correct tool for the job needs some careful consideration and planning. Ruby makes a lot of sense for getting applications off the ground quickly and reinforcing good practices like testing, code separation, and readability that I find important in forming new digital humanities programmers. Web development over the last decade has become exceptionally complex (AJAX, web services, web standards, multiple browsers, etc.) and the real hope is that by using Ruby and Rails as an approach, people will be inspired to continue down a development path to both enrich their own scholarship and impact the larger digital humanities community without becoming frustrated by syntax. This is a bit of an experiment which we and other digital humanities shops are undertaking, and in which we’re inviting everyone to participate. No matter the language, we should all be engaged in teaching best practices in project design and management, in software development techniques, in the construction of usable and elegant interfaces, and in the application of these things to humanities scholarship, through which everyone wins!

Other Resources

• Rails for PHP Developers
• 7 Reason I switched back to PHP after 2 years on Rails
• RubyGuides
• JRuby

Jean Bauer, former Scholars’ Lab Graduate Fellow in Digital Humanities announces: “I have just released my first open source project. HUZZAH!”

DAVILA is a database schema visualization/annotation tool that creates “humanist readable” technical diagrams. It is written in Processing with the toxiclibs physics library and released under GPLv3. DAVILA takes in the database’s schema and a pipe separated customization file and uses them to produce an interactive, color-coded, annotated diagram similar in format to UML. There are many applications that will create technical diagrams based on database schema, but as a digital humanist I require more than they can provide.

Technical diagrams are wonderfully compact ways of conveying information about extremely complex systems. But they only work for people who have been trained to read them. If you design a database for a historian, and then hand him or her a basic E-R or UML diagram, you will end up explaining the diagram’s nomenclature before you can talk about the database (and oftentimes you run out of time before getting back to the research question underlying the database). This removes the major advantage of technical diagrams and can also create an unnecessary divide between the technical and non-technical members of a digital humanities development team.

I have become fascinated by how documenting a project (either in development or after release) can build community. I’m not just talking about user generated documentation (ala wikis), but rather the feeling created by a diagram or README file that really takes the time to explain how the software works and why it works the way it does. There is a generosity and even warmth that comes from thoughtful, helpful documentation, just as inadequate documentation can make someone feel stupid, slighted, or unwanted as a user/developer. I will be writing on this topic more in the months to come (perhaps leading up to an article). In the meantime, check out DAVILA and let me know what you think.

Project homepage: http://www.jeanbauer.com/davila.html

First a little digression to explain latitude and longitude, and why computer GISs generally prefer decimal degrees when expressing lat/lon as a coordinate pair.

Latitude and Longitude is a spherical coordinate system for describing location upon a sphere (or upon an object that’s approximately a sphere, like the Earth).  Just as there are 360 degrees in a circle, there are 360 degrees of longitude (numbered 180 W (-180) to 0 to 180 E (+180) on either side of the Greenwich Prime Meridian) and 360 degrees of latitude (numbered 90 S (-90) to 0 to 90 N (+90) from the south Pole to the Equator (the Prime Parrallel, so to speak) to the North Pole… and back again on the other side of the globe, to complete the circle).  Each degree can be subdivided into 60 minutes, and each minute subdivided into 60 seconds.  Further subdivision of seconds is expressed as fractions or decimals.  Thus you could express the geographic location of the Scholars’ Lab at UVA as being at 38º 02’ 12.3540” N, 78 º 30’ 19.7928” W (or +38º 2’ 12.3540”, -78 º 30’ 19.7928”).

Computer GIS programs all want the northing and easting coordinate pair saved as just two numbers (one number for latitude, one number for longitude) instead of three different fields to contain the degrees, minutes, and seconds for latitude and another three fields for longitude.  This makes it much easier for the computer to plot location.  Many GIS programs also prefer the coordinates to be ordered longitude, latitude since that mimics X, Y coordinates.  Since there are 60 minutes in a degree and 60 seconds in a minute (or 3600 seconds in a degree [60 x 60]), you could write the location of the Scholars’ Lab as -78 – (30/60) – (19.7928/3600), 38 + (02/60) + (12.3540/3600) which is -78.505498°, 38.036765° in decimal degrees.

Now back to the emergent tartan-plaid problem.  What would happen if instead of converting to decimal degrees, you simple wrote out the degrees-minutes-seconds numbers in the format DD.MMSSssss ?  The Scholars Lab location would become -78.30197928, 38.02123540.  But the computer GIS would still interpret this as decimal degrees, and would compress all points falling within a 1×1 degree box into just the first 6/10ths x 6/10ths of the box, with a gap without points filling the rest of the box.  Spread over a large region, this would result in a tartan plaid-like pattern emerging.

So what is supposed to look like this:

Would end up looking like this:

And both the correct and incorrect versions together:

Not that this is a common mistake people using GIS make by any stretch, but when someone has a question about why their points are all coming out misaligned with strange empty striping patterns, it can take a little while to deduce what’s going on if you’ve never seen the results of such a mistake before.

Enter Capistrano…If you’ve not used this before, essentially this automates the deployment of web applications to your server environment.  It’s written in Ruby, but allows you to deploy ANY type of web application (we use it for Cocoon, Rails, Java, and PHP applications in the Scholars’ Lab). If you’ve got a larger shop, you may also take a look at a web interface called Webistrano which allows non-programmer types to deploy software through a web interface.

To show off the power of this software, I thought I’d write up how we use capistrano to deploy Omeka in various environments. The setup can be a little complex, but there are some good tutorials for getting started (see Setting up a Rails Server and Deploying with Capistrano on Fedora from Scratch and the Capistrano Getting Started). The following code snips assume you have successfully installed capistrano and use Subversion as your SCM (if you need SVN hosting, you can start a new project on Google Code; you can also use Github if you declare the git scm in the code).

The first step in getting your Omeka project automated for capistrano is ensuring both the capistrano and railsless-deploy gems are installed (if you’re not a ruby-ist, gem is a package manager for Ruby applications and libraries):

sudo gem install capistrano railsless-deploy

Capistrano installs a new command on your system called “capify” which sets up the boilerplate for capistrano. Just execute the capify script in your project directory:

cd /path/to/project/trunk
capify .

This will create two files, Capfile in your root directory and a config/deploy.rb. You’ll need to edit the Capfile ever so slightly to add the requirement for the railsless-deploy gem. It should read as follows:

load 'deploy' if respond_to?(:namespace) # cap2 differentiator
# Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) }

require 'rubygems'
require 'railsless-deploy'
load    'config/deploy'

Now we just need to do some setup in the config/deploy.rb file to tell capistrano about Omeka. This is where you need to know a little about how your server is set up and you may need to slightly change your server set up in order to use capistrano. The way capistrano works is that it creates a releases directory on your path that holds “deployments” of you project. The latest version of the project is then symlinked the project directory into the releases. This allows you to very quickly undo a deployment that goes awry.

As an example, we deploy projects to /usr/local/projects, so our omeka project would get deployed to /usr/local/projects/omeka. Capistano will create a few directories in /usr/local/projects/omeka:

• releases (timestamped directories of your application)
• shared (for log files, SCM cache, files you don’t want to be overwritten)
• current (symlink to latest directory in the releases directory)

If you’re setting up Omeka, you will need to redirect the base Directory to the “current” symlink. Here’s the vhost entry we use for Omeka as an example (this is an Ubuntu server; you may need to change the log file path if you are deploying to another operating system).

<VirtualHost *:80>
ServerName your.server.org
DocumentRoot /usr/local/projects/omeka/current/
<Directory "/usr/local/projects/omeka/current">
Options FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>

ErrorLog /var/log/apache2/omeka_error.log
TransferLog /var/log/apache2/omeka_access.log
</VirtualHost>

Now, to edit the config/deploy.rb file to set things up for automated deployments.

# You must always specify the application and repository for every recipe. The
# repository must be the URL of the repository you want this recipe to
# correspond to. The deploy_to path must be the path on each machine that will
# form the root of the application path.

set :application, 'omeka'
set :repository, 'http://your.svn.path/repo/trunk'

set :deploy_to, "/usr/local/projects/#{application}"
set :deploy_via, :remote_cache
set :user, 'deployer'
set :runner, user
set :run_method, :run

default_run_options[:pty] = true

ssh_options[:username] = user
ssh_options[:host_key] = 'ssh-dss'
ssh_options[:paranoid] = false

role :app, 'www.coolomekaapp.org'
role :web, 'www.coolomekaapp.org'
role :db, 'www.coolomekaapp.org'

# ===============
# Custom Tasks
# ===============
namespace :deploy do
desc 'Make sure the archives directory has the proper permissions'
task :chmod_archive_dir, :except=>{:no_release => true} do

run "chmod g+rw #{current_path}/archives"
end
desc 'Sets up the intitial db.ini config'
task :upload_database_config, :except=>{:no_release => true} do
db_config = <<-INI
[database]
host     = "your.db.host"
username = "db_user"
password = "db_password"
name     = "db_name"
prefix   = "omeka_"
;port     = ""
INI

put db_config, "#{current_path}/db.ini"

end

desc 'Move archives directory so it doesn\'t get overwritten during deployments'
 task :move_archive_dir, :except=>{:no_release => true} do
 run "mv #{current_path}/archives #{shared_path}"
 end

 desc 'Just svn up the directory'
 task :svn_up, :except => {:no_release => true} do
 run "svn up #{current_path}"
 end

 desc 'Link archives folder to shared directory'
 task :link_archives_dir, :except=>{:no_release => true} do
 run "cd #{current_path} && ln -snf #{shared_path}/archives archives"
 end

end

# ======================
# Task Event Hooks
# ======================

after 'deploy:symlink', 'deploy:upload_database_config', 'deploy:link_archives_dir'
after 'deploy:cold', 'deploy:chmod_archives_dir', 'deploy:move_archives_dir'
after 'deploy', 'deploy:cleanup'

Ok, there’s a lot going on here. I’ll briefly explain what’s going on, but there should be enough here for you to start hacking. But let’s see what tasks capistrano know about and you can call. If you are still in your project directory, just type cap -T to list all the capistrano tasks. Your output should look similar to this:

cap deploy                        # Deploys your project.
cap deploy:check                  # Test deployment dependencies.
cap deploy:chmod_archive_dir      # Make sure the archives directory has the ...
cap deploy:cleanup                # Clean up old releases.
cap deploy:cold                   # Deploys and starts a `cold' application.
cap deploy:migrate                # Run the migrate rake task.
cap deploy:migrations             # Deploy and run pending migrations.
cap deploy:pending                # Displays the commits since your last deploy.
cap deploy:pending:diff           # Displays the `diff' since your last deploy.
cap deploy:restart                # Restarts your application.
cap deploy:rollback               # Rolls back to a previous version and rest...
cap deploy:rollback:code          # Rolls back to the previously deployed ver...
cap deploy:setup                  # Prepares one or more servers for deployment.
cap deploy:start                  # Start the application servers.
cap deploy:stop                   # Stop the application servers.
cap deploy:symlink                # Updates the symlink to the most recently ...
cap deploy:update                 # Copies your project and updates the symlink.
cap deploy:update_code            # Copies your project to the remote servers.
cap deploy:upload                 # Copy files to the currently deployed vers...
cap deploy:upload_database_config # Sets up the intitial db.ini config
cap deploy:web:disable            # Present a maintenance page to visitors.
cap deploy:web:enable             # Makes the application web-accessible again.
cap invoke                        # Invoke a single command on the remote ser...
cap log:tail_log                  # Tail the rails log file
cap shell                         # Begin an interactive Capistrano session.

Some tasks were not listed, either because they have no description,
or because they are only used internally by other tasks. To see all
tasks, type `cap -vT'.

Extended help may be available for these tasks.
Type `cap -e taskname' to view it.

To use a specific capistrano task, you just type the command listed. But let’s get back to the actual script and go over that briefly. Part of the installation process of Omeka requires you to reset the permissions of the archives directory. This is handled by the chmod_archive_dir. However, because of the way that capistrano deploys applications, the archives folder would get overwritten in every deployment. To get around this, we move the archives directory to the shared folder, then create a symlink from the current directory to the shared/archives directory.

There’s a task to upload_database_config that you can have in your cap script (we deploy out of private repos), but if you’re deploying out of a public repo, you may want to just put the db.ini file on the server in the shared directory and symlink it into the current_path. Lastly, there are times where you just need to do an “svn up” (or git pull) to update something small and not need to do a full deployment. This is where the cap deploy:svn_up helps out….guess what it does

Capistrano also provides task even hooks to execute specific tasks after specific events. In this script, when you do a cold deployment (when you are setting things up for the first time), the script will change the permissions on the archives directory, then move the archives directory to the shared directory. When you do a normal deploy (after the directory has gotten a proper symlink), the script will upload the database config, then symlink the archives directory and run a cleanup (keep the last 5 versions you deployed).

While there’s still a bit of up-front set up to do on your server, capistrano significantly speeds up your ability to to consistently deploy software, quickly roll-back if (that changes to “when”, if you write code long enough) problems occur, and reinforces a development process that involves SCM!

Resources

• Capistrano
• Automated PHP Deployment with Capistrano
• Railsless-deploy
• Capistrano Tutorials

The specifications for the Timeline plugin are wonderfully documented and explained on the Omeka wiki. This actually illustrates a great practice that is all too often ignored…explicitly stating what some software should do. Taking the time to think through the “what” a piece of software should do will save  you time in the long run as it forces you to think about how everything fits together, alleviating ambiguity and allowing you to focus on the task at hand.

For this specification, there were two requirements:

• The plugin should create a helper function for creating a SIMILE Timeline widget from an array of items. The helper function should allow you to specify which metadata elements the time data should come from, as well as the element that specifies the caption (by default, the Dublin Core Title element).
• The timeline should allow for time intervals (start and end dates) and points in time (singe date).

While there are two requirements, it’s a good idea to break this us a little more into individual (atomic) tasks. First, we need to take a look at the SIMILE Timeline Widget documentation. Taking a quick look at their Getting Started guide, this seems pretty straight forward

1. Include the Timeline javascript
2. Create a div container in the HTML view (e.g. <div id=”timeline”></div>
3. Format the items from Omeka as Timeline Events
4. Add the Timeline.create() call to the Omeka HTML view

The specification states that this needs to be a helper function. If you’re not familiar with this term, a helper function is a block of code that does some of the computation for another piece of code. In many frameworks, helper functions aren’t actual objects, but pieces of procedural code that can be accessed from across the application that help add functionality that isn’t exactly proper to place in a model or controller; essentially these are statically accessible functions (in coding terms) that can be called from properly instantiated objects (in this case a view).

Now that we have good idea of what the code should do, let’s turn to some actual code. Omeka uses the Zend Framework to keep from doing a lot of repetitive programming, so most of the syntax of what we need to do is driven by how Zend handles PHP. On top of the Zend Framework, Omeka implements its own plugin infrastructure, so there are a few things we need to take in to account in our design.

The Zend Framework is a model-view-controller (MVC) framework designed to organize code for maintainability and DRY-ness (Don’t Repeat Yourself). One the the hallmarks of most MVC applications is its physical separation of files and functionalities. In the case of Omeka plugins, the hierarchy is generally split into model, view, controller, and tests directory. For the Timeline plugin, since we are developing a helper function, we use a slightly modified directory structure:

Timeline
|__helpers
|__tests
|__views

We also need some mechanism to tell Omeka about a plugin. This metadata is currently provided in a file called plugin.ini. This file is pretty straight forward, but  let’s go over it briefly:

[info]
name="Timeline"
author="Scholars' Lab"
description="SIMILE Timeline for Omeka"
link="http://omeka.org/codex/Plugins/Timeline"
omeka_minimum_version="1.0"
omeka_tested_up_to="1.2"
version="1.1"
tags="Timeline, simile, chronology, time, temporal"

This file is what the Omeka admin interface uses to display information about your plugin. Two things that may not be initially obvious are the omeka_minimimum_version and omeka_tested_up_to lines. One fact you’ll learn about software development, especially with API development, is as projects mature, the needs of the API grow along with them. You want to be able to mitigate potential issues should the plugin API change by explicitly setting the minimum revision number that your plugin is tested against (you can get older revisions from the SVN tags repo at https://omeka.org/svn/tags/).

Note: you can run multiple versions of Omeka on your machine for testing by checking out separate versions of the software in you web tree. For instance, you can have localhost/omeka1.0, localhost/omeka1.1, localhost/omeka_trunk. Setting this up is beyond the scope of this post (be sure to set up separate databases), but if you have questions, leave a comment.

The next thing we need to do is tell Omeka what to do with our plugin. The top-level plugin.php file contains instructions (hooks) to tell the Admin interface what to do when a user installs or uninstalls the plugin. This is where we let Omeka know that items tagged with “Timeline” should use the Timeline Plugin, to set up some logging to help us debug when something goes wrong, and some default routes to help make “pretty” URLs.

Now, with all the preliminary setup taken care of, now we can start developing the helper function. First, let’s examine the Controller which tells Omeka what to do when an action is requested by the framework. This is actually a fairly straight forward:

<?php
class Timeline_TimelinesController extends Omeka_Controller_Action

{
	private $logger;

	public function init()
	{
		$this->_modelClass = 'Item';
		$writer = new Zend_Log_Writer_Stream(LOGS_DIR . DIRECTORY_SEPARATOR . "timeline.log");
		$this->logger = new Zend_Log($writer);
	}

	public function showAction()
	{
		$this->view->item = $this->findById();
	}

}

Let’s go over briefly what’s going on here. There are some semantics in the way in which these Controller objects are named which are inherited from the way in which Zend handles Controllers. The Timeline_TimelinesController follows the convention of the “package” (the plugin) name, underscore, plural controller name (to handle multiple controllers), and finally “Controller” (which explicitly tells a programmer what function the Object performs). Because this is a Framework, we also want to be able to inherit a lot of behaviors without needing to code them ourselves, which is handled by the “extends Omeka_Controller_Action” (this is the base CRUD class for Omeka which overrides and extends the Zend_Controller_Action object). The “important” part of the Controller code is really the showAction function which sets a variable named “item” in the view which contains a reference to the ID of an Omeka object. The rest just sets up a logger to keep track of what’s going on.

Note: If you run in to problems with this plugin, it is most likely related to logging. For more on this, see the documentation on Retrieving Error Messages.

Now we can get into the guts of the actual helper function. When you boil down the code, most of this is JavaScript with some strategically placed PHP. What we did is create a helper function named “createTimeline” which actually does the work for us. This takes two required items, a div reference to associate the Timeline on your page, and an array of Omeka Items with which to populate the Timeline.

function createTimeline($div, $items = array(), $captionElementSet = "Dublin Core", $captionElement =  "Title", $dateElementSet = "Dublin Core", $dateElement =  "Date" ) {
		echo js("prototype");
		global $mets;
		$mets = array($captionElementSet, $captionElement, $dateElementSet, $dateElement);
		?>
		<!--  we have to load the script in this funny way because we need to get the tag into the head of the doc
			because of the the funky way Simile Timeline loads its sub-scripts  -->
		<script type="text/javascript">
			scripttag = document.createElement("script");
			scripttag.src = "http://static.simile.mit.edu/timeline/api-2.3.0/timeline-api.js?bundle=false";
			scripttag.type = "text/javascript";
			$$("head")[0].insert(scripttag);

			if (typeof(Omeka) == "undefined") {
				Omeka = new Object();
			}

			if (!Omeka.Timeline) {
				Omeka.Timeline = new Object();
			}

		</script>

		<script type="text/javascript" defer="defer">
			Omeka.Timeline.timelinediv = $("<?php echo $div;?>");

			Omeka.Timeline.events = [
			<?php
				function event_to_json($item) {
					global $mets;
					return "{ 'title' : '" . getMet($item, $mets[0], $mets[1]) . "',
					'start' : '" . getMet($item, $mets[2], $mets[3]) . "',
					'description' : '" . getMet($item, "Dublin Core", "Description") . "',
					'durationEvent':false }";
				}
				echo implode(',',array_map('event_to_json', $items));
				?>
				];

		</script>
		<?php
	     echo js("createTimeline");
		?>
		<script type="text/javascript">
			Event.observe(window, 'load', onLoad);
			Event.observe(document.body, 'resize', onResize);
		</script>

		<?php
}

There’s a lot going on here, and there is a mix of PHP in the JavaScript. The first thing is making sure the prototype.js library is included, then declaring a variable named “mets” in the global scope (to make it available to other variable scopes). After we’ve declared $mets, get in to the JavaScript to include on the page and introducing a new JavaScript Namespace (Omeka.Timeline) which allows you to extend this code in other views.

The second script block actually formats Omeka items that you’ve called as Timeline Events in the JSON format calling a helper method we also include in the code:

function getMet($item, $elementSet, $element) {
	 $tmp = $item->getElementTextsByElementNameAndSetName($element, $elementSet);
	 return addslashes( $tmp[0]->text ) ;
}

This function returns the metadata for an Omeka item, which is then used the createTimeline’s sub-method of event_to_json to properly construct an event for Timeline. After all the JSON strings are created, we “glue” all the array elements with a comma with the implode function.

As you can see, not a lot of code actually needs to be written to add functionality to Omeka. With a little research, and some pointers on syntax, extending Omeka can be done quite quickly and doesn’t require a degree in computer science. If you’re interested in getting started on a plugin, I highly recommend the Omeka dev list; the community is growing and questions are answered quickly (usually by folks on the Omeka development team) and is a great way to learn about the technical issues surrounding developing software using the Omeka platform.

Resources

• Omeka Plugin API
• Omeka Developer List
• Timeline Source Code
• Timeline Documentation
• Zend Framework
• Zend Framework Documentation

Scholars’ Lab News for April 2010 (PDF version)

Scholars’ Lab News April 2010 (text-only version)

We’re developing a new web-based tool for visualizing these historic boundary changes and it’s nearly ready for prime time.  We’ll announce the beta release here soon.

So with the knowledge that US state boundaries have already been subject to drastic change over time, let’s have some fun with geographic information systems to visualize drastic mathematically-induced changes to those familiar US state boundaries.

For our experiment, let’s keep all our current state capital cities right where they are since they are laden with the necessary infrastructure of government.  But we’ll move the state boundary lines Voronoi-style so anywhere you travel in each of our new states you’ll be closer to the state capital than any other state capital.  In other words, when you’re standing anywhere inside our newly outlined Virginia, you will always be closer to the Virginia state capital, Richmond, than any other state capital.  That seems very efficient.  Let’s have a look.

Here’s that familiar grade-school wall map of the lower 48 US states and their capital cities.   Now let’s tweak the map with GIS software to reconfigure the states, Voronoi-style.

Wow, what a difference Voronoi makes.

Let’s measure just how much the states have changed in our new layout.   In absolute terms, Utah and New Mexico make the biggest land grabs while Texas and California lose the most real estate.  But as a percentage of their current area, Rhode Island is the big winner ballooning in size by over 240% while Massachusetts shrinks 60%.

To visualize the state-by-state changes, Todd Burks from neighboring Clemons Library overlayed the two maps.

Intrigued?  Read more about Voronoi and Thiessen polygon GIS techniques.

Hello. In my last blog, I began my discussion of Pandora.com, the streaming audio website which offers a new kind of web radio to listeners. Enter a “seed” song into Pandora’s search engine, and the site will create a streaming “station” composed of songs that resemble your seed song. This process is powered by the Music Genome Project, a massive research endeavor which began in the early 2000s and is based out of the company’s Oakland, California headquarters.

How is Pandora’s song-recommendation engine different than web radio platforms that came before it? Well, the majority of other online radio stations, such as last.fm, operate off a system called collaborative filtering. What is collaborative filtering? In layperson’s terms, collaborative filtering involves matching one user’s taste to another’s (or a series of other people). On a site like last.fm, over time a user amasses a playlist of songs they’ve expressed a preference for—a sort of musical taste profile. Last.fm’s search tools automatically identify other users with whom your tastes seem to overlap, and uses this information to power “radio” stations you can stream on the site. The process is pretty simple, and based on personal intuition and the data existing users have already entered into the system. Collaborative filtering powers aspects of many media websites, such as Amazon.com’s personal recommendation feature for shoppers.

It does have some limitations for online radio listeners, however. As Pandora’s founder Tim Westergren pointed out in a 2006 interview with Leo Laporte and Amber MacArthur, collaborative filtering-powered online radio stations have a tendency to only recommend what is broadly popular in contemporary pop music. While independent-label music certainly has a strong presence on last.fm, a quick scan of various users’ profiles on the site may suggest that Westergren has a point. Even among “indie” users on last.fm, there’s a whole lot of Death Cab for Cutie and Modest Mouse ruling the playlists (nothing against either of these bands). Collaborative filtering doesn’t necessarily ensure that the site’s users will discover truly obscure stuff they hadn’t heard of before. And in keeping with my interest in genre boundaries vis-à-vis Internet radio, in interviews Westergren has attributed the problem to the mainstream music business’ interest in keeping consumers bracketed into genre-specific niches. In the aforementioned chat with Laporte and MacArthur, Westergren cited the “age-old problem in the music industry” wherein a tiny percentage of music released by a given label typically accounts for nearly all its sales—a problem codified by genre boundaries.

Pandora, through its Music Genome Project, aims to circumvent this problem, by offering its users a new kind of recommendation engine. As I mentioned in my earlier post, the Music Genome is a systematic endeavor to deconstruct and analyze individual pop songs using over 400 “musical attributes” that the company has identified. These attributes include everything from tempo, to vocal timbre, to harmonic movement—even sound production aspects like echo and reverb. In other words, it is essentially a musicological approach in the strictest sense of the word. The focus is on sound itself, rather than a band’s cultural associations with other bands (as is the case in collaborative filtering). Indeed, Westergren bragged in the aforementioned interview that “when we recommend to you a piece of music, we don’t even know how popular it is.”

Instead, what the Music Genome Project entails is the company’s roughly fifty analysts sitting down in the Oakland, CA headquarters and methodically tagging a given song using these 400+ attributes. Westergren has described the process in ways akin to the scientific method, noting that a percentage of songs the analysts deconstruct are reviewed twice for quality. The songs, categorized by attributes, are added to the Project’s over 500,000 songs (and counting) accumulating in the company’s database. Songs sharing a similar musical “DNA” are then automatically matched and linked by Pandora’s search engine when you enter in a “seed” song. Westergren has called the Genome “kind of like a musical taxonomy,” and I don’t think this language is accidental. As Fabian Holt has pointed out about musical genres, “Discourse on the temporal dimensions of categories is saturated with organicist metaphors, as in discussions of how genres are born, how they grow, mature, branch off, explode, and die.”1 Even though Pandora in fact aims to get around genre, it seems to me that this biologic language informs the company’s mission and direction.

In any case, as a Pandora user, I have often benefited from the happy accidents occasioned by the way the Music Genome Project works. For instance, I entered in Pandora as a “seed” Bob Dylan’s song “Tonight I’ll Be Staying Here With You”, a lilting, mid-tempo country-rock stroll. The Genome built a streaming station for me that included folk-rocky chestnuts by relatively obscure ‘60s and ‘70s groups like UFO and Earth Opera. It’s likely that I would not have heard about these groups without Pandora, or at least that I would’ve heard about them years from now in another context. In this regard, it seems that Westergren does have something to boast about regarding his claim that Pandora’s search engine connects listeners with “invisible” music in a way that mainstream, genre-bound, multinational music corporations just can’t.

On the other hand, there are several notable gaps in the logic and execution of Pandora and the Music Genome Project model. The first gap I feel compelled to point out is a very practical one. Returning to my example of the station based around “Tonight I’ll Be Staying Here With You,” Pandora is skilled in giving a user a lot of what they like. Enter in a twangy rock song like Dylan’s and you’ll get a station with loads of twangy rock songs. But there can be too much of a good thing; namely, I find homogeneity of songs’ tempo an issue on Pandora stations. “Tonight I’ll Be Staying Here With You” is a bit plodding, and I’ve found that over a few hours of playing this station, I mostly get one plodding song after the next. This can be useful in terms of finding hidden gems, but makes for monotonous, even frustrating listening over a span of a few hours.

I do know that Pandora makes much of its “Thumbs Up/Thumbs Down” feature, which allows the user to indicate her or his preference for a given song. Pandora’s algorithms will adjust the playlist’s direction (ever so slightly) upon a “Thumbs Down” for a song you don’t care for. In a 2006 interview with the New York Times, Westergren describes this feature as a concession to human subjectivity (within an otherwise “objective” platform), and I agree. The “Thumbs Up/Thumbs Down” feature requires active listening and participation on the user’s part—generally a good thing, I’ll admit. But what if I want to just sit back with a cold beverage and let the music play? The Genome’s platform, as it currently works, seems unable to deliver the ebbs and flows in tempo and musical texture which I enjoy in a good mixtape or college radio show.

These kind of practical gaps in Pandora’s service point me toward a larger theoretical problem worth discussing. In its insistence upon musical sound as the key ingredient for making song recommendations, the Music Genome Project willingly suspends belief in some basic social facts about the way music works. Music is undeniably social, cultural, and political. It’s the soundtrack to our lives as we dance, eat dinner, exercise, commute to work, fall in love, and so on. Music blasts out of loudspeakers at political rallies. We argue with friends over drinks about the relative merit of this or that musical group. And music is always part of a commercial marketplace, even in this age of file-sharing. Given all this, I find Westergren’s claim that a band’s marketplace popularity is “completely irrelevant to what we do” a wee bit disingenuous, or at the very least requiring a willing suspension of disbelief regarding music’s social and marketplace role.

The Music Genome Project’s near-exclusive focus on sound itself, coupled with its organicist rhetoric regarding “musical DNA”, seems to suggest the company believes it can map out music in its totality—that it can “crack the code” of music, so to speak. As an aspiring musicologist, this reminds me a bit of another massive scholarly endeavor which worked toward a similar goal of cataloging music: Alan Lomax’s Cantometrics project. Developed by Lomax in the late 1950s and into the ‘60s, Cantometrics was a project wherein Lomax and several co-researchers analyzed the performance styles of (mostly traditional “folk”) songs from hundreds of different cultures around the world, tagging them with a variety of traits. These performance traits, such as vocal timbre, were organized into a computerized system wherein elements of the different musics could be compared. Lomax made the bold claim that one could draw conclusions about the social structure of a given society based on some of these performance traits (societies noted for a certain style of singing were sexually repressive, for instance). Of course, this claim was quite controversial, and has been challenged by other scholars since as overly reductive and essentialist. Fortunately, the Music Genome Project doesn’t attempt to make the connection between music and social structures the way Cantometrics did; indeed, as I said, the Genome Project’s rhetoric seems to deny aspects of the social world, if anything.

However, in the desire to systematically categorize and compare different aspects of music, one could say that the Genome Project and Cantometrics spring from a shared wellspring of human curiosity. One issue with this categorizing mission, though, is the problem of sample size. Lomax’s research was criticized for not casting a broad enough net in collecting these comparable performance traits. One could ask similar questions about the Music Genome Project’s scope. As I mentioned, the company’s website points out that its database currently features over 500,000 songs, and counting. This seems like a lot, but how useful is that number, when one considers the thousands and thousands of songs which are released commercially every year? And how can one ensure that multiple varieties, styles, and (yes, even) genres of music are adequately represented within those 500,000 songs?

Additionally, Pandora shares potentially problematic assumptions with Cantometrics regarding humans’ ability to fully categorize and catalog the world, to reduce music to its essence. This descends from the Enlightenment idea that our natural world is fully knowable through empirical and objective observation. That bedrock assumption has been the basis for the natural sciences, and one can see its influence on a project like Alan Lomax’s. The problem is: while an empirical observation approach might work well for classifying different varieties of tree frogs, when one wades into the murky waters of human behavior, it’s a lot more difficult to claim objectivity. Indeed, for myself and for a growing number of musicologists and humanities scholars more broadly, it is basically impossible to claim objectivity in one’s understanding of the world.

This is not to say that Pandora explicitly makes a claim of total objectivity, on their website or elsewhere. But as with Cantometrics, the fact that the company breaks songs down into discrete components and then makes comparisons and connections based on those components suggests that they believe music is knowable in some objective way. Pandora hasn’t made public a list of its over 400 “musical attributes”, but shares a handful of them on their website’s Frequently Asked Questions page. Some of the attributes they share make a lot of sense, and could even be called “objective”: major or minor key tonality, for instance. But consider an attribute like “headnodic beats”: in its FAQ entry, Pandora’s analysts admit they created the term themselves (it describes hip-hop beats which are strong, but not forceful enough to dance to). Given that probably almost no one outside of the Pandora offices uses this term, it can’t reasonably be called objective. This is not to say that an identification of some subjectivity within Pandora’s research model makes the whole enterprise come crashing down. Rather, I just wish to point out that while company prides itself on the cold objectivity of a computer algorithm choosing your music for you, human beings with subjective viewpoints created the components which power that algorithm.

Related to this, both Pandora and Cantometrics raise questions regarding musical gatekeepers, tastemakers, and their authority. Indeed, as ethnomusicologist Steven Feld has mused in response to Lomax’s work, “What are the sources of authority, wisdom, and legitimacy about sounds and music? Who can know about sound? Is musical knowledge public, private, ritual, esoteric?”.2 Many researchers of pop music, pop culture, and genre agree that this issue of who is doing the classifying, categorizing, and ranking is a really important question. For instance, snobby clerks at your local independent record store may decide that Gillian Welch’s music belongs in the “folk” rather than “rock” section of the store. But where does the authority behind their judgment come from?   Their judgment is informed by their life experiences and backgrounds as (mostly) well-educated middle-class white males.

These gatekeepers’ judgments are also informed by a deep knowledge of various musical genres: the ability to distinguish glam from punk from grunge, and so on. Since Pandora’s fifty music analysts perform a similar function, I find this aspect of their job paradoxical: though the company seems to pride itself on getting beyond musical genre, these analysts must be extremely well-versed in genre in order to do their jobs well. In a recent video post on Pandora’s blog, Westergren states that the purpose of the Music Genome Project is ultimately to connect musicians with audiences, in ways the traditional music business can’t.3 This is notably egalitarian rhetoric; it works off the assumption that consumers and musicians are empowered enough to seek each other out, and that they don’t need tastemakers dictating what music they should like.

As I noted above, however, the computerized system that listeners use to connect with musicians is designed and maintained by a group of (relatively) elite tastemakers. And in Westergren’s public statements about these analysts’ qualifications, I read a certain degree of anxiety over what kind of authority is vested in that role of analyst. In his 2006 interview with Leo Laporte and Amber MacArthur, Westergren pointed out that while all their analysts are regularly-gigging musicians, in order to carry out the depth of analysis required for the Music Genome Project, one really “need[s] an academic background”. Thus, in addition to being a working musician, an analyst employed by Pandora also needs at least a four-year undergraduate degree in music theory.

On a practical level, this makes a lot of sense to me. If you’re going to employ folks to analyze songs for you, wouldn’t you want them to have an understanding of musical principles on several different levels? On the other hand, on a theoretical level, Pandora’s insistence on both “street” and “book” smarts from its analysts demonstrates an unresolved subliminal conflict over whether “brains and corporate no-how” or “gut, ‘Id’ feelings” are what shape the music we listen to. Thus, in this way, Pandora and the Music Genome Project struggle with these issues of taste and knowledge hierarchies just like other public pop prognosticators, even as their seemingly objective research platform denies this social fact.

This may read as though I am beating up on Pandora, but I hope the position I’m staking out is subtler than that. Rather, I have simply been attempting to point out some slight contradictions of logic within the Music Genome Project’s overall research platform. On a practical, user’s level, I enjoy the site. And to be fair to Pandora’s employees, on a certain level they seem to recognize the issues I am brining up here. For instance, in a recent post on Pandora’s official blog by one of its music analysts, Michael Zapruder likens evaluating songs to judging a baby beauty contest, and then points out,

“The idea that all music is equal and deserves equal rights is somehow fundamentally a democratic idea; as is the corresponding idea that the public, and not some small cadre of experts, is the best judge of musical quality. But the fact that some music not only attracts more listeners, but also seems to mean more to more people over a longer period of time, indicates that there is actually something fundamentally unequal about music as well.”4

In other words, perhaps this issue of taste isn’t an “either/or” problem, but rather a “both/and” one. And by its nature, it’s most likely a problem with no definitive answer.

It seems that in his blog entry, Pandora employees like Zapruder are trying to find a practical, everyday way of working around and through this problem—and I can’t fault them for that. Certainly, the academic in me bristles when I see Pandora present something like “headnodic beats” as some kind of objective criteria for judging music. But on a practical level, it seems that these classifications, even if they’re vague (such as “vinyl ambience,” or what have you) are perhaps vague at least partly in the service of the listener’s experience—of trying to match users to interesting new music. It doesn’t seem that the point of the Genome is to categorize musical attributes simply for the sake of categorization. Rather, the point seems to be to put that information to use, making musical connections for the listener. So perhaps it’s a utilitarian reason why the Music Genome cuts certain logical corners on the “objective vs. subjective” question.

Ultimately, Pandora’s service rests upon the assumption that sound itself is the only aspect which really matters when analyzing different forms of music. It assumes that sound automatically trumps the sociocultural boundaries of genre, taste, and marketplace. This isn’t true, of course: in the real world we live in, rhetoric surrounding genre and taste guide the musical choices we all make, from Walmart AC/DC lovers to bebop nerds. But the Music Genome Project’s fiction regarding the supremacy of sound is an important, if very one-sided, position to have out there in the world. In fact, it’s almost counter-cultural in a way, because journalists and advertisers often focus so much on image when considering contemporary pop music. Pandora’s vision is a kind of imagined musical utopia, making a particularly 21st-century-specific stand for the importance of musical sound—a stand made possible by the shared cultural resource of the Internet.

Finally, closing with an idea my professor Fred Maus pointed out to me, when you’re confronted with enjoying a song you didn’t think you would like on Pandora (you enter in a Turbonegro song as your “seed” and are rewarded with a Poison song, for example), that tells us something important about genre boundaries. Your bemusement proves that musical genres exist. They’re cultural; they don’t hold up to objective scrutiny. And they’re based on something more than just musical sound; they’re built around assumptions that have to do with hierarchies of taste and class. Thus, paradoxically, we can learn quite a bit about the rules of genre from a website devoted to transcending those rules.

The 1907 Massie Map of Albemarle Co., VA

While going through our archives of scanned maps, we recently ran across a copy of Frank A. Massie’s 1907 “A new and historical map of Albemarle County, Virginia” [Special Collections, University of Virginia Library], commonly referred to as the Massie map, which contains a wealth of detailed historical information for the county in which the University of Virginia sits.

After obtaining a more recent scan of the map from Special Collections and Andrew Curly (of the Library’s Digital Production Services), we georectified the digital map and added it to our geospatial data portal. Accessing the map through our portal (here or here) allows you to not only view the map at high resolution in your browser, but also to view the Massie map in Google Earth, as well as being able to overlay the Massie map over other base maps and other geospatial data layers (for example, by making a WMS call to our portal’s server from your desktop GIS; or by pulling the Massie map into your webpage-embedded dynamic map using Open Layers).

Among the many interesting historical features on the maps, are the locations of:

• The residences and named estates of prominent county landowners, including the oldest house in the county, plus various churches and schools, numerous dams, mills, and quarries, and local geologic and mineral resources;
• Where Revolutionary war prisoners where held (the Barracks that held Hessian troops and from which Barracks Road was named, as well as where Baron and Baroness de Reidesel resided) plus old militia rendezvous points;
• Routes and encampments of various military campaigns (notably those of British Lieutenant Colonel Banastre Tarleton’s 1781 raid on Charlottesville and Gov. Thomas Jefferson’s subsequent flight, General T. J. “Stonewall” Jackson’s Confederate Army’s march in 1862, General George A. Custer’s unsuccessful Union Army raid in 1864, and General Philip Sheridan’s Union Army’s march in 1865 along with locations of many of the mills and bridges Sheridan had destroyed);
• Existing and proposed rail lines of many now-nonexistent railroads;
• Locations where various murders and river drownings occurred.

Additionally, the map contains tables showing city and county data from the 1900 census and a 1906 household survey (listing, among other things: total property values and taxes; numbers of livestock (1000’s of cattle, horses, hogs, and sheep, but only 24 goats!), wagons, watches, clocks, sewing machines, and pianos; number of manufacturers with their capital, number employees, and total wages; farm acreage; and enrollment figures for the white and colored public schools.

There’s also an inset showing the major rail lines and railroad junctions for the state of Virginia.

GIS software creates, manages, analyzes, and visualizes geographically referenced data.  Environmental Systems Research Institute (ESRI) in Redlands, California, produces ArcGIS desktop, a GIS software suite.  The following examples use ArcGIS ArcMap software version 9.3.1 at the ArcInfo product level.  Instructions for accessing GIS software at the University of Virginia are here: http://guides.lib.virginia.edu/gis.  If you are not affiliated with UVA, contact your local IT support person or ESRI for information on accessing this GIS software.

Calculating polygon centroids – When working with polygon features (like county boundaries) in GIS it is often necessary to locate the geographic center or centroid of each polygon as a point feature.  In ESRI’s ArcGIS desktop software the ‘Feature To Point’ tool creates polygon centroids.

For more information see the ArcGIS online help for the ‘Feature To Point’ tool:

http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?id=1798&pid=1790&topicname=Feature_To_Point_%28Data_Management%29

Creating a point distance table – Given a set of point features, GIS software can calculate the straight-line distance from each point in the set to every other point in the set.   The output distance table contains one row for each point-to-point combination along with the calculated distance.  In ESRI’s ArcGIS desktop software the ‘Point Distance’ tool creates a point distance table.

For more information see the ArcGIS online help for the Point Distance tool: http://webhelp.esri.com/arcgisdesktop/9.3/index.cfm?id=1353&pid=1347&topicname=Point_Distance_%28Analysis%29

Example using the ‘Feature To Point’ and ‘Point Distance’ tools – Given a polygon dataset representing boundaries of a group of US counties, calculate the distance between each county in the group and every other county in the group.

1. After ensuring the county polygon boundary file is projected using a distance-preserving projection, select the county polygons for the study area.
2. Convert the selected county polygons to county polygon centroid points using the ‘Feature To Point’ tool.
3. Generate the point distance table for all county centroid points created in step 2 using the ‘Point Distance’ tool.  Distance is expressed in the linear unit of the input dataset, which is meters in our example.

Since Omeka runs on PHP, this is actually a PHP configuration issue and not something you can currently tweak in Omeka. Basically, you just need to tell PHP to allow larger files sizes that are larger than the default. A very easy way to do this is to edit the .htaccess file that Omeka ships with along the following lines:

php_value upload_max_filesize 20971520
php_value post_max_size 20971520

I’ll note here that doing things this way only affects your Omeka project. Another way to go about this is to add the above to the Apache configuration that defines from where Omeka should be served. For example:

<VirtualHost *:80>

ServerName www.coolomeka.org
DocumentRoot /var/www/omeka

<Directory "/var/www/omeka">
    Options FollwSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all
</Directory>

    ErrorLog logs/omeka_error_log
    TransferLog logs/omeka_transfer_log

    php_value upload_max_filesize 20M
    php_value post_max_size 20M
</VirtualHost>

Lastly, you can edit the php.ini file (usually in /etc/php.ini or /etc/php5/apache2/php.ini). Just do a search in the file and change the following settings:

  memory_limit = 32M
  post_max_size = 20M
  upload_max_size = 20M

You typically don’t need to reload Apache (as long as you did not edit the Apache configuration file) to get these settings to work.

For more info on this, check out these resources

• Description of core php.ini directives
• PHP Upload Configuration
• PHP Directives Related to (Large) File Upload

Huh?

This project will allow scholars (and other stewards of cultural heritage) to create Web-based geospatial and temporal visualizations that build on the rich EAD metadata libraries produce in describing their archival collections and making them more discoverable — but the crucial twist is that we didn’t want to think of our Neatline visualizations as products of the metadata. They’re not brain-dead algorithmic output or some kind of thoughtless expression of the archivist’s (nuanced, but necessarily broad) stance toward historical or literary documents of interest. (Yes, I’m asking for it; bring it on!) In other words, Neatline isn’t about the parsing of placenames and automated population of timelines with data. Rather, we’ve conceived this tool (really, as development proceeds, this approach, because Neatline is emerging as an arrangement of instruments and an attitude toward their use) as a kind of playspace for the scholarly interpretative act. In future posts, we’ll describe our development effort and I’ll delve a little into the conceptual background for Neatline in the Temporal Modelling project I undertook several years ago with Johanna Drucker.

In the meantime, you can read about how we’re employing the Omeka plugin framework as a way to handle GIS services for scanned historical maps on the ScholarsLab.org project pages for Neatline-in-progress and our larger Omeka plugin work, or you can check out our dedicated project blog.