The current stable version of Perl is 5.18.0 … but for very good reasons, Koha doesn’t require the latest and greatest. For a very long time, Koha required a minimum version of 5.8.8. It wasn’t until October 2011, nearly four years after Perl 5.10.0 was released, that a patch was pushed setting 5.10.0 as Koha’s minimum required version.

Why so long? Since Perl is used by a ton of core system scripts and utilities, OS packagers are reluctant to push ahead too quickly. Debian oldstable has 5.10.1 and Debian stable ships with 5.14.2. Ubuntu tracks Debian in this respect. RHEL5 ships with Perl 5.8 and won’t hit EOL until 2017.

RHEL5 takes it too far in my opinion, unless you really need that degree of stasis — and I’m personally not convinced that staying that far behind the cutting edge necessarily gives one much more in the way of the security. Then again, I don’t work for a bank. Suffice it to say, if you must run a recent version of Koha on RHEL5, you have your work cut out for you — compiling Perl from tarball or using something like Perlbrew to at least get 5.10 is a good idea. That will still leave you with rather a lot of modules to install from CPAN.

But since we, as Koha hackers, can count on having Perl 5.10, we can make the most of it. Here are a few constructs that were added in 5.10 that I find particularly useful for hacking on Koha.

Defined-OR operator

The defined-or operator, //, returns its left operand unless its value is undefined, in which case it returns the right operand. It lets you write:

my $a = get_a_possibly_undefined_value();
$a //= '';
print "Label: $a\n"; # won't throw a warning if the original value was undefined

or

my $a = get_a_possibly_undefined_value() // '';

rather than

my $a = get_a_possibly_undefined_value();
$a = '' unless defined($a);

or (horrors!)

my $a = get_a_possibly_undefined_value();
$a ||= ''; # if $a started out as 0...

Is this just syntactical sugar? Sure, but since Koha is a database-driven application whose schema has a lot of nullable columns, and since use of the Perl warnings pragma is mandated, it’s a handy one.

Named capture buffers

This lets you give a name to a regular expression capture group, allowing you to using the name rather than (say) $1, $2, etc. For example, you can write

if ($str =~ /tag="(?[0-9]{3})"/ ){
    print $+{tag}, "\n"; # %- is a magic hash that contains the named capture groups' contents
}

rather than

if ($str =~ /tag="([0-9]{3})"/ ){
    print $1, "\n";
}

There’s a bit of a trade-off with this because the regular expression is now a little more difficult to read. However, since the code that uses the results can avoid declaring unnecessary temporary variables and is more robust in the face of changes to the number of capture groups in the regex, that trade-off can be worth it.

UNITCHECK blocks

The UNITCHECK block joins BEGIN, END, INIT and CHECK as ways of designating blocks of code to execute during specific points during the compilation process for a Perl module. UNITCHECK code is executed right after the module has been compiled. In the patch I’m proposing for bug 10503, I found this handy to allow module initialization code to make use of functions defined in that same module.

Warning, warning!

There are some constructs that were added in Perl 5.10, including the given/when keywords and the smart match operator ~~, that are deprecated as of Perl 5.18. Consequently, I will say no more about them other than this: don’t use them! Maybe the RHEL5 adherents have a point after all.

Libraries are sneaky, crafty places.  If you walk into one, things may never look the same when you walk out.

Libraries are dangerous places.  If you open your mind in one, you may be forever changed.

And, more mundanely, university libraries are places that employ a lot of work-study students.  I was one of them at Ganser Library at Millersville University.  Although I’ve always been a bookish lad, when I started as a reference shelver at Ganser I wasn’t thinking of the job as anything more than a way to pay the rent while I pursued a degree in mathematics.  And, of course, there were decidedly limits to how much fascination I found filing updated pages in a set of the loose-leaf CCH tax codes.  While some of the cases I skimmed were interesting, I can safely say that a career in tax accountancy was not in my future, either then or now.

Did I mention that libraries are crafty?  Naturally, much of the blame for that attaches to the librarians. As time passed, I ended up working in just about every department of the library, from circulation to serials to systems, as if there were a plot to have me learn to love every nook and cranny of that building and the folks who made it live.  By the time I graduated, math degree in hand, I had accepted a job with an ILS vendor, directly on the strength of the work I had done to help the library migrate to the (at the time) hot new ILS.

While writing this post, it has hit me hard how much I owe an incredible debt of gratitude to my mentors at Ganser.  To name some of them, Scott Anderson, Krista Higham, Barbara Hunsberger, Sally Levit, Marilyn Parrish, Elaine Pease, Leo Shelley, Marjorie Warmkessel, and David Zubatsky have each taught me much, professionally and personally.  To be counted among them as a member of the library profession is an honor.

Today I have an opportunity to toot my horn a bit, having been named one of the “Movers and Shakers” this year by Library Journal.  I am grateful for the recognition, as well as the opportunity to sneak a penguin into the pages of LJ.

Original image by Larry Ewing
Original image by Larry Ewing
Why a penguin? In part, simply because that’s how my whimsy runs. But there’s also a serious side to my choice, and I’m happy that the photographer and editors ran with it. Tux the penguin is a symbol of the open source Linux project, and moreover is a symbol that the Linux community rallies behind. Why have I emphasized community? Because it’s the strength of the library open source communities, particularly those of the Koha and Evergreen projects, that inspire me from day to day. Not that it’s all sunshine and kittens — any strong community will have its share of disappointments and conflicts. However, I deeply believe that open source software is a necessary part of librarians (I use that term broadly) building their own tools with which to share knowledge (and I use that term very broadly) with the wider communities we serve.

The recognition that LJ has given me for my work for Koha and Evergreen is very flattering, but for me it is at heart an opportunity to reflect, and to thank the many friends and mentors in libraryland I have met over the years.

Thanks, and may the work we share ever continue.

Both Koha and Evergreen use memcached to cache user sessions and data that would be expensive to continually fetch and refetch from the database. For example, Koha uses memcached to cache MARC frameworks, while Evergreen caches search results, bibliographic added content, search suggestions, and other data.

Even though the data that gets cached is transitory, at times it can be useful to look at it. For example, you may need to check to see if some stale data is present in the cache, or you may want to capture some statistics about user sessions that would otherwise be lost when the cache expires.

The library libMemcached include several command-line utilities for interrogating a memcached server. We’ll look at memcdump and memccat.

memcdump prints a list of keys that are (or were, since the data may have expired) stored in a memcached server. Here’s an example of the sorts of keys you might see in an Evergreen system:

memcdump --servers 127.0.0.1:11211
oils_AS_21a5dc5cd2aa42ee7c0ecc239dcb25b5
ac.toc.html.0531301990
open-ils.search_9fd0c6c3553e6979fc63aa634a78b362_facets
open-ils.search_9fd0c6c3553e6979fc63aa634a78b362
oils_auth_8682b1017b7b27035576fecbfc7715c4

The --servers 127.0.0.1:11211 bit tells memcdump to check memcached running on the local server.

A list of keys, however, doesn’t tell you much. To see the value that’s stored under that key, use memccat. Here’s an example of looking at a user session record in Koha (assuming you’ve set the SessionStorage system preference to use memcached):

memccat --servers 127.0.0.1:11211 KOHA78c879b9942dee326710ce8e046acede
---
_SESSION_ATIME: '1363060711'
_SESSION_CTIME: '1363060711'
_SESSION_ID: 78c879b9942dee326710ce8e046acede
_SESSION_REMOTE_ADDR: 192.168.1.16
branch: CPL
branchname: Centerville
cardnumber: cat
emailaddress: ''
firstname: ''
flags: 1
id: cat
ip: 192.168.1.16
lasttime: '1363060711'
number: 51
surname: cat

And here’s an example of an Evergreen user session cached object:

memccat --servers 127.0.0.1:11211 oils_auth_8682b1017b7b27035576fecbfc7715c4
{"authtime":420,"userobj":{"__c":"au","__p":[null,null,null,null,null,null,null,null,null,"186",null,"t",null,"f",119284,38997,0,0,"2011-05-31T11:17:16-0400","0.00","1-888-555-1234","1923-01-01T00:00:00-0500","user@example.org",null,"2015-10-29T00:00:00-0400","User","Test",186,654440,3,null,null,null,"1358890660.7173220299.6945940294",119284,"f",1,null,"",null,null,10,null,1,null,"t",654440,"user",null,"f","2013-01-22T16:37:40-0500",null,"f"]}}

We’ll let the YAMLites and JSONistas square off outside, and take a look at a final example. This is an excerpt a cached catalog search result in Evergreen:

memccat --servers 127.0.0.1:11211 open-ils.search_4b81a8a59544e8c7e9fdcda357d7b05f
{"0":{"summary":{"checked":630,"visible":"546","excluded":84,"deleted":0,"total":630,"complex_query":1},"results":[["74093"],["130197"], ...., ["880940"],["574457"]]}}

There are other tools that let you manipulate the cache, including memcrm to remove keys and memccp to load key/value pairs into memcached.

For a complete list of the command-line tools provided by libMemcached, check out its documentation. To install them on Debian or Ubuntu, run apt-get install libmemcached-tools. Note that the Debian package renames the tools from ‘memdump’ to ‘memcdump’, ‘memcat’ to ‘memccat’, etc., to avoid a naming conflict with another package.

“It’s just politics.”

This is a common enough phrase, and the usual implication of it is dismissive: if it’s just politics, it’s not about anything really important. It’s grandstanding, it’s just more sound and fury, it’s a sausage factory. At best, it’s the domain of the politicians; let them worry about it. There’s a long post in me about how the attitude behind “it’s just politics” contributes to poor participation in democracy and bad policymaking.

This is not that post.

The inspiration for the post before you was somebody making a comment to more or less that effect the other day in regards to the past and ongoing controversy regarding Koha, its licensing, its trademarks, and its forks. My position on the matter should come as no surprise. If you want Koha, go to http://koha-community.org/. If you’re a librarian using it, please contribute back where you can and participate in the Koha community. If you’re a vendor supporting, implementing, or hacking it, know that it is not just yours, you should give back, obey both the letter and the spirit of the GPL, be a good community member, and don’t worry: you can do all that and still make money! Look Ma! No monopoly!

But dragging myself back on topic, one thing to clear up first: this post is not about the comment that inspired it. I am going after a generality here, not any particular throwaway comment.

What can “it’s just politics” mean when talking about a dispute concerning an open source project and its licensing? Quite a few things:

  1. (Re)opening this can of worms is going to derail any discussion of the software itself for weeks. This can be a very real concern: disputes about the license or the direction of the project can take years to resolve, can become very acrimonious, and frankly can be terribly boring. I, for one, personally don’t find license disputes inherently interesting, and I strongly suspect that most participants in F/OSS projects don’t either. But bitter experience has shown me that sometimes it is necessary to participate anyway and not leave it just to the language lawyers. What can make resolving disputes even more difficult is that email and IRC as communication media have weaknesses that can exacerbate conflict.
  2. Less talk, more code! What doesn’t get done if you’ve just spent an hour fisking the latest volley in the GPL2+ vs. AGPL3 debate? There’s an opportunity cost — that hour wasn’t spent writing some code, or testing, or proofreading the latest documentation edits. That opportunity cost can compound — if you don’t get the kudos for the results of that fisking and miss the warm feeling you get seeing a longstanding bug get closed because of your patch, you may end up disengaging.
  3. Can’t we all get along? It can be very unpleasant being in the middle of an important dispute. While I do think that the Koha community has come out of this stronger than ever, I also mourn the opportunities for human connection and friendships that have been permanently sundered as a result of the conflict.
  4. Newbie here. What is going on?!? It can be very disorienting checking out the mailing list of a F/OSS project you’re considering using only to find that everybody apparently hates each other. It can be even worse if you find yourself making an innocent statement that gets interpreted as firmly putting yourself in one camp or another. Tying this back to the previous point, is the Koha community stronger? Yes. Has it also developed a few shibboleths that cause project regulars to sometimes come down a little too hard on new members of the community? Unfortunately, yes.
  5. From the point of view of an external observer, it’s hard to make sense of what’s going on. It’s all too easy to lose the thread of what’s is being disputed, and the definitive histories of the war tend to come out only after the last bodies have been buried. On the other hand, particularly if you’re an external observer who has some external or self-imposed reason to make judgements about the dispute, do your research: a snap conclusion will almost certainly be the wrong one, or at least lack important nuance.
  6. The noise is getting in the way of figuring out if this software is useful. Fair enough — and if you’re a librarian evaluating ILSs, obviously a key part of your decision should be based on the answer to the following question: will a given ILS solve my problem, either now or in the the realistically foreseeable future. But read on, since that isn’t the only question to be answered.

The outcome of a big dispute in a F/OSS project can be positive, but there’s no question that it can be tremendously messy and painful. But just like in the realm of the elephants, donkeys, and greens, politics informs policy. And policy consequences matter. And there’s no royal road to success:

  • Software doesn’t write itself. People are always involved, and unless you’ve just fired-and-forgotten some code into the wild, any F/OSS project worth thinking about involves more than just one person.
  • The invisible hand is still here. The economics of a F/OSS project may not be based on cash money (though there’s a place for both money and passion), but the fundamental economic problem of resource allocation and human motivation is inescapable.
  • Communities don’t build themselves, and they certainly don’t maintain themselves without effort. In the case of library-related F/OSS projects, there are special considerations: both the library profession and F/OSS hackerdom value sharing. However, there are significant differences in the ways that libraries and hackers tend to communicate and collaborate, and those differences can’t be negotiated without a lot of communication.
  • Regardless of whether you fall more on the “F” side or the “OS” side of the divide in the acronym, F/OSS works for a combination of baldly pragmatic and ethical reasons. But as the very structure of the “F/OSS” acronym implies, there’s are many disagreements and differences of emphasis among F/OSS contributors and users.

What’s missing from these bullet points? The One True Path to Open Source Success. Why is it missing? Because it doesn’t exist: free software has been around long enough that there’s a good body of recommendations on how to do it well, but there’s no consensus.

And if there’s no consensus, then what? It has to be found — or created — or not found, leading to a hopefully amicable parting of the ways. But that can’t happen without discussion, conflict, and resolution. While it certainly isn’t necessary for everybody to participate in every single debate, (constructive!) engagement with the discussion can be a valuable contribution in its own right. If you can help improve how the discussion takes place, even better.

If you or your institution has a stake in the outcome, participating is part of both the duty and promise of F/OSS for libraries: owning our tools, without which our collections will just gather dust.

Put another way, politics, in its broadest and most noble meaning, can’t be avoided, even if engaging means spending some time away from the code. You may as well embrace it.

By the way, I suspect that if you did get manage to get software to write itself, you still couldn’t escape politics. I doubt that an artificial intelligence creative enough to code can be built without incorporating a sufficient degree of complexity that it would be able to avoid all moments of indecision. AI politics may well end up looking rather bizarre to humans, but they’d still be faced with the basic political problem of resolving conflict and allocating resources.

There are lot of good changes coming in Koha 3.4.0, which will be released tomorrow. Check out the current draft of the release notes. But this release of Koha includes some major architectural changes, and although the upgrade process is simple, it definitely pays to read the instructions first.  

In particular, there are two upgrade steps that should not be missed:

Install Template::Toolkit

Koha 3.4.0 uses the Template::Toolkit Perl module instead of HTML::Template::Pro for the OPAC and staff interface templates.  Template::Toolkit must be installed before trying to run the web updater, as the web installer itself now uses TT.  If you run Koha on Debian or Ubuntu, run apt-get install libtemplate-perl. On other Linux and Unix platforms, install the packaged version of TT if available; if a packaged version isn’t available, run cpan Template.

Note that if you’re following the instructions, running ./koha_perl_deps.pl -u -m will catch the TT dependency requirement. Just don’t forget to actually install it.

Run scripts to update your bib records

Koha 3.4.0 will no longer store copies of the item record data as MARC fields in the bibliographic records. This resolves a long-standing performance issue where changing an item record (even just to change its status when it is checked out) required that Koha update the bibliographic record as well. However, this means that during upgrade it is necessary to touch all of the bib records in order to remove the item tags. To do this, run the following steps:

misc/maintenance/remove_items_from_biblioitems.pl --run
misc/migration_tools/rebuild_zebra.pl -b -r

This can take several hours on a large database, so plan accordingly.

I noticed that a patch recently submitted for Koha adds the following line to one of Koha’s Perl modules:

use utf8;

utf8 is a perfectly fine and proper Perl pragma, right? Indeed it is. The problem is that the purpose of the patch is to try to fix an issue with reading and displaying UTF-8 characters from a file. So what does use utf8; contribute to that patch?

Nothing.

The only thing that the utf8 pragma does is signal to the Perl interpreter that the Perl source code is in UTF-8. If you’re using non-ASCII characters in your source code, you’ll need the pragma. For example,

use utf8;
my $str = "La lluvia en España se mantiene principalmente en el llano!";

or even

use utf8;
my $str_en_inglés = "The rain in Spain falls mostly on the plain!";

If what you’re actually trying to do is ensure that the script is handling UTF-8 input and output correctly, use utf8; won’t help. This tutorial will.

With the recent move of the Koha mailing lists to http://lists.koha-community.org/, the full email address for submitting a patch is now koha-patches@lists.koha-community.org. Since that’s a bit long to type every time you submit a patch, here’s one way to save a few keystrokes.

First, create a text file file to define a short alias for the email address. I chose to call the file ~/.git_aliases. The file should contain a line like this:

alias kpatches koha-patches@lists.koha-community.org

Next, tell Git that you want to use this address book file:

$ git config --global sendemail.aliasesfile ~/.git_aliases
$ git config --global sendemail.aliasfiletype mutt

Now, all you have to do is use kpatches in place of the full address. For example:

$ git send-email 0001-bug-4801-fix-paging-in-display-of-staged-bibs-and-im.patch
0001-bug-4801-fix-paging-in-display-of-staged-bibs-and-im.patch

Who should the emails appear to be from? [Galen Charlton ]

Who should the emails be sent to? kpatches

or

$ git send-email -to kpatches 0001-bug-4801-fix-paging-in-display-of-staged-bibs-and-im.patch

I am glad to see that PTFS and its LibLime division have contributed the developments that PTFS has been working on for the past year or so, including features commissioned by the East Brunswick and Middletown Township public libraries and others. The text of LibLime’s announcement makes it clear that this is meant as a submission to Koha 3.2 and (more so) 3.4:

The code for the individual new features included in this version has also been made available for download from the GIT repository. The features in this release were not ready for 3.2, but, pending acceptance by the 3.4 Koha Release Manager, could be included in release 3.4.

Chris Cormack (as 3.4 release manager) and I (as 3.2 release manager) have started work on integrating this work in the Koha. Since 3.2 is in feature freeze, for the most part only the bugfixes from Harley will be included in 3.2, although I am strongly considering bringing in the granular circulation permissions work as well. The majority of the features will make their way into 3.4, although they will go through QA and discussion like any other submission.

So far, so good. As a set of contributions for 3.2 and 3.4, “Harley” represents the continuation of PTFS’ ongoing submissions of code to Koha in the past year. Further, I hope that if PTFS is serious about their push for “agile” programming, that they will make a habit of submitting works in progress for discussion and public QA sooner, as in some cases “Harley” features that were obviously completed months ago were not submitted until now.

But here is where the mixed messages come in: “Harley” is prominently listed on koha.org as a release of Koha. Since no PTFS staff are among the elected release managers or maintainers for Koha, that is overreaching. Ever since Koha expanded beyond New Zealand, no vendor has hitherto unilaterally implied that they were doing mainstream releases of Koha outside of the framework of elected release managers.

Before I go further, let me get a couple things out of the way. If somebody wants to enhance Koha and create installation packages of their work in addition to contributing their changes to the Koha project, that’s fine. In fact, if somebody wants to do that without formally submitting their changes, that’s certainly within the bounds of the GPL, although obviously I’d prefer that we have one Koha instead of a bunch of forks of it. If any library wants to download, install, test, and use “Harley”, that’s fine as well. Although there could be some trickiness upgrading from “Harley” to Koha 3.2 or Koha 3.4, it will certainly be possible to do so in the future.

What I am objecting to is the overreach.  Yes, “Harley” is important.  Yes, I hope it will help open a path to resolve other issues between PTFS/LibLime and the rest of the Koha community.  Yes, I thank PTFS for releasing the code, and in particular publishing it in their Git repository.  That doesn’t make it an official release of Koha; it is still just another contribution to the Koha project, the same as if it came from BibLibre, software.coop, Catalyst, Equinox, one of the many individual librarians contributing to Koha, or any other source.

“Harley” is available for download from LibLime’s website at http://www.liblime.com/downloads.  This is where it belongs.  Any vendor-specific distribution of Koha should be retrievable from the vendor’s own website, but it should not be presented as a formal release.  Perhaps there is room to consider having the Koha download service also offer vendor-specific distributions in addition to the main releases, but if that is desired, it should be proposed and discussed on the community mailing lists.

Updating koha.org to remove the implication that “Harley” is an official release is a simple change to make, and I call upon PTFS to do so.

Please see my disclosure statement. In particular, I am release manager for Koha 3.2 and I work for a competitor of PTFS. This post should not be construed as an official statement by Equinox, however, although I stand by my words.

Earlier today Chris Cormack and I were chatting on IRC about various ways to manage patches, and decided to stage a little tutorial about how to pull from remote Git repositories:

<chris> speaking of public repo’s … i have been pushing to github for a while
<chris> but i have set up git-daemon on my machine at home too
<gmcharlt> chris: anything you’re ready to have me look at to pull?
<chris> not really
<chris> one interesting thing is the dbix_class branch
<chris> http://git.workbuffer.org/cgi-bin/gitweb.cgi?p=koha.git;a=summary
<gmcharlt> even if it’s trivial, it occurs to me that doing it and writing up how we did it might be useful material for a tutorial blog post or maiilings to koha-devel
<chris> lemme check
<chris> tell ya what
<chris> ill do a history.txt update
<chris> and commit that, and we can pull that
<chris> gmcharlt: http://git.workbuffer.org/cgi-bin/gitweb.cgi?p=koha.git;a=shortlog;h=refs/heads/documentation
<chris> so you can a remote for my repo
<chris> git remote add workbuffer.org git://git.workbuffer.org/git/koha.git
<chris> then git checkout -b documentation –track workbuffer.org/documentation
<chris> (probably need a git fetch workbuffer.org too(
<chris> then you can cherry-pick that commit over
<chris> thats one way to do it
<chris> or you could just checkout a branch
<gmcharlt> chris: yeah, I think I’ll do it as a pull
<chris> checkout -b mydocumentation
<chris> git pull workbuffer.org/documentation
<chris> i think that will do it anyway
<gmcharlt> yeah, then into my staging branch
<gmcharlt> git checkout test
<gmcharlt> git merge mydocumentation/documentation
<gmcharlt> or directly
<gmcharlt> git merge workbuffer.org/documentation
<chris> yep
<chris> i think the pull will do fetch + merge for ya
<gmcharlt> it does indeed
<gmcharlt> fetch first, though
<gmcharlt> lets you do git log –pretty=oneline test workbuffer.org/documentation
<chris> good point
<gmcharlt> chris: well, let’s make it official – send a pull request to the patches list
<chris> will do
<gmcharlt> e.g., Subject: PULL – git://git.workbuffer.org/koha.git – documentation – history changes
<gmcharlt> brief description of changes in body
<gmcharlt> something like that
<chris> works for me
<gmcharlt> “Welcome, all, to DVCS performance theatre”
<chris> off it goes
<chris> this was our first git tutorial right there .. quick someone take photos or something 🙂