Sakai Development: Post Ten

So now to work on the presentation side of things, using the velocity files. This is something new to me, so I'm going to start off by reading through a tutorial suggested by Andrew Martin, with reference to the Sakai content tool. The first thing to do is to find the velocity configuration, which is a file listed in the relevant web.xml file, and which turns out to be called (unsurprisingly) velocity.properties. This in turn tells me that the velocity templates are stored where I would expect them to be, in the sakai-src/content/content-tool/tool/src/webapp/vm directory, sorted into two sub-directories, content and resources (which is a directory containing templates for common fragments, judging by the file names). One which we probably want to emulate is content/chef_resources_deleteConfirm.vm, so I'll start by opening that. Basically, the intention is to extend it by adding a metadata form. (I'll also need to make an archiving version equivalent to content/sakai_resources_deleteFinish.vm, I think.)

This is also where I come across the Sakai language support mechanism:

$tlang.getString("del.del")

I'll have to follow that through and add an appropriate entry for archiving, at least for the English language - there aren't all that many languages I could add entries to apart from that one. This, "del.del1", and "gen.cancel" appear several times in the two files. The values in different languages are set in files in config/localization/bundles/src/bundle/org/sakaiproject/localization/bundle/content/ (according to grep -R), so I'll need to sort things there. The file content.properties is the list of default phrases to use, which effectively means American English. So I add in content.properties, which usefully also contains a set of Dublin Core term descriptions:

#Archive Vm
archive.confirm = Archive confirmation and metadata creation
archive.sure = Enter information which describes the collection you have selected, and check the resources listed are correct.
archive.archive = Archive
archive.form = Enter information which describes the collection you have selected, and check the resources listed are correct.
archive.metadatatable = Table allows the input of metadata to describe the collection of items for archiving.
archive.resourcetable = Table holds information about resources selected for archiving. Column headers contain descriptions of contents and links for sorting.

I may need to add more later, when the form itself takes shape properly. Next, the form for confirmation itself. I'll also need to alphabeticise, as that's how the properties file is organised. This will start out as a copy of the delete confirmation form. Most of the changes amount to altering the word "delete" to "archive", to match equivalent updates to the ResourcesAction script which processes these files.

Then the main part of the change, to add the metadata input table. The metadata table needs to come before the file listing, simply because we don't know how many files might be in the listing, and so how far down the page the listing table might reach; confusing if the submitter needs to scroll down several times to find it. It starts with a table tag, like that for the listing, but instantly I see a problem. The file listing table has class "listHier" (presumably hierarchical list), but that is obviously not going to be appropriate for the metadata part of the form. The easiest way to solve this (as the vm file is a fragment of HTML and doesn't have the CSS declarations in it, so finding the right CSS file might be awkward - especially as there are almost 300 of them in the Sakai source code) is to find a table which is the right type, and copy the style information from that. Looking at the tables in the content-tool VMs, there are three types: listHier, itemSummary and no declared class, the first two with or without classes "lines" (not sure what this will do) and labelindnt (indent labels, presumably). I'll try not declaring a class first, so that the table will just inherit the styles current at this point in the page where it is displayed. So I add a table and appropriate DC terms - noticing that Sakai's interpretation of the meaning of a couple of them appears to be slightly different from that of the SWORD library developers, judging by the descriptions of the fields in the vocabulary list.

On to the second form. The delete version of the form seems to be identical, which is sensible - it takes the reader back to the file listing, which would now be missing the deleted items. At least, I think that's what happens; the form has submission buttons, which is a bit strange if this is the case. What I think I want to see displayed here for the archiving is the result of the submission (and a summary of what has been entered by the user on the previous page). So I add appropriate context.put statements to ResourcesAction to display the metadata which has been sent off, and display the results as text.

It would be possible to display the status of the submission in real time, using JQuery and Ajax. But I decided that I probably don't have the time to set this up, and most deposits should be pretty quick. So I just want to get the receipt, check the status, and then (assuming that the status indicates a successful deposit) display the information in the receipt - or at least, some of it, as there are quite a lot of fields which might be included in a returned receipt by the SWORD2 server. The swordapp lists the following fields which may be set in a DepositReceipt object:

receipt.getStatusCode();
receipt.getLocation();
receipt.getDerivedResourceLinks();
receipt.getOriginalDepositLink();
receipt.getEditMediaLink();
receipt.getAtomStatementLink();
receipt.getContentLink();
receipt.getEditLink();
receipt.getOREStatementLink();
receipt.getPackaging();
receipt.getSplashPageLink();
receipt.getStatementLink("application/rdf+xml");
receipt.getStatementLink("application/atom+xml;type=feed");
receipt.getSwordEditLink();
receipt.getTreatment();
receipt.getVerboseDescription();

Without going back to the SWORD2 specification it is not immediately apparent what all these fields mean. First, though, is a field which is not strictly speaking part of the receipt, but tells us whether the deposit itself was successful (if it has the value 200, 201, or 202), or not. The SWORD2 specification lists a small number of error codes which a server could send, so I only need to process these (and provide a default response, just in case - I could understand the web server returning 500 or 404, neither of which are listed part of the specification but would indicate respectively a problem with the way the SWORD server is set up on the web server or an incorrect URL respectively). Some are provided for SWORD2 clients, rather than direct access by a human being using a web browser - to allow, for example, the editing of the metadata for the deposited item at the location of the client rather than needing to go through the repository interface; such things would not be relevant for our purposes. So I need to add code to the script to set appropriate information into the context, so it be available to Velocity, and then add code to the archiveFinish Velocity template to display them.

I'm now going to finally attempt a compilation. This is a make or break moment, as the outcome of the process will be used to decide whether it is worth going on to finish off the work or whether it will need to be left in an incomplete state while I go on to other things that the Research360 project needs.  First time, it complains about the missing swordapp library, which I installed. Then there's the irritating missing end bracket:

sakai-src/content/content-tool/tool/src/java/org/sakaiproject/content/tool/ResourcesAction.java:[6796,73] error: ')' expected

Easily fixed. At least, once I realise I'm looking at line 7696 not line 6796...

And then again, and I get hundreds of errors from files I've not even changed -

could not parse error message:   symbol:   variable Menu
  location: class AttachmentAction
/home/simon/work/sakai-src/content/content-tool/tool/src/java/org/sakaiproject/content/tool/AttachmentAction.java:339: error: cannot find symbol
  state.setAttribute(MenuItem.STATE_MENU, bar);
                     ^

Sakai Development: Post Nine

Before actually starting writing the code to do the deposit itself, I need to set up and include the SWORD2 Java client libraries. If you're not used to github, you might take a while to see the  button, which you can use to download the library as a zip file. Unzip it, cd to the created directory, and run

mvn clean package

to compile (and download a large number of new library files). This should hopefully end up with:

[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: /home/simon/work/swordapp-JavaClient2.0-420485d/target/sword2-client-0.9.2.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3 minutes 24 seconds
[INFO] Finished at: Wed Sep 26 15:14:32 BST 2012
[INFO] Final Memory: 27M/121M
[INFO] ------------------------------------------------------------------------

and then it's a question of copying the jar file to somewhere where it can be picked up by Sakai. This requires two things (assuming my understanding of how maven POM files work is correct):

• Add a dependency to the relevant pom.xml file, which will be picked up on compilation, so that maven will attempt to download the relevant jar file, and, if it can't, will ask for it to be installed locally by hand. The relevant file is in the content/content-tool/tool directory, and needs the following added (with line numbering):

• Import the necessary classes into the ResourcesAction.java file so that the library can be used in this script. This is a simple pair

import org.swordapp.client.SWORDClient;

import org.swordapp.client.SWORDClient.*;

at line 135 of the file.

The code which will use this is based on doFinalizeDelete (lines 6545-6675), which follows doDeleteConfirm when the confirmation is selected. I haven't yet worked out where the actual display of the confirmation request happens, so this is not the last change by any means. The confirmation step could also include the opportunity to select from a list of repositories, and from the collections which exist at the chosen repository (as obtained from the repository SWORD2 service document). But that is a complication I don't really want to get into at this stage, I just want to be able to carry out a simple deposit. So I'm going to have both the repository and the collection set in the configuration file for the servlet.

The process is relatively simple; however, I should point out that I have just noticed in the SWORD2 client documentation that multi-part deposit is not supported, and the way I have been thinking has assumed that it is. So I will have to make a zip file or something of the items to be deposited (as their nature as a collection is important). java.util.zip is a core package, but not one I've ever used before; I'll start by adding the import for the package at the top of the file (new line 52).

The steps for producing a SWORD2 deposit from the selected files are:

• Get archive details from configuration (some will always need to be obtained from configuration, but a service description could be downloaded on the fly to get information about collections etc. - just not in this version as I'm already overrunning the schedule);
• Prepare metadata, using data from the confirmation form, which should basically be a subset of DC terms for use in the deposit (bearing in mind that it's possible for the depositor to go to the repository and change the metadata later if necessary);
• Prepare files - create a zip as a file stream;
• Authenticate to repository using userid/password provided in confirmation form;
• Make deposit and (hopefully) get back a receipt.

While the information given in the swordapp documentation at first looks pretty complete, it is missing some details which I need, as I discover on starting to put the code together. I'll need to look at the source code for the app to get them.

The first issue is with the metadata. Dublin Core is not the only metadata used in SWORD2; there is some basic information which is part of the Atom profile: title, author, last modification date, and so on, as seen in the example here. The documentation gives no information about how to set this, and in fact I can't find anything useful in the source code (the string "updated", which is the element name for the last modification date, does not appear anywhere in the client code). I'm not particularly familiar with Atom, so it is possible that these are optional. I'll ignore this for the moment and see what happens. I'm also going to assume that in order to give multiple entries, I just repeatedly add a term: this needs to be supported for DC contributor - I think this should work, but I haven't actually gone through the apache Abdera library .which the swordapp code uses to check this.

Just at this point Andrew Martin put up a useful blog post which details his journey to working with the SWORD2 java library. He's not doing exactly the same thing, though we have already been in contact. While I need to go deeper into the coding, his post is probably a very useful resource for anyone reading this one.

The next thing to sort out is creating a ZIP file (virtually) to contain the items selected for archiving. I've not done this before, and the ZIP creation stuff I can find online, as well in my ancient Java book,   concentrates on making a ZIP from files (this looks pretty  useful for that, and is likely to form the framework I'll use) rather than from the Sakai content platform where the content may not even be stored in a filesystem. So I need to work out how to get the content out of the files as a Java input stream. I'll start by looking through the ResourcesAction.java code, and then move on to other files in the same directory if I can't find anything. All the input streams in ResourcesAction are for making changes to content rather than reading it - makes sense, as reading is not an action whcih affects the resource. But this code from FilePickerAction.java (lines 1710-13) makes it look very simple:

InputStream contentStream = resource.streamContent();
String contentType = resource.getContentType();
String filename = Validator.getFileName(itemId);
String resourceId = Validator.escapeResourceName(filename);

I just need to work back through the context to be sure that this code is doing what it appears to be doing. Although it doesn't appear to be (because it's in a method for up dating the resource), this is what is is in fact doing, as I eventually discover when I find the relevant javadocs (ContentHostingService, BaseContentService, and ContentResource - not from the current release, though). To re-use this code, the ContentResource class needs to be loaded, which it already is, and the content service needs to be set up (outside the loop which runs through the selected items):

ContentHostingService contentService = (ContentHostingService) toolSession.getAttribute (STATE_CONTENT_SERVICE);

The first problem, then, is that what I have is a ListItem object, when what I want is the itemID (which is a String); this is simple, as id is a property of the ListItem object, so I can just get it. I'll also need to protect against the itemid being null, which I don't think should happen. I'm not quite sure what the correct thing would be to do if it does, so I'll just log a warning if that happens. So the code I add is (lines 6741-4):

String itemid = item.id;

ContentResource resource = null;
if (itemid != null)
{

and then in the conditional block (6752-6774),

resource = contentService.getResource(itemId);
InputStream contentStream = resource.streamContent();

byte[] buf = new byte[1024];

//get filename and add to zip entry
String fileName = item.getName();

if (fileName != null)
{
  zip.putNextEntry(new ZipEntry(fileName);
}
else
{
  zip.putNextEntry(new ZipEntry("Unnamed resource with ID " + itemid);
}

int len;
while ((len = contentStream.read(buf)) > 0) {
  zip.write(buf, 0, len);
}
zip.closeEntry();     
contentStream.close(); 

Some preparation has to happen before this, using piped streams to turn the zip output into the input for the SWORD library, calculating the MD5 hash we want for the deposit stage on the way:

MessageDigest md = MessageDigest.getInstance("MD5");
DigestInputStream dpins = new DigestInputStream(pins, md);

And now we should be in a position to set up the deposit with the SWORD2 library..

It's also occurred to me that the solution to the problem of multiple archives is to embed them in the confirmation web form - the user selects the archive there from a drop down list, and the script reaps the URL used for deposit. So the URL to use is then just a form parameter. Except - the sword client readme file suggests that a collection object (derived from the collection description in a service document) is needed for deposit, so I need to check in the source code to see if there's a method with a deposit URL as an alternative. Turns out that there is, so I'll use that. So we have (ignoring a whole load of exceptions which will surely need to be caught, for the moment):

// Set up authentication
AuthCredentials auth = new AuthCredentials(params.getString("userid"),params.getString("password"));

// Make deposit and (hopefully) get back a receipt
Deposit deposit = new org.swordapp.client.Deposit();
deposit.setFile(dpins);
dpins.close();
pins.close();
pouts.close();
byte[] fileMD5 = md.digest();

deposit.setMimeType("application/zip");
deposit.setFilename(params.getString("title") + ".zip");
deposit.setPackaging(UriRegistry.PACKAGE_SIMPLE_ZIP);
deposit.setMd5(fileMD5);
deposit.setEntryPart(ep);
deposit.setInProgress(true);

// And now make the deposit
DepositReceipt receipt = client.deposit(params.getString("depositurl"), deposit, auth);

The next issue is what to do with the receipt. And how to alert the user to the success or failure of the deposit. The confirmation web page should still be available (especially if it has an indication that the status of the archiving will be displayed there). So it be displayed there, dynamically. So for the moment, I'll just hold on to the receipt and revisit this code when I've written the appropriate Velocity file.

There's just one final bit of code to add to this file, which is to add a call to the confirm delete method   as a new state resolution function (lines 7083-6):

else if(ResourceToolAction.ARCHIVE.equals(actionId))
{
  doArchiveconfirm(data);
}

Then I can start work on the Velocity file. I should say at this point that I don't expect this code to compile without error. I'm absolutely certain there will be exceptions which haven't been caught, and I may well have confused some of the variable names in the course of this. But I want to get on to the next stage before coming back here.

Sakai Development: Post Eight

This post follows straight on from the last, especially as I've missed a constant setting which goes with the ones listed at its end, a new line 857:

ACTIONS_ON_MULTIPLE_ITEMS.add(ActionType.ARCHIVE);

Looking at the next section I might need to change, where permissions are sorted out at lines 1721-1801, I don't think anything needs to be altered, because this draws in constant values which I have already altered. However, things do need to be changed where permissions are set for items. This is done with new lines 2203 and 2209:

boolean canArchive = ContentHostingService.allowArchiveResource(id);
item.setCanArchive(canArchive);

Of course, there will need to be corresponding alterations in the ContentHostingService class. I just need to find it - there is no ContentHostingService.java file in the source code. And searching online doesn't find anything useful. It's time to email those in the know, but meanwhile I can carry on with other bits of code. But I got a really quick answer - before I had a chance to do so, which tells me that I'm on the right lines:

"Yeah this one is in the kernel. The CHS Api lives there.
Sounds like what you are doing is correct. You'll need to add a method there that, most likely, checks some permission and returns true or false. If you want it set via a permission of course. You could just make it always available if that is what you wanted to, then you may not need to mod the kernel.

Then you could do item.canaddarchive(true)"

Thinking a bit more about this, I feel that perhaps I don't need a new kernel method, but I can be more sophisticated than making the service always available. To archive an item is basically making a copy, so what I really should do is to tie the archiving permission to the copy permission. So instead of the lines 2203 and 2209 above, I'll just check the canRead permission which is already there. (It strikes me, though, that in this world of DRM, read and copy are not necessarily the same thing, but never mind.) At least, I'm now at the end of the setting of permission booleans - the next bit of code should actually do something. (And no, I still have no idea why none of the kernel appears to be in the source code I downloaded, but this difficulty is one of the factors in my decision.)

What I want to work on now is to build the three archiving intermediate pages, which should be almost precisely like existing pages. The code for this starts at line 3961, which is where the delete confirmation page building code begins. To remind you, the three intermediate pages are the confirmation, collection metadata entry, and finish; the first and third will basically be copies of the equivalent delete pages and will therefore be built much in the same way. (I expected that most of the work for this little project would consist of copying and then amending existing code, nothing terribly difficult, and this is exactly what is being done here.) Since the code being copied is quite long, I'm not going to quote it all here. The two routines are very similar, so it looks as though copy and paste has already been used. I don't think I have time to look into the lines which are commented "//%%%% FIXME". I'll also need a third copy for MODE_ADD_ARCHIVE_METADATA. The new code becomes lines 4063 to 4207.

The code to call one of these new routines comes next, also a simple copy and modification of existing code. This comes in the context for the main page being re-displayed, as it will be on MODE_ARCHIVE_FINISH: lines 4823-7:

else if (mode.equals (MODE_ARCHIVE_FINISH))
{
  // build the context for the basic step of archive confirm page
  template = buildArchiveFinishContext (portlet, context, data, state);
}

This gives a total of about 200 lines of boilerplate code copied, modified slightly, and inserted back into the class. From this point onwards, we start getting into more exciting development (though there is still a little bit more to add, to call the code to create the context for the metadata form and for the confirmation page - which, it has just occurred to me, could sensibly be the same thing...I may want to revisit some of the above changes to do this, but for the moment I'll just leave things as they are, as it shouldn't do any harm to create constants but not use them).

The model I have used so far for these changes is the existing code for item deletion. Now, there is a slight problem: the actual deletion code appears to delete items one at a time when more than one is selected for deletion, and we can't do this for the archiving; we want one SWORD2 transaction whether there are one or more items. We now need to copy and modify two routines - doCopy, and doDeleteconfirm - which set the application state to doCopy or Deleteconfirm respectively. The first will set the doArchive state, and the second will set Archiveconfirm state, in both cases processing the list of items selected for archiving into a vector format, and these states should then be processed to make the actual archiving or the display of the confirmation page happen. This gives another hundred or so lines of code modified which doesn't really do much.

The next place where something needs to be added is now line 6288. The doDispatchItem method, of which this switch block forms part, is, like most of the rest this class, sparsely commented, but appears to be the part which determines what to do - hence the switch block, with cases corresponding to the different actions. The question is, whether the ARCHIVE case should be like the COPY case or the DELETE case? It's hard to tell without more documentation. The COPY case basically adds the ID of a selected item to a list of items to copy, while the DELETE case actually calls deleteItem - a method we have already decided isn't appropriate to copy directly (as we don't want to break down the archiving of a collection of items into a sequence of archiving actions on the individual items). So the ARCHIVE case needs to be something in between, something like this (I hope):

case ARCHIVE:
  List items_to_be_archived = new ArrayList();
  if(selectedItemId != null)
  {
    items_to_be_archived.add(selectedItemId);
  }
  state.removeAttribute(STATE_ITEMS_TO_BE_MOVED);
  state.setAttribute(STATE_ITEMS_TO_BE_ARCHIVED, items_to_be_archived);
  if (state.getAttribute(STATE_MESSAGE) == null)
  {
    // need new context
    state.setAttribute (STATE_MODE, MODE_ARCHIVE_FINISH);
  }
  break; 

which forms the new lines 6288 to 6301.

Sakai Development: Post Seven

The plan

How exactly do I expect the SWORD2 application to work? From the point of view of a Sakai user, I think that the first version should do the following:

1. add a new action to the drop down menu which appears in a resources page, which would have a configurable name but basically be "Submit to archive";
2. when files and/or directories are selected, this action may display a metadata form which would need to be completed to continue to submit;
3. the items chosen are submitted using SWORD2.

(I'll think about error reporting requirements when I work on each of these in more detail.) The clear limitation of this version is that only one archive can be set, whereas in practice an institution may well have several which are available: a publications archive and a research data archive, for example. So if time permits, a second version will be created which enables a series of archives to appear as actions on the resources pages.

So the development tasks consist of:

1. work out how to add a new action and give it configuration information;
2. work out how to pick up pointers to the selected items and display an error if none are selected (and a list of what has been selected and a confirmation that the list is correct);
3. work out how to create a new form for the metadata and configure its contents and when it is displayed;
4. integrate with the SWORD2 java library for deposit using configured information about the archive (and obtain any other information about the submitted items which is required by SWORD2 for deposit);
5. install patch on server as well as on laptop;
6. test against at least EPrints and DSpace.

It seems to me that the first three tasks are likely to be quite common development requirements for people working on Sakai, so I hope that my description of how to carry them out will be useful.

Figuring Things Out and the First Code Updates

The first thing to do is to figure out where the specific action drop down is actually set up. The easiest way I can think of to do this is to carry out a very simple search of the source code. There are 15,826 files in the source code, according to eclipse, so there may well be lots of hits for words likely to be common in the source code of any application, such as "action". Eclipse does provide a comprehensive interface to search the contents of the files, but it does't appear to do anything really useful for a large number like this, such as indexing them. (It's probably possible to set this up, maybe with an eclipse interface to a tool such as apache's solr, but I don't know how and I'm not sure it would be worth it for a small task like this.) Running a search goes through about 8,000 files and then creates a pop-up which tells me "Problem occurred with search", with a button "Show details" which freezes eclipse when pressed. So this is less useful than it appears to be.

Linux has a large number of command line tools which can be used to search files. There are file indexers around, but I don't run one on my desktop as the indexing process is usually slow and memory intensive. Other tools are rater complicated to use - the find command, for example, has a syntax I can never remember how to use properly even though I first came across it in 1990 in a Unix version which is close to that still available in Linux in 2012 and which appears in several shell scripts I have written over the years. I know what the name of the graphics file which is used to display the button which expands into the drop-down menu I want to change is (because I've found it in the web page source code): icon-dropdn.gif. So I can search for that in the source code, as it's likely to appear near the place I want to work on. (I actually did this before, when trying to get the resources tool to work properly.) Then a simple series of greps finds the file I want in tomcat:

% cd $CATALINA_HOME/webapps
% grep "icon-dropdn.gif" */vm/*
% grep "icon-dropdn.gif" */vm/*/*
sakai-content-tool/vm/content/sakai_filepicker_select.vm: img alt="$tlang.getString(" border="0" class="dropdn" gen.add="gen.add" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_filepicker_select.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.actions="button.actions" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /span class="Apple-tab-span" style="white-space: pre;"    /span>
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.actions="button.actions" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /

(I searched the vm directories because they contain the files which generate the interface for Sakai; I've removed some of the HTML markup from the output lines as a quick way to get the results to display properly in blogger.) The results make it clear that the file we want ends up as sakai-content-tool/vm/content/sakai_resources_list.vm. But what is the file which is used to generate this one in the source code? There is no sakai-content-tool directory there. For this, a very simple find command will do the job perfectly:

% find work/sakai-src -name "sakai_resources_list.vm"

./work/sakai-src/content/content-tool/tool/target/sakai-content-tool-2.8-SNAPSHOT/vm/content/sakai_resources_list.vm

./work/sakai-src/content/content-tool/tool/src/webapp/vm/content/sakai_resources_list.vm

and there we have it: work/sakai-src/content/content-tool/tool/src/webapp/vm/content/sakai_resources_list.vm. Now to actually look at this file in eclipse - but it turns out that the listing in eclipse doesn't follow the directory/file structure of the source code itself and doesn't contain the vm file at all (I should perhaps have expected this, but it's about five years since I last used the IDE). And even on a fairly new laptop bought specifically to have as much RAM as financially possible (3 GB), it's using 50% of the available memory just sitting there open but doing nothing. For reading the source code files, it's definitely going to be easier to use a normal text editor, of which my choice tends to be gedit. So that's where I now open the file. (Note: I will want to create a patch eventually, so I'm going to need to have two copies of the source code, as well as the one in the Eclipse workspace so that the changes can be extracted.)

Now, vm files are not a type of HTML generating script I've used before (the older JSPs being more familiar in my Java development experience), so I want to do a little work understanding the format before going on with this. A quick search on filesuffix.com tells me that the program used to parse them is Apache Velocity, and on that site is a user guide.

While the graphic was useful to find the file, and indeed the part of the file which generates the drop down list which I want to alter, it doesn't do any work of itself. The graphic appears several times on each row of the table containing the directories and files managed by the resource tool, with the various menus being different types of actions, such as ones which are only applicable to folders (e.g. create new folders) and some to both files and folders (copy/paste, etc). Two of the drop down lists apply to single items at a time (Add, Actions) and appear next to each item in the list, rather than letting the user select a group of items and then carry out an action on all of them at once. However, what I want is more an action which is applied to every selected item in the table, whether it is a file or a folder, and these appear at the top of the table, greyed out unless at least one item is selected. (This set of buttons consists of text, rather than a drop down list.) The buttons are "Remove", "Move" and "Copy", by default.

A bit of fiddling with "Inspect Element" in the web browser (a really useful feature for understanding complex web pages), and I see that the relevant lines of code are those near the HTML class definition "navIntraToolLink viewNav" (line 178), which displays a button for each member of an array called listActions.  This is not itself set in this vm file, but comes from the relevant Java class which is listed at the top of the file in a useful comment: "sakai_resources_list.vm, use with org.sakaiproject.content.tool.ResourcesAction.java". So that is the next file to look at, and here it is possible to see the array being created and populated (lines 4315-4336). It's a little difficult to work out what these lines do without a significant amount of context, as they basically turn one type of list into another, and the file is really too big to be readily understood without a great deal more commentary (I've removed much of the spacing to make the code display better in a fairly narrow style of blog post):

ContentCollection collection = ContentHostingService.getCollection(collectionId);

ListItem item = ListItem.getListItem(collection, null, registry, need_to_expand_all, expandedCollections, items_to_be_moved, items_to_be_copied, 0, userSelectedSort, false, null);
Map listActions = new HashMap();
List items = item.convert2list();
for(ListItem lItem : items){
  if(lItem.hasMultipleItemActions()){
    for(String listActionId : lItem.getMultipleItemActions().keySet()){
      ServiceLevelAction listAction = registry.getMultiItemAction(listActionId);
      if(listAction != null){
        listActions.put(listActionId, listAction);
      }
    }
  }
}

This then indicates that the I need to understand the ListItem class to work out how this is populated, because this is what is used to create the initial list which is then manipulated. This class is defined in  the sakaiproject/content/tool/ListItem.java file, another 4000+ lines of code with fairly minimal commenting. I worked through the relevant code (the getListItem method starts on line 151) adding temporary comments to ensure I remembered what I had already worked out. The first important bit is to work out what permissions the user has over the items in the collection, which in fact has an extremely useful comment:

* calculate permissions for this entity.  If its access mode is 
* GROUPED, we need to calculate permissions based on current user's 
* role in group. Otherwise, we inherit from containing collection
* and check to see if additional permissions are set on this entity
* that were't set on containing collection...

and it's a sensible way to do it, dynamically changing the list of actions depending on what the user is allowed to do, but isn't something that I'd thought of. Stupid of me, especially as access control is my main field of expertise... So the place to look is not here at all, but the code which lists the permissions available to be set. The form which does this looks like this in a default setup (presumably the creation of custom groups of users would add new columns):

So to find the vm file which generates the table, I need to find a fairly distinctive looking piece of the HTML source for this table. The name attribute of the form tag, "permissionForm" is an obvious one, and finds me the authz/authz-tool/tool/src/webapp/vm/helper/chef_permissions.vm VM file. The line which generates the rows of the table, which correspond to actions and which we want to add to, is line 65:

#foreach($lock in $abilities)

so next I need to find out where the list $abilities is set. Unlike the VM I looked at earlier, this one doesn't have a useful comment indicating which java file is responsible for its processing, but I think I can assume it will be authz/authz-tool/tool/src/java/org/sakaiproject/authz/tool/PermissionsHelperAction.java. Having done so, I can see that it is from line 120:

private static final String TEMPLATE_MAIN = "helper/chef_permissions";

The abilities variable is exported to the VM in line 385, which is set between lines 368 and 382 from a list of functions (potentially filtered, which is why it takes 14 lines of code). The code is set by lines 357-363:

// in state is the list of abilities we will present
List functions = (List) state.getAttribute(STATE_ABILITIES);
if (functions == null)
  {
    // get all functions prefixed with our prefix
    functions = FunctionManager.getRegisteredFunctions(prefix);
  }

which will first get data from the class attribute STATE_ABILITIES, and, if this is empty, it will ask the FunctionManager for a list of registered functions. I suspect that it is the latter which is needed, as the script then goes on to write the list of functions back into STATE_ABILITIES. The function manager is defined in the class org.sakaiproject.authz.cover.FunctionManager, so that is where I need to look next. However, finding it is more problematic. There is no file in the source code named FunctionManager.java, and no file containing a class definition for FunctionManager. However, I did eventually find some documentation on the use of the FunctionManager which seems to explain how I should add new functions (it appears when searching using google for "sakai FunctionManager" but not when searching the Sakai WIKI through Confluence's own search box...). It's clearly out of date (linking to non-existent files), but I hope still helpful. I also found a document in the reference documentation section of the source code, reference/docs/architecture/sakai_security_reg.doc, which explains how the function manager works. However, what I really need is an overview of how the security in Sakai works, and I eventually found a useful collection of WIKI pages, https://confluence.sakaiproject.org/display/~markjnorton/Sakai+Security+Model and the four linked to from it.

My understanding of the Sakai security model is now this. There are four components: users, (permission) groups, functions, and objects. New groups can be created (using the Security Service); users can be placed in groups (using a GroupProvider); functions can be associated with objects (using the function manager); and users or groups can be granted permission to carry out a function (using a Permission Helper). Resolution of a user's permission to carry out a function on an object is done by the AuthzGroup service.

So what I need to do is to add a new function via the ListItem.java script (not the Permissions Helper as I previously thought) and work out how to add that permission to the owner of a collection of files.  The function manager service documentation tells me that the preferred method to do this is by using Spring to inject the service, via an XML file; however, I quickly discover (with my old friend grep - in this case, the command "grep -R --include *.xml FunctionManager .") that there are no XML files which include the string FunctionManager in the Sakai source. So: should I be good and follow the documentation (last edited in 2010), or should I do this the way that everyone else seems to have done? In fact, it looks on close inspection as though the content manager tool doesn't do things like this at all, because it inherits its permissions from elsewhere (and I've now basically gone round a circle to ResourcesActions.java again). But this time I have more idea what I'm looking for, and can actually start making changes. I'm not sure how or indeed if I can create a configurable name for it, however.

In what follows, line numbers refer to the line number displayed in the editor as the file is changed, so once a line has been added, all those with higher numbers are one greater than they were in the original file. Don't worry: at the end of this process, I'll publish a patch file with all the changes included.

Change line 348 to have ARCHIVE as a member of the list of actions:

CREATE, DELETE, READ, REVISE, SITE_UPDATE, ARCHIVE

Add new line 415 to be similar to the other actions here:

public static final List CONTENT_ARCHIVE_ACTIONS = new ArrayList();

There probably need to be some constants set in this section to govern the behaviour of the archiving process, but at the moment I'm not entirely sure what they should be. Perhaps one to indicate the requirement for metadata, and another to give the status of the archiving process (similar to line 510, "private static final String MODE_DELETE_FINISH = "deleteFinish";" - if that is indeed what this constant indicates!). I have to revisit this. I could really do with finding proper javadocs for this part of the code, but Sakai is exceptionally unhelpful for this. For example, https://confluence.sakaiproject.org/display/DOC/Sakai+CLE+2.8+Release+Notes lists some sources for javadocs, but there are separate javadoc locations for each of the projects which make up sakai, and chasing links in from this list ends up at 404 not found errors in many cases. The Sakai project is desperately in need of a tidier documentation collection for developers, but creating one will be an enormous job. Looking at way that MODE_DELETE_FINISH is used later on, it is used to set up what is displayed after the deletion has occurred, and I need some sort of equivalent, a message indicating the archive submission has been made, as well as a similar confirmation message. So I add a new line 512 and 514 (with a blank line between them):

private static final String MODE_ARCHIVE_FINISH = "archiveFinish";
private static final String MODE_ARCHIVE_CONFIRM = "archiveConfirm";

There is already a "MODE_REVISE_METADATA" constant, which appears to make Sakai display a metadata form, though I presume that this is for a single resource at a time rather than for a collection. So I'm going to want to have a mode for adding archive metadata, which forms a new line 531:

protected static final String MODE_ADD_ARCHIVE_METADATA = "add_archive_metadata";

Further down, there are more constants to add, with comments similar to those already there (lines 605-11):

/** The name of the state attribute containing a list of ListItem objects corresponding to resources selected for submission to the archive */
private static final String STATE_ARCHIVE_ITEMS = PREFIX + REQUEST + "archive_items";
/** The name of the state attribute containing a list of ListItem objects corresponding to nonempty folders selected for submission to the archive */
private static final String STATE_ARCHIVE_ITEMS_NOT_EMPTY = PREFIX + REQUEST + "archive_items_not_empty";
protected static final String STATE_ARCHIVE_SET = PREFIX + REQUEST + "archive_set";

This is proving quite complicated. I think that, if I had more time, I'd probably want to build a generic method for adding new actions, by configuring the action name and a Java class (implementing some interface, say) to carry it out. The difficulty with that approach would presumably be how to handle actions which need to take the control away from the current page, as happens with deletion confirmation and with the archive metadata I'll need.

Next, I will set the VMs to use to handle the deposit confirmation, completion, and the metadata form, at line 785.

private static final String TEMPLATE_ARCHIVE_FINISH = "content/chef_resources_archiveFinish";
private static final String TEMPLATE_ARCHIVE_CONFIRM = "content/chef_resources_archiveConfirm";
private static final String TEMPLATE_ARCHIVE_METADATA "content/sakai_resources_archiveMetadata";

The need to set these in the code rather than making them configurable details seems rather poor design to me, but never mind.

Still more constants need to be added. Archiving should make no change to the resources themselves, so it is an action which should be in the same category as copy. So at line 816, I add:

CONTENT_READ_ACTIONS.add(ActionType.ARCHIVE);

and at 836:

ACTIONS_ON_FOLDERS.add(ActionType.ARCHIVE);

and at 850:

ACTIONS_ON_RESOURCES.add(ActionType.ARCHIVE);

There's a lot more to do, but that will have to be in the next post, I think.

Sakai Development: Post Three

At this point we can start to install Sakai itself. I decided to start with the source archive, mainly because I don't really want to have to debug any serious issues with the development source code. Because of the restrictions on my VM, I saved http://source.sakaiproject.org/release/2.8.2/artifacts/sakai-src-2.8.2.tar.gz to my desktop and then used scp to copy it to the server. I now have sakai-src-2.8.2.tar.gz in my home directory.

Once the source is downloaded, the first new step is to set up some environment variables. The instructions say that the java executables (java, javac, etc.) need to be in the path; in fact, the way that Java is set up on Debian makes this unnecessary, as /usr/bin/java (etc.) are symbolic links to the java executables, which do not need themselves to be in the path. Some work is needed, though; to increase the memory available to Sakai, create the file /usr/local/tomcat/bin/setenv.sh with the contents:

export JAVA_OPTS='-server -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dsun.lang.ClassLoader.allowArraySyntax=true'

Make the new file executable:

$ sudo chmod a+x /usr/local/tomcat/bin/setenv.sh

The required version of subversion is already installed.

Next, install maven 2.2. This is the version of maven in the Debian software repositories as maven2, but this is a package which has a very large number of dependencies, some of which will clash with the already installed Sun java (openjdk, for one). Although java binaries and compilation is supposed to be compatible across the different implementations, it feels wrong to use this maven - and having multiple java implementations on the server when its not necessary seems silly, a recipe for confusion later on. It makes me feel happier to install maven by downloading from apache. (Though in the end, for the sake of time, I chose to download the binary, with no idea about which java was used to compile it...) The installation process is basically the same as that for tomcat.

$ sudo mv apache-maven-2.2.1-bin.tar.gz /usr/local
$ cd /usr/local
$ sudo tar zxvf apache-maven-2.2.1-bin.tar.gz
$ sudo ln -s apache-maven-2.2.1 maven

maven needs some setting up, and I choose to do this globally, so that every new session picks up the home and options for maven. In /etc/bash.bashrc, add the lines:

MAVEN_HOME=/usr/local/maven

export MAVEN_OPTS='-Xms512m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=128m'
PATH=$PATH:$JAVA_HOME/bin:$CATALINA_HOME/bin:$MAVEN_HOME/bin
export JAVA_HOME CATALINA_HOME MAVEN_HOME PATH

at the end of the file (the last two lines replacing the ones given in the previous post). To do the same things for a single user, add settings to .profile in the home directory. When this has been done, run

$ source /etc/bash.bashrc

to import these settings into the current session and then test to be sure that this all worked:

$ mvn --version

Apache Maven 2.2.1 (r801777; 2009-08-06 20:16:01+0100)
Java version: 1.6.0_26
Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux" version: "2.6.32-5-amd64" arch: "amd64" Family: "unix"

Create a local maven repository directory:

$ cd $HOME
$ mkdir -p .m2/repository

and add a settings file to the m2 directory also created by this command, as .m2/settings.xml, containing:

Additional information may need to be added if the server uses a proxy to communicate to the Internet, as maven doesn't check the http_proxy system variable (but go here rather than trying to follow the instructions in the Sakai guide to add the instructions to the settings.xml file). The instructions now tell you to configure Sakai, which is confusing as Sakai has not yet been installed (and the instructions tell the installer to edit as yet non-existent files). There is a minor step missing from the instructions, which is to open the archive of the Sakai source code already downloaded. In the user's home directory,

$ tar zxvf sakai-src-2.8.2.tar.gz
$ cd sakai-src-2.8.2

Next, set up the mysql database to use. The right JDBC driver for the database you intend to use needs to be downloaded and put into /usr/local/tomcat/shared/lib; for mysql, this is available from http://www.mysql.com/products/connector/. The database will also need to be set up as per the configuration entries you set up. For mysql this entails the following activities (with some parts of the server response omitted, and the actual password I used replaced by the word "password"):

$ mysql -u root --password
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.

Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

mysql> create database sakai;
Query OK, 1 row affected (0.00 sec)
mysql>  grant usage on *.* to sakai@localhost identified by 'password';Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on sakai.* to sakai@localhost;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye

Note that I had already set a root password; often, new installations of mysql do not have the password set. To test this worked, I then ran:

$ mysql -u sakai -p 'password' sakai
mysql>; exit
Bye

Now it's possible to carry out some pre-compilation configuration. First, make a new properties file to configure the compilation-time information, such as the server domain name; I felt that the best start was with a copy of the existing exemplar, default.sakai.properties. There is also a file, sample.sakai.properties, which contains all the many possible items it is possible to configure at this point - since the default.sakai.properties file is already 924 lines long, to work through the longer file was a job I hoped would be unnecessary.

$ cd config/configuration/bundles/src/bundle/org/sakaiproject/config/bundle
$ cp default.sakai.properties sakai.properties

Then edit the new sakai.properties. Most of these changes from the default were because I decided to forward only access requests for /portal from apache to the tomcat listener, and so the URLs for navigation all need to start /portal. My expectation at this point is that the changes I make won't matter - they all seem to be in the nature of the sort of local customisation which should easily be changeable later (and, if not, I can always change the properties and compile again).. The contents of a sakai.properties file are documented in a 90 page Microsoft Word file found at sakai-src-2.8.2/reference/docs/architecture/sakai_properties.doc - useful, if a bizarre choice for a documentation format for an open source project. To summarise the main points, as relevant to the compilation I want to carry out:

• The edited file should end up at /usr/local/tomcat/sakai/sakai.properties, and further changes can indeed be made later. The compilation process does not copy this file across, so it will need to be sorted out by hand. One of the consequences of not doing this is that Sakai will attempt to use the default hsqldb database, which has not been installed, and this will prevent the installation from working (the error in this case, which appears in catalina.out, is "java.sql.SQLException: Table not found in statement [select COUNT(*) from SAKAI_SESSION]").
• The connection details for the database to use need to be set up, and the existing hsqldb values commented out.
• The property portalPath is used for adding a pathway to the URL (which should not be used to contain the full path to the portal where there is some extra needed, as here where URLs ending /sakai are forwarded to tomcat); this is not in the default properties file.

Now comes the first moment of truth: compilation. The first stage of this is to compile the master project for Sakai.

$ cd ~/sakai-src-2.8.2/master
$ mvn clean install

This is followed by a lengthy list of missing libraries, followed by downloads and installation of these libraries - which is far nicer than having to fulfil all the dependences by hand. Eventually, there is the magic message:

[INFO] [install:install {execution: default-install}]
[INFO] Installing /home/sjm62/sakai-src-2.8.2/master/pom.xml to /home/sjm62/.m2/repository/org/sakaiproject/master/2.8.2/master-2.8.2.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 14 seconds
[INFO] Finished at: Wed Jul 18 14:15:09 BST 2012
[INFO] Final Memory: 16M/495M
[INFO] ------------------------------------------------------------------------

Then, the actual compilation and deployment to tomcat of Sakai itself. I'm aiming to compile the cafe build, recommended for new developers.

$ cd ~/sakai-src-2.8.2
$ mvn -Pcafe clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat

I set this going and then immediately realised that I don't have write permission to /usr/local/tomcat. As expected, this ends with

[INFO] [sakai:deploy {execution: default-cli}]
[INFO] Copy /home/sjm62/.m2/repository/javax/servlet/jstl/1.1.2/jstl-1.1.2.jar to /usr/local/tomcat/shared/lib/jstl-1.1.2.jar
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Fialed to deploy to container :/usr/local/tomcat/shared/lib/jstl-1.1.2.jar (Permission denied)
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6 minutes 36 seconds
[INFO] Finished at: Wed Jul 18 14:26:33 BST 2012
[INFO] Final Memory: 76M/495M
[INFO] ------------------------------------------------------------------------

What doesn't occur to me at this point is that the quick way to fix this problem is to give myself write permission to /usr/local/tomcat and its sub-directories, at least temporarily. Instead, I try using sudo to allow the installation to occur.

$ sudo mvn clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat

Predictably, this also fails (the reason being that I created the sakai environment in  /home/sjm62/.m2, so if I run the command as sudo, the information fails to be found). So now try:

$ sudo mvn -s /home/sjm62/.m2/settings.xml clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat

This seems to work better, though I'm not at all sure what it might be doing to the ownership and permissions in ~/.m2. Download checksums failed for a fair number of the required resources, with no obvious effect (It looks from this as though the checksum of the downloaded file is compared to the stored checksum, but all that happens if they are different is that a warning is given, but the downloaded file is used anyway.) After twenty minutes downloading and compilation, this happens:

[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 9 source files to /home/sjm62/sakai-src-2.8.2/osp/wizard/tool/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
Failure executing javac, but could not parse the error:

The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
...
(tedious details omitted)
...
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18 minutes 30 seconds
[INFO] Finished at: Wed Jul 18 15:01:39 BST 2012
[INFO] Final Memory: 102M/243M
[INFO] ------------------------------------------------------------------------

This suggests that the maven options set the memory size too low, even though I used the values from the documentation. I doubled the values, and ran the compilation again, with the same result. (It's very frustrating to have the compilation run for almost twenty minutes, then run out of memory.) It looks as though more thought is needed - perhaps the virtual machine has too little memory. Indeed, with no interactive processes running, the machine has 584 Mb used out of a total of 1002 Mb available of virtual memory: so setting the maximum memory usage of maven above 512 Mb will be pointless (the information comes from running free -m). According to top, the most memory intensive process running is tomcat, so I now propose to run the compilation once more, with tomcat shut down (must remember to restart it later...). This time I will also monitor the memory usage in another shell, using top and also running tail -f /var/log/messages, which should indicate problems if the machine's memory is full (the page at http://rimuhosting.com/howto/memory.jsp is a very helpful introduction to the diagnosis of linux memory issues, BTW). There is no obvious message, though it is possible that

Jul 19 09:56:48 float mpt-statusd: detected non-optimal RAID status

is connected, as Linux memory usage is increased by the caching of disk blocks in spare memory. (I would expect Linux memory management to be sophisticated enough to stop caching the blocks at pretty short notice when an interactive process wants the memory, however.) So my conclusion now is - I should increase the memory allocation for maven and try again. This time, MAVEN_OPTS is set to "-Xms1024m -Xmx2048m -XX:PermSize=1024m -XX:MaxPermSize=2048m" - and the build still fails. It's time to hit the Sakai dev mailing list. Posting at 1.45AM Pacific time unfortunately makes it likely I'll need to wait for an answer - so I'll do something else for a while. The answer comes fairly quickly, but apart from fixing obvious typos, leads to circling round whether or not I should be using root or sudo or neither. (Note: I can't just create new users on this VM in the obvious way, because it's set up to use an LDAP server for authentication, requiring userIDs other than root to match ones on the LDAP. There's probably a way round this, but it's not worth the time to sort it out, I don't think.) Also, the suggestion that tomcat needs to be run as a non-root user is something I thought about, but then didn't do, as apache's tomcat documentation says that there shouldn't be issues. (And applications writing to files outside the tomcat tree seems to me to be the application's security problem not tomcat's - especially as it's not likely to be possible if selinux is enabled...) Anyway, self-justification apart, the solution is to get more memory for the virtual machine. Even with 2Gb virtual memory, the compilation process takes up 90% of it, so it is clear that the maven options suggested in the documentation are rather on the low side of what is needed, at least on Debian. But with the upgrade and write permission to /usr/local/tomcat, the compilation now runs to a successful conclusion:

$ ls /usr/local/tomcat/webapps/
access.war         profile2-tool.war  sakai-rutgers-linktool.war
accountvalidator        profile-tool   sakai-rwiki-tool.war
accountvalidator.war        profile-tool.war   sakai-search-tool
authn.war         providers   sakai-search-tool.war
balancer         providers.war   sakai-sections-tool.war
courier.war         ROOT    sakai-siteassociation-tool.war
dav.war          sakai-alias-tool.war  sakai-site-manage-group-helper.war
direct          sakai-announcement-tool.war sakai-site-manage-group-section-role-helper.war
direct.war         sakai-archive-tool.war  sakai-site-manage-link-helper.war
emailtemplateservice-tool      sakai-assignment-tool.war sakai-site-manage-participant-helper.war
emailtemplateservice-tool.war  sakai-authz-tool.war  sakai-site-manage-tool.war
imsblis          sakai-axis   sakai-site-pageorder-helper.war
imsblis.war         sakai-axis.war   sakai-site-tool.war
imsblti          sakai-calendar-summary-tool.war sakai-syllabus-tool.war
imsblti.war         sakai-calendar-tool.war  sakai-tool-tool-su.war
jsf-resource         sakai-chat-tool.war  sakai-usermembership-tool.war
jsf-resource.war        sakai-citations-tool.war  sakai-user-tool-admin-prefs.war
jsp-examples         sakai-content-tool.war  sakai-user-tool-prefs.war
library.war         sakai-editor.war   sakai-user-tool.war
login-render.war        sakai-fck-connector.war  sakai-web-tool.war
messageforums-tool        sakai-gradebook-testservice.war samigo-app
messageforums-tool.war        sakai-gradebook-tool.war  samigo-app.war
osp-common-tool.war        sakai-help-tool.war  savecite.war
osp-glossary-tool.war        sakai-login-tool.war  scheduler-tool
osp-jsf-example.war        sakai-mailarchive-james.war scheduler-tool.war
osp-jsf-resource.war        sakai-mailarchive-tool.war servlets-examples
osp-matrix-tool.war        sakai-memory-tool.war  sitestats-tool
osp-portal-tool.war        sakai-message-tool.war  sitestats-tool.war
osp-portal.war         sakai-metaobj-tool.war  tomcat-docs
osp-presentation-tool.war      sakai-news-tool.war  tool.war
osp-wizard-tool.war        sakai-podcasts.war  webdav
podcasts.war         sakai-postem-tool.war  web.war
polls-tool         sakai-presence-tool.war  wiki.war
polls-tool.war         sakai-reset-pass   x
portal-render.war        sakai-reset-pass.war  xsl-portal.war
portal.war         sakai-rights-tool.war  x.war
profile2-tool         sakai-roster-tool.war

Now on to the final stage of getting Sakai up and running. Restart tomcat:

$ sudo /usr/local/tomcat/bin/startup.sh 
[sudo] password for sjm62: 
Using CATALINA_BASE:   /usr/local/tomcat
Using CATALINA_HOME:   /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME:        /usr/lib/jvm/java-6-sun/
Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar

and then it should be visible in the web browser. However, it isn't, and there are large numbers of errors in the tomcat logs.

I found that recompiling and installing over an existing, failed, sakai build caused huge numbers of problems in tomcat (the re-installed version is basically missing libraries which installed fine the first time around, even if it does have some which were originally missing). This seems to be the problem discussed in http://collab.sakaiproject.org/pipermail/sakai-dev/2009-August/002909.html and the solution is to re-install tomcat completely and remove the existing maven project in .m2/repository/org/sakaiproject directory. However, even then I was still getting errors like the following (note that the full error only appears in the localhost.[date].log file, not in catalina.out, which has an exceptionally unhelpful short version of the first line only):

SEVERE: Error configuring application listener of class org.sakaiproject.util.ToolListener
java.lang.ClassNotFoundException: org.sakaiproject.util.ToolListener
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1438)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1284)
(etc.)

(This particular error is repeated a large number of times, presumably once for each time a servlet tries to load the class.) I'm not sure what the cause is - permissions, or not removing enough of failed installations, perhaps - but it's fixed by getting rid of the entire sakai and tomcat installation and starting again from scratch. Once I remember to:

• ensure that CATALINA_HOME is correctly set;
• copy the setenv.sh file to $CATALINA_HOME/bin;
• copy the mysql JDBC driver JAR file to $CATALINA_HOME/shared/lib;
• copy the sakai.settings file to $CATALINA_HOME/sakai;

then I end up having an installation I can see from a web browser. There are obviously still some fairly serious problems, as can be seen from the screenshot to the right, but I feel that I'm starting to get somewhere.

Sakai Development: Post Two

The virtual host is now operational, so it is time to start setting up the environment for installing the Sakai CLE (2.8.2) from source. The machine is running debian 6.0.5.

In the past, I haven't generally installed java and tomcat on linux machines in the standard ways, because when I started doing that, the easiest way to do the installs was to download the packages and install in /usr/local, setting up symlinks where necessary. But now I think would be a good time to start doing things the standard ways, which should in 2012 be much better than they were in 1998. At least, I hope so!

Note that I am doing all this using sudo. It is not always installed on debian systems, and you may prefer to do this as the root user.

Preferred versions of java and tomcat are Oracle's Sun java SE 6 (the JDK) and apache tomcat 5.5.35 respectively (even though end of life for 5.5.35 is September 2012). So to start with, install java from the debian packages, following the useful instructions at http://www.thegeekstuff.com/2009/09/how-to-install-java-on-ubunt-debiau/:

$ sudo apt-get install sun-java6-jdk
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Package sun-java6-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'sun-java6-jdk' has no installation candidate

Looks as though the repository which contains sun-java6-jdk is missing. It's in the non-free repository, which can be added by adding "non-free" at the end of the first two uncommented lines in /etc/apt/sources.list:

deb http://ftp.uk.debian.org/debian squeeze main non-free
deb-src http://ftp.uk.debian.org/debian squeeze main non-free

(I used vi to do this.) Then update the list of packages:

$ sudo apt-get update
$ sudo apt-get install sun-java6-jdk

and indicate acceptance of the license which is displayed (press tab to select, return to activate it) - you should have to do this for two different screens. Check that this is now working, with:

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Success. Now set up the environment variables for to enable tomcat to find this java when started using sudo. Edit /etc/bash.bashrc and add

JAVA_HOME=/usr/lib/jvm/java-6-sun/

export JAVA_HOME

as a new last line. (This method will also work if you wish to start and stop tomcat while logged in as root, but will only be picked up the next time you log in, not in the current session.) To make this value be set each time sudo is used, edit /etc/sudoers and add the line

Defaults env_keep += "JAVA_HOME"

after the existing Defaults line(s).

So now I need to install tomcat. The problem here is that the debian packages are now version 6, and Sakai 2.8 is not compatible with this version, despite an end of life date of September 2012. This is not the only problem, as according to http://www.crazysquirrel.com/computing/debian/servers/debian-tomcat-package.jspx, installation from the debian package is not recognised as valid by Eclipse, and this could well cause problems later if I want to use Eclipse for development - though I would be running Eclipse on my laptop, not on the server, hopefully. It's also worrying that the release notes for tomcat 5.5.35 indicate that it requires the Java JRE 5.0 by default, rather than Java 6 as recommended by the Sakai WIKI.

However, the strongest statement seems to me to be that Sakai 2.8 is not compatible with tomcat version 6. So I think that I'll need to install tomcat 5.5.35 from source. (The best option, given infinite time, would be to wait for the release of Sakai CLE 2.9, but my timetable does not permit that.) I'll follow the instructions on the crazysquirrel link already listed, as I can never keep the details of how to set up the Apache-to-tomcat connectors in my head.

Download the binary core distribution files from http://tomcat.apache.org/download-55.cgi?Preferred=http%3A%2F%2Fmirrors.ukfast.co.uk%2Fsites%2Fftp.apache.org%2F. The VM I'm using has been set up so that I can't connect from it to random sites on the Internet, so I download the file to my desktop and copy it to the VM.. I now have apache-tomcat-5.5.35.tar.gz sitting in my development space home directory. Actually installing tomcat is easy:

$ sudo mv apache-tomcat-5.5.35.tar.gz /usr/local

$ cd /usr/local
$ sudo tar zxvf apache-tomcat-5.5.35.tar.gz
$ sudo ln -s apache-tomcat-5.5.35 tomcat

$ cd

The symlink created in the last step is intended to make life easier: when upgrading tomcat, it shouldn't then be necessary to re-configure everything to point to the new location of tomcat.

As with java, system-wide environment variables should be set up. Edit /etc/bash.bashrc and add

CATALINA_HOME='/usr/local/tomcat'
PATH=$PATH:$CATALINA_HOME/bin
export JAVA_HOME CATALINA_HOME PATH

(The last line replaces the last line from the java installation step.)

Third step - set up mod_proxy_ajp. This is the simplest way to link apache and tomcat, though some people prefer mod_jk. mod_proxy_ajp is compiled into the debian apache2 package, but is not loaded. The same is true of the other proxy modules, some of which are needed for mod_proxy_ajp to work. To fix this, add symlinks for each proxy module configuration file from /etc/apache2/mods-available to /etc/apache2/mods-enabled:

$ cd /etc/apache2/mods-enabled
$ sudo ln -s ../mods-available/proxy* .
$ cd

Now configure mod_proxy_ajp. There should already be a connector configured at the tomcat end listening on port 8009 (look in /usr/local/tomcat/conf/server.xml to check). Apache will also need to be configured. I decided that access to Sakai should be via the URL /sakai, so my extra configuration looks like this:

  
   AddDefaultCharset Off
   Order deny,allow
   Allow from all

   ProxyPass /sakai ajp://localhost:8009/]
   ProxyPassReverse /sakai ajp://localhost:8009

This needs to go in /etc/apache2/sites-available/default between the last  and the ErrorLog directive.

Finally, restart apache:

$ sudo /etc/init.d/apache2 reload
Reloading web server config: apache2.

and start tomcat:

$ sudo /usr/local/tomcat/bin/startup.sh

The remaining component to the server environment (load balancing being unnecessary in the development context) which needs to be set up is the database. The Sakai 2.8 release notes give a choice of Oracle or MySQL, and in the development environment, MySQL is clearly the right choice. Run

$ sudo apt-get install mysql-server

and all the needed mysql packages will be installed, including the command line client. You will be asked to change the root password for the installation as part of the installation process, and at the end of it the MySQL server will be started.