Sakai Development: Post Seven

The plan

How exactly do I expect the SWORD2 application to work? From the point of view of a Sakai user, I think that the first version should do the following:

1. add a new action to the drop down menu which appears in a resources page, which would have a configurable name but basically be "Submit to archive";
2. when files and/or directories are selected, this action may display a metadata form which would need to be completed to continue to submit;
3. the items chosen are submitted using SWORD2.

(I'll think about error reporting requirements when I work on each of these in more detail.) The clear limitation of this version is that only one archive can be set, whereas in practice an institution may well have several which are available: a publications archive and a research data archive, for example. So if time permits, a second version will be created which enables a series of archives to appear as actions on the resources pages.

So the development tasks consist of:

1. work out how to add a new action and give it configuration information;
2. work out how to pick up pointers to the selected items and display an error if none are selected (and a list of what has been selected and a confirmation that the list is correct);
3. work out how to create a new form for the metadata and configure its contents and when it is displayed;
4. integrate with the SWORD2 java library for deposit using configured information about the archive (and obtain any other information about the submitted items which is required by SWORD2 for deposit);
5. install patch on server as well as on laptop;
6. test against at least EPrints and DSpace.

It seems to me that the first three tasks are likely to be quite common development requirements for people working on Sakai, so I hope that my description of how to carry them out will be useful.

Figuring Things Out and the First Code Updates

The first thing to do is to figure out where the specific action drop down is actually set up. The easiest way I can think of to do this is to carry out a very simple search of the source code. There are 15,826 files in the source code, according to eclipse, so there may well be lots of hits for words likely to be common in the source code of any application, such as "action". Eclipse does provide a comprehensive interface to search the contents of the files, but it does't appear to do anything really useful for a large number like this, such as indexing them. (It's probably possible to set this up, maybe with an eclipse interface to a tool such as apache's solr, but I don't know how and I'm not sure it would be worth it for a small task like this.) Running a search goes through about 8,000 files and then creates a pop-up which tells me "Problem occurred with search", with a button "Show details" which freezes eclipse when pressed. So this is less useful than it appears to be.

Linux has a large number of command line tools which can be used to search files. There are file indexers around, but I don't run one on my desktop as the indexing process is usually slow and memory intensive. Other tools are rater complicated to use - the find command, for example, has a syntax I can never remember how to use properly even though I first came across it in 1990 in a Unix version which is close to that still available in Linux in 2012 and which appears in several shell scripts I have written over the years. I know what the name of the graphics file which is used to display the button which expands into the drop-down menu I want to change is (because I've found it in the web page source code): icon-dropdn.gif. So I can search for that in the source code, as it's likely to appear near the place I want to work on. (I actually did this before, when trying to get the resources tool to work properly.) Then a simple series of greps finds the file I want in tomcat:

% cd $CATALINA_HOME/webapps
% grep "icon-dropdn.gif" */vm/*
% grep "icon-dropdn.gif" */vm/*/*
sakai-content-tool/vm/content/sakai_filepicker_select.vm: img alt="$tlang.getString(" border="0" class="dropdn" gen.add="gen.add" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_filepicker_select.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.actions="button.actions" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /span class="Apple-tab-span" style="white-space: pre;"    /span>
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.add="button.add" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /
sakai-content-tool/vm/content/sakai_resources_list.vm: img alt="$tlang.getString(" border="0" button.actions="button.actions" class="dropdn" icon-dropdn.gif="icon-dropdn.gif" nbsp="nbsp" sakai="sakai" src="#imageLink(" /

(I searched the vm directories because they contain the files which generate the interface for Sakai; I've removed some of the HTML markup from the output lines as a quick way to get the results to display properly in blogger.) The results make it clear that the file we want ends up as sakai-content-tool/vm/content/sakai_resources_list.vm. But what is the file which is used to generate this one in the source code? There is no sakai-content-tool directory there. For this, a very simple find command will do the job perfectly:

% find work/sakai-src -name "sakai_resources_list.vm"

./work/sakai-src/content/content-tool/tool/target/sakai-content-tool-2.8-SNAPSHOT/vm/content/sakai_resources_list.vm

./work/sakai-src/content/content-tool/tool/src/webapp/vm/content/sakai_resources_list.vm

and there we have it: work/sakai-src/content/content-tool/tool/src/webapp/vm/content/sakai_resources_list.vm. Now to actually look at this file in eclipse - but it turns out that the listing in eclipse doesn't follow the directory/file structure of the source code itself and doesn't contain the vm file at all (I should perhaps have expected this, but it's about five years since I last used the IDE). And even on a fairly new laptop bought specifically to have as much RAM as financially possible (3 GB), it's using 50% of the available memory just sitting there open but doing nothing. For reading the source code files, it's definitely going to be easier to use a normal text editor, of which my choice tends to be gedit. So that's where I now open the file. (Note: I will want to create a patch eventually, so I'm going to need to have two copies of the source code, as well as the one in the Eclipse workspace so that the changes can be extracted.)

Now, vm files are not a type of HTML generating script I've used before (the older JSPs being more familiar in my Java development experience), so I want to do a little work understanding the format before going on with this. A quick search on filesuffix.com tells me that the program used to parse them is Apache Velocity, and on that site is a user guide.

While the graphic was useful to find the file, and indeed the part of the file which generates the drop down list which I want to alter, it doesn't do any work of itself. The graphic appears several times on each row of the table containing the directories and files managed by the resource tool, with the various menus being different types of actions, such as ones which are only applicable to folders (e.g. create new folders) and some to both files and folders (copy/paste, etc). Two of the drop down lists apply to single items at a time (Add, Actions) and appear next to each item in the list, rather than letting the user select a group of items and then carry out an action on all of them at once. However, what I want is more an action which is applied to every selected item in the table, whether it is a file or a folder, and these appear at the top of the table, greyed out unless at least one item is selected. (This set of buttons consists of text, rather than a drop down list.) The buttons are "Remove", "Move" and "Copy", by default.

A bit of fiddling with "Inspect Element" in the web browser (a really useful feature for understanding complex web pages), and I see that the relevant lines of code are those near the HTML class definition "navIntraToolLink viewNav" (line 178), which displays a button for each member of an array called listActions.  This is not itself set in this vm file, but comes from the relevant Java class which is listed at the top of the file in a useful comment: "sakai_resources_list.vm, use with org.sakaiproject.content.tool.ResourcesAction.java". So that is the next file to look at, and here it is possible to see the array being created and populated (lines 4315-4336). It's a little difficult to work out what these lines do without a significant amount of context, as they basically turn one type of list into another, and the file is really too big to be readily understood without a great deal more commentary (I've removed much of the spacing to make the code display better in a fairly narrow style of blog post):

ContentCollection collection = ContentHostingService.getCollection(collectionId);

ListItem item = ListItem.getListItem(collection, null, registry, need_to_expand_all, expandedCollections, items_to_be_moved, items_to_be_copied, 0, userSelectedSort, false, null);
Map listActions = new HashMap();
List items = item.convert2list();
for(ListItem lItem : items){
  if(lItem.hasMultipleItemActions()){
    for(String listActionId : lItem.getMultipleItemActions().keySet()){
      ServiceLevelAction listAction = registry.getMultiItemAction(listActionId);
      if(listAction != null){
        listActions.put(listActionId, listAction);
      }
    }
  }
}

This then indicates that the I need to understand the ListItem class to work out how this is populated, because this is what is used to create the initial list which is then manipulated. This class is defined in  the sakaiproject/content/tool/ListItem.java file, another 4000+ lines of code with fairly minimal commenting. I worked through the relevant code (the getListItem method starts on line 151) adding temporary comments to ensure I remembered what I had already worked out. The first important bit is to work out what permissions the user has over the items in the collection, which in fact has an extremely useful comment:

* calculate permissions for this entity.  If its access mode is 
* GROUPED, we need to calculate permissions based on current user's 
* role in group. Otherwise, we inherit from containing collection
* and check to see if additional permissions are set on this entity
* that were't set on containing collection...

and it's a sensible way to do it, dynamically changing the list of actions depending on what the user is allowed to do, but isn't something that I'd thought of. Stupid of me, especially as access control is my main field of expertise... So the place to look is not here at all, but the code which lists the permissions available to be set. The form which does this looks like this in a default setup (presumably the creation of custom groups of users would add new columns):

So to find the vm file which generates the table, I need to find a fairly distinctive looking piece of the HTML source for this table. The name attribute of the form tag, "permissionForm" is an obvious one, and finds me the authz/authz-tool/tool/src/webapp/vm/helper/chef_permissions.vm VM file. The line which generates the rows of the table, which correspond to actions and which we want to add to, is line 65:

#foreach($lock in $abilities)

so next I need to find out where the list $abilities is set. Unlike the VM I looked at earlier, this one doesn't have a useful comment indicating which java file is responsible for its processing, but I think I can assume it will be authz/authz-tool/tool/src/java/org/sakaiproject/authz/tool/PermissionsHelperAction.java. Having done so, I can see that it is from line 120:

private static final String TEMPLATE_MAIN = "helper/chef_permissions";

The abilities variable is exported to the VM in line 385, which is set between lines 368 and 382 from a list of functions (potentially filtered, which is why it takes 14 lines of code). The code is set by lines 357-363:

// in state is the list of abilities we will present
List functions = (List) state.getAttribute(STATE_ABILITIES);
if (functions == null)
  {
    // get all functions prefixed with our prefix
    functions = FunctionManager.getRegisteredFunctions(prefix);
  }

which will first get data from the class attribute STATE_ABILITIES, and, if this is empty, it will ask the FunctionManager for a list of registered functions. I suspect that it is the latter which is needed, as the script then goes on to write the list of functions back into STATE_ABILITIES. The function manager is defined in the class org.sakaiproject.authz.cover.FunctionManager, so that is where I need to look next. However, finding it is more problematic. There is no file in the source code named FunctionManager.java, and no file containing a class definition for FunctionManager. However, I did eventually find some documentation on the use of the FunctionManager which seems to explain how I should add new functions (it appears when searching using google for "sakai FunctionManager" but not when searching the Sakai WIKI through Confluence's own search box...). It's clearly out of date (linking to non-existent files), but I hope still helpful. I also found a document in the reference documentation section of the source code, reference/docs/architecture/sakai_security_reg.doc, which explains how the function manager works. However, what I really need is an overview of how the security in Sakai works, and I eventually found a useful collection of WIKI pages, https://confluence.sakaiproject.org/display/~markjnorton/Sakai+Security+Model and the four linked to from it.

My understanding of the Sakai security model is now this. There are four components: users, (permission) groups, functions, and objects. New groups can be created (using the Security Service); users can be placed in groups (using a GroupProvider); functions can be associated with objects (using the function manager); and users or groups can be granted permission to carry out a function (using a Permission Helper). Resolution of a user's permission to carry out a function on an object is done by the AuthzGroup service.

So what I need to do is to add a new function via the ListItem.java script (not the Permissions Helper as I previously thought) and work out how to add that permission to the owner of a collection of files.  The function manager service documentation tells me that the preferred method to do this is by using Spring to inject the service, via an XML file; however, I quickly discover (with my old friend grep - in this case, the command "grep -R --include *.xml FunctionManager .") that there are no XML files which include the string FunctionManager in the Sakai source. So: should I be good and follow the documentation (last edited in 2010), or should I do this the way that everyone else seems to have done? In fact, it looks on close inspection as though the content manager tool doesn't do things like this at all, because it inherits its permissions from elsewhere (and I've now basically gone round a circle to ResourcesActions.java again). But this time I have more idea what I'm looking for, and can actually start making changes. I'm not sure how or indeed if I can create a configurable name for it, however.

In what follows, line numbers refer to the line number displayed in the editor as the file is changed, so once a line has been added, all those with higher numbers are one greater than they were in the original file. Don't worry: at the end of this process, I'll publish a patch file with all the changes included.

Change line 348 to have ARCHIVE as a member of the list of actions:

CREATE, DELETE, READ, REVISE, SITE_UPDATE, ARCHIVE

Add new line 415 to be similar to the other actions here:

public static final List CONTENT_ARCHIVE_ACTIONS = new ArrayList();

There probably need to be some constants set in this section to govern the behaviour of the archiving process, but at the moment I'm not entirely sure what they should be. Perhaps one to indicate the requirement for metadata, and another to give the status of the archiving process (similar to line 510, "private static final String MODE_DELETE_FINISH = "deleteFinish";" - if that is indeed what this constant indicates!). I have to revisit this. I could really do with finding proper javadocs for this part of the code, but Sakai is exceptionally unhelpful for this. For example, https://confluence.sakaiproject.org/display/DOC/Sakai+CLE+2.8+Release+Notes lists some sources for javadocs, but there are separate javadoc locations for each of the projects which make up sakai, and chasing links in from this list ends up at 404 not found errors in many cases. The Sakai project is desperately in need of a tidier documentation collection for developers, but creating one will be an enormous job. Looking at way that MODE_DELETE_FINISH is used later on, it is used to set up what is displayed after the deletion has occurred, and I need some sort of equivalent, a message indicating the archive submission has been made, as well as a similar confirmation message. So I add a new line 512 and 514 (with a blank line between them):

private static final String MODE_ARCHIVE_FINISH = "archiveFinish";
private static final String MODE_ARCHIVE_CONFIRM = "archiveConfirm";

There is already a "MODE_REVISE_METADATA" constant, which appears to make Sakai display a metadata form, though I presume that this is for a single resource at a time rather than for a collection. So I'm going to want to have a mode for adding archive metadata, which forms a new line 531:

protected static final String MODE_ADD_ARCHIVE_METADATA = "add_archive_metadata";

Further down, there are more constants to add, with comments similar to those already there (lines 605-11):

/** The name of the state attribute containing a list of ListItem objects corresponding to resources selected for submission to the archive */
private static final String STATE_ARCHIVE_ITEMS = PREFIX + REQUEST + "archive_items";
/** The name of the state attribute containing a list of ListItem objects corresponding to nonempty folders selected for submission to the archive */
private static final String STATE_ARCHIVE_ITEMS_NOT_EMPTY = PREFIX + REQUEST + "archive_items_not_empty";
protected static final String STATE_ARCHIVE_SET = PREFIX + REQUEST + "archive_set";

This is proving quite complicated. I think that, if I had more time, I'd probably want to build a generic method for adding new actions, by configuring the action name and a Java class (implementing some interface, say) to carry it out. The difficulty with that approach would presumably be how to handle actions which need to take the control away from the current page, as happens with deletion confirmation and with the archive metadata I'll need.

Next, I will set the VMs to use to handle the deposit confirmation, completion, and the metadata form, at line 785.

private static final String TEMPLATE_ARCHIVE_FINISH = "content/chef_resources_archiveFinish";
private static final String TEMPLATE_ARCHIVE_CONFIRM = "content/chef_resources_archiveConfirm";
private static final String TEMPLATE_ARCHIVE_METADATA "content/sakai_resources_archiveMetadata";

The need to set these in the code rather than making them configurable details seems rather poor design to me, but never mind.

Still more constants need to be added. Archiving should make no change to the resources themselves, so it is an action which should be in the same category as copy. So at line 816, I add:

CONTENT_READ_ACTIONS.add(ActionType.ARCHIVE);

and at 836:

ACTIONS_ON_FOLDERS.add(ActionType.ARCHIVE);

and at 850:

ACTIONS_ON_RESOURCES.add(ActionType.ARCHIVE);

There's a lot more to do, but that will have to be in the next post, I think.

Sakai Development: Post Six

Get Sakai source code and set up Eclipse IDE

There is a really useful guide to setting up a development environment on the Sakai WIKI. I found it by accident when searching for a solution to one of the problems I encountered: if only it had been linked to from the Sakai website's "Getting Started/Technical Contributors" page. I think I would have saved a lot of time and effort over the last few weeks.

To download the source from the repository (rather than a bundled release), use subversion to add the code to the development environment (as opposed to a server to use for testing, which is what most of the preceding work to this was about). A new enough version of subversion is already installed on Debian:

$ svn --version
svn, version 1.6.12 (r955767)
   compiled May 31 2011, 16:12:12
(etc)

so now download the code this way:

$ svn checkout https://source.sakaiproject.org/svn/sakai/branches/sakai-2.8.x/ sakai-src

This takes a while. If you need to use a web proxy, it should be set up in the global section of /etc/subversion/servers (uncomment the existing http-proxy lines and add appropriate values). Note that HTTP proxies may not enable every subversion function, though this checkout should be fine.

I have a chequered history with IDEs, partly because they are memory intensive applications, and the times I've needed to use them I've  been near the end-of-useful-life for the computer I was using, so there was never enough memory to run them properly, and partly because I've never really used one intensively, so my programming habits have remained the way they were before IDEs became popular tools. But this time, there should be no real excuse, as the computer I'm using is just 16 months old, and I deliberately bought one with as much virtual memory as I could afford. Following the instructions at https://confluence.sakaiproject.org/display/BOOT/Install+Eclipse+WTP, I downloaded eclipse (installation just consists of unzipping the packaged archive, basically) then installed Webtools, subclipse (for which http://subclipse.tigris.org/update_1.8.x needs to be added to the download sites), and the maven eclipse plugin (ditto http://download.eclipse.org/technology/m2e/releases) through the Eclipse updater, though the main component was already installed in the base package. Then I set eclipse to ignore "bin" and "target" directories when running svn - from the Window-Preferences-Team-Ignored Resources menu of Eclipse.

Some settings need to be changed. Eclipse doesn't run with memory settings high enough for Sakai (even given what I said about IDEs in the last paragraph). So edit eclipse/eclipse.ini and upgrade -Xms and Xmx to 128m and 1024m  respectively, and add "-XX:+UseParallelGC" as a new line.

According to the instructions (but don't do this until you have read the rest of this paragraph!!), to prepare this for use in Eclipse, cd to the sakai-src directory and run:

$ mvn eclipse:clean

$ mvn eclipse:eclipse

which removes any existing eclipse files in the source code (presumably there shouldn't be any anyway) and then creates new ones - which takes a while and has several failed dependencies which have to be resolved manually (the error message helpfully tells you how - the probelm is basically that some library files are not found where expected). Next, create a new workspace for Sakai in Eclipse, using the File-Workspace-Other menu to enter a new workspace name (I used "ws-sakai"); slightly disconcertingly (even when warned this will happen), Eclipse immediately shuts down and restarts when you click OK. Then, add the source code to this workspace. Switch to the Java perspective (Window-Open Perspective-Java), turn off automatic builds (checkbox in Project menu), and import the Sakai source code (File-Import-General-Existing Projects into Workspace and browse to the Sakai source code directory). This fails, because eclipse thinks that this the source code is not a project file. This issue (and the missing dependencies) has been raised on the mailing list before I had got round to doing so, and the response from Steve again is not to do things this way; you only need to have the code in eclipse when you want to modify it, and even then only the specific project which is to be modified. Missing dependencies are then solved by adding the shared library directory of the tomcat installation to the classpath in Eclipse. With a large project like Sakai, this approach makes sense, but it really needs to be spelt out in the documentation! What's a bit annoying about this is that I now need to install tomcat on my laptop, not something I really want to do - I was hoping to write code on the laptop and test it on the server.

So I carry out the steps for an actual Sakai install which I haven't already done for this: setting up tomcat, creating the mysql DB, add the mysql connector to tomcat, edit sakai.properties, and compile with mvn. Of course, this gives a new error:

[INFO] Failed to resolve artifact.

Missing:
----------
1) com.sun:tools:jar:1.5.0

  Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=com.sun -DartifactId=tools -Dversion=1.5.0 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=com.sun -DartifactId=tools -Dversion=1.5.0 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

  Path to dependency: 
   1) org.sakaiproject.kernel:sakai-component-manager:jar:1.4.0-SNAPSHOT
   2) com.sun:tools:jar:1.5.0

----------
1 required artifact is missing.

for artifact: 
  org.sakaiproject.kernel:sakai-component-manager:jar:1.4.0-SNAPSHOT

from the specified remote repositories:
  default (http://repo1.maven.org/maven2),
  central (http://repo1.maven.org/maven2),
  sakai-maven (http://source.sakaiproject.org/maven2),
  sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

So I found the files which would fix this problem, carried out the mvn commands suggested in the error, and tried again, only to end up with the same to missing files. It ended up with me realising that I was using the wrong java implementation for this - I have several installed on the laptop, and /usr/bin/java is pointing to openjdk. So I tried again with the (Oracle) Sun java SDK, and this time the compilation and installation proceeded without error. However, sakai itself was inaccessible to the web browser, and this is caused by missing libraries:

SEVERE: Error configuring application listener of class org.sakaiproject.portal.charon.velocity.PortalRenderEngineContextListener
java.lang.NoClassDefFoundError: org/sakaiproject/portal/api/PortalRenderEngine
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
        at java.lang.Class.getConstructor0(Class.java:2716)
        at java.lang.Class.newInstance0(Class.java:343)
        at java.lang.Class.newInstance(Class.java:325)
(and so on)

The right jar file has been created in the source tree, just not deployed to tomcat. So having found it (sakai-src/portal/iportal-render-engine-impl/mpl/target/sakai-portal-render-engine-impl-2.10-SNAPSHOT.jar - checking it has the missing class using jar -tf sakai-portal-render-engine-impl-2.10-SNAPSHOT.jar) and copied it to $CATALINA_HOME/shared/lib, try again - but this does not fix the problem.

My thought at this point is that there is a possibility that symbolic links may be the cause of a lot of the problems I had earlier, both on the laptop and on the test server. If tomcat is installed from the debian repositories, it is distributed across the filesystem in accordance with Linux standards which the tomcat project itself does not follow (libraries go under /usr/lib, configuration under /etc, log files under /var/log, and so on). This is problematic because many tomcat applications need a single directory which is $CATALINA_HOME, which has all the tomcat components in it, and the debian package solution is to set up a directory at /var/lib/tomcat6 which contains symbolic links to the real locations of the distributed files. If bits of Sakai are not clever enough to follow these symbolic links, it is not surprising that there are large number of inaccessible jar files. Similarly, on the server I followed my usual practice of creating a symbolic link to the actual tomcat installation directory (this makes life much easier when upgrading tomcat, or installing new versions of Sakai, because this can all be done invisibly to the users of the site, who only see something when the symbolic link is recreated to point to a new tomcat installation), and it is possible that the problem with the missing image files is caused by this too. I'm not going to bother having another go at the source installation on the server, but I will try a tomcat as downloaded from apache for the laptop.

OK, the next compilation caused some serious laptop problems - it crashed it. And this is without updating the code from the last compilation, which went fine. Time to stop for the day.

A re-install later (I was thinking about changing my distro anyway) and a reset maven, tomcat and re-downloaded source later, and I'm ready to compile. And again I end up with the dreaded missing library error:

[INFO] snapshot org.sakaiproject:sakai-announcement-help:2.8-SNAPSHOT: checking for updates from sakai-maven2-snapshots
Downloading: http://source.sakaiproject.org/maven2-snapshots/org/sakaiproject/sakai-announcement-help/2.8-SNAPSHOT/sakai-announcement-help-2.8-SNAPSHOT.jar
[INFO] Unable to find resource 'org.sakaiproject:sakai-announcement-help:jar:2.8-SNAPSHOT' in repository sakai-maven2-snapshots (http://source.sakaiproject.org/maven2-snapshots)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Fialed to deploy to container :Unable to download the artifact from any repository

(and at the same time notice a typo in the mvn output). Now, I have a good source for the missing files in the working installation on the server, so I can download it and install it, as per the instructions from the mvn output. And this is just the first of 8 - until it starts going round in circles.

The solution for me was to change the deployment, and just run

% mvn -Pcafe sakai:deploy

which is then installing a cut-down version of Sakai which seems not to include any of the modules which have these dependency issues. And it works:

So now back to importing Sakai code into eclipse. I re-ran the Maven eclipse commands above, this time without error. I created a ws-sakai workspace; as before, sakai the restarts. The .m2/repository directory is already in the class path in eclipse, so no need to add it (presumably this was done on the installation of the maven eclipse plugin). I thought I'd try just once to import the whole of the sakai-src tree into eclipse. This resulted in the following error, that the java libraries to interface to subversion could not be found:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Usefully, eclipse (or more specifically, the subclipse plugin) suggested a site for a fix: http://subclipse.tigris.org/wiki/JavaHL (a page on the subclipse WIKI), which also explained why the libraries are not built into the subclipse distribution (too complicated due to differing installation methods on different operating systems). Finding the instructions for the redhat based linux distribution (after rebuilding my laptop, I'm using fedora - temporarily, it turns out to be too irritating to keep as my main desktop), I downloaded the file I needed from http://www.open.collab.net/downloads. The list of available files doesn't quite match the subclipse wiki page description (there doesn't seem to be an rpm available any more, for instance), and CollabNet required me to register before downloading, neither of which seem ideal. Howeve, CollabNet Subversion Edge does include the required library, as the file csvn/lib/libsvnjavahl-1.so.0 (csvn being the name of the directory that the downloaded tar file expands into). It's then probably sensible to update the JAVA_OPTS so that the jvm loads the new library each time it is started adding the following to the user's .profile file

JAVA_OPTS=-Djava.library.path=/lib

or amending an existing JAVA_OPTS entry, then adding JAVA_OPTS to the list of exported environment variables. The same path needs to be added to the eclipse configuration as well, by shutting down eclipse, editing the eclipse.ini file in the eclipse home directory, adding the same information (no need to specify JAVA_OPTS, this time), and then re-starting. What's now annoying is that I now need to switch workspace, which means that eclipse will shut and restart - I could put the sakai workspace in the shortcut call to start eclipse, I suppose, if I'm going to need to keep doing this. The projects from the source directory now seem to be loaded completely - success!

All this effort required to get the source code into a tool to make it easier to work with. I seriously think I'd have been better off just downloading the source code and working with a simple text editor directly - the old school method of programming, before these time-consuming products were invented to save developer time and effort. But I now feel something of a sense of accomplishment: I have the sakai source code imported into eclipse. Now for the real work to begin...

Sakai Development: Post Five

So now there is a (mostly) working Sakai environment. What next? There are three tasks which I want to carry out, two covered in this post and the third in the next.

Create Some Basic Content

Since the project is to develop a tool which acts on material in the resources tool, some test material will be needed stored in Sakai. There is a useful guide for new Sakai administrators at http://www.freesoftwaremagazine.com/articles/create_your_online_project_site_with_sakai which I used to create some basic content (though it was not entirely helpful, as the Programmer's Cafe version of Sakai has far fewer resources available than the default version of Sakai which most people will install). The content uses material from http://www.identity-project.org, which is a website where the content is not only Creative Commons, but mostly written by me, so something I can re-use without any qualms and which saves me having to create some documents, web pages and so on myself. The resources tool is the one I need for the work, so I'll concentrate on adding stuff to that.

Trying to do this reveals a new problem with the built version of Sakai - all the controls for adding new files are missing from the resources tool. This is quite possibly for the same reason that I was unable to access Sakai webdav from my desktop - something is clearly wrong with this area of Sakai. See the email I sent to the sakai-dev mailing list.

The screenshot was taken while logged in as admin, but it makes no difference (except to the webdav access error) when I accessed Sakai as a non-admin user.

Trying to recompile just makes the problem worse, as tomcat now fails to restart properly, with a very lengthy error, the apposite part of which appears to be:

2012-08-15 12:14:31,306 ERROR main org.sakaiproject.util.NoisierDefaultListableBeanFactory - Failed to preinstantiate the singleton named org.sakaiproject.warehouse.service.ChildWarehouseTask.wizard.support.item. Destroying all Spring beans.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.warehouse.service.ChildWarehouseTask.wizard.support.item' defined in file [/home/sjm62/apache-tomcat-5.5.35/components/osp-warehouse-component/WEB-INF/wizard-components.xml]: Cannot resolve reference to bean 'org.sakaiproject.warehouse.service.PropertyAccess.id' while setting bean property 'fields' with key [0]; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'org.sakaiproject.warehouse.service.PropertyAccess.id' is defined

tomcat then proceeds to destroy a long list of Java beans (several hundred of them), before reporting successful start-up, erroneously. And this is with the same codebase, same tomcat, same maven, and same java as previously. This appears to be a seriously retrograde step in the Sakai installation - days wasted, and no actual development work possible so far.

It's beginning to look like time to start over again from scratch, and this is basically the eventual suggestion from Sakai guru Steve Swinsburg, which is to deploy into a clean tomcat installation (which can be done by deleting the tomcat webapps and shared/lib directory). The first reload runs into database issues - producing the following error in the tomcat logs (followed by others):

SEVERE: Exception sending context initialized event to listener instance of class org.sakaiproject.util.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.profile2.tool.entityprovider.ProfileEntityProvider#0' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Cannot resolve reference to bean 'org.sakaiproject.profile2.logic.ProfileLogic' while setting bean property 'profileLogic'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.profile2.logic.ProfileLogic' defined in file [/usr/local/tomcat/components/profile2-pack/WEB-INF/components.xml]: Cannot resolve reference to bean 'org.sakaiproject.profile2.logic.SakaiProxy' while setting bean property 'sakaiProxy'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.sakaiproject.profile2.logic.SakaiProxy' defined in file [/usr/local/tomcat/components/profile2-pack/WEB-INF/components.xml]: Invocation of init method failed; nested exception is org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not insert: [org.sakaiproject.emailtemplateservice.model.EmailTemplate]; uncategorized SQLException for SQL [insert into EMAIL_TEMPLATE_ITEM (LAST_MODIFIED, OWNER, SUBJECT, emailfrom, MESSAGE, HTMLMESSAGE, TEMPLATE_KEY, TEMPLATE_LOCALE, defaultType, VERSION) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; SQL state [HY000]; error code [1366]; Incorrect string value: '\xE4\xBB\x8E${l...' for column 'SUBJECT' at row 1; nested exception is java.sql.SQLException: Incorrect string value: '\xE4\xBB\x8E${l...' for column 'SUBJECT' at row 1

which is, I would guess, caused by not having deleted the existing database in order to start again. Sakai sets up the database automatically, but this is part of the compilation process, so I will need to re-compile - again. But this time, it's my fault for not zapping the database first. This is more successful, at least as far as log files go. But there is nothing in tomcat/webapps/portal, until I recompile just that directory from the source code, and then I just get a page saying that an unknown error has been encountered when I access sakai through a web browser, with no further information being logged. Yet another rethink needed now.

In the end, I just junked everything (starting from an empty tomcat and an empty mysql database, with no .m2/repository directory) and started again, which led to a finished compilation, but with the same problem in Sakai I had at the start of this post with missing resources hooks. Round in a circle we go. The only thing I can think of is that the documents specify 5.5.33, and the current version of tomcat-5, which I installed, is 5.5.35. So I will try again with 5.5.33.

After another day of fiddling around, I get a working version of Sakai 2.8.2 in tomcat 5.5.33 - but it is still without file manipulation hooks in the resources tool. A suggestion from Andrew Martin is that it could be an invalid file path; looking at a working Sakai installation shows me that the drop down menu uses a gif image, specifically /library/image/sakai/icon-dropdn.gif - and that file exists on my attempted install. Other files in the same directory are properly displayed - /library/image/sakai/sortascending.gif for instance. and this one will display too if the path to the file itself is opened in a browser. However, the string "icon-dropdn.gif" does not appear in the source code for the page, so looking at where this is generated is probably the next thing to try. The gif is referenced in two files in the webapps/sakai-content-tool/vm/content/ directory: sakai_filepicker_select.vm and sakai_resources_list.vm.

At this point, I've decided that I'm just wasting my time trying to get a demo site working by compiling code. If I need to have it on my laptop too, I might as well concentrate on that. It should be possible to update a demo site produced using a binary with test code I produce. So that's what I'll do, following the instructions here (but using MySQL). Fifteen minutes later, after forgetting to install a MySQL java connector, we have a working Sakai installation, with resource tool file management hooks.

What have I gained from this extended effort to try to get Sakai to compile and work? A lot of frustration, and the feeling that Sakai is far too fragile as a collection of software considering that it's supposed to be a mature product. Some of this is a consequence of the servlet architecture - it's a trait shared with a number of tomcat-using products I have installed in the past. But Sakai itself has a complex structure, with a large number of separate servlets, and it could do with a re-organisation of its architecture to create more simplicity. After all, it basically just consists of several interfaces to the outside world (HTTP, WebDav, and API, so far as I can tell so far), code to generate the interface as required, back end stuff (database interface, for example), and plugins. I'd like to see a lot more being hidden away in libraries, with webapps to service the various interfaces and a standardised plugin structure; however, I suspect that this would require a major re-write of the code, and is unlikely to happen.

And now I can finally add the material. I noticed a blog post from the OpenExeter project about the limitations of the DSpace interface for SWORD, which is going to help me to think about the requirements for the tool for Sakai.

Add LDAP authentication

Instructions are found at https://confluence.sakaiproject.org/display/~steve.swinsburg/LDAP+in+Sakai+2.5 - this requires recompilation of the providers project, and perhaps might have been better done as part of the initial deployment if compilation had worked out. Overwriting a binary webapp with the new providers code works fine (it's probably sensible to take a copy of providers.war before overwriting it, to make it possible to go back to the old version if necessary). The instructions were combined with local values supplied by Jez Cope from his existing Sakai installation. Editing XML with vi (as I was doing) requires strict attention to ensuring correct XML syntax - failure to do so results in a failure of the LDAP connector, while leaving the rest of Sakai working (as it's only one of the webapps which is replaced). Setting this up is easily the best experience I've had with Sakai so far: apart from a couple of typos I made which needed fixing, it worked first time exactly the way I interpreted the documentation. One small downside - this change seems to make Sakai start up even more slowly - 108.789 seconds when I restarted tomcat to test the installation.