Sakai Development: Post Six

Get Sakai source code and set up Eclipse IDE

There is a really useful guide to setting up a development environment on the Sakai WIKI. I found it by accident when searching for a solution to one of the problems I encountered: if only it had been linked to from the Sakai website's "Getting Started/Technical Contributors" page. I think I would have saved a lot of time and effort over the last few weeks.

To download the source from the repository (rather than a bundled release), use subversion to add the code to the development environment (as opposed to a server to use for testing, which is what most of the preceding work to this was about). A new enough version of subversion is already installed on Debian:

$ svn --version
svn, version 1.6.12 (r955767)
   compiled May 31 2011, 16:12:12
(etc)

so now download the code this way:

$ svn checkout https://source.sakaiproject.org/svn/sakai/branches/sakai-2.8.x/ sakai-src

This takes a while. If you need to use a web proxy, it should be set up in the global section of /etc/subversion/servers (uncomment the existing http-proxy lines and add appropriate values). Note that HTTP proxies may not enable every subversion function, though this checkout should be fine.

I have a chequered history with IDEs, partly because they are memory intensive applications, and the times I've needed to use them I've  been near the end-of-useful-life for the computer I was using, so there was never enough memory to run them properly, and partly because I've never really used one intensively, so my programming habits have remained the way they were before IDEs became popular tools. But this time, there should be no real excuse, as the computer I'm using is just 16 months old, and I deliberately bought one with as much virtual memory as I could afford. Following the instructions at https://confluence.sakaiproject.org/display/BOOT/Install+Eclipse+WTP, I downloaded eclipse (installation just consists of unzipping the packaged archive, basically) then installed Webtools, subclipse (for which http://subclipse.tigris.org/update_1.8.x needs to be added to the download sites), and the maven eclipse plugin (ditto http://download.eclipse.org/technology/m2e/releases) through the Eclipse updater, though the main component was already installed in the base package. Then I set eclipse to ignore "bin" and "target" directories when running svn - from the Window-Preferences-Team-Ignored Resources menu of Eclipse.

Some settings need to be changed. Eclipse doesn't run with memory settings high enough for Sakai (even given what I said about IDEs in the last paragraph). So edit eclipse/eclipse.ini and upgrade -Xms and Xmx to 128m and 1024m  respectively, and add "-XX:+UseParallelGC" as a new line.

According to the instructions (but don't do this until you have read the rest of this paragraph!!), to prepare this for use in Eclipse, cd to the sakai-src directory and run:

$ mvn eclipse:clean

$ mvn eclipse:eclipse

which removes any existing eclipse files in the source code (presumably there shouldn't be any anyway) and then creates new ones - which takes a while and has several failed dependencies which have to be resolved manually (the error message helpfully tells you how - the probelm is basically that some library files are not found where expected). Next, create a new workspace for Sakai in Eclipse, using the File-Workspace-Other menu to enter a new workspace name (I used "ws-sakai"); slightly disconcertingly (even when warned this will happen), Eclipse immediately shuts down and restarts when you click OK. Then, add the source code to this workspace. Switch to the Java perspective (Window-Open Perspective-Java), turn off automatic builds (checkbox in Project menu), and import the Sakai source code (File-Import-General-Existing Projects into Workspace and browse to the Sakai source code directory). This fails, because eclipse thinks that this the source code is not a project file. This issue (and the missing dependencies) has been raised on the mailing list before I had got round to doing so, and the response from Steve again is not to do things this way; you only need to have the code in eclipse when you want to modify it, and even then only the specific project which is to be modified. Missing dependencies are then solved by adding the shared library directory of the tomcat installation to the classpath in Eclipse. With a large project like Sakai, this approach makes sense, but it really needs to be spelt out in the documentation! What's a bit annoying about this is that I now need to install tomcat on my laptop, not something I really want to do - I was hoping to write code on the laptop and test it on the server.

So I carry out the steps for an actual Sakai install which I haven't already done for this: setting up tomcat, creating the mysql DB, add the mysql connector to tomcat, edit sakai.properties, and compile with mvn. Of course, this gives a new error:

[INFO] Failed to resolve artifact.

Missing:
----------
1) com.sun:tools:jar:1.5.0

  Try downloading the file manually from the project website.

  Then, install it using the command: 
      mvn install:install-file -DgroupId=com.sun -DartifactId=tools -Dversion=1.5.0 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there: 
      mvn deploy:deploy-file -DgroupId=com.sun -DartifactId=tools -Dversion=1.5.0 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

  Path to dependency: 
   1) org.sakaiproject.kernel:sakai-component-manager:jar:1.4.0-SNAPSHOT
   2) com.sun:tools:jar:1.5.0

----------
1 required artifact is missing.

for artifact: 
  org.sakaiproject.kernel:sakai-component-manager:jar:1.4.0-SNAPSHOT

from the specified remote repositories:
  default (http://repo1.maven.org/maven2),
  central (http://repo1.maven.org/maven2),
  sakai-maven (http://source.sakaiproject.org/maven2),
  sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots)

So I found the files which would fix this problem, carried out the mvn commands suggested in the error, and tried again, only to end up with the same to missing files. It ended up with me realising that I was using the wrong java implementation for this - I have several installed on the laptop, and /usr/bin/java is pointing to openjdk. So I tried again with the (Oracle) Sun java SDK, and this time the compilation and installation proceeded without error. However, sakai itself was inaccessible to the web browser, and this is caused by missing libraries:

SEVERE: Error configuring application listener of class org.sakaiproject.portal.charon.velocity.PortalRenderEngineContextListener
java.lang.NoClassDefFoundError: org/sakaiproject/portal/api/PortalRenderEngine
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2406)
        at java.lang.Class.getConstructor0(Class.java:2716)
        at java.lang.Class.newInstance0(Class.java:343)
        at java.lang.Class.newInstance(Class.java:325)
(and so on)

The right jar file has been created in the source tree, just not deployed to tomcat. So having found it (sakai-src/portal/iportal-render-engine-impl/mpl/target/sakai-portal-render-engine-impl-2.10-SNAPSHOT.jar - checking it has the missing class using jar -tf sakai-portal-render-engine-impl-2.10-SNAPSHOT.jar) and copied it to $CATALINA_HOME/shared/lib, try again - but this does not fix the problem.

My thought at this point is that there is a possibility that symbolic links may be the cause of a lot of the problems I had earlier, both on the laptop and on the test server. If tomcat is installed from the debian repositories, it is distributed across the filesystem in accordance with Linux standards which the tomcat project itself does not follow (libraries go under /usr/lib, configuration under /etc, log files under /var/log, and so on). This is problematic because many tomcat applications need a single directory which is $CATALINA_HOME, which has all the tomcat components in it, and the debian package solution is to set up a directory at /var/lib/tomcat6 which contains symbolic links to the real locations of the distributed files. If bits of Sakai are not clever enough to follow these symbolic links, it is not surprising that there are large number of inaccessible jar files. Similarly, on the server I followed my usual practice of creating a symbolic link to the actual tomcat installation directory (this makes life much easier when upgrading tomcat, or installing new versions of Sakai, because this can all be done invisibly to the users of the site, who only see something when the symbolic link is recreated to point to a new tomcat installation), and it is possible that the problem with the missing image files is caused by this too. I'm not going to bother having another go at the source installation on the server, but I will try a tomcat as downloaded from apache for the laptop.

OK, the next compilation caused some serious laptop problems - it crashed it. And this is without updating the code from the last compilation, which went fine. Time to stop for the day.

A re-install later (I was thinking about changing my distro anyway) and a reset maven, tomcat and re-downloaded source later, and I'm ready to compile. And again I end up with the dreaded missing library error:

[INFO] snapshot org.sakaiproject:sakai-announcement-help:2.8-SNAPSHOT: checking for updates from sakai-maven2-snapshots
Downloading: http://source.sakaiproject.org/maven2-snapshots/org/sakaiproject/sakai-announcement-help/2.8-SNAPSHOT/sakai-announcement-help-2.8-SNAPSHOT.jar
[INFO] Unable to find resource 'org.sakaiproject:sakai-announcement-help:jar:2.8-SNAPSHOT' in repository sakai-maven2-snapshots (http://source.sakaiproject.org/maven2-snapshots)
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Fialed to deploy to container :Unable to download the artifact from any repository

(and at the same time notice a typo in the mvn output). Now, I have a good source for the missing files in the working installation on the server, so I can download it and install it, as per the instructions from the mvn output. And this is just the first of 8 - until it starts going round in circles.

The solution for me was to change the deployment, and just run

% mvn -Pcafe sakai:deploy

which is then installing a cut-down version of Sakai which seems not to include any of the modules which have these dependency issues. And it works:

So now back to importing Sakai code into eclipse. I re-ran the Maven eclipse commands above, this time without error. I created a ws-sakai workspace; as before, sakai the restarts. The .m2/repository directory is already in the class path in eclipse, so no need to add it (presumably this was done on the installation of the maven eclipse plugin). I thought I'd try just once to import the whole of the sakai-src tree into eclipse. This resulted in the following error, that the java libraries to interface to subversion could not be found:

Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Usefully, eclipse (or more specifically, the subclipse plugin) suggested a site for a fix: http://subclipse.tigris.org/wiki/JavaHL (a page on the subclipse WIKI), which also explained why the libraries are not built into the subclipse distribution (too complicated due to differing installation methods on different operating systems). Finding the instructions for the redhat based linux distribution (after rebuilding my laptop, I'm using fedora - temporarily, it turns out to be too irritating to keep as my main desktop), I downloaded the file I needed from http://www.open.collab.net/downloads. The list of available files doesn't quite match the subclipse wiki page description (there doesn't seem to be an rpm available any more, for instance), and CollabNet required me to register before downloading, neither of which seem ideal. Howeve, CollabNet Subversion Edge does include the required library, as the file csvn/lib/libsvnjavahl-1.so.0 (csvn being the name of the directory that the downloaded tar file expands into). It's then probably sensible to update the JAVA_OPTS so that the jvm loads the new library each time it is started adding the following to the user's .profile file

JAVA_OPTS=-Djava.library.path=/lib

or amending an existing JAVA_OPTS entry, then adding JAVA_OPTS to the list of exported environment variables. The same path needs to be added to the eclipse configuration as well, by shutting down eclipse, editing the eclipse.ini file in the eclipse home directory, adding the same information (no need to specify JAVA_OPTS, this time), and then re-starting. What's now annoying is that I now need to switch workspace, which means that eclipse will shut and restart - I could put the sakai workspace in the shortcut call to start eclipse, I suppose, if I'm going to need to keep doing this. The projects from the source directory now seem to be loaded completely - success!

All this effort required to get the source code into a tool to make it easier to work with. I seriously think I'd have been better off just downloading the source code and working with a simple text editor directly - the old school method of programming, before these time-consuming products were invented to save developer time and effort. But I now feel something of a sense of accomplishment: I have the sakai source code imported into eclipse. Now for the real work to begin...