Once the source is downloaded, the first new step is to set up some environment variables. The instructions say that the java executables (java, javac, etc.) need to be in the path; in fact, the way that Java is set up on Debian makes this unnecessary, as /usr/bin/java (etc.) are symbolic links to the java executables, which do not need themselves to be in the path. Some work is needed, though; to increase the memory available to Sakai, create the file /usr/local/tomcat/bin/setenv.sh with the contents:
export JAVA_OPTS='-server -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dsun.lang.ClassLoader.allowArraySyntax=true'
Make the new file executable:
$ sudo chmod a+x /usr/local/tomcat/bin/setenv.sh
The required version of subversion is already installed.
Next, install maven 2.2. This is the version of maven in the Debian software repositories as maven2, but this is a package which has a very large number of dependencies, some of which will clash with the already installed Sun java (openjdk, for one). Although java binaries and compilation is supposed to be compatible across the different implementations, it feels wrong to use this maven - and having multiple java implementations on the server when its not necessary seems silly, a recipe for confusion later on. It makes me feel happier to install maven by downloading from apache. (Though in the end, for the sake of time, I chose to download the binary, with no idea about which java was used to compile it...) The installation process is basically the same as that for tomcat.
$ sudo mv apache-maven-2.2.1-bin.tar.gz /usr/local $ cd /usr/local $ sudo tar zxvf apache-maven-2.2.1-bin.tar.gz $ sudo ln -s apache-maven-2.2.1 maven
maven needs some setting up, and I choose to do this globally, so that every new session picks up the home and options for maven. In /etc/bash.bashrc, add the lines:
MAVEN_HOME=/usr/local/maven export MAVEN_OPTS='-Xms512m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=128m' PATH=$PATH:$JAVA_HOME/bin:$CATALINA_HOME/bin:$MAVEN_HOME/bin export JAVA_HOME CATALINA_HOME MAVEN_HOME PATH
at the end of the file (the last two lines replacing the ones given in the previous post). To do the same things for a single user, add settings to .profile in the home directory. When this has been done, run
$ source /etc/bash.bashrc
to import these settings into the current session and then test to be sure that this all worked:
$ mvn --version Apache Maven 2.2.1 (r801777; 2009-08-06 20:16:01+0100) Java version: 1.6.0_26 Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre Default locale: en_GB, platform encoding: UTF-8 OS name: "linux" version: "2.6.32-5-amd64" arch: "amd64" Family: "unix"
Create a local maven repository directory:
$ cd $HOME $ mkdir -p .m2/repository
and add a settings file to the m2 directory also created by this command, as .m2/settings.xml, containing:
Additional information may need to be added if the server uses a proxy to communicate to the Internet, as maven doesn't check the http_proxy system variable (but go here rather than trying to follow the instructions in the Sakai guide to add the instructions to the settings.xml file). The instructions now tell you to configure Sakai, which is confusing as Sakai has not yet been installed (and the instructions tell the installer to edit as yet non-existent files). There is a minor step missing from the instructions, which is to open the archive of the Sakai source code already downloaded. In the user's home directory,
$ tar zxvf sakai-src-2.8.2.tar.gz $ cd sakai-src-2.8.2Next, set up the mysql database to use. The right JDBC driver for the database you intend to use needs to be downloaded and put into /usr/local/tomcat/shared/lib; for mysql, this is available from http://www.mysql.com/products/connector/. The database will also need to be set up as per the configuration entries you set up. For mysql this entails the following activities (with some parts of the server response omitted, and the actual password I used replaced by the word "password"):
$ mysql -u root --password
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
mysql> create database sakai;
Query OK, 1 row affected (0.00 sec)
mysql> grant usage on *.* to sakai@localhost identified by 'password';Query OK, 0 rows affected (0.00 sec)
mysql> grant all privileges on sakai.* to sakai@localhost;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye
Note that I had already set a root password; often, new installations of mysql do not have the password set. To test this worked, I then ran:
$ mysql -u sakai -p 'password' sakai mysql>; exit ByeNow it's possible to carry out some pre-compilation configuration. First, make a new properties file to configure the compilation-time information, such as the server domain name; I felt that the best start was with a copy of the existing exemplar, default.sakai.properties. There is also a file, sample.sakai.properties, which contains all the many possible items it is possible to configure at this point - since the default.sakai.properties file is already 924 lines long, to work through the longer file was a job I hoped would be unnecessary.
$ cd config/configuration/bundles/src/bundle/org/sakaiproject/config/bundle $ cp default.sakai.properties sakai.propertiesThen edit the new sakai.properties. Most of these changes from the default were because I decided to forward only access requests for /portal from apache to the tomcat listener, and so the URLs for navigation all need to start /portal. My expectation at this point is that the changes I make won't matter - they all seem to be in the nature of the sort of local customisation which should easily be changeable later (and, if not, I can always change the properties and compile again).. The contents of a sakai.properties file are documented in a 90 page Microsoft Word file found at sakai-src-2.8.2/reference/docs/architecture/sakai_properties.doc - useful, if a bizarre choice for a documentation format for an open source project. To summarise the main points, as relevant to the compilation I want to carry out:
- The edited file should end up at /usr/local/tomcat/sakai/sakai.properties, and further changes can indeed be made later. The compilation process does not copy this file across, so it will need to be sorted out by hand. One of the consequences of not doing this is that Sakai will attempt to use the default hsqldb database, which has not been installed, and this will prevent the installation from working (the error in this case, which appears in catalina.out, is "java.sql.SQLException: Table not found in statement [select COUNT(*) from SAKAI_SESSION]").
- The connection details for the database to use need to be set up, and the existing hsqldb values commented out.
- The property portalPath is used for adding a pathway to the URL (which should not be used to contain the full path to the portal where there is some extra needed, as here where URLs ending /sakai are forwarded to tomcat); this is not in the default properties file.
$ cd ~/sakai-src-2.8.2/master $ mvn clean installThis is followed by a lengthy list of missing libraries, followed by downloads and installation of these libraries - which is far nicer than having to fulfil all the dependences by hand. Eventually, there is the magic message:
[INFO] [install:install {execution: default-install}]
[INFO] Installing /home/sjm62/sakai-src-2.8.2/master/pom.xml to /home/sjm62/.m2/repository/org/sakaiproject/master/2.8.2/master-2.8.2.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1 minute 14 seconds
[INFO] Finished at: Wed Jul 18 14:15:09 BST 2012
[INFO] Final Memory: 16M/495M
[INFO] ------------------------------------------------------------------------
Then, the actual compilation and deployment to tomcat of Sakai itself. I'm aiming to compile the cafe build, recommended for new developers.
$ cd ~/sakai-src-2.8.2 $ mvn -Pcafe clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat
I set this going and then immediately realised that I don't have write permission to /usr/local/tomcat. As expected, this ends with
[INFO] [sakai:deploy {execution: default-cli}]
[INFO] Copy /home/sjm62/.m2/repository/javax/servlet/jstl/1.1.2/jstl-1.1.2.jar to /usr/local/tomcat/shared/lib/jstl-1.1.2.jar
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Fialed to deploy to container :/usr/local/tomcat/shared/lib/jstl-1.1.2.jar (Permission denied)
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6 minutes 36 seconds
[INFO] Finished at: Wed Jul 18 14:26:33 BST 2012
[INFO] Final Memory: 76M/495M
[INFO] ------------------------------------------------------------------------
What doesn't occur to me at this point is that the quick way to fix this problem is to give myself write permission to /usr/local/tomcat and its sub-directories, at least temporarily. Instead, I try using sudo to allow the installation to occur.
$ sudo mvn clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat
Predictably, this also fails (the reason being that I created the sakai environment in /home/sjm62/.m2, so if I run the command as sudo, the information fails to be found). So now try:
$ sudo mvn -s /home/sjm62/.m2/settings.xml clean install sakai:deploy -Dmaven.tomcat.home=/usr/local/tomcat
This seems to work better, though I'm not at all sure what it might be doing to the ownership and permissions in ~/.m2. Download checksums failed for a fair number of the required resources, with no obvious effect (It looks from this as though the checksum of the downloaded file is compared to the stored checksum, but all that happens if they are different is that a warning is given, but the downloaded file is used anyway.) After twenty minutes downloading and compilation, this happens:
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Compiling 9 source files to /home/sjm62/sakai-src-2.8.2/osp/wizard/tool/target/classes
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Compilation failure
Failure executing javac, but could not parse the error:
The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: PermGen space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
...
(tedious details omitted)
...
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18 minutes 30 seconds
[INFO] Finished at: Wed Jul 18 15:01:39 BST 2012
[INFO] Final Memory: 102M/243M
[INFO] ------------------------------------------------------------------------
This suggests that the maven options set the memory size too low, even though I used the values from the documentation. I doubled the values, and ran the compilation again, with the same result. (It's very frustrating to have the compilation run for almost twenty minutes, then run out of memory.) It looks as though more thought is needed - perhaps the virtual machine has too little memory. Indeed, with no interactive processes running, the machine has 584 Mb used out of a total of 1002 Mb available of virtual memory: so setting the maximum memory usage of maven above 512 Mb will be pointless (the information comes from running free -m). According to top, the most memory intensive process running is tomcat, so I now propose to run the compilation once more, with tomcat shut down (must remember to restart it later...). This time I will also monitor the memory usage in another shell, using top and also running tail -f /var/log/messages, which should indicate problems if the machine's memory is full (the page at http://rimuhosting.com/howto/memory.jsp is a very helpful introduction to the diagnosis of linux memory issues, BTW). There is no obvious message, though it is possible that
Jul 19 09:56:48 float mpt-statusd: detected non-optimal RAID statusis connected, as Linux memory usage is increased by the caching of disk blocks in spare memory. (I would expect Linux memory management to be sophisticated enough to stop caching the blocks at pretty short notice when an interactive process wants the memory, however.) So my conclusion now is - I should increase the memory allocation for maven and try again. This time, MAVEN_OPTS is set to "-Xms1024m -Xmx2048m -XX:PermSize=1024m -XX:MaxPermSize=2048m" - and the build still fails. It's time to hit the Sakai dev mailing list. Posting at 1.45AM Pacific time unfortunately makes it likely I'll need to wait for an answer - so I'll do something else for a while. The answer comes fairly quickly, but apart from fixing obvious typos, leads to circling round whether or not I should be using root or sudo or neither. (Note: I can't just create new users on this VM in the obvious way, because it's set up to use an LDAP server for authentication, requiring userIDs other than root to match ones on the LDAP. There's probably a way round this, but it's not worth the time to sort it out, I don't think.) Also, the suggestion that tomcat needs to be run as a non-root user is something I thought about, but then didn't do, as apache's tomcat documentation says that there shouldn't be issues. (And applications writing to files outside the tomcat tree seems to me to be the application's security problem not tomcat's - especially as it's not likely to be possible if selinux is enabled...) Anyway, self-justification apart, the solution is to get more memory for the virtual machine. Even with 2Gb virtual memory, the compilation process takes up 90% of it, so it is clear that the maven options suggested in the documentation are rather on the low side of what is needed, at least on Debian. But with the upgrade and write permission to /usr/local/tomcat, the compilation now runs to a successful conclusion:
$ ls /usr/local/tomcat/webapps/ access.war profile2-tool.war sakai-rutgers-linktool.war accountvalidator profile-tool sakai-rwiki-tool.war accountvalidator.war profile-tool.war sakai-search-tool authn.war providers sakai-search-tool.war balancer providers.war sakai-sections-tool.war courier.war ROOT sakai-siteassociation-tool.war dav.war sakai-alias-tool.war sakai-site-manage-group-helper.war direct sakai-announcement-tool.war sakai-site-manage-group-section-role-helper.war direct.war sakai-archive-tool.war sakai-site-manage-link-helper.war emailtemplateservice-tool sakai-assignment-tool.war sakai-site-manage-participant-helper.war emailtemplateservice-tool.war sakai-authz-tool.war sakai-site-manage-tool.war imsblis sakai-axis sakai-site-pageorder-helper.war imsblis.war sakai-axis.war sakai-site-tool.war imsblti sakai-calendar-summary-tool.war sakai-syllabus-tool.war imsblti.war sakai-calendar-tool.war sakai-tool-tool-su.war jsf-resource sakai-chat-tool.war sakai-usermembership-tool.war jsf-resource.war sakai-citations-tool.war sakai-user-tool-admin-prefs.war jsp-examples sakai-content-tool.war sakai-user-tool-prefs.war library.war sakai-editor.war sakai-user-tool.war login-render.war sakai-fck-connector.war sakai-web-tool.war messageforums-tool sakai-gradebook-testservice.war samigo-app messageforums-tool.war sakai-gradebook-tool.war samigo-app.war osp-common-tool.war sakai-help-tool.war savecite.war osp-glossary-tool.war sakai-login-tool.war scheduler-tool osp-jsf-example.war sakai-mailarchive-james.war scheduler-tool.war osp-jsf-resource.war sakai-mailarchive-tool.war servlets-examples osp-matrix-tool.war sakai-memory-tool.war sitestats-tool osp-portal-tool.war sakai-message-tool.war sitestats-tool.war osp-portal.war sakai-metaobj-tool.war tomcat-docs osp-presentation-tool.war sakai-news-tool.war tool.war osp-wizard-tool.war sakai-podcasts.war webdav podcasts.war sakai-postem-tool.war web.war polls-tool sakai-presence-tool.war wiki.war polls-tool.war sakai-reset-pass x portal-render.war sakai-reset-pass.war xsl-portal.war portal.war sakai-rights-tool.war x.war profile2-tool sakai-roster-tool.war
Now on to the final stage of getting Sakai up and running. Restart tomcat:
$ sudo /usr/local/tomcat/bin/startup.sh [sudo] password for sjm62: Using CATALINA_BASE: /usr/local/tomcat Using CATALINA_HOME: /usr/local/tomcat Using CATALINA_TMPDIR: /usr/local/tomcat/temp Using JRE_HOME: /usr/lib/jvm/java-6-sun/ Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar
and then it should be visible in the web browser. However, it isn't, and there are large numbers of errors in the tomcat logs.
I found that recompiling and installing over an existing, failed, sakai build caused huge numbers of problems in tomcat (the re-installed version is basically missing libraries which installed fine the first time around, even if it does have some which were originally missing). This seems to be the problem discussed in http://collab.sakaiproject.org/pipermail/sakai-dev/2009-August/002909.html and the solution is to re-install tomcat completely and remove the existing maven project in .m2/repository/org/sakaiproject directory. However, even then I was still getting errors like the following (note that the full error only appears in the localhost.[date].log file, not in catalina.out, which has an exceptionally unhelpful short version of the first line only):
SEVERE: Error configuring application listener of class org.sakaiproject.util.ToolListener
java.lang.ClassNotFoundException: org.sakaiproject.util.ToolListener
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1438)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1284)
(etc.)
(This particular error is repeated a large number of times, presumably once for each time a servlet tries to load the class.) I'm not sure what the cause is - permissions, or not removing enough of failed installations, perhaps - but it's fixed by getting rid of the entire sakai and tomcat installation and starting again from scratch. Once I remember to:- ensure that CATALINA_HOME is correctly set;
- copy the setenv.sh file to $CATALINA_HOME/bin;
- copy the mysql JDBC driver JAR file to $CATALINA_HOME/shared/lib;
- copy the sakai.settings file to $CATALINA_HOME/sakai;
then I end up having an installation I can see from a web browser. There are obviously still some fairly serious problems, as can be seen from the screenshot to the right, but I feel that I'm starting to get somewhere.
Posts
