Overview

Please see ASF Content Management System for general information about the design, layout, and motivation for the CMS.

See CMS Adoption for a list of projects currently using the CMS.

The CMS wiki page contains additional useful information.

Markdown format

The mdtext files are converted to html with python markdown.

The markup syntax is described in http://daringfireball.net/projects/markdown/syntax

The following extensions are enabled for the CMS:

The extensions are part of the 'Markdown Extra' syntax documented in http://michelf.com/projects/php-markdown/extra/

Information flow

A user creates a mdtext file with the webclient in his/her own workspace, or in a working copy of the site's source. After a commit the following stages occur:

Staging

Staging is initiated by a commit to the svn repository.

  • the mdtext file is converted to html by the build and written to the staging site.
  • the converted html file is committed from the staging site to the staging repo.

Publishing

Publishing is initiated by clicking the [publish] link in the webclient, or by running http://s.apache.org/cms-cli from the command line.

  • the changes from the staging site are merged into the production repository by the POST to the publish link.
  • subsequently svnpubsub notifies the main webserver of the change and the webserver svn up's the site.

Setting up your content

Easiest way to get setup quickly is to use the boilerplate site setup. It has the right files in the right structure. Just export it and add it to svn.

svn export https://svn.apache.org/repos/infra/websites/cms/template site
svn add site
svn ci site -m "initial setup for the cms"

The location of your 'site' dir in svn is what Infra needs to get you setup in the CMS. Make sure to include this in your JIRA when requesting to use the CMS.

Once setup, you'll want to change the site/trunk/templates/skeleton.html file to use the look and feel you want for your project.

FAQ

What version of Django's template library does the CMS use?

The CMS uses a fork of Dotiac::DTL version 0.8, which seems to implement this version of django templates.

How do I use the code highlighter?

Some questions have arisen regarding the syntax hilighter on code blocks. Basically the guessing algorithm isn't too great, so to give it help explicitly tell it which lexer to use from the list of short names here:

http://pygments.org/docs/lexers/

What you do is prefix your code/text block with an indented ":::shortname" or "#!shortname", where shortname is the name of the lexer to use. Often you want to use ":::text" just to tell the codehilighter to do nothing special to the indented block. The ":::shortname" variant will not supply line numbers whereas the "#!shortname" variant will.

How do I add a new file or directory to my site?

To add a resource you first edit a directory by clicking on the Edit link to a directory. Then you add the name of the new file or directory to the form input field at the top of the page; directories are signified by a trailing '/' in their name. Hit enter and a form will be provided to create the new file or directory.

How do I publish generated docs (eg. doxygen)?

First place all your generated docs into a local directory, let's call it foo/. Next create a compressed archive of foo/ by running

$ tar -czf foo.tar.gz foo

Then in the CMS navigate to the directory you'd like to add that directory to. Pull up the Edit screen and type "foo/" into the form field. Then hit enter. Fill in the file upload widget by pointing it at foo.tar.gz, click on "Quick Commit", add a log message, and hit Submit. Wait while the staging site builds the docs, and when that process is completed click Publish Site and Submit. Please note that the name of the local directory you archived must match the name of the directory to be added to the CMS.

The way to maintain this setup is to use the CMS to delete stale doc trees and replace them with fresh versions.

More sophisticated projects should be capable of scripting this into their release packaging process. The requests to the CMS would look something like this (in perl):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ua = LWP::UserAgent->new(requests_redirectable => ["GET"]);  
$ua->default_header(Accept => "application/json");
$ua->credentials("cms.apache.org:443", "ASF Committers", $user, $passwd);
$response = $ua->head("https://cms.apache.org/redirect?action=delete;uri=$urlencoded_uri_of_website_doctree");
$delete_uri = $response->header("Location")
    or die "Missing Location header";
$delete_uri .= "/" unless $delete_uri =~ m!/$!;
## be careful here, if you screwed up the uri above you could wind up deleting your entire site.
$delete_uri =~ m!/trunk/content/$! and die "Won't delete entire site!";
$response = $ua->post($delete_uri, [ submit => "Submit" ]);
die $response->status_line unless $response->is_success;
$commit_uri = $delete_uri;
$commit_uri =~ s!/delete/!/commit/!;
$response = $ua->post($commit_uri, [ submit => "Submit", message => "deleting stale doc tree" ]);
die $response->status_line unless $response->is_success;
$add_uri = $delete_uri;
$add_uri =~ s!/delete/!/add/!;
## put new doctree into foo.tar.gz
$response = $ua->post($add_uri, Content_Type => 'form-data', Content => [
   submit => "Submit",
   message => "new doctree",
   commit => 1,
   file => [ "foo.tar.gz" ],
]);

Then use the http://s.apache.org/cms-cli script to publish the site (once the staging build has completed).

How do I run the CMS build scripts?

You can check out the scripts which can be used to build your site from https://svn.apache.org/repos/infra/websites/cms/build/. The perl script that generates the *.html site files from *.mdtext is build_site.pl. The build_site.pl script has some Python dependencies. The easiest easiest way to install these is to use the Python setuptools. First check your version of Python python --version and follow the instructions to install the appropriate set of tools from here http://pypi.python.org/pypi/setuptools.

After the python setuptools are installed other python dependencies can be installed as follows:

$ sudo easy_install Pygments
$ sudo easy_install ElementTree (Not completely sure that this is needed)
$ sudo easy_install Markdown

Depending on your Perl installation, you may need to install some CPAN modules first. A few users have had to install the following CPAN modules due to the use of the ASF::Value module:

  • install XML::Atom::Feed
  • install XML::RSS::Parser::Lite
  • install XML::Parser::Lite
  • install YAML
  • install SVN::Client

You may also need to do a force install depending on your setup.

Once everything is installed you'll need to run

$ cd /path/to/cms/build/tree
$ export MARKDOWN_SOCKET=`pwd`/markdown.socket PYTHONPATH=`pwd`
$ python markdownd.py

to get the markdown processing daemon up.

How come my nested lists don't render properly?

The python implementation of markdown insists on 4-space indents per nesting level.

How do I verify compatibility if I write a client app for the CMS?

Assuming you're using json, you can verify API compatibility by checking https://cms.apache.org/compat?version=$VERSION where $VERSION is the API version of the CMS that your code was written against.

What is Quick Commit and why would I use it?

If you are making a simple edit to a single page, Quick Commit will save you the trouble, after you've submitted your edits, of pulling up the Commit screen, filling in the Log Message, and hitting Submit. It is a convenient feature that probably should be used for 80% of your work in the CMS.

Why doesn't the bookmarklet take me directly to the Edit screen for a page?

It takes you to a Browse screen for 2 reasons:

  1. to get people familiar with the features of the CMS
  2. because the bookmarklet sometimes doesn't take you to the resource you want.

An example of 2 is the following url: http://www.apache.org/licenses/LICENSE-2.0. Instead of taking you to the .mdtext representation it takes you to the .txt one, because httpd's mod_negotiation prefers the shorter representation when other factors are equal.

However if you still prefer to have the bookmarklet take you to the Edit screen, simply hack the bookmarklet to add an action=edit parameter to the query string. Like this for example: ASF CMS.

How come the publish link doesn't show diffs of newly added files?

svn diff doesn't show diffs of copied files, which is what the [Publish Site] link does with added files. There is an option available in subversion 1.7 that enables diffs of copied files.

How do I verify the markdown -> html conversion by the python implementation, before committing my changes?

All you need to do is Browse your changes after submitting your edits (in this case you want to avoid using Quick Commit).

I'm not a committer on $project, which uses the CMS. Can I still use the CMS?

If you are currently an Apache committer and want to submit a patch to the $project dev list using the CMS, you can do that right now. Simply use the bookmarklet and pull up an Edit session for the page in question and submit your changes. Then view the subsequent Diff link, which will provide a link to the Mail Diff feature. Fill in the form and your changes will be mailed to any apache.org address you choose (but only one address per mailout please).

Non-Apache committers can still checkout the site in svn and submit patches to the list. Otherwise they may access the CMS using username anonymous and an empty password and use the Mail Diff feature mentioned above to send patches to the list from the CMS.

The main idea is to {% include navigation %} in your site template, where navigation is a Django variable name. How you construct this variable and pass it along to the site template is up to you- you could break out separate paths in path.pm and add the per-path navigation to the argument list of each path. Alternately you could use view.pm to process the path and generate a navigation argument using regexp captures from path.pm.

How do I force the CMS to do a full site build?

Simply make a trivial change to a file in the templates/ or lib/ directory.

How do I use the CMS with an external build system?

Ensure your build system is supported on the cms.apache.org host and supply the proper @system_args to build_external.pl. Currently we are targetting Ant, Maven, Forrest, and hybrids of same, once the community figures out the proper @system_args for those build environments.

We've completely restructured our site and now there are a lot of leftover files on the live site. How do we clean things up?

Simply delete the content and cgi-bin subdirs of your staging tree within https://svn.apache.org/repos/infra/websites/staging and then trigger a full site build. Publishing your changes will clean things up on the live site.

Is there any way to turn off all these annoying buildbot commit messages appearing on our commit lists?

No- they are a core part of the CMS's internal bookkeeping and cannot be routed elsewhere. However they are not essential reading for projects whose site builds are well-understood, so feel free to route those messages to /dev/null.

Help! I made a massive commit to my CMS source tree and now I cannot use the CMS webgui, not even to publish the site!

If your browser/router/whatever times out the connection to cms.apache.org while using the bookmarklet, just try back in a few minutes and it should work quickly the next time round. If you find that your changes prevent you from publishing using the website, feel free to use the publish.pl script on people.apache.org to avoid timeouts. It is important to be considerate to the other users of your site and follow thru to publish the changes produced by massive commits, even if it takes you an hour or so to complete the process. Once your changes have made their way into production, the CMS will perform normally until the next massive changeset is applied.

The CMS's publish operation has been recoded to be based on svnmucc and is now much faster. TLP's using a hybrid CMS/svnpubsub type arrangement for their sites will need to restore any svnpubsub-based directories manually after doing the Lazy Publish.

We've seen some sites experience long publication times with very minor changes due to a certain atypical performance problem with merges in svn 1.6, but this situation appears to have been resolved in the 1.7 line which we plan to upgrade to shortly.

If you think having massive trees is a problem for svn, investigate the --depth=immediates option to svn checkout. You can checkout the base dir of a very large tree quickly and simply and modify/commit any files directly contained therein. Let the CMS's build system do its job and worry about managing the full site sources so you don't have to.