Weceem Tip: How to render items from an RSS feed

Posted by: on Nov 10, 2010 | No Comments

Last night I was working on a Weceem site and had cause to render some links to articles from an external RSS feed. I have a Weceem Widget node already for this “news” panel, but the RSS comes from another site. Turns out even without direct support for rendering feeds (Weceem does support producing feeds out of the box), using Groovy’s magic methods on java.net.URL and the XmlSlurper you can easily do this.

Here’s a complete example:

This will render five news articles from the BBC news feed. It is incredibly naïve:

  1. It doesn’t make any attempt at error handling
  2. It will likely blow up if there are less than 5 items in the feed
  3. It doesn’t do any caching – the feed is retrieved (by the server) every time the widget is rendered

However with a few tweaks this can be turned into something more robust. I imagine we will add a specific tag for this that hides the error handling and caching issues into a future build of Weceem.

Weceem 0.8 Released – Highlights and background

Posted by: on Jan 12, 2010 | 4 Comments

I’ve just managed to push out version 0.8 of the Weceem CMS for Grails.

This is a pretty cool if slightly unglamourous release because it has focussed on some performance and security stuff – oh and compatibility with the latest and greatest Grails 1.2.0 release.

Let me first apologise for the ropey typography on weceem.org – we haven’t yet had the time/resources to fix up the CSS styles so things are a little ugly in places. We will fix it as soon as we get time!

Anyway, we added a security policy. This is a groovy script (currently only loaded once at startup) that lets you define what different users can do. This uses a DSL that lets you declare roles and then say what they can do in different spaces, including whether they can admin the space, edit/view/delete/create content, and even do this by URI requested. (see more info and example here)

Because we believe strongly that Weceem should not force you to use a particular authentication library, we had to decouple the policy mechanism from authentication. As a result these roles are completely uninterpreted by Weceem. To integrate and authentication system all you have to be able to do is provide the name, email, login and list of roles for the currently logged-in user. (an example here)

This has enabled some cool stuff in WeceemSecurityService which relies on the security policy. The service has utility methods for implementing our security logic, e.g. isUserAllowedToViewContent(Content c). For example in previous versions of Weceem you could not preview content if it was not in a publicContent status (eg not Published).

Thankfully from version 0.8, anybody who has the EDIT permission can view non-published content. the default security policy ensures that the default administrator account has EDIT permission, so you can preview away as much as you like. We plan future updates to this to allow the security policy to control who can manipulate different types of content, which will be really powerful for people using custom content types.

On the performance front, it became obvious that something needed to be done because the default “index” page installed by Weceem into new spaces was resulting in huge amounts of SQL queries for a single page.

This is because the page is made up like this:

  • It pulls in a Template node for the styling
  • It pulls in three Widgets for reusable HEAD section tags and header and footer
  • The header widget iterates over all root content nodes and their children to render the menu with the wcm:menu tag
  • The page itself links to various StyleSheet and JavaScript nodes to pull in styling and scripts – these are processed on separate requests but still add to the overall burden of the page

This can result in a lot of SQL chatter because we have (rightly so) made no effort to optimize this until now.

There are a couple of areas here that would make a big difference to the SQL traffic.

It is very important to realise that turning on the 2nd level cache in your Grails app’s GORM configuration does not magically give you major performance improvements. My understanding of this rather complex area (which frankly I found very disappointing) is:

  • The 2nd level cache is only used for retrieval by object id. This is very important
  • The query caches are used to cache the ids of objects returned for a given query
  • Query caches are invalidated frequently by Hibernate if your model is not primarily read-only (and can cause some threading contention)

Luckily a CMS is pretty read-only in terms of number of requests that actually read vs update content, so the 2nd level cache is a good candidate for us here.

One of the major SQL hits for Weceem is resolving a URI to the ultimate content node to render. Due to the model we need to query for each part of the URI, so a request for /a/b/c results in three selects. So that’s an easy one – we added URI path -> content id caching (and some other smarts) into the ContentRepositoryService. So once content has been hit, it will always be retrieved by id in future, via the 2nd level cache.

Another issue is iterating over child nodes. This is less trivial. We are using some query caching but I have noticed that some of the criteria were not hitting the caches despite this – it needs further investigation. I think that due to the polymorphic nature of the content model and query cache invalidation issues, we may stop using these in future (think blog comments being submitted and invalidating ALL your caches).

Next up: Template and Widget nodes are GSP pages that we compile and evaluate. It turned out that due to issues in Grails GSP handling (that persist in 1.2 to my knowledge), there is no internal caching of compiled GSP classes built from non-Resource content e.g. strings. This results in a leak of PermGen space which ultimately results in VM collapse. So we now have a simple cache of compiled Template and Widget GSP pages, which is automatically invalidated as necessary when templates and widgets are edited, so it is transparent to the end user that there is a cache.

Finally with regard to performance, we introduced a nice simple wcm:cache tag. This lets you cache fragments of a Template/Widget and hence get major performance improvements. The cache is currently fixed at 1 hour, but its great for anything that pulls in remote content or for any expensive node iteration tags you might be using. More enhancements will come in future.

A couple of nice little things we squeezed in:

  1. The Cancel button is back on the content editor screen, in the “right” place for Windows users (meh) and browsers (who made return always select the first button argh!)
  2. The wcm:link tag now passes through any unused attributes eg class=”whatever”
  3. We added a JS syntax highlighting script to the default space that you can use to render code snippets in your content easily

Anyway, please enjoy this release. Soon time to get started on 0.9 which should see the Blog functionality completed and other refinements.

Weceem 0.1 to 0.2 – changes under the hood

Posted by: on Aug 20, 2009 | No Comments

We released Weceem CMS 0.2 just before I went away on holiday a couple of weeks ago, so I haven’t had time to write anything about it until now.

Weceem is a CMS written purely in Grails, using Grails plugins as much as possible. It has some history at jCatalog AG who have worked on the code and core concepts in various forms for a while now, and for the last 10 months or so I’ve been helping them out with Grails approaches, architecture and a little coding.

We didn’t want to write another CMS just for the sake of it. Lord knows there are so many out there, its almost painful to write another. However we believe we can bring simplicity and novel functionality to the CMS space that is missing in other Java-based CMSes. (Perhaps this is the folly of all who re-invent the wheel?)

Release 0.1 was a line in the sand, whereas 0.2 includes some major overhaul. It took a lot longer to get to 0.2 than we intended, but there were some major changes behind the scenes, aside from bringing it up to Grails 1.1.1 compatibility and splitting it out into a Grails plugin and an app, so you can embed it in your own Grails apps or use it standalone.

Here’s a brief write-up of some of the changes we made, and why. There is also a short screencast that shows these changes here.

Reworking the admin UI

The admin UI desperately needed some love, and we want to move to a much more contemporary and less “Windows app” like feel. Something that is bright, bold and clear.

We also greatly simplified it, reducing the number of screens. We are for the moment at least using BlueprintCSS for the structural layout of the admin UI. It has been a major timesaver for us coders who, if we are honest are not the best cross-browser CSS designers.

What we’ve done so far with the admin styling is begin the migration away from the old UI. The work is by no means complete yet, and the UI will be more polished in time. We’re very much focusing on getting the bare bones functionality working well first before we do anything too fancy with the visuals.

Switching to jQuery and jQuery UI

A wide mix of tech was used in 0.1 for legacy/RAD/prototyping reasons, resulting in a fair amount of bloat, and slow page load times.

As a result we ditched all of the Dojo/Dijit/Dojox stuff that was there and complete rewrote the UI in pure jQuery and jQuery UI. Our code is wonderfully lean now.

We looked at all the libraries out there and decided on jQuery. Its “where its at” as far as we’re concerned. Unfortunately we have had to use a “non-core” tree table widget and some custom behaviours for the draggable content repository, which is currently a bit ugly in places – but it is a work in progress that improves greatly on the old tree UI.

We still have some cross-browser glitches eg badly floated button icons in Safari and a few layout “blow outs” that we will fix in time. Any help with these jQuery UI style issues would be much appreciated – I haven’t received any help on the jquery ui mailing list about the Safari icon issue in particular.

Dynamic content editor

In 0.1, which had effectively evolved from a prototype, each content type had its own controller and views for editing, with lots of duplication. Much of this was modified scaffolding, and we all know that while scaffolding is great for helping you to build a house, nobody lives in a house and leaves the scaffolding up once the work is done!

Extensibility of content types in Weceem is very important to us. So in 0.2 we introduced a dynamic editor framework. There is a single controller and view set for editing all content types in Weceem now.

We achieved this by using some convention properties on the content classes to declare which fields are to be presented in the content editor, and how they should be rendered.

Because of Groovy’s wonderful dynamic abilities and Grails taglib support, we are able to render any custom editor you may wish to use on a content field – you just have to supply a Grails tag that follows a specific naming convention, and use the correct name in the editor declaration of the field in the content class.

For example, rich html editing as used for the “content” properties of HTMLContent and WikiItem, is achieved by specifying the “RichHTML” editor:

static editors = {
    template(group:'extra')
    content(editor:'RichHTML')
    keywords()
}

The editor then invokes the editorFieldRichHTML() tag when rendering the “content” field of a HTMLContent node in the editor. Its as simple as that. We also have hooks for custom editors to pull in extra resources in the <head> section of the page, such as JS and CSS files.

We have grouping of fields into the “Extra” section, and perhaps more in future, and you can force the hiding of certain properties.

This mechanism works with inheritance, which we use heavily in the content model – although the details of precedence rules are yet to be finalized.

Reduced plugin dependencies

We removed our dependency on RichUI and a couple of other plugins, which were either not fully used yet by the CMS’s current features, or were providing unnecessary bloat.

In fact we don’t use any Grails plugins for JS-related UI work now, only the FCKEditor plugin for the rich editor for HTML editing.

Separation into Grails Plugin and Application

This was a bit of an upheaval what with all the refactoring going on too, but Weceem which was formally a Grails application is now 99% encapsulated in the Grails WeceemCMS plugin which you can install into your own Grails applications to instantly add a great extensible CMS with a ready to use admin interface.

The plugin is hosted in the central Grails plugin repository (eg: grails install-plugin weceem).

The other 1% of the original Weceem application functionality remains a standalone application that uses the Weceem CMS plugin and simply adds some config mechanisms for the DataSource, and adds Acegi security and a default content repository, a copy of the weceem.org site. Its literally a run-and-go CMS, the full source of which is under 250KB zipped!

The plugin has a very simple abstraction of security which may be subject to change in coming releases until we hit 1.0, but this allows the plugin to not care about what authentication and authorisation mechanisms you want to use. We simply abstracted this functionality into a service where you can inject your own handlers to obtain the currently logged in user id etc.

Navigation and extensible admin

More work remains to be done on this but we have utilized the Grails Navigation plugin for the admin menus, including the items in the Administration page.

What this means is that applications that use the Weceem CMS plugin can inject their own admin screens into the UI provided by Weceem. This is absolutely trivial to achieve with the navigation plugin, and exactly this approach is used by the Weceem application to add user management screens which are specific to the app, into the Administration screen:

class UserController {
  static navigation = [group:'weceem.app.admin',
    action: 'list', title: 'users']
...
}

Chowing down

Eating your own dog food is always recommended (who invented that phrase… who eats dog food anyway, I know I wouldn’t recommend it) and we do this in a big way:

  1. Weceem.org is self hosted with Weceem CMS application
  2. Weceem CMS application uses the Weceem CMS plugin, and proves that the abstractions work
  3. The people that are developing Weceem are using it for real world sites other than weceem.org

Next up…

We’re already working on version 0.3 which should bring full blog functionality in, clean up some UI niggles and bugs and flesh out the Wiki functionality too – time permitting. Oh and more test coverage and functional tests. The gap to the next release has to be a lot shorter!

I’ve put together a quick screencast showing a quick run through the admin parts of Weceem from a developer perspective. You can download it here.