Marc Palmer

Bean-Fields plugin 0.6 released

Marc Palmer — Sat, 13 Mar 2010 19:40:20 +0000

I have released Bean-Fields plugin 0.6. This extremely handy plugin for Grails applications makes your data form GSPs DRY by centralizing the rendering and styling of your fields, handling rendering, rendering appropriate HTML field based on property type, application of HTML max length constraints, rendering “required field” indicators, and rendering per-field errors. Rendering a whole bean’s worth of fields can be as simple as:

Version 0.6 fixes a bunch of bugs related to rendering fields for nested property paths e.g. propertyName=”book.author.firstName” and introduces support for list / array properties eg “book.authors[3].firstName” (This was really quite painful to implement!). Radio groups are working properly now, and test coverage much improved – thanks to contribs from Antony Stubbs.

It also adds a user-definable threshold for whether a radio group or select list should be used for a field with an inList constraint.

Full list of resolved issues is here.

Weceem 0.8 Released – Highlights and background

My JAX London 2010 talk now online in video form

Marc Palmer — Thu, 25 Feb 2010 14:29:16 +0000

I delivered my talk “Yes you CAN use Grails” yesterday at JAX London 2010, and it was well received. Unfortunately nobody was videoing it, and as the slides are “Presentation Zen” style you won’t get much from the official slides from JAX. It talks about Grails basics and how Grails is the perfect fit for existing Java development shops, and the things you need to know to make it possible to use it in your workplace. Well, at least you can try.

So I recorded myself doing the talk here at home and have uploaded a screencast of it all for those of you who are interested. Its not so good the second try – you had to be there for the best version

Click the thumbnail to download the MOV. Its about 47 minutes. Its probably going to cost me a fortune having it on Amazon S3!

Also available as a torrent.

People interested in making the case for Grails may be interested in an older post of mine that I’d completely forgotten about – “10 common misconceptions about Grails”.

Data sync on iPhone, iPod, iPad – the missing link?

Marc Palmer — Sun, 07 Feb 2010 13:05:15 +0000

Users and particularly developers of Palm’s old line of PalmOS devices will keenly remember that Palm were the only people to get syncing right at the time.

Aside from all the basics, they allowed 3rd party applications on the device AND the desktop to talk to each other directly to sync custom data. I’ve bitched about this before.

As an avid Mac, iPhone, MobileMe and soon to be iPad user, I have to wonder what is happening with this at Apple. My real-world gripe is this:

I was just about to open OmniFocus on my iPhone specifically so that it would sync with the latest data on my MacBook Pro, which is set to sync via MobileMe (using a pretty ugly file based solution). Why am I even doing this? Why isn’t this data synced (a) when I dock my iPhone to sync all the other iTunes stuff, and (b) why can’t it automatically sync wirelessly

Well part (b) is easier to answer, although it is a three-fold answer. First, there’s no background app support to allow automatic sync of the OmniFocus app on the phone. That should be addressed by the Push API functionality except that OmniFocus doesn’t support Push API (server cost to them to do so) and even if they did support Push, iPhone SDK Push is not able to automatically pass the data to the application to force it to sync – the user must acknowledge the event and run the app on the phone manually. It’s a pile of suck, surprisingly, with a real feel of “disconnected device”.

Part (a) is more tricky to answer. It must be trivial for Apple to add this kind of support for direct-to-app syncing. They already have/had Sync APIs for OS X for a long time. Lack of support for this apparently makes no sense.

In conclusion I am very surprised that Apple has not updated the OS X Sync APIs so that:

Third party apps can sync any data they like to/from the iPhone/iPod/iPad with iTunes as the conduit (that was the concept’s name in PalmOS if I recall)
The transport for sync is completely hidden from the applications such that sync will happen transparently via Dock, Wifi (direct between devices on local Wifi network), and via MobileMe cloud if the device is not on the same Wifi network.

This is not rocket science after all. And yet we still have to know / think about what networks our devices are connected to, manually make sure we run them frequently etc. It is pretty lame, Mr. Jobs.

My Apple tablet predictions – for what its worth

Marc Palmer — Wed, 20 Jan 2010 14:09:06 +0000

Might as well join in the fun eh.

Ironically, I think that there will not be that much hardware “frill” with the Apple tablet. I think that actually this is really all about software and the masses of computer users who are not “power users”.

Follow the logic:

Net books are very popular.

The iPhone and the new breed of beyond-smart phones is incredibly popular.

What is the common thread here? Both devices are very portable and offer most of the basic computing that people need day to day. What is it that most people – which I’m afraid guys means non-geeks, the truly massive market beyond geekdom – need?

That’s it. That’s what most people who are not raving geeks need. In fact some geeks may need only that. After all there isn’t much you can’t do with web apps now (e.g. bespin). Google OS/Chrome stuff has been geared to this from the get-go, its not a novel idea.

Functionally, most people also need to be able to write/edit documents that can be read by MS Word – not that they need MS Word, they just need to write out .doc files. This can be done via web apps or via lightweight local apps.

However Apple would not do something like this unless it also offered uniquely integrated stuff.

So on the back of this I reckon the tablet will:

Not be that revolutionary hardware wise – eg physically this probably is like a giant iphone
To include some web-hosted (with local offline usage) iWork for Pages (= docs & spreadsheets writing. maybe keynote too)
Full access to all your iTunes audio and video media and photos (cloud or not) – I would be surprised if this is 100% cloud done at this stage, what with the awful 3G coverage and slow speeds to sync photos and videos. Access to this done “ipod style”, which is a killer recipe as the market has shown
A first class large-form factor email app, geared to multitouch
And as suspected the delivery of formerly-print media, possibly opening up iTunes marketplace to any author who wants to prepare and sell content. Who knows perhaps you will even be able to create new textual/mixed content on the iPad and sell it via iTunes. This content provision is probably the one really new thing that helps make such a pad a really attractive proposition.

In a nutshell, a beautiful portable computer that is most definitely NOT a laptop because a great deal of people will never see themselves as the laptop carrying kind. However they are likely to part with cash for something that is much smaller than that but a true lifestyle accessory that “just works”. Obviously it will support custom apps and app store too – which has already shown on iPhone that a lot of people just want little stuff that makes life easier.

Several programmers including myself have wondered “Why do I need something like that?”. The answer is if you have an iPhone and laptop, you don’t. The big market win here is not people like us, its everyone else in the real world! Laptops are complete overkill for a lot of people and the netbook market has sort of shown that. They’re not so much winning against laptops as a result of price, they’re winning on form factor and simplicity. If people really needed high-end laptop features, they’d still buy a laptop instead of a netbook.

I’m pretty sure netbooks aren’t aimed at programmers either – although I am confident some masochists code Perl on them and swear that its the best calculator computer they’ve ever had. The tablet on the other hand, is squarely aimed at attacking the netbook and light-use laptop market.

Let’s see what Wednesday brings!

Weceem 0.8 Released – Highlights and background

Marc Palmer — Tue, 12 Jan 2010 14:01:02 +0000

I’ve just managed to push out version 0.8 of the Weceem CMS for Grails.

This is a pretty cool if slightly unglamourous release because it has focussed on some performance and security stuff – oh and compatibility with the latest and greatest Grails 1.2.0 release.

Let me first apologise for the ropey typography on weceem.org – we haven’t yet had the time/resources to fix up the CSS styles so things are a little ugly in places. We will fix it as soon as we get time!

Anyway, we added a security policy. This is a groovy script (currently only loaded once at startup) that lets you define what different users can do. This uses a DSL that lets you declare roles and then say what they can do in different spaces, including whether they can admin the space, edit/view/delete/create content, and even do this by URI requested. (see more info and example here)

Because we believe strongly that Weceem should not force you to use a particular authentication library, we had to decouple the policy mechanism from authentication. As a result these roles are completely uninterpreted by Weceem. To integrate and authentication system all you have to be able to do is provide the name, email, login and list of roles for the currently logged-in user. (an example here)

This has enabled some cool stuff in WeceemSecurityService which relies on the security policy. The service has utility methods for implementing our security logic, e.g. isUserAllowedToViewContent(Content c). For example in previous versions of Weceem you could not preview content if it was not in a publicContent status (eg not Published).

Thankfully from version 0.8, anybody who has the EDIT permission can view non-published content. the default security policy ensures that the default administrator account has EDIT permission, so you can preview away as much as you like. We plan future updates to this to allow the security policy to control who can manipulate different types of content, which will be really powerful for people using custom content types.

On the performance front, it became obvious that something needed to be done because the default “index” page installed by Weceem into new spaces was resulting in huge amounts of SQL queries for a single page.

This is because the page is made up like this:

It pulls in a Template node for the styling
It pulls in three Widgets for reusable HEAD section tags and header and footer
The header widget iterates over all root content nodes and their children to render the menu with the wcm:menu tag
The page itself links to various StyleSheet and JavaScript nodes to pull in styling and scripts – these are processed on separate requests but still add to the overall burden of the page

This can result in a lot of SQL chatter because we have (rightly so) made no effort to optimize this until now.

There are a couple of areas here that would make a big difference to the SQL traffic.

It is very important to realise that turning on the 2nd level cache in your Grails app’s GORM configuration does not magically give you major performance improvements. My understanding of this rather complex area (which frankly I found very disappointing) is:

The 2nd level cache is only used for retrieval by object id. This is very important
The query caches are used to cache the ids of objects returned for a given query
Query caches are invalidated frequently by Hibernate if your model is not primarily read-only (and can cause some threading contention)

Luckily a CMS is pretty read-only in terms of number of requests that actually read vs update content, so the 2nd level cache is a good candidate for us here.

One of the major SQL hits for Weceem is resolving a URI to the ultimate content node to render. Due to the model we need to query for each part of the URI, so a request for /a/b/c results in three selects. So that’s an easy one – we added URI path -> content id caching (and some other smarts) into the ContentRepositoryService. So once content has been hit, it will always be retrieved by id in future, via the 2nd level cache.

Another issue is iterating over child nodes. This is less trivial. We are using some query caching but I have noticed that some of the criteria were not hitting the caches despite this – it needs further investigation. I think that due to the polymorphic nature of the content model and query cache invalidation issues, we may stop using these in future (think blog comments being submitted and invalidating ALL your caches).

Next up: Template and Widget nodes are GSP pages that we compile and evaluate. It turned out that due to issues in Grails GSP handling (that persist in 1.2 to my knowledge), there is no internal caching of compiled GSP classes built from non-Resource content e.g. strings. This results in a leak of PermGen space which ultimately results in VM collapse. So we now have a simple cache of compiled Template and Widget GSP pages, which is automatically invalidated as necessary when templates and widgets are edited, so it is transparent to the end user that there is a cache.

Finally with regard to performance, we introduced a nice simple wcm:cache tag. This lets you cache fragments of a Template/Widget and hence get major performance improvements. The cache is currently fixed at 1 hour, but its great for anything that pulls in remote content or for any expensive node iteration tags you might be using. More enhancements will come in future.

A couple of nice little things we squeezed in:

The Cancel button is back on the content editor screen, in the “right” place for Windows users (meh) and browsers (who made return always select the first button argh!)
The wcm:link tag now passes through any unused attributes eg class=”whatever”
We added a JS syntax highlighting script to the default space that you can use to render code snippets in your content easily

Anyway, please enjoy this release. Soon time to get started on 0.9 which should see the Blog functionality completed and other refinements.

Apple iTunes and NAS usage – please fix it Steve!

Marc Palmer — Tue, 12 Jan 2010 11:20:22 +0000

This is such a nightmare and it only grows with time.

Many many people have this desire: a single place for all their media: music, videos etc.

A NAS device is the place, backed up suitable. The problem is iTunes / Front Row just does not fit with this strategy if you have more than one computer / Apple device.

The core issue is that iTunes does not automatically rescan the media folder for new files. So you can point as many iTunes as you like to a shared location but they will only see the files they add to their libraries via purchasing or importing media.

So you end up with many devices in your house, all with a different view onto your shared media.

To access that new album you imported, you have to manually Add To Library on that iTunes instance, hoping it isn’t set to duplicate the files on the server. When this happens, as it certainly used to for me, you then end up with many copies of albums as you re-create your library from scratch on the various computers over the years.

Add to this the fact that if for some reason your connection to the NAS goes down when you run iTunes, it will revert the media path to the local folder, and you end up with a total mess – a bunch of machines with files only on some of them, their media split between local and NAS, and only the ones that added those files having them listed in their library.

The experience is so un-Apple it is shocking, and it causes daily pain.

It is important to remember that the iTunes Media folder is where it PUTS files that you buy/import – and that is ALL. (There is a new “Automatically add to iTunes” folder there, which seems half-assed to me). The iTunes Library is specific to each computer and is the list of media and the file path to each. This, unlike the media folder that actually stores the media files, is something you do not want to share between computers in many cases.

Now, iTunes 9 added Home Sharing. But guess what, this sucks and blows! Why? It (a) only shares iTunes-purchased content and (b) it duplicates the files to your local HD. Home sharing, I believe has a lot more to do with the iSlate/tablet and their new datacenter – music in the cloud crap – so you can sync your iSlate content without a cable.

Please Apple, it needs to work like this:

How it should work

This is relatively simple:

First, never change the media path in iTunes if the previous path is not reachable. Tell the user what is happening so they can fix it
When new files are added, make a bonjour announcement to any other iTunes running (perhaps even wide-area bonjour to make iPhones/Slates pick it up) so that they can instantly add the file to their iTunes library
When a file is not located on the local disks, have a local cache for stuff that the user ACTUALLY PLAYS. My wife doesn’t like all that heavy metal I listen to, so let’s not fill her hard disk with a clone of it eh? iSlate and Apple TV / mac mini media hubs etc can pick up just the files that are needed.
For the occasion when not all the machines / iTunes are running, have iTunes do daily rescan in the BACKGROUND for any new files in the media folder and AUTOMATICALLY add these to the Library. This is not rocket science.
Maintain an “excludes” list on each iTunes library so the user can remove items from their local itunes Library (without deleting it from the NAS) and they will not be offered the file again in a future background sync.

Don’t give me stuff about non-purchased media not having ISRC codes to identify them and de-dupe. You can dedupe on SHA hashes of the media (calculate once and embed in the metadata of the file) and failing that trackdata, and then failing that – USER INTERVENTION eg “There are some new media files added and we don’t know if they are duplicates or not – help me”.

You can even put all these newly discovered files into a special “Newly discovered” place in iTunes where the user can yay or nay them – or have it set to auto-accept (default).

The more and more macs and related devices are sold to households the more shitty this problem becomes and you REALLY REALLY need to fix it Apple. Without the cloud. The cloud is not a solution for terabytes of media being instantly accessible in your living room.

Please. Just do it. iTunes Home Sharing was nearly it, but sadly failed completely to address this.

This is spot on.

Marc Palmer — Thu, 17 Dec 2009 12:13:02 +0000

From http://www.mcclatchydc.com/cartoons/gallery/80178-a80399-t3.html

No related posts.

Why aren’t you using Grails at work?

Marc Palmer — Mon, 14 Dec 2009 15:34:57 +0000

As mentioned earlier today, I’m going to be talking at JAX London 2010 about how you CAN use Grails.

I would like to hear however, the reasons why some of you may not have been able to use Grails at your place of work. I’m sure there are many amusing and probably frustrating stories.

OK, I’m looking for soundbites for the talk. Come on, feed me!

No related posts.

Speaking at JAX London 2010

Marc Palmer — Mon, 14 Dec 2009 13:18:06 +0000

They’ve firmed up the timetable now and it seems I’m on the last day where the Groovy & Grails track is happening – at the same time that there’s a Java EE 6 talk. So I’m hoping that all the smart people will be over at my talk

Full details of the JAX London 2010 program are here http://jaxlondon.com/

…and here’s the overview of what my talk (which is not written yet, but is roughly fleshed out) will be about:

Grails is the hot property in rapid web application development frameworks on the Java platform, attracting around 90,000 downloads per month. It uses the Groovy language but is nonetheless primarily a Java-based stack. This makes it a perfect fit for Java development teams. Most programmers I talk to want to be working with Grails more in their day job because it is actually great fun and incredibly productive. However organisations are often resistant to change and may not be as excited as they should be about the opportunities Grails offers their business. This talk will give you some insights into the power of Grails and ways to ease your organisation into a more productive and enjoyable era of development – for the good of all concerned!

The talk is listed as “basics” , which I suppose is fair but I expected to present some reasonably advanced concepts for non-Grails users in terms of ways to use Grails and plugins.

I’m crapping myself to be honest, no idea how big the conf rooms will be etc. Gulp. It should be fun though. Knowing how high or low to pitch it at a general-purpose Java conf is tricky though. I see the Rails crowd have a “twitter in about an hour”. I’m hoping to pitch something a bit more “get grails working within your existing enterprise systems” which I think may be a bit more relevant to the crowd.

Comments on this appreciated, especially from JAX veterans – I’ve never been myself.

No related posts.

Do we need a commercial market for Grails Plugins?

Marc Palmer — Fri, 11 Dec 2009 14:02:14 +0000

Right at the end of the Groovy & Grails Exchange (AKA #ggx) final park bench session I came up with a question that has been brewing in my mind for a long time. Thing is, it is quite a big and difficult question, so I thought it would be better to shut the hell up and deal with it later after I’d clarified things in my mind a bit, i.e. now.

Here it is in a nutshell: Unless I’ve missed something, there’s no commercial market for Grails plugins. And this could be a problem because what percentage of all free open source dev projects out there survive in the long run with a vibrant user base? It must be pretty low by my estimation.

As a developer of many Grails plugins I am most definitely not having a whine about how I can’t make money out of my plugins (at least directly). I will always be able to get enough work one way or another – this is about the health of the plugins themselves which suffers because it is not currently possible to make money from them. Unfortunately this is really related to the general contentious issues surrounding the whole free (no cost) software movement. Paying for support works for large applications/frameworks, but for the smaller stuff it doesn’t work as a model in my opinion.

Anyway, its great that we have over 300 grails plugins now. Some of them are already stagnating though – including some of mine. Keeping plugins current and of good quality is fundamental as we know from experience (not just with Grails) that old plugins – which may still be relevant – stop working with newer builds of the “mother” framework. This leaves users in the cold over time, and can cause the attractiveness of a framework (eg Grails) to decay over time.

Conversely the number of stable releases of Grails that are in use by users only increases over time, which means all plugin developers have an increasing burden of support and testing for every plugin release.

When a big part of the value of a framework comes from the reusable components you can pull into your apps, if you end up with a high percentage of these components becoming mothballed over time, your ecosystem starts to look like a graveyard rather than the productive paradise it was when you started.

Let’s look at a possible plugin lifecycle – it happens to be very similar to my situation which will be no surprise, but I’m sure I’m not alone:

Inception: Either a lone developer has a great idea and a lot of spare time, or their employer has a great need that is reusable and they agree to open source it.
Dash to meet needs: Implementation steams ahead, the plugin is released
Documentation: high quality documentation is written for the first release (if you’re lucky)
Honing: in response to other users’ feedback, bugs are fixed, new features added
Steady state: the plugin works. No new major features are needed by the core developer/company
Stagnation begins: Minor bugs and feature requests begin to build up in JIRA. Some users submit patches or contrib minor fixes
Infrequent maintenance: Original devs return when they have precious spare time, usually to fix the issues that affect them most, but maybe with a couple of user-related fixes.
Stagnation sets in: Documentation drifts out of sync with point releases, builds stop working with newer versions of Grails. Developer doesn’t relish re-testing all their plugins with all stable releases of Grails in circulation (Grails 1.0.3/4, 1.1.1, 1.1.2, 1.2… and beyond) and so hides head in sand. This holds back Grails users who rely on such plugins who then do not upgrade to newer Grails versions.

It might look something like this:

It doesn’t look that bad does it? Try multiplying it by 10 plugins.

So what can be done about this? I don’t have the answer.

Would an iPhone App Store like mechanism for premium plugins work? There’s certainly people out there who would pay for plugins – but whether there are enough to pay a non-punitive price and make plugin development a viable career for people so they don’t let the rot set in because they can devote their real paid time to development?

Would a “Plugin development co-op” work, where people pay a low-ish monthly subscription fee to use and receive priority support for all participating plugin developers’ plugins? I rather like this idea, but again it remains to be seen if this could generate enough revenue to make it possible for enough developers to dedicate non-spare time to the development.

It just bothers me – why must plugin development be a “spare time” thing? These are very serious and important components in peoples applications. Will big enterprises really be happy to move to using Grails to dev apps that use a bunch of free plugins by individuals who write them on the train or when their wives have gone to sleep? Something that has Apache behind it may be an easy sell, but “some guy in a shed in the UK somewhere” doesn’t have the same ring to it does it.

I think if this problem is solved well, it may well open up the enterprise market that much quicker. Surely we can’t expect SpringSource to “buy” every enterprise-vital plugin that pops up, to give it credibility?

There is also another issue – that of licensing. Serious businesses need to know about the licenses that plugins are released under. This is not yet clearly identified, we could do more work there. I started tagging my plugins on grails.org with license tags such as “license-apache2″. However you can’t currently browse plugins by tag as far as I can tell on grails.org

Your thoughts appreciated.

No related posts.