My iOS and general software development articles, reviews and design stuff is going to be at the new transition.io site. There’s also a new twitter account to follow for post notifications, and a new app.net account too. If you don’t yet have an app.net account you can get a limited free one from me by asking me for an invite. Can recommend it, it’s like Twitter minus the disregard for the users.

My Grails blog posts will now only appear on [grailsrocks.com]. If you’re not already following grails_rocks on Twitter you should do so as new posts will be notified that way.

I hope a lot of you will stay with me in one or both subject areas!

Thanks to the Wayback machine we can see the full 1990s horror of my old website and the app, and lack of understanding that bullet lists should stop at or before 5 items.

I totally forgot that my app got a 4-star review on ZDNet too.

Tips on Channel Design: Interview with Marc Palmer of AnyWare Limited

Channel Maker, a channel creation and maintenance tool designed by Marc Palmer, simplifies the task of generating a CDF file to represent site contents for an Active Channel. Chapter 10 of Guide to Creating Web Channels provides a case study based on Marc’s experiences launching his channel development tool on the Internet. In this interview, Marc brings us up to date on his current thoughts about channel design and the future of Web channels on the Internet.

What elements go into good Web channel design?

Basically, channels that don’t have any ITEMs or sub-channels SUCK! The “New Scientist” channel is a case in point-it is terrible for offline viewing. Also, 500K+ is an unnacceptable level of precaching for dial-up use, in my opinion. I’ve looked at IDG’s Active Channel site and I like the approach of having 2 different channels, one for modem and one for fast links. I imagine the difference is in the pre-caching.I would say that the page referred to by the top level CHANNEL tag (the page it goes to when you first open that channel) should load very quickly… otherwise, it can take ages before you are able to select an item within the channel. I would also say that people should try to keep the text content of their channels changing every day or so, even if there’s not much news – I think people will unsubscribe from channels that don’t seem to change (it encourages the idea that channels don’t work too). If you just change the text content and not the images, the precaching will be very quick. You can read some more of my thoughts on the topic at: http://www.anyware.co.uk/anyware/cm/guide.html

Have you made any IE4 discoveries recently?

I discovered something yesterday-you can drag and drop a subscription from your Subscriptions folder (i.e. a subscription to an HTML page, not a channel) onto your Channel bar. I never knew that… there are few channels I subscribe to, but there are some plain WWW sites that I like to track, as well. Why do you think companies have been so slow to adopt Web channels as a way to move info to their customers?

I think the general problem has been the diversity of the technology-there is no clear winner. Netscape’s Netcaster, PointCast, Castanet, IE Channels… all slightly different technologies with no easy way of providing cross-browser support.

I think the two main reasons that Microsoft Active Channels have not taken off are:

1) Buggy implementation (caching problems etc.) 2) Poor design of the channels that do exist – puts people off the technology

…and a third reason that seems to be emerging is Microsoft’s apparent lack of direction with Channels. They are said to be dropping the channel bar from IE5, but I am told that channels will definitely still be supported and exploited.

Where do you see Web channels going in the next couple of years?

I think they’ll probably become a tool for TV-delivered Web content, perhaps just for navigating the content that is available, rather than controlling downloads. Microsoft and WebTV may well use them.

Automatic software updates are interesting too… IE 4.01 introduced this but nobody really knows about it. You can create “blessed” shortcuts to your applications that use a CDF file to check if there is a newer software release available-if there is, it may already have been precached (if the CDF file specifies it) and you are taken to a WWW page (also precached) that tells you about the new version, with a link to the setup file. This is all automatic, so every time you run your application it will quickly scan the cached .CDF to see if there is a new version. Could be very useful for keeping staff up to date, or home users who don’t see the importance of downloading the latest software patch (i.e. WebTV).

What are the best Active Channels you’ve discovered?

I thought that What’s New Too was nice because it used DHTML as soon as the technology was there (when it was rare for things to fly around a WWW page!). Nowadays however, I have yet to find a well-thought out channel. It disappoints me that companies like CNN remain on the fence, opting for PointCast, which is something I don’t want to download; after all, I already have a browser with Channel facilities!

Any suggestions as to useful potential applications for Web Channel technology?

It’s possible to use channels simply for pre-caching – for example one user of Channel Maker uses a Channel to keep large numbers of staff up to date with their latest online forms for customer queries and so on, so when they come in in the morning all the latest info is instantly available. I imagine this can improve morning start-up time as the intranet WWW server will not have to handle with 1000s of HTTP requests at 9 a.m. when everyone logs in.

You can also use a channel instead of using an HTTP-EQUIV REFRESH meta tag in pages to provide more sensible updating at specific times rather than at X seconds since the user accessed the page. CNN take note…

Marco Arment’s post about Google Reader also ties into this. The tide has been turning for some time, but it is clear that the big guys want to own bespoke APIs. Given the famous “openness” of web-based apps why aren’t we seeing real efforts by these guys to create open interoperable protocols for things like Twitter-like messaging? In the old days we’d be expecting such a protocol to allow a single client app to post “tweets” to Twitter, Facebook, App.Net all using the same wire format. Those days are gone.

In Google Reader’s case they had a bespoke API, but all the original content is exposed with RSS, not an API Google control. Thus the incremental value they could add (and revenue they could derive) was minimal relative to their corporate goals.

What is more interesting to me is how this need for control affects the current mobile market. Manufacturers are differentiating by adding their own APIs on top of Android, just like all the handset operators including Samsung did back in the days of J2ME mobile apps (remember the Samsung FUN Club anyone? It wasn’t FUN in any way).

It has become very obvious over the last 12 months that Samsung is succeeding in the smartphone market, and this means Android has a lot of users, but it does not mean that Android itself is succeeding. As has been mentioned before, this is putting Android into a difficult place where Samsung can call the shots as they have all the market volume for Android. This appears to be happening because they are introducing their own APIs and implementations of services and features, moving away from Google services.

Why are custom Android APIs important? Well if you’re a mobile developer with a clue and you need to target non-Apple platforms, the only one that matters is Samsung*. If you want to use the latest features of these mass market phones or provide the users with the experience they come to expect from an integrated platform, you will likely have to use Samsung’s exclusive APIs. This brings us back round to the custom API lock-in of the big net services.

I have no problem with that at all – after all my preferred platform of iOS / OS X is total platform lock in, in both language and API terms. In fact [I have argued that APIs are the killer requirement for a thriving mobile ecosystem] and this is why Android’s low O/S upgrade rates are a massive problem for app developers who want to make money selling apps (separate from large services trying to make ubiquitous free clients).

Handsets need to differentiate and add features over time, and they are already stabilising just like Mac laptops, which see solid but incremental refinements in hardware. It is the software and API that changes more. This time is arguably upon us already with phones – battery life and form factor are really the only hardware variables that will see continued tweaks. The odd sensor hear and there, the odd experimental new input method.

The thing is that everyone creating these platforms knows APIs are critical now. Google pretended that the poor Android O/S upgrade rates didn’t matter, but only because their entire business model for Android prevents them having any control over this. OEM manufacturers naturally tend towards short product life cycles and low margins which preclude long term update programs for existing customers.

I’m pretty sure Samsung don’t even want to build a long term platform, they just want differentiation and developer lock-in by offering custom APIs. Horace muses in the podcast that Samsung may one day have their own developer event like WWDC or Google I/O. I am not convinced by this because I don’t believe they have coherent long term plan for the platform. Its all just about this season’s hardware and appearing to be competitive with Apple. They would likely outsource the setup of this event and frankly I can’t see many people queueing up to talk to Samsung engineers or standing in a crammed room to learn how to code some new eyeball movement tracking API, which is the kind of thing that people pay money to do at Apple’s WWDC.

With the recent ditching of Andy Rubin, it would appear that Google is realising that Android is out of their control now and that they need to move to something new. This is probably the Android/Chrome hybrid that seems implicitly on the cards (it at least feels more likely than iOS and OS X merging).

As Horace Dedieu intimates in the podcast, maybe Google went with Android as an interim solution until the hardware, standards, connectivity and market “matured” enough to tolerate pure web O/S like Chrome. It was after all a peculiarly non-web solution for them to choose, and yet only those with blinkers on would have thought a pure web solution would work for mobile 5 years ago, let alone today.

(*)This was not how it used to be. Before Apple saved us from the mobile operator despots and brought us real smartphones, you couldn’t get an app distributed on any mobile operator’s network unless you ported it to support their own specific set of typically really bad and poor performing handsets like… oh the Samsung E700 (FUN club remember!), Sharp GX10 (screw you Vodafone!) or the O2 X1 (BenQ deserve punishment for that at some point).

This episode covered some recent debacle about journalists being asked to write for free by companies like The Atlantic who should pay for work. If you’ve developed open source for free, this might ring some bells.

What was very interesting for me was the part (contributed by Leah I think) about people thinking that maybe you don’t need to be paid to do something you love doing, as well as the part about doing it to get exposure to lead on to further work. These are very common justifications given for doing your “craft” for free. Again this sounds very familiar to me.

This attitude is blatant nonsense the moment you stop to think about it. Of course you should be paid to do what you love. Its what we all should strive for – making our living from something we enjoy – its a basic way to be happy which is what we all want. Then add to the mix something else: for many of us, there is more than one thing we love doing. This means that if people don’t want to pay you for this thing that you love, the smart people shift to that thing that they love, where people will pay.

The other very interesting element was the talk about Facebook and social and photo services like Path and Flickr. Their guest Matt Honan, a writer for WIRED rightly says about ailing online or service communities:

“Sorry, but fuck that. The community doesn’t have any obligation to keep this place going. The people who are running the community do.”

He goes on to talk about how Flickr dropped the ball by not having any sane mobile strategy which had them cede the social photo market to Instagram.

This is incredibly brutal and truthful. Businesses or products that rely on a community for their survival actually cannot expect their community to make the business or product thrive. Such activity by the community is a fig leaf over underlying failures and deficiencies. Furthermore and perhaps counterintuitively, people who haven’t paid anything have no reason to be loyal. People who have paid have made an investment they are less likely to turn their back on. Still, if you aren’t delivering the goods for your community, even those that pay will go.

The community has to enjoy the service or product and be served well by it. The onus is on the business or provider to make and continue to make it compelling and keep it contemporary. If you make Open Source stuff, however you are funded (or not), you have to keep making it compelling for people to participate. You cannot expect the community to step up and fill the gaps you leave if it is “your” product – which in truth is usually the case one way or another with all Open Source projects. There’s usually a company with ownership of the team, or an individual who is the leader of the project, and maybe some contributors whose employers are happy to bankroll some time.

If your product or service is not compelling enough, people will try other things and often they will not come back.

Even if they loved what you do, they will go where the pain is less and reminisce about how they enjoyed what your project did, and maybe mourn it a little, but they need to get on with doing stuff and removing the pain points, and they need to be where they can find the other people and features they need to interact with. Failing to change your course when things are causing you problems is that oft-mooted definition of madness.

On a complete side-note, in the podcast Mike Monteiro also lays it down beautifully about the evil that is Facebook:

Settings are only as good as the ethics of the people behind them – Mike Monteiro

However it is completely unsustainable for me to work unpaid on this stuff, in fact I should have called this a year or two ago. Releasing plugins used in many peoples’ apps involves a high degree of responsibility, as well as ongoing support and maintenance.

I have tried over the last 2-3 years to fund my free open source development with meaningful amounts of money; support subscriptions, selling e-books, and the Resources 1.2 Kickstarter. The Kickstarter may yet reach its goal, and that would be great. Many people have already pledged very generously.

Personally I invested a lot of energy, time and money to try to achieve a sustainable development model. These attempts have not received the level of support from the community needed to make them viable. This is arguably inevitable if you create free product in a relatively small market. Getting money from developers is hard, getting money from your users is close to impossible after you’ve already given your product to them for free.

For years I thought I could create free open source in the manner of a “product” to a high standard. In the end you can’t escape the fact that you have to either have a lot of time to kill or the financial resources to be able to achieve that. I wanted to create lots of great new stuff that was missing from the ecosystem because I needed it. I released these to share them with others, without considering the workload that would create over time.

The upshot is that I’m afraid I can no longer do any development or support work on Grails plugins unless I am paid to do so. In effect this is no real change over the situation in recent months, it is merely a formalisation of my intentions.

This affects the following plugins:

• Resources
• Platform Core
• Platform UI
• Email Confirmation
• Cached Resources
• Zipped Resources
• Cache Headers
• Feeds
• Bean Fields
• Taxonomy
• Invitation Only
• One Time Data
• Authentication
• Navigation
• Update I forgot Functional-Test

…as well as the work-in-progress plugins:

• Bootstrap UI
• Bootstrap Theme
• Fresh Security

Any future changes I do make to these plugins will strictly be on the basis of sponsorship by clients or my personal needs. I do not plan to release any new plugins publicly on anything other than these same terms.

There are likely to be some minor Platform Core, Platform UI and Weceem updates in the coming months because of a kind offer of sponsorship for this work by jCatalog AG. If your company wants to sponsor a block of development time please get in touch.

I will begin the process of updating the grails.org plugin portal to clearly state that I am no longer supporting the plugins. I will no longer respond to JIRA issues. I may still post to the mailing list when I have time.

All of the code for the plugins is already in Github. You are welcome to fork it. If you want to release new versions of these plugins please:

1. Discuss this intention on the Grails user mailing list
2. Go ahead and get the relevant permissions to do so
3. Follow sane release processes i.e. good versioning, tracking of fixes and features in JIRA (or github issues) and publish good release notes, tagging in git etc
4. Make it clear that you’re taking on support issues and that I won’t be

Update: for clarity, this means I will not review, merge or publish new releases. Someone else will have to take responsibility for that. Fork my Grailsrocks repos to somewhere else and release the plugins under the same name to the Grails plugin repo, after getting permission to do the release from the Grails team.

Effectively I am going to become a some-time contributor to my own plugins. If people from the community want to run with them, they are welcome provided my credit is retained and the license adhered to.

I want to re-iterate that I am still committed to Grails, but my time is going to be spent working with Grails for my paying clients.

I do remain concerned that there seems to be a distinct lack of large companies that use Grails giving back to the community in any tangible way. I can’t understand this, given the fantastic benefits that Grails provides. Perhaps people think that because VMWare own it that there is no need to provide extra resources?

Anyway, I’m looking forward to working with some new clients from March 2013. If you’re interested get in touch.

I ended up launching a new free utility for The Co-operative Bank customers.

As a long time customer of The Co-operative Bank, where your money goes is rather tedious to ascertain. There’s a lot of personal finance software out there that imports your statements as CSV or OFX files, and lets you do some analysis on them, and this is what you need to use to stay sane. Rifling through page upon page of statements is for the birds.

The problem for us The Co-operative Bank customers is that there is no way to export or download your personal banking statements from their very long-in-the-tooth online banking service. Its unbelievable but true. Also, googling seems to indicate that The Co-op have in excess of 20 million account holders. None of whom can use financial planning software without manually inputting all the data from their statements. Or… using some of the online services that claim to “connect” to The Co-op to download your financial data.

In reality those services ask you for all of your confidential login information, which arguably violates your Co-op bank terms and may open you up to claims if your account is used fraudulently. These services are all I believe using a service called Yodlee which stores your confidential login details and uses them to connect as necessary to get your data. This is not something I am willing to do, especially given the fact that most of these services are not open about their use of Yodlee and the risks connected to this.

Anyway, I came up with a new and very elegant solution. Its taken three days to polish it up, but I now present you with uncoop.me.

All you do is visit uncoop.me once, drag the “Save Co-op Statement” button to your bookmarks, and when viewing your co-op statement of choice, press the “Save Co-Op Statement” button/link in your browser. It “scrapes” the information out of the current page, generates both CSV and OFX format data for it, and lets you save it to your computer with a smart automatically-generated filename like “Statement_PRIVILEGE_12345678_01_02_2013.csv” (or .ofx).

I’m pretty pleased with it. I’m hoping the 20 million plus Co-operative bank customers will enjoy this new freedom until such a time The Co-op actually implement it themselves.

We’ve already hit 10% funding in less than two days, with less than thirty backers. I know at least 110 people were prepared to buy the Resources LeanPub book (if it gets made) so there’s a lot more people out there interested in supporting this work with funds. The Kickstarter has lots of info about the project, but in terms of why I’ve used Kickstarter it may help to give some more details.

Basically, I just can’t keep up with the Open Source workload. The opportunity cost to me is too great to spend many hours per week keeping up with all the issue and pull requests on my plugins, let alone develop new features. Development has become purely “needs based” for me because I really do have to pay the bills.

I have pursued other options, at not insignificant cost in time and cash:

• Paid support subscriptions – no interest and cost a fortune in time and Merchant account & recurring payment API fees
• E-books – some people will happily pay for books but its hard enough to make money on them let alone enough money to allow you to improve the code itself, which is what people really want

I hope the Kickstarter reaches its goal. In reality I think this is pretty unlikely unless we get some corporate backers. There are surely a lot of medium/large companies out there benefitting from the features Resources provides, and the corporate pledge levels include promo slots on this blog which gets a good amount of Grails specific traffic.

I’ve also created (and almost completed) a Lean Publishing e-book as a high quality user guide for the plugin.

You can get the PDF, ePub and Kindle versions of this for just a few dollars and you will continue to get updates as I finish, refine and update it over time. This has been an interesting experience and means we now have guide docs (as well as new free reference docs) for the plugin.

If you are interested in using the plugin or just want to support my work, please buy it!

There is however one aspect of this exploit that could cause problems for Grails apps. I’m not aware of any critical security breaches that would be possible but some app code may be vulnerable to bad behaviour.

The issue we do have is that Grails automatically creates maps of params that are found with dot notation in their names, to provide the nested parameter maps feature.

This means that if you directly use params values, your code could unexpectedly receive a Map instead of a String value if someone crafts a malicious request. In most cases this will simply cause some kind of exception. This isn’t ideal but on the basis this will only be happening if someone tries to hack your box, it’s probably not a big deal. There’s perhaps a denial of service possibility here.

You can see this at work on the Grails website, and examples of the ways apps may react to it. If you craft a plugin search query to include a sub-parameter:

http://grails.org/plugins/search?q.meaningOfLife=42

What you see is an error message. Chances are this means the app 500′d due to an exception. This is different to the case where you query with no value for “q”:

http://grails.org/plugins/search?_____meaningOfLife=42

In this latter case it sees there is no “q” at all and so redirects you. This indicates that getting a Map for “q” in the first case caused some obscure problem in the code. If this was a system that updated tables, if it wasn’t using transactions it would be possible that it left the database in a bad state.

The other way to see this at work in grails.org is to do a site-wide text search using a frigged Map argument:

http://grails.org/search?q.meaningOfLife=42

If you click that, will see how it was turned into a map and displayed back to you. So it actually tried to search for “[meaningOfLife:42]” but the syntax of the search was rejected by Lucene.

The more insidious problem could theoretically arise if you pass a params value as an argument to a method. Groovy’s dynamic method resolution might resolve to a different form of the method to the one intended. This is similar in some ways to the real Rails exploit.

Here’s an example in a security controller:

private doUpdate(String id) {
}

private doUpdate(Map properties) {
}

def update = {
   doUpdate(params.id)
}

In this case you can hopefully see how, if the request was for /update?id.roles=admin that we might be in for some trouble as the Map variant of the method may be called and have a totally different outcome, depending on your code path.

I don’t know of any real vulnerabilities of this kind with dynamic finders or other Grails or Groovy methods. If you can come up with any please do let us know as soon as possible. Binding whitelists are still important in this scenario to prevent such a Map binding to one of your domain object associations, but if you didn’t want that property to bind you were excluding it already I presume.

The simple fix for this is to do the right thing in the first place: never used unchecked params from requests. All such data is inherently untrustworthy even if a user is authenticated (authorised accounts can still craft bad requests). This is “Computer Science 101″ really. However it is currently just too easy to be lazy and use params direct. We’ve all done it despite the new mechanisms available.

You probably know that Grails provides two fantastic mechanisms to avoid using params un-checked. Command objects and typed action parameter arguments allow you to enforce the correct type and validation and so this kind of attack cannot bite you.

I have suggested to the other Grails core devs that in a future Grails release we disable access to params by default, unless an @UnsafeParams annotation is added to the action. Details of how this affects plugins remain to be fleshed out.

This service, as you can see in the side bar on the right of this page, lets you book time with me for a call for which you pay $2 per minute. This means you can limit the time of your call and how much you spend, but you can also get professional help from me without requiring contracts or any longer term commitments.

I don’t expect this to generate a lot of revenue, but who knows it could help. Perhaps you will be my first caller?