Thinking Outside the Platform

It is the mantra of our times that we live in a connected world.  People connect to each other socially; companies connect to their customers and suppliers to transact business as well as to their employees and shareholders; and groups of all sizes from loose collections of individuals with a common interest to multinational charities, connect with their members.  This landscape of connection is nothing new and existed even before the internet, but the internet is the quintessential technology of connection and it has become an indispensable mode of communication between people whenever meeting face-to-face is not convenient.

The web is today the most visible face of the internet and you would expect the pattern of connectivity on the web to roughly match that in the offline world and for the most part it does, but the web is still a long way from replacing more traditional means of communication in a number of areas. I'm still shocked and astonished by the number of companies that, even today, do business with each other by sending faxes! Even in a world we like to think of as modern you don't have to look far to find organisations using technologies that force its recipients to manually re-key information. (Did you know that facsimile transmission was a Victorian invention?).

The bulk of web-based communication is concerned with information retrieval, for the web allows us to view the content of its numerous servers as a single distributed but connected information resource. The amount of information that is available on the web is truly mind-boggling and yet there are vast amounts of data that are not yet online and much of what is online is not yet interconnected. But the web is not just about information storage and retrieval. As more businesses and organisations migrate to the web, increasing numbers of business processes are being web-enabled. This is perhaps most obviously seen in the rise of online shops where customers can purchase goods and services in an automated environment. With the product selection and payment processes being handled by a web application, the shop owner simply receives notifications of completed sales. Yet many common business and organisational processes are still only rarely carried out online. Despite the web's obvious success, there is still a long way to go before it becomes a truly universal medium.

Joomla came into this already connected world with its mission "to provide a flexible platform for digital publishing and collaboration" as a true web application. For most people the word Joomla is synonymous with the Joomla CMS; for them Joomla simply

the CMS. There is no denying that the CMS is the most important product currently in the Joomla stable, but as the web itself evolves, our ideas of what a content management system is expected to do must evolve in parallel and it is becoming more and more apparent that the idea of the "platform" goes beyond what is represented by the CMS. This is something that the original authors of Joomla's mission statement, perhaps presciently, had in mind when it was hammered out at the Joomla Leadership Summit in 2008. This article explores what "thinking outside the platform" might mean for the future of Joomla and why hypermedia-based web services are expected to be a central feature of this larger view of Joomla's mission.

The web as it stands today is built largely around people; people with web browsers. For many years it has been common practice to publish content on a website with scant regard for how others might want to consume or re-use that content beyond pointing a web browser at it. This assumption that people will surf the web with a web browser has led to websites being optimised for that mode of use, with the usually unintended consequence that most websites have become information silos. You know the data is in there but getting it in or out of the silo is restrictive and non-trivial. If the only way to retrieve data from a CMS is to parse the HTML that you get when you visit one of its pages then making serious use of that data is going to remain a struggle.

And the website is often just the tip of the iceberg when it comes to the information silo problem. You don't have to go far to find examples of organisations, or departments within organisations, that are sitting on mountains of data that would be much more useful if only it could be shared or connected. Furthermore, if you are having to manually key information into your business system, or if you are expecting your trading partners to rekey the information you provide them into their business systems, then that is a symptom of the information silo problem. It's also a huge waste of human resources.

But what about the non-human web? The development of web services was one response to the problem of getting data into and out of web information systems, in particular when the systems concerned were owned by some other group or organisation. Despite some early design mistakes, such as XML-RPC, RESTful web services APIs are now firmly established as an architectural choice when connecting disparate information systems, especially when humans are not directly involved.

The growth of the web is slowly but surely changing the way people think about the information that is locked inside their organisations and about how that information is exchanged with external entities. If organisations are to publish and share more of their data and more of their business processes then it needs to be super easy for them to do so. After all, exchanging data between information systems, whether inside or outside corporate boundaries, has been possible since long before the internet came along, using EDIFACT, X12 and other international standards, but it has usually required expensive custom development to make it work. What is different now is that the internet and more specifically the web has given us ubiquitous connectivity and potentially a uniform interface to all our information systems. And that eliminates a large part of the reason why those information silos came into existence in the first place.

The barriers that have isolated our information silos are crumbling. Imagine a future where establishing a new trading partnership in which at least the basic documents are transferred automatically and seamlessly between partners, can be as simple as following a link on a website and answering a few simple questions. The rise of REST APIs can be seen as being part of this trend towards leveraging web technologies to liberate data and expose it to access by many different devices on many different platforms; allowing it to play a more dynamic role in business and organisational processes. Once you realise that even such a staid business document as a purchase order can be represented as hypermedia content, then you are on the path towards more efficient and effective automated business processes.

The web as a whole is a single, huge, loosely coupled, distributed hypermedia application and the reasons for its success were analysed some 15 years ago by Roy Fielding, one of the architects of HTTP, in his now famous doctoral dissertation in which he described REST. The success of the web is in large part down to the REST architectural style that underpins it and if we want our applications to have the same characteristics of scalability, loose coupling, evolvability and so on, that the web itself has then it makes sense to adopt that same architectural style. Since the web itself is RESTful that shouldn't be difficult, so all we're doing is making sure that our applications use the same style as the platform on which they're built. In other words, we should swim with the flow instead of against it.

It turns out that most of the REST constraints are pretty easy to comply with. Indeed, we comply with most of them by simply using HTTP correctly. However, there is one word that stands out as being particularly important: "hypermedia". If we don't implement hypermedia and the constraints that apply to it, then we are not RESTful and some of the most important benefits of the web's architecture are lost. Hypermedia is the key to web services.

Fielding refers to one of the most important REST constraints as "hypermedia as the engine of application state", which essentially means that when a server sends a resource representation to a client it is exposing the current application state and the set of possible state transitions that the client is allowed to request. By ensuring that the resource representations are hypermedia, client software can drive the application by following links or completing embedded forms. It's a concept we're all familiar with because it is exactly what happens when a server sends an HTML page (which represents the application's state) to a client which renders it to a user, who clicks on a link or fills out a form and in doing so moves the application's state through one of the allowed state transitions.

Web services are a foundational technology that should lie at the heart of what we mean by content management. But for Joomla websites to become first class citizens in a world of distributed hypermedia content we first need to construct a comprehensive standardised, hypermedia-based REST API that will enable any website owner to easily provide access to content that has hitherto been obscured behind a wall of HTML. That is the task that the Web Services Working Group has set itself and you can read about its work here: http://docs.joomla.org/Web_Services_Working_Group

But what does having a comprehensive set of hypermedia-based web services available in the core Joomla distribution mean for site owners and developers?. Here are some of the possibilities that will be opened up:

• It will be much easier to create client code that accesses data in the CMS and which manipulates that data. For example, it will be easier to write those little Ajax calls that you need from time to time, for example when you need to pull in some article content within a module or grab a menu in some other component.
• Services can be hierarchical so you can create new services by combining other services to provide new behaviours. One example suggested recently was to create a new service that makes it easy for a user to create a new article and create an associated menu item and category on-the-fly as required.
• It will be straightforward to offer users access to your content using a modern hypermedia-based API with only minimal coding. This is ideal for organisations with an open data policy.
• It shouldn't be too hard to add API key management capabilities so you'll be able to deliver a fee-based API solution to your users if you want to.
• Third-party developers will be able to add hypermedia-based web services to their extensions which will blend seamlessly with other web services available from the same site.
• Third-party developers will also be able to create new installable services; a whole new class of extension with consequent new sales opportunities.
• Increased opportunities for inter-system communication; breaking down information silos; bringing the world a step closer to point-and-click EDI (Electronic Data Interchange).

I believe that this technology is vitally important to the future of the Joomla CMS and I've been working hard to move this project forward, first by writing a detailed specification for a web services API, then by creating proof-of-concept code that demonstrates how it can be built in to Joomla 3.x and what it could look like.

But I have only limited volunteer time available to me and I'd like to see this project accelerate. I want to see this technology incorporated into Joomla 3.2, due to be released in September of this year and to that end I created a Kickstarter project to try to raise funds so that I can work full-time on achieving the goal of getting comprehensive hypermedia-based RESTful web services support into Joomla by then.

At the time of "going to press" for this magazine article there are only a few days left before the funding period ends and if it fails to reach its target then it will not go ahead. So if you share my vision of getting hypermedia-based web services into the next Joomla release and you'd like to help me get it there then please go to Kickstarter right now and pledge as much as you can.

Here's the link: http://www.kickstarter.com/projects/964120480/joomla-web-services

Thank you.