Content Log

CMIS one year on - soon in public review

Autumn marks the new school year, business year and CMIS year

It's that time again. Autumn and the new school year and we are about to take another major step with CMIS, the Content Management Interoperability Services proposed standard at OASIS, and we will be that much closer to having CMIS become an official standard. As of today, a majority of the technical committee has voted to put the CMIS specification to public review with zero no votes.

I was recently reminded that I wrote about the public introduction of CMIS around the same time last year. This was the time that IBM, Microsoft and EMC decided it was time to announce the Content Management Interoperability Services that they started October 2006 and release it to OASIS to become an official standard. At that time Alfresco was fortunate enough to have been involved in the process for the previous year before joining the original three, along with Open Text, BEA and SAP. For the year before that, IBM, Microsoft and EMC broke away from AIIM iECM committee to speed up a process that itself had been going since Autumn 2005. Much of the groundwork was laid with iECM and the parallel, but Java-focused JSR-283 committee. All in all, this will not be a fly-by-night standard, but a standard that has been carefully deliberated, considered and grounded in real world use cases.

This last year has seen the number of vendors participating increase substantially with plenty of beneficial results. We have had a few face to face meetings that have triggered some great collaboration in a very competitive industry. The number of eyes looking at the spec has increased, but so have the minds contributing new ideas and expertise. SAP presented real world requirements that needed security in the form of ACLs, so something that we had been willing to punt on before, we added in. Microsoft, IBM, SAP and other owners of search engines participated in a working group to define APIs to get changes to a repository to keep search indexing up to date. Also, Day and Nuxeo set up an Apache project to share Java API and SPI components along with Gabriele Columbro from Alfresco and our friends at SourceSense. David Caruana, Alfresco's chief architect, contributed interfaces to handle renditions and specifically thumbnail previews.

More importantly, committee members were building real life implementations of the specification at each stage to really test its implementability and completeness. As Ethan Gur-esh from Microsoft once said, "I relish the fact that CMIS is retrospective." It can work with existing repositories, not be overly prescriptive, but still describe a powerful content application platform. Regular plug-fests ensured that the systems interoperated with each other. This alone distinguishes it from all other content management standardization initiatives - real life integration matching the current state of the specification. Dave Caruana's Java test harness, now in Chemistry, provided a tool for other vendors to test the completeness of their server implementations.

Developers started creating new applications based upon the early specs, such as CMIS Spaces and CMIS Explorer, which tested its usability as a platform interface for content-centric applications. CMIS's initial focus on real-life use cases has so far paid off as the level of functionality seems to match new applications being built and existing applications integrate content management capability, such as Drupal, Joomla and Confluence have with Alfresco through CMIS. I certainly encouraged anyone who wanted to integrate with Alfresco to do so using CMIS.

Today, unless some catastrophic event occurs, CMIS will go into public review. This is the last step in OASIS before becoming an official standard. As the process is laid out, it could take a maximum of 6 months for completion of the step and CMIS becoming a standard. It also means that only typos and minor corrections will change in the spec, currently at version 0.7. The determination of the spec leads, David Choy from EMC as Chairman and Al Brown from IBM and Ethan Gur-esh from Microsoft as Editors, has ensured that we got to this point and they have done an excellent job marshalling the process along. Everyone on the committee seems determined to see this to as speedy a completion as possible. It seems likely that CMIS will become a standard in the first quarter of 2010. I can't see anything getting in the way of this other than the US government disapproving or Google, whichever is more powerful. All the rest of the major players backing the standard. (Craig Randall, formerly of EMC Documentum, just popped up at the last call representing Adobe.)

Being the eternal optimist, I have very high hopes for CMIS, but I believe I have reason to be optimistic. I have seen it before first hand with SQL. Without vendor lock-in, it will now be possible to build content-rich applications that can run against many different systems. Customers will be able to move content from repository to another. It will be much easier to find stuff in random repositories without having to build multiple connectors. As the breadth of applications and utilities becomes available against these repositories, usage and deployment of content repositories will become more widespread and grow a much bigger pie for all vendors. As Alan Pelz-Sharpe reports in CMS Watch, CMIS is already gaining traction in 15% of AIIM members before it has reached standard status.

I'm really looking forward to this new year!

New Alfresco 3.2 was designed for the Great Recession

Back at the height of the market downturn in October, we looked at how Alfresco should address the rapidly changing economic situation. Rather than being pessimistic, we believed that this was a real opportunity for us. Times like these wipe the decks clean and provide openings for companies that provide value for money and can replace existing older technologies. This is the fourth recession in my career and what past experience has shown is that management and IT are under pressure to do more with less, cut costs, make remaining people more productive, and implement new technology if there is a clear zero-sum gain in cost reduction. Regulation also always comes after the disaster hits, but the new regulatory regime must be addressed with fewer resources. It seemed like a perfect time to be in open source!

Today we are releasing Alfresco Community Edition 3.2 and it really is an ECM designed for the Credit Crunch. We have been expanding the capabilities that generally been out of reach of anyone who could not afford a traditional ECM system, but who can now use one to reduce costs, improve productivity, reduce long-term costs of development of content applications or prove compliance. This release tackles  records management capabilities, handling and archiving of emails, mobile access for the worker on the go, the latest and greatest implementation of CMIS, and new extranet collaboration capabilities. All of these are targeted at what we felt would be important factors in a lean economic environment. All are also available as open source to help reduce the cost of managing content in enterprises struggling to do more with fewer resources.

New records management capabilities are very important for us, because this is the platform with which we will be going to the US government to certify for DoD 5015.2, but also because it provides a level of control that any organization facing regulatory requirements will find useful, such as life cycle management, retention policies, review process and disclosure and transparency controls. Built upon the new Share and SURF platforms these records management capabilities are the basis for a new records management application that is planned for certification at the end of September. To support this, we have added a new interactive forms system based upon the Yahoo YUI Ajax library and now allows both types and aspects to be applied and used in Share, including new records or regulatory metadata. By basing these capabilities on Share, we get a lot of the benefits of Share, including in browser viewing without downloading the record, URL-addressability of records information, and collaborative capabilities such as commenting, tagging and discussions. A new import and export capability is designed to simplify archiving records sets and import them to separate records repositories if necessary. A new records life cycle management automatically handles the physical storage of records to offline or tertiary storage. This is also the first records management system designed to be queryable by the proposed OASIS CMIS standard.

New records capabilities are destined for DoD 5015.2 certification

Related to and required by records management is a new ability to manage and archive emails using the IMAP email protocol. Virtually any email client can access, archive and categorize documents, records, attachments and other content with no plug-in required, because the Alfresco repository supports the IMAP protocol natively just as it does CIFS, WebDAV and NFS. This email integration is designed for two purposes. The first is for archiving and managing email, especially records. This interface allows you to manage email according to your organizational policies. If the policy is to archive everything and figure out organization later, the Alfresco rules can accept all content and can invoke rules to help organize, classify and apply the appropriate retention policies. However, if it is important for users to help classify the email as records, then users can drag and drop emails into the appropriate repository folders within their standard email client.

Manage and archive emails as records or access Alfresco from your email client

The second purpose for IMAP integration is to allow users who live in email to be able to access content from the repository without leaving email. Most browsers are IMAP capable, such as my Mac Mail client, and we have done extensive testing on Microsoft Outlook. Metadata and context of content is presented to the user with Freemarker script templates, which are very easy to configure. This metadata appears as email text in folders from the repository and the actual content appears to be an attachment to the email. This makes it easy to forward or send documents as either simple attachments or who content with metadata. It is also easy to use Alfresco from devices designed specifically for email such as the Blackberry, iPhone, Palm Pre or other mobile mail devices.

Every recession seems to create a step change in technology usage and this recession is probably no different. Smart phones now outnumber the number of laptops sold and for many tasks they can be just as effective. That is why we felt that content will be increasing consumed, processed and created on these small devices. However, the smaller form factor means that you can’t just take a big app and make it smaller. With Alfresco 3.2, we have looked at the tasks that people perform today and what content management tasks they could perform on mobile devices. The result is a version of Alfresco Share designed to fit the form-factor of these new smart phones, starting first with the iPhone. According to StatCounter, between the iPhone and iTouch, Apple has approximately 37% of the mobile browser market. It is also the first ECM application designed for business processes on the go. We have focused not only on browsing and access of content, but also on the business processes by allowing users to start and track workflows and activities around content collaboration.

Alfresco Share is now designed for use on the go with the Apple iPhone

This release also has the most complete implementation of the latest release of CMIS, version 0.61 of the OASIS CMIS Technical Committee, of which David Caruana and I are members. We have implemented both the REST-based Atom Pub and SOAP Web Services protocols. Dave has spent a lot of time on these capabilities and it is the future of our API. A lot of the work has gone into the new query language that provides SQL-like query capabilities along with other capabilities that had previously required using Lucene. Alfresco’s implementation of CMIS has been the basis of integrations with Joomla, Drupal, Atlassian’s Confluence, CMIS Spaces and CMIS Explorer and we expect more in the future. Dave and Gabriele Columbro are planning on contributing some our experience to the  Apache Chemistry project. Dave is hoping to contribute our CMIS client test harness, which may be used against any CMIS and currently contains over 100 tests covering all aspects of the spec including schema validation.

Some of the CMIS integrations to Alfresco include Joomla, Drupal and Confluence

Following release of Share last year and updates earlier this year, we have added a number of capabilities to support the use of Share in an extranet in order to provide content collaboration outside the enterprise firewalls or in the cloud. We have been testing scaling Share to tens of thousands of concurrent users. Since extranet use cases are more people-oriented, we have extended the contextual information available about users in the users profiles. We have also simplified the administration of users, groups and sites from a new administration framework integrated into Share. Share can also take advantage of some of the advanced metadata management capabilities of the Alfresco repository with the new forms system mentioned earlier and explicit support for types and aspects in Share. Some of the new user interface components available include Content Favorites and a new Image Gallery. This release of share is Cloud-ready for EC2 and other cloud services.

Alfresco’s Web Content Management Platform has been improved to support larger authoring and deployment environments better. A new parallel deployment engine uses multi-threaded updates to web farms for higher performance updates of web sites. A new web clustering architecture allows authoring installations to scale to more users and allows the deployed servers to also be clustered for shared services. A change to the architecture of previewing means that changes to either content or even code can be instantaneously previewed in a test environment. Improved rendering and transformation of web content provides better support for XML includes, XSL transforms and execution of Freemarker and Web Scripts in the web tier. A new pluggable deployment architecture allows you to deploy to multiple delivery environments such as to file systems, other Alfresco servers or external and web-edge delivery channels.

New WCM capabilities include better scaling of authoring environment and new parallel deployment to web farms

Sorry for the long post, but there is a lot that we have been working on while the Great Global Recession has been raging on. Our goal has been to help you cut costs of traditional ECM, improve your productivity through mobility and efficiency in handling content, address new compliance issues with the first open source records management system, help tame the great email beast, and engage your customers, partners and employees with new WCM and extranet capabilities. We also hope to future-proof your content applications by delivering the first and best with CMIS implementations as they appear from the OASIS technical committee. Last year, we showed you how we can save costs by providing apples to apples comparisons between ECM vendors using the US GSA Schedule 70 pricing. Now we hope to help you beat even those saving with new capabilities that we are releasing with the Alfresco Community 3.2 download. We hope you give it a try at http://wiki.alfresco.com/wiki/Download_Community_Edition

Follow me on Twitter @johnnewton

CMIS Face to Face at Microsoft in Redmond

Dave Caruana and I just finished participating the in the first Face to Face of the OASIS CMIS Technical Committee here in Redmond, WA. Companies participating in person were IBM, Oracle, EMC, Alfresco, Exalead, OpenText, SAP, Day, Nuxeo, Dennis Hamilton (representing himself), and of course, Microsoft. Gary Gershon, Julian Reshke from Greenbytes and Betsy Fanning from AIIM participated by phone. (Sorry if I missed anyone.) Virtually everyone in the Enterprise Content Management market. The tone of the meeting was very cooperative and collaborative. David Choy from EMC is the chair of the committee and did a good job of staying neutral on the issues. Al Brown from IBM/FileNet is the secretary for the bindings and Ethan Gur-esh from Microsoft is the secretary for the data model. All three did an excellent job keeping things on track and discussing all the open issues. Most of us have all worked together before in both the JCR meetings and iECM meetings and that made collaboration a lot easier. Dennis and I even go back to ODMA and DMA - over 15 years ago.

(BTW, I micro-blogged what was going on Twitter. CMIS Tweets are below. You can follow me on http://twitter.com/johnnewton)

The CMIS effort so far has been use case driven with the main use cases being collaborative content management, integration into portals, mashups and search. There are a number of use cases that explicitly out of scope, such as records management and web content management. Some use cases lie in between where the main application may not be addressed, but we want to allow developers to create applications that may access the information that may be created by other out-of-scope applications. An example that we discussed a lot was records management. We don't want CMIS to be so complex that you can build a records management system, but you may want to access records information for the purposes of eDiscovery.

These use cases that are in-scope seem to be clear and compelling enough that we all agreed that we urgently want to get CMIS to market to get people building these applications. Although this was one of the last thing that we discussed, it is the most impressive outcome of the meeting. There was so much agreement and consensus in the meeting that we all felt it is important to meet the deadline of having CMIS complete before the end of the year. David Choy did a great job outlining the OASIS process and what must be done. The bureaucracy (and I mean that in the nicest possible way) of the OASIS process is such that the minimum time that it takes to get from public review to standard is 6 months. So we all agreed a timeline that would get this done. This means that the spec will be ready for public review by the end of Spring. That's a mean feat, but all agreed doable with few obstacles in the way. It also means that the 0.5 spec going into the OASIS process was pretty solid.

That doesn't mean that there wasn't plenty to discuss. It was a three day event after all and we used it all up. The most controversial issues were around security and what is truly RESTful. The Security proposal was presented by SAP who have a real need to secure documents consistently with the SAP system. This was an excellent use case and provided a real-world example from which to work and focused on Access Control Lists. Most of us have been involved in round and round discussions on security and I'm not quite sure how we got to a general consensus. We are not quite done, but we all seem to have a great deal of agreement. A subcommittee will work on this some more. I had proposed a more abstract Policy interface prior to OASIS submission based upon work that we have done in iECM and JSR-283. After discussing this with David Pitfield of Oracle, I think we need to think about that one some more. (Sorry James McGovern, but XACML was shot down in flames. For the reasons why, please see this IBM article on implementing ACLs with XACML and look how horrible ACLs look. http://www.ibm.com/developerworks/xml/library/x-xacml/)

David Nuescheler, who is the chair of the JSR-283 committee and CTO of Day Software, presented a couple of discussions in collaboration with Julian Reschke. Julian is also on the WebDAV committee and brings a unique perspective. David presented some of the thoughts that originated with Roy Fielding, also from Day, on whether the REST binding is really RESTful. Although there are arguments to not call the Atom Pub protocol REST bindings, we all felt it was important to communicate our intentions toward REST. In the end, we will call the protocol support RESTful Atom Pub bindings or something like that. David suggested perhaps exploring protocols other than Atom Pub, such as WebDAV. Apparently, these were ruled out quite a while ago and there are already WebDAV standards bodies. Decision is to continue with Atom Pub. Julian, who also seems to know HTTP extremely well, logged many issues around how CMIS uses / abuses HTTP and most of his recommendations are being incorporated into the spec.

David also raised the issue of reference implementations. In the end, we all agreed that we are all making progress on our own implementations and that a reference implementation wasn't necessary. There was interest and probably continues to be interest in an open Test Compliance Kit (TCK), but we all agreed that a clearer spec should be the primary point of reference rather than a TCK. It may be up to us, the open source vendors, to come up with a TCK. We have already created both a Java and .NET test harness to test CMIS access. Even Microsoft said that if you need a free CMIS test platform, you can get Alfresco. :-)

As the spec has developed, we have all been prototyping our implementations. The early implementors (Alfresco, EMC, IBM, Microsoft, OpenText, Oracle and SAP - in alphabetical order :-)) participated in an interoperability session or "plug-fest" at a Redmond interop lab in August prior to making the spec public. And it worked! It is a real testament to the fact that this is becoming a real standard. In other words, it works. This is part of the reason that there is not as much demand for a reference implementation. It is also part of the reason that it continues to develop and have such momentum. By working with each other to test each others implementations, we are discovering what needs to be tweaked in our software and what needs to be fixed in the spec. Another interop session is planned sometime around AIIM as well as a demo that the AIIM conference is organizing and sponsoring.

In addition, we were able to demonstrate our integrations with CMIS. IBM showed Quickr accessing CMIS. Day showed a prototype that they are working on that sits on Jackrabbit. Dave showed some new components that we are working on with Share and the Joomla and Drupal integrations with Alfresco using CMIS. Microsoft and EMC had shown their integrations in previous interop sessions.

There were a few other issues that we discussed. OpenText presented hierarchical properties. I presented a proposal on how to deal with Aspects or Mixins. A little more than half the systems represented, some like Oracle and IBM had more than one, support some notion similar to Aspects - think of them as data type plug-ins. The vendors that have broad search requirements had a great deal of interest in Universal Search (really a new use case) and want to see access to change logs to simplify indexing new or changed documents in ECM systems. However, I think we all agree that we don't want any of these to hold up a 1.0 of the specification and getting CMIS out to market.

CMIS has momentum. It will be a real specification. We all seem to agree on what we want and what our customers need. Companies are implementing it. IBM, EMC and Alfresco have released 0.5 implementations. Microsoft can't talk about what they are doing, but they will. (They would probably have to kill me if I knew exactly what they are doing. ;-)) It could be more and it can probably be better, but CMIS that is out there and implemented will be better than the perfect spec coming out much later.

Microsoft were gracious hosts and provided with good facilities and food. (Just replace the projector bulb. :-)) Thanks very much Ethan!

If you are interested, CMIS Tweets follow:

• Discussing CMIS use cases: Collab Content Mgmt, Portals, Mashups, Search
• Open use case issues: Observation, tagging, data dict., better REST bindings - less verbose
• Can't use MSN Messenger inside of Microsoft on their guest network. Bizarre.
• Going thru introductions, including favorite color. (Hmm.) Should note that SAP and OpenText are also here.
• Interesting discussion on reference implementation. Decision that Tech Committee will not implement ref impl.
• TC agrees that we need well documented test cases. Lots of interest in a Test Compat Kit.
• Show and tell coming up. Day and Lotus.
• Tomorrow will be REST design (with input from Roy Fielding via David Nuescheler), schema design and various issues.
• Now discussing REST issues. David Nuescheler is bringing up the issues first raised by Roy Fielding who first described REST.
• Discussed REST at length. Staying with Atom Pub, but will look at browser-based Javascript libraries.
• Gary Gershon is reviewing the SOAP bindings for CMIS.
• David Nuescheler: "There is no such thing as a REST binding. REST is an architecture."
• It is very difficult to concentrate on the minutiae of SOAP web services. (zzzzzz)
• Ethan (MSFT) now discussing unified search.
• Long discussion on universal search and relationship to observation. Universal search indexing looks like a MUST. Obs is SHOULD.
• Dave Caruana is demonstrating Alfresco SURF and CMIS.
• Dave Caruana just demonstrated Joomla and Drupal integrated with Alfresco thru CMIS.
• Just got caught out. I'm supposed to talk about data dictionaries and I don't remember what the issue was. :( Jet Lag!
• Long discussion on ACLs proposed by SAP. No appetite for XACML. Sorry James McGovern. Keeping it very simple.
• I was very surprised at Microsoft's interest in content crawling and Universal Search. Lumped into Observation discussion y'day.
• Just finished presenting how to do Mixins / Aspects in CMIS. Some want to model as related objects. How would you do it?
• Just finished OpenText proposal on "Hierarchical Properties"
• Question on the table - What do we need to do to finish the proposal?
• After lunch, group photo! Courtesy of Dennis Hamilton.
• Now batching.
• Discussing timeline for CMIS and flowcharted process. We are remarkably close! But min 6 months.
• Let me clarify. Get to public review in under 6 months.
• Closing out most open issues on spec from the Technical Committee.
• Would love to see Adobe here.
• Successful CMIS face to face complete! :-)