Video Labs: Kaltura HTML5 Sequencer available on Wikimedia Commons

I am happy to invite the Wikimedia community to try out the latest Kaltura HTML5 video sequencer as part of a Wikimedia/Kaltura Video Labs project that can now be used on Wikimedia Commons with resulting sequences visible on any Wikimedia project. For those that have been following the efforts, it has been a long road to  deliver this sequence editing experience within the open web platform and within the MediaWiki platform. This blog post will highlight the foundational technologies in use by the sequencer in its present state and outline some of the upcoming features in Firefox 4, and enhancements to the sequencer itself that are set to improve the editing experience.
If you want to just jump into editing, please check out the commons documentation page and play around with the editor and let us know what you think. This project is early on in its development. Your bug reports,  ideas, feedback and participation will help drive future features and how these tools are used within Wikimedia projects.
If you’re interested in Video on Wikipedia in general, please consider joining the Wikivideo mailing list which will cover a wide range topics, including the sequencer, collaborative subtitles, timed text, video uploading, video distribution, format guidelines, and campaigns to increase video contributions to the site.
And finally, if you are in the New York area consider checking out the Open Video Conference coming up October 1st to the 3nd, which will be a great space to hack on open video and work on ideas for the future of video on Wikimedia projects.

The Kaltura HTML5 sequencer Edit Decision list format ( EDL ) is based on an old standard called SMIL. The SMIL spec has evolved over time and has become famous for being very large and complicated to implement in a real world player. The javascript based SMIL player in the Kaltura sequencer is no different in this respect as it implements just enough of the spec to be useful for collaborative video editing and exposes an HTML5 video style API for controlling playback and timing. Another nice feature of SMIL xml is that it is extensible for custom components. So far we have added some tools for using the editor with MediaWiki templates. This gives sequence authors the dynamic power of wikitext and html css to layout elements in sequences.
By choosing HTML as the definitive compositor for the editing format it allows the sequencer to rapidly evolve and add new features along side the HTML platform. In other words if we can display an svg graphic on top of a video in HTML, we can do that in the sequencer.
This of course runs into the issue of cross browser compatibility; not all browsers are going to support ogg video and svg graphics at the same time. How do you play those video sequences in those browsers? Additionally once you start stacking many video elements with many sub section seek requests, transitions, effects, image pan zooms and audio it becomes difficult to give a user a consistent playback experience and keep the playback of all these elements in sync.
To address these issues we have a browser extension called Firefogg. Firefogg a project we have been working on with open source video hacker j^. It is used by many on the web today to convert video into free formats ( Ogg Theora or WebM ). We use that same in-browser video encoder technology and some custom Firefox DOM -> canvas features to grab the sequence frame by frame and build a flat video file that gets uploaded to commons so anyone[1] can play back the sequence. This provides a full end-to-end editing platform so that video edited with the html5 sequencer can be rendered out directly by the client. This is useful as the sequencer can also work as a stand alone video editor.

While this technology have been in development for sometime there are still many issues that prevent a smooth video editing experience. Fortunately the community driven web browser platform is rapidly maturing. One of the first issues you may notice when using the editor is the slow seek times on video. Ogg was originally designed as a streaming media format,  so it is not very fast to seek. Fortunately Chris Pearce from Mozilla has been hard at work on improving ogg with an index so that seeking will be blazingly fast with Firefox 4 over plain http connections. This will help enormously for sequences with lots of clips with lots of subsections. Another thing you may notice is the slow render time when publishing video. This is also slated to be improved with Firefox4, with more direct access to canvas renderings, improved buffer control and improved seeking within cached media.
Audio is also a problem area in the current browser model. Fortunately, the Mozilla community has been working on an audio data api that will enable standard audio features like cross dissolves,  and multi track volume control. These features also landing in Firefox 4 are crucial for audio support that goes beyond basic audio cuts, and will enable high quality control over track volumes as background music comes and goes within a given sequence.
Another feature you may find missing is cross dissolves transition and key frame based pan and zoom. These two features should be making their way into the sequencer soon. I have started a roadmap page to help outline a feature timeline.
You may find that it would be really convenient if the sequencer had X feature. It is important that you share what X feature is. There is a wide range of possibilities from automated multilingual timed text building from source assets to whiz-bang features like hardware accelerated WebGL transitions and effects, to collaborative tool helpers like sequence transclusion. I look forward to working with people’s ideas as we build a powerful platform for collaborative knowledge production in the visual medium.
(1) Well almost anyone … Google chrome needs to fix a theora 1.1 rendering bug 😉
Peace,
Michael Dale, Open Source Video Collaboration Technology