FLV Cuepoints in Flash CS3 with Flash Video Encoder versus FLV and F4V Cuepoints in Flash CS4 with Adobe Media Encoder Best Practice

This article is aimed to clarify:

How Cuepoints and which cuepoints type are presented to the user in Flash CS3 for FLV

and
How Cuepoints and which cuepoints type are presented in Flash CS4 for FLV and F4V.

View Cuepoint In Flash CS3
==========================

FLV:

Cuepoints when available are inported at the end of the Import Video Wizard workflow.
User are able to see their cuepoints by selecting the FLVPlayback Control on stage and after showing the Component Inspector panel (dockable panel). The cuepoints are visible in the cuepoint property of the Component Inspector. Cuepoints are sorted based on the SMPTE time at which they have been created. User can insert Event or Navigation Cuepoints using Flash Video Encoder. Cuepoints are embedded in the FLV.

Codewise, Flash Developers have the option to use Action Script 2 or Action Script 3 in order to interpret Cuepoints (Event and Navigation) at runtime (when the SWF is playback by Flash Player 9 or Flash Player 10). I am not going to cover the onCuepoint Callback (timed based event cuepoint callback) which applies to FLV only but rather focus on the use case where all cuepoints are preloaded/received at once. The following code is Action Script 3 code using NetStreem to attach a _metadata class in order to receive Metadata through onMetaData event callback. In onMetaData Cuepoints are returned in an Array.


connection.connect(connectURL);
stream = new NetStream(connection);
//assign metadata client
stream.client = _metadata;
var video:Video = new Video();
video.attachNetStream(stream);
stream.seek(0);

........

_metadata.onMetaData = function(eventObj:Object):void {
var cuePoints:Array = eventObj.cuePoints;

//cuepoints are returned as an Array
}

Callback Sequence
=================

For FLV the callback sequence will be

Callbacks: onMetaData


View Cuepoints In Flash CS4
===========================

FLV:

FLVs now contains two types of Cuepoints. Two types of Cuepoints are encoded in the FLV. "Flash CS3" Cuepoints and Cuepoints encoded as a subset of an XML Track


first < has been replace by { and > by } because google blog text editor is retarded..

{xmpDM:trackName}AME Markers{/xmpDM:trackName}
{xmpDM:frameRate}f254016000000{/xmpDM:frameRate}
{xmpDM:markers}
{rdf:Seq}
{rdf:li rdf:parseType="Resource"}
{xmpDM:startTime}7695905817600{/xmpDM:startTime}
{xmpDM:name}Title1{/xmpDM:name}
{xmpDM:type}FLVCuePoint{/xmpDM:type}
{xmpDM:cuePointType}Navigation{/xmpDM:cuePointType}
{xmpDM:cuePointParams}
{rdf:Seq}
{rdf:li rdf:parseType="Resource"}
{xmpDM:key}Title{/xmpDM:key}
{xmpDM:value}Star Trek{/xmpDM:value}
{/rdf:li}
{rdf:li rdf:parseType="Resource"}
{xmpDM:key}Color{/xmpDM:key}
{xmpDM:value}Blue{/xmpDM:value}
{/rdf:li}
{/rdf:Seq}
{/xmpDM:cuePointParams}
{/rdf:li}
{/rdf:Seq}
{/xmpDM:markers}
{/rdf:li}
{/rdf:Bag}
{/xmpDM:Tracks}

The cuepoint type are the same than the one defined in the FLV spec part of Flash CS3. The FLV Cuepoint Track (AME Markers in the above sample XMP block) follows
a "well defined" XML syntax, namely the XMP subset of RDF. The RDF spec uses a BNF-like grammar, not an XML DTD. The XML syntax described above is compliant with Flash CS4 integration of Cue Points through the Cue Points dialog and the Component Parameters Cue Points Property. This is assuming you are calling Adobe Media Encoder from Flash CS4 at encoding time.
For FLVs to ensure backward compatibility with Flash CS3, Flash CS4 presents the FLV Cuepoints encoded in the "non XMP form", like wise they were encoded in Flash CS3 and earlier, and WON'T present FLV XMP Cuepoints in the Component Inspector at all if the FLVs have been encoded by Adobe Media Encoder. If Flash Designers and Developers change XMP Cuepoints in their FLVs they have to bare in mind to maintain the non XMP cuepoints as well encoded in the file if they want to see them in Flash CS4 using Adobe SDK and Adobe Software. If Flash Developers prefer to preload XMP Cuepoints only in their SWFs for FLVs, they can rely on Adobe Bridge CS4 or the Import and Export XMP feature available in Adobe Media Encoder CS4 to edit/replace their Cuepoints. They need to preload their Cuepoints at runtime all the time through the onXMPData callback. As far as synchronisation of Cuepoints is concerned they can use a timer and video polling techniques when loading Cuepoints through onXMPData in order to provide accurate synchronization of Event and Navigation Cuepoints with Action Scripts. If they don't want to preload XMP Cuepoints but still want to use Cuepoints in their SWF they need to preload Cuepoints at runtime through onMetadata and proceed with timer and video polling techniques to perform the synchronization of their Action Scripts Code or rely on OnCuepoint which is an SMPTE timed based event callback notification available in Flash CS3 as well and Flash Player 9 but only for FLV.

F4V:

F4Vs also contains XMP Metadata in order to represent "FLV Cuepoints" as well when the F4Vs are encoded by Adobe Media Encoder

For F4Vs Flash CS4 present XMP Cuepoints in the Component Inspector in the order they appears in the RAW XMP Metadata block embedded in the F4V and in the Cuepoints dialog, Cuepoints params are listed in the order they appears in the RAW XMP Metadata block. User can alter XMP Cuepoints using Software and Human Interface such as Adobe Media Encoder or Adobe Bridge CS4 by exporting/modifying/importing XMP files.

Codewise, Flash Developers have the option to use Action Script 2 or Action Script 3 in order to interpret FLVs and F4Vs "FLV Cuepoints" at runtime (when the SWF is rendered by Flash Player 10). As far as synchronisation of Cuepoints is concerned they HAVE to use a timer and video position polling techniques when loading Cuepoints through onXMPData and they have to do their own Action Script synchronization for event and navigation cuepoints.


The following code is Action Script 3 code using NetStream to attach a _metadata class in order to receive Metadata through onMetaData event callback as well as the new callback onXMPData.

Callback Sequence
=================

For F4V the callback sequence is

Callbacks: onMetaData -> onXMPData -> onMetadata

For FLV the callback sequence is

Callbacks: onMetaData -> onXMPData


connection.connect(connectURL);
stream = new NetStream(connection);
//assign metadata client
stream.client = _metadata;
var video:Video = new Video();
video.attachNetStream(stream);
stream.seek(0);

........

_metadata.onMetaData = function(eventObj:Object):void {
var cuePoints:Array = eventObj.cuePoints;

// For F4V First and Third callback call in the callback sequence listed above
// For FLV First call in the callback sequence listed above
}

_metadata.onXMPData = function(info:Object):void {
trace("raw XMP =\n");
trace(info.data);
// For F4V Second call in the callback sequence listed above info.data contains raw XMP block
// For FLV Second call in the callback sequence listed above info.liveXML contains raw XMP block
}

Note: Flash CS4 will import Cue Points stored in the info.data object only not info.liveXML but your AS3 code in your custom player will receive the data as listed above with Flash Player 10

Overall : It is possible with both FLV and F4V cuepoints to have frame accuracy with Event and Navigation Cuepoint

XMP Resources and spec

XMPCore http://partners.adobe.com/public/developer/xmp/topic.html
XMP http://www.adobe.com/devnet/xmp/

Flash CS4 Video Playback http://www.adobe.com/devnet/flash/articles/flvplayback_fplayer9u3_04.html

FLV and F4V http://en.wikipedia.org/wiki/Flash_Video

XMP http://en.wikipedia.org/wiki/Extensible_Metadata_Platform

http://www.adobe.com/products/xmp/

Some useful resource regarding RDF and triple store

http://www.w3.org/TR/rdf-schema/
http://simile.mit.edu/reports/stores/

and finally a great use case of what can be done using XMP cuepoint : Interactive FLV and F4V Video

http://younsi.blogspot.com/2008/12/interactive-video-in-flash-using-f4v.html