Interlinking Multimedia

How to Apply Linked Data Principles to Multimedia Fragments

Becze Szabolcs, VUB, 1st year master of computer science.

1. Abstratct

This report introduces the interlinking multimedia (iM), a new way to apply linked data
principles on the current web data: pictures, videos and audio items. There will be an
introduction about the current solutions to address and retrieve multimedia content together with
its semantic representation. Finally, I will give a brief overview about the current XML database
systems, which allows us to efficiently store and query metadata about multimedia.

2. Motivation

Millions of multimedia documents are produced every day, by uploading a picture to

Flickr or Facebook, video to Youtube, or audio file or creating a RIA, so we can say that it is
easy to produce, but much harder to reuse. These systems allow their users to manually tag
them, comment them and share these comments, but this information is locked into the system

Figure 1 Tagging somebody on Facebook

[1]
 i.e cannot be used by other external applications. To solve this issue, the multimedia semantics
community has focused on developing so-called multimedia ontologies.

3. IM principles

The motivation for introducing iM comes from the fact that we currently do not have proper
means to address and describe fragments of multimedia resources in the Web of data. The goal
of iM is to derive both semantic and audio-visual representations from multimedia resources on
the Web.

Tim Berners-Lee outlined the principles of the Linked Data as follows [1]:

1. All resources should be identified using URIs;

2. All URIs should be dereferenceable, that is HTTP URIs, as it allows looking up the
identified resources;
3. When looking up an URI, it leads to more (useful) data about that resource;
4. Links to other URIs should be included in order to enable the discovery of more data.

To explain it, let us have a look at a concrete scenario. Someone wants to download an audio
file, but she is only interested the content between second 15 and second 45. The following
GET command will therefore be issued:

Figure 2 Request for the audio content Figure 3 Response to the first request

We assume that on server side a module can cut out only that interval from the audio file.
Further on, we also need the semantics attached to this file, so we will send another get request
but this time we are interested in the RDF description of the media content.[1]

Figure 4 Request for RDF description of the media Figure 5 The requested RDF description
content
[2]
 4. Methods for interlinking multimedia

So far we described how to connect the semantic meaning with the media content. The next
question is how to create these semantics, and how to annotate the already existing resources.
Four different methods can be distinguished.

1. Manual method
The manual method relies on the end user’s qualitative feedback about the multimedia.
One proposal is CaMiCatzee, which allows people to semantically annotate picture on
Flickr and query for persons using their FOAF documents, URIs or person names.

Another more interactive approach is Game Based Interlinking (GBI), which tries to
make the interlinking more fun, and hide the task of interlinking behind a game. Such an
approach is for example realized by OntoGames, OntoTube or Peakaboom just to.

2. Collaborative interlinking
A Collaborative approach to interlink resources could be followed using Semantic Wikis.
Semantic Wikis extend the principle of traditional Wikis such as collaboration, linking and
versioning are provided and the possibility to type links and articles via semantic
annotations.

3. Semi-automatic method
Semi-automatic interlinking methods consist of combining multimedia-analysis
techniques with human feedback. Analysis techniques can process the content itself or
the context in which the content is expressed in order to suggest potential interlinking.
The user would need to accept, reject, modify or ignore those suggestions. [1]

4. Automatic methods
It’s achieved by purely analyzing the content of the media with image/audio processing
methodologies.
For example in case of an music file, the content can be divided into segments, and can
be automatically linked to musical concepts like: vers, chorus, refrain etc. In the below
picture you can see such a segmentation. We have semantic data about the total
duration of the audio, about the first chorus and the first verse. Each segment stores
information about the start time, and duration about the segment, but can contain other
meta-information too, like instruments, artists etc...

[3]
 5. Handling the metadata
Figure 6 Segementation of an audio / music file

The next section is added by me, because I felt that storing, querying and managing this
large amount of amount of RDF / XML data is or will be an another very important topic.
In further I will discuss briefly what are the advantages of using XML DBMS. I looked for
some XML database management system, instantly on Wikipedia I found 11 free or
commercial implementation of an XML-based DBMS. Two main types can be
distinguished [4]:

a. XML Database types

1. XML-enabled: these map all XML to a traditional database (such as

a relational database), accepting XML as input and rendering XML as output.
2. Native XML (NXD): the internal model of such databases depends on XML and uses
XML documents as the fundamental unit of storage, which are, however, not necessarily
stored in the form of text files.

b. Why to use XML database?

There are several reasons why an XML based database is more favored over a traditional
relational database. First XML has the advantage that it stores semi structured data in contrary
to relational database data where you have records. When a record item does not contain
anything it will have a null value, but in case of an XML node, it just simply omits that attribute.

[4]
 According to Ronald Bourret, the retrieval speed is faster than the relational one.
Depending on how the native XML database physically stores data, it might be able to retrieve
data much faster than a relational database. The reason for this is that some storage strategies
used by native XML databases store the entire documents using physical (rather than logical)
pointers between the parts of the document. This allows the documents to be retrieved either
without joins or with physical joins, both of which are faster than the logical joins used by
relational databases. The obvious drawback in this case is that the increased speed applies
only when retrieving data in the order it is stored on disk. If you want to retrieve a different view
of the data, such as a list of customers and the sales orders that apply to them, performance will
probably be worse than in a relational database. Thus, storing data in a native XML database
for performance reasons applies only when one view of the data predominates in your
application. [5]

6. Related work

The iM is not the only approach to link multimedia content. Other attempts were introduced
before it, like the Synchronized Internet Markup Language (SMIL) a W3C recommendation that
enables the integration of independent multimedia objects such as text, audio, graphics or
videos into a synchronized multimedia presentation. Within this presentation, an author can
specify the temporal coordination of the playback, the layout of the resentation and hyperlinks
for single multimedia components. The latest version of SMIL provides a rich MetaInformation
module which allows the description of all elements of a SMIL document using RDF.

7. Personal opinion

I found the article very well structured and well explained. They started from theory (the
requirements, with the presentation of iM principles), and they finished with more practical
solutions (CaMiCatzee, Ramm.x). The examples what they shown were illustrative and
understandable, with a lot of pictures. I would like to highlight also the reference list, which
helped me a lot to understand in more depth certain technologies or products (ramm.x or
OntoTube).
I can’t really mention any negative points, but maybe I can mention that I wanted to hear more
about how this metadata is handled, queried and stored efficiently, because if in the future
everything will have some XML based metadata than I think the management of this will be an
important issue.

The topic itself is a very interesting and I think very important too. Some issues related with the
multimedia interlinking have been solved, and new ones came. Two years ago, when I searched
for a semantic wiki software I could only find one or two possible implementations. Now, I can
see that there are tens of them: KiWi, Kiki Wiki, OntoWiki, PhpWiki, Freebase and a lots of
more, just to mention a few of them. The situation is the same with the XML database software,
just within a few years the number of (free) products has doubled or tripled. Personally, I will
look forward to the changes in this field in the next few years because I expect huge changes in,
how media content is handled and interlinked.

[5]
 8. References

[1] Hausenblas, M., Troncy, R., Bürger, T., & Yves, R. (2010). Interlinking Multimedia, How to
Apply Linked Data Principles to Multimedia Fragments.

[2] Bhowmick, S., Madria, S., & Ng, W. ( 2004). Web Data Management. Springer Verlag.

[3] Abiteboul, S., & Suciu, D. (n.d.). Data on the Web- from Relations to Semistructured Data
and XML. Morgan-Kaufmann.

[4] Gruen, C. (2011). Wikipedia. Retrieved from http://en.wikipedia.org/wiki/XML_database.

[5] Bourret, R. (2005). http://www.rpbourret.com. Retrieved from

http://www.rpbourret.com/xml/XMLAndDatabases.htm#isxmladatabase.

[6]