FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |1

Report on the Features and Opportunities for

Networked Multimedia Systems

FeaturesandOpportunitiesforNetworkedMultimediaSystems

Authors

MichaelBoniface,StefanoModafferi
ITInnovationCentre

AthanasiosVoulodimos
TheInstituteofCommunicationsandComputerSystems
NationalTechnicalUniversityofAthens

DavidSalamaOsborne
Atos

SandraMurg
JoanneumResearch

Version1.1

P a g e |2

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |3

Foreword
Newmediaapplicationsandservicesarerevolutionisingsocialinteractionanduserexperiencein
bothsocietyandinwiderangingindustrysectors.Therapidemergenceofpervasivehumanand
environmentsensingtechnologies,novelimmersivepresentationdevicesandhighperformance,
globallyconnectednetworkandcloudinfrastructuresisgeneratinghugeopportunitiesfor
applicationproviders,serviceproviderandcontentproviders.
Thesenewapplicationsaredrivingconvergenceacrossdevices,clouds,networksandservices,and
themergingofindustries,technologyandsociety.Yetthedevelopersofsuchsystemsfacemany
challengesinunderstandinghowtooptimisetheirsolutions(QualityofServiceQoS)toenhance
userexperience(QualityofExperienceQoE)andhowtheirdisruptiveinnovationscanbe
introducedintothemarketwithappropriatebusinessmodels.
Inthisreport,wepresenttheresultsofanewmultidisciplinarycollaborativeapproachtoproduct
andserviceinnovationthatbringstogetherusers,technologyandliveeventsinaseriesof
experimentsconductedinrealworldsettings.Throughexperimentationwehaveexploredabroad
rangeoftechnical,societalandeconomicchallengesfacedbytechnologyproviderseachaimingto
createandexploitnewmultimediavaluechainsinmarketssuchasleisureandtourism,culturaland
heritage,andsportsscienceandtraining.
Theexperimentshighlightthefeaturesofmultimediasystemsandthefutureopportunitiesfor
companies,astheInternetcontinuestotransitiontowardstheincreasinglyconnectedworldof
InternetofThingsandBigData.Weknowthatputtinguservaluesattheheartofdesigndecisions
andevaluationisthekeytosuccess,andthatlongtermbenefitstoprovidersoftechnology,services
andcontentmustderivefromenhanceduserexperience.Engagingusersinrealworldsettingsto
codesignandassesshowtechnologycanbeusedisnowmoreimportantthantestinghow
technologywillbeoperated.
Wehaveonlyscratchedthesurfaceofpossibilityinnovelnetworkedmultimediasystemsyetwe
believethattheindividualandcollectiveresultsinthereportaresignificantastheyaregroundedin
realworldevidence.Anewwayofconductingresearchandinnovationhasbeencreatedthat
maximisesthepotentialforcommercialexploitationandsocietalimpact.Wethinkthisisextremely
importantandwhenadoptedwillleadtogreaterbenefitsforall.

MichaelBoniface
EXPERIMEDIAAssociation
Nov2014

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |4

NetworkedMultimediaSystems
Multimediaisthecombinationofmultipleformsofcontentandisafundamentalelementof
applicationsinareassuchascommunication,entertainment,education,researchandengineering.
Theconvergenceoftechnologiesfordistributedmultistakeholdersystems,dataanalyticsanduser
experienceisdramaticallychangingthewaymultimediasystemsneedtoproduce,deliverand
consumecontent.
Providersofmultimediasystemsarenowlooking
tocreatevaluebylinkingpeopletoeachother
andtolocations(bothrealandvirtual)insucha
wayastocapturethepopularimagination,and
exploitthedesiresofconsumerstosharetheir
experiences,thuscreatingnewchannelsfor
revenuecreationandadvertising.
Tocreatesuchexperiencesrequiresinnovative
applicationsthatfocuson:enhanced
personalisation,nonlinearstorytelling;
interactiveimmersiveexperiences;creationof
socialcommunitieswhichallowpeopletouse3D
environmentstocommunicateandinteractwith
eachother;thecaptureandreproductionofthe
realworldin3D;andthecreationofperceptual
congruitybetweenrealandvirtualworlds.
Ofcourse,theseinnovativeapplicationswillplacesignificantdemandsonnetworkandcontent
managementinfrastructuresasprovidersattempttodeliverguaranteedQualityofServiceand
enhancedQualityofExperiencetocommunitiesthatdynamicallyorganisethemselvesaround
sociallydistributed,fixedandmobilecontent.Theseadditionaldemandswillrequireinvestmentin
infrastructurebuttheexpectationisthatbylinkingmultimediaandenhancedrealworld
experiences,consumerswillbepreparedtomakelonglastingcommitments.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |5

AMultiVenueMediaExperimentationService
EXPERIMEDIAisamultivenueexperimentationserviceforresearchanddevelopmentofnovel
Internetproductsandservicesaimingtodelivernewformsofsocialinteractionanduserexperience.
EXPERIMEDIAwasdevelopedaspartofaEuropeanresearchprojectofthesamenamewithinthe
FutureInternetResearchandExperimentationinitiative(FIRE)[1].

TheEXPERIMEDIAprojectsetouttodevelopandoperateauniquefacilityofferingresearchersand
companieswhattheyneedtogaininsightintohowFutureInternettechnologiescanbeusedand
enhancedtodeliveraddedvaluemediaexperiencestoconsumers.Theapproachaimedtodeliver,
reusable,costeffectivetestingandexperimentationfacilities,platforms,toolsandservicesforsocial
andnetworkedmediasystems.TheEXPERIMEDIAprojectdevelopedfourfoundationelements
necessaryforexperimentationofmultimediasystemsconductedinrealworldenvironments:

Smartvenues:attractivelocationswherepeoplegotoexperienceeventsandwhere
experimentscanbeconductedusingsmartnetworksandonlinedevices;
Smartcommunities:onlineandrealworldcommunitiesofpeoplewhoareconnectedoverthe
Internetandavailableforparticipationinexperiments;
Liveevents:excitingrealworldeventsthatprovidetheincentivesforindividualsandsmart
communitiestovisitthesmartvenuesandtobecomeparticipantsinexperiments;
ServicePlatform:stateoftheartFutureInternettestbedinfrastructureforsocialand
networkedmediaexperimentssupportinglargescaleexperimentationofusergenerated
content,3Dinternet,augmentedreality,integrationofonlinecommunitiesandfullexperiment
lifecyclemanagement.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |6

Thecombinationofliveevents,venues,usercommunitiesandanadvancedtechnologyplatform
acceleratesproductandserviceinnovationbyallowingcompaniestococreatesolutionsinreal
contextswithendusers.EXPERIMEDIAcharacterisesliveeventsasanycooperativehumanactivity
thatcanbeenhancedthroughaccesstorealtimeinformationdeliveredbytheInternet.Examples
liveeventsinclude:

1000spectatorsattendingatwodayskichampionshipataskiresort
Anathleteparticipatinginaonehoursportstrainingsessionwithacoachandsportsscientist
Agroup50studentsattendingaonehourinteractivevirtualrealitypresentationaboutancient
Greece
Asmallgroupofhikersonadaytriponamountain,aroundofgolforatrailrun

Therearemanysociotechnicalandeconomicbenefitstoexperimentersofusingliveeventsasthe
basisoftrialsandexperimentalstudies.Eachliveeventcapturesadistinctuserexperiencetobe
enhancedalongwithprovidingtemporalandspatialconstraintsassociatedtheactivitysuchas
location,technicalconstraintsassociatedwithavailableinfrastructureandsocioculturalconstraints
associatedwiththeusercommunities.Dealingwithcontextualfactorsisamajorchallengefor
experimentersaimingtodevelopgenericsolutionsforInternetdeploymentsandtounderstandhow
toaddressbarrierstoadoptionoftechnology.Inadditiontheabilityofmediatechnologiesto
connectpeopleinrealtimeacrossdistantlocationscancreatenewopportunitiesforinteraction
withliveevents.Fromaneconomicperspective,liveeventsprovidetechnologyproviderswith
accesstoanentrypointtoapotentialmarket.Thisentrypointcanleadtosignificantdirectand
indirectsales.
SocioTechnicalbenefitsforexperimenters:
TestingOpportunities

Economicbenefitsforexperimenters:
ExploitationOpportunities

observationofindividualandcommunitybehaviours

accesstoapotentialmarket,direct
sales

experienceofscalingforlargescaleshortlived
communities

workingwithacustomerscustomers

adaptationtotheenvironment,consideringphysical,
socialandethicalconstraints

creationofhighimpactshowcases,
indirectsales

adaptationofcontentaccordingtoindividualand/or
grouppreferences

engagementandcollaborationwith
stakeholders,potentialpartners/
suppliers

realtimeorchestrationallowingforadaptivenarratives
sensorsanddevicesfordetectionandtrackingof
featurepoints
devicecapabilitiesbothremoteandatavenue
cooperativeorcollaborativeframeworksincluding
dealingwithselfishormalicioususers

Table1:BenefitsandOpportunitiesforExperimenters

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |7

TheEXPERIMEDIAServicePlatformconsistsofasetofmediaservicesthathavebeeninstrumented
fordeeplevelsofobservabilityforusewithinexperimentationandtechnologytrials.Eachservice
hasacorrespondingservicemodelwithQoSmetricsthatarereportedandavailabletothecustomer
duringexperimentation.Suchdetailedmetricsarenecessaryforcustomerstoexplorethe
relationshipbetweenQoSandQoE.Thesetypesofmetricsaretypicallynotavailablefrom
equivalentcommercialservices.Inaddition,aprovenancemodelisofferedthatallowsusercentric
activitiesandinteractionstobetrackedandlinkedtothedetailedmetricsreportedbytheother
entitiesinvolved.Thiscapabilityisimportanttoallowexperimenterstotrackusersinopenstudies
andtoexplorecorrelationsbetweenQoE,systeminteractionandsystemperformance.

Fromtheplatformpointofviewthereusabilityacrossexperimentsisakeypointenablingmulti
domainapplications.Themediaservicesaretechnologyenablerswhosecapabilityallowsusersto
achieveaddedvaluethroughuse,eitherbydesign(i.e.thepurposeisknowninadvance)ormore
frequentlybyopenness(i.e.thepurposeisopportunisticallyestablishedbytheuser).Technology
enablersareakeypartoffutureinnovationinprogrammessuchasFIREandtheFutureInternet
PublicPrivatePartnership.NetworkedMultimediatechnologyenablersmustaddresstheneeds
novelapplicationsandservicesallowingthemtoexploitarangeofsocial,audio/visual,pervasive
contentand3Dcontent.Theplatformoffersservicestosupportdifferenttypesofcontent
consideringthedistinctcharacteristicsandlifecycles(authoring,managementanddelivery).

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |8

SmartVenuesandExperiments
SmartVenuesarerealworldlocationsthatofferliveevents,communities,infrastructureand
relevantdataassetstoexperiments.Smartvenueshavedistinctcharacteristicsandprovidecontext
forexperimentation.EXPERIMEDIAhasthreesmartvenuescoveringimportantapplicationsectors
formultimediasystemsincludingoutdoorsandleisure,culturallearningandsportstrainingand
science.

CentredaltRendiment(CAR),Spain,isahighperformancesportstrainingcentrewhichgives
supporttoathletescompetingataninternationallevel.CARoffersaprofessionalenvironment
forsmallscale(5participant)controlledexperimentsaimingtoimprovetrainingprogrammesfor
students,athletes,coachesandsportsfederationswithinadedicatedsmartbuildingwitha
privatecloudandhighperformancefixedandwirelessnetworkconnectivity.
TheFoundationoftheHellenicWorld(FHW),Greece,isaculturalcentrethatoffersrealand
virtualexhibitions,congressesandperformingartseventsaimingtoeducatepeopleaboutthe
HellenicWorld.FHWoffersapublicenvironmentformediumscale(30participant)experiments
aimingtoimprovevisitorexperienceandthequalityoflearningthroughmultimediaexhibitions,
virtualandimmersivereconstruction,andseriousgames.FHWoffersa3D,domeshapedvirtual
realitytheatre,exhibitionplaces,andcavesystems.
Schladming,Austria,isoneoftheleadinginternationalskiresortsinAustriaandpartoftheSki
Amadnetworkcovering28skiareasandtownsthatmakeupthelargestskiareainEurope.
Schladmingoffersapublicenvironmentformediumscale(50participant)opentrialsof
technologyaimingtoimprovevisitorexperiencewithintheregion.Theecosystemiscomplex
andpotentialactivitiesarebroadbutmostrelevantarewinterandsummeroutdoorsportssuch
asskiing,hiking,mountainbiking.

Wefundedaseriesof16experimentsthroughtwoopencalls.Theexperimentswereconductedby
researchersandSMEsatthreeSmartVenuesthroughoutEuropecoveringabroadrangeand
complimentarymultimediatopics.

SchladmingSmartVenue
DigitalSchladming:hyperlocalsocialcontentsyndicationandfiltering
MediaConnect:ubiquitousinteractiveandpersonalisedmedia
PinPointSchladming:augmentedrealitymobileapplications
iCaCoT:interactiveUHDcamerabasedcoachingandtraining
SmartSkiGoggles:realtimeinformationdeliveredtowearabledatagoggles

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |9

CARSmartVenue
LiveSynchro:accurateanalysisofchoreographedteamsports
3DMediainSports:noninvasivereconstructionofbiomechanics
CONFetti:interactive3Dvideoconferencingforcollaborativesportstraining
3DAcrobatics:wirelesssensormotioncaptureand3Dvisualisation
3DRSBA:remote3Dsportsbiomechanicsanalysis
CARVIREN:multifactorathletetrackingusingrealtimevideoandsensorinformation
AugmentedTableTennis:automaticnotationanalysissystembasedonvibrationsensors
andontablesurfaceprojection

FHWSmartVenue
NextGenDigitalDomes:learning,interactionandparticipationusingsocialandaugmented
content
REENACT:seriousgamesandimmersivemedia
BLUE:personalisedmuseumexperiencesusingcognitiveprofiling
PLAYHIST:seriousgameswithrealtime3Dreconstructionofmovinghumans

Asignificantdilemmaisbalancingresearchversusinnovationactivities.GeoffNicolsonof3Monce
saidResearchisturningmoneyintoknowledge,whereasInnovationisturningknowledgeinto
money.Veryfeworganisationscompletethefulllifecycleinthescopeofanexperiment.Inmany
cases,impactisachievedmuchlatereitherinhouseby
othergroups(e.g.industryorganisation)orbyothers
Crossingtheinnovationchasm
exploitingknowledgepublishedresearchinstitutions.In
requiresbuildingeffective
factforresearchinstitutionsthelinkbetweenknowledge
multidisciplinaryteams
generationandexploitationininnovativeservicesis
significantlyweaker.However,bycreatingmulti
disciplinaryteamsincludingdomainexperts,socialscientists,legalexpertsandtechnologists
workingwithendusersitispossibletoovercomebarriersandaccelerateadoptionintargetmarkets

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |10

Smartvenuesareconcernedwithofferinginnovativeservicesthatdeliverenhanceduser
experience.Knowledgeisonlyaroutetothatgoal.Thefirstopencallexperimentshadanemphasis
onknowledgecreationratherthaninnovationduetothecharacteristicsofthepartnersperforming
thework.Asaconsequence,theimpactofthoseexperimentswasfarlessandtheprojectstrategy
waschangedtocreateexperimentsdrivenbySMEs
Significantcommercialopportunities
forthesecondopencall.Overallsixexperiments
wereexecutedbySMEs,ninebyresearch
havebeendeliveredbyexperiments
institutionsandonebyindustry.18technology
highlightingtheinnovationpotentialof
outcomeswhereidentifiedfromtheexperiments
EXPERIMEDIA
withimpactclassifiedasfollows:

Commercialisation(5of18):benefitisexploitableinrevenuegeneratingproductsandservices.
FurtherTrials(4of18):promisingoutcomesjustifyingfurtherinvestmentintrialstoscaleupto
producequantitativeresultsortoexplorequalitativelyinanewapplicationdomain.
FurtherResearch(8of18):benefitlooksfeasiblebutcouldnotbesustainedwithoutsignificant
researchanddevelopment
Barrier(1of18):benefitcouldnotbedelivered

Significantcommercialopportunitieshavebeendeliveredtoexperimentershighlightingthe
innovationpotentialofEXPERIMEDIA.SmartSkiGoggleswilllaunchacommercialserviceintheSki
Amaderegionforthe2014/2015skiseasonandthereareongoingnegotiationsforthe
commercialisationoftheassociatedliftwaitingtimeservice.CARVIREN,3DAcrobatSportsand
3DRSBAresultedincommercialcontractswiththeCARSmartVenue.DigitalSchladmingMyMeedia
serviceremainsoperational12monthsaftertheexperimentandispartofIN2sstagingstrategy
andbusinessmodel.iCaCoTisinnegotiationwithSchladmingSkiSchoolforuseofinteractiveUHD
videoandannotationsystemaspartoftheirskiertrainingoffering.3DMediainSportshasreceived
significantcommercialinterestfromweightliftingandcyclingcommunitiesfollowingalargescale
trialwiththeMovistarcyclingteam.AugmentedTableTennishascreatedsignificantcommercial
interestfromTVbroadcastersandtheInternationalOlympicCommittee.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |11

NetworkedMultimediaSystems
FeaturesandOpportunities
Multimediasystemsarecharacterisedbythosethatacquire,processanddelivermultipleformsof
contentinservicesandapplicationswhereuserexperienceisasignificantfactorfortheirsuccess.
Thefeaturesofmultimediasystemsareextremelybroadcoveringallaspectsofcontentlifecycles
suchaslowlevelsignalandimageprocessing,datafusion,transcoding,compressionand
decompression,networktransmission,andrendering.Multimediasystemsevolveandare
intrinsicallylinkedtocontentformsthattheysupport.
Inrecentyears,theformsofcontentavailableandwaycontentisproducedandconsumedhas
changedsignificantly.Mobiledevices,wearabletechnologies,sensors,camerasandonlineservices
areacquiringanincreasingarrayofpervasive,social,audiovisualand3Dcontentaboutrealworld
environmentsandhowindividualandcommunitiesbehave.Inaddition,novelimmersive
environments,augmentedrealitydevicesandhighdefinitiondisplaysaretransforminguser
experiences.
Wenowexplorethefeaturesandopportunitiesformultimediasystemsasdefinedbythe16
experimentsconductedattheEXPERIMEDIAfacility.Throughamultidomainapproachwehave
identifiedandexploredacrosssectionofchallengesthatareassociatedwithmultimediafeatures
andtheirapplication.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |12

UsersattheHeartoftheSystem
Usercentricityisacriticalelementinthe
Userswhoparticipateinobservationsmust
designanddevelopmentofmultimedia
alsobethesameusersthatrealisetheprimary
systemsaimingtoenhanceuserexperience.
Understandingtheneeds,wantsand
benefits
limitationsofendusersmustbegiven
extensiveattentionthroughoutthedesignprocess.Wehaveadoptedtwomainprinciplesinouruser
centricdesignprocesses:

usersaretheprimarybeneficiaries,andotherbenefitstoprovidersofservicesandtechnology
willfollowfromuserbenefits
userswhoparticipateinobservationsarealsothosesameusersthatrealisetheprimary
benefits.

TheseprinciplesreflecttheshifttowardsthedemocratisationofInternetserviceswhereusersplaya
greaterroleingeneratinginformationandtheneedtorecogniseexplicitlythecostandbenefitof
participation.Ingeneralterms,designersmustconsideramultistakeholderdatavaluechainwhere
observationsareacquired,dataareprocessedbymultimediacapabilitiesanddataaretransformed
intobenefitspresentedtousers.
Observationistheprocessofcloselywatchingandmonitoringusersandtheircontext.User
observationsareprocessedasaninherentpartofcontentdelivery(e.g.locationandactivitytracking
ingeolocationservices)orareusedtounderstandtheexperienceitself(e.g.ausersatisfaction
survey).Fromausersperspective,observationshaveacosteitherdirectlyintermsoftimeand
attentionduringanexperience,orindirectlyintermsoflossofrighttoselfdetermination(i.e.
privacy).ContextobservationsareprocessedtogiveadditionalmeaningtoQualityofExperience
(e.g.auserhadagoodtimeinagroupof15closefriends)andimportantlytooptimisetheQualityof
Servicedeliveredbyserviceproviders.AscontextplaysasignificantinfluentialroleinQualityof
Experienceitistypicallythecasethatserviceprovidershavetomanagecontext,includingbothreal
world(e.g.howmanypeopleparticipating)andmultimediacontext(e.g.howmuchinfrastructure
resource,qualityofvirtualpresentations,etc.).

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |13

Analysingtheexperimentswecandefinesixcategoriesofuserobservationsfromatotalof95
differentuserobservations:

Satisfaction(32of95):feedbackaboutrelativesatisfactionwiththeirexperiencecovering
aspectssuchasutility,emotional,subjective,economic,usabilityandusefulness.
OnlineActivities(32of95):directinteractionwithanapplication(e.g.interactionlogs,website
statistics)thatcomplementstherealwordactivities,andisstronglyrelatedwiththenatureof
theexperiments.
RealWorldActivities(16of95):activityrecognition,forexample,biomechanicsrepresenting
thepositionofbodycomponents(e.g.theangleformedbybonesinanathletewhile
performing),higherlevelhumanactivities(e.g.weightlifting,skiing).
Collaboration(7of95):therelationshiptoagroup,intermsofinterpersonalrelationships,
socialinteraction,groupdynamics(e.g.questionsinagrouppresentation),groupenhancement.
Location(6of95):theabsoluteorrelativepositionofauserwhererelativemeanswithrespect
toexternalelements(e.g.askirun).
Cognitive(1of95):thecapacitytoprocessinformationandapplyknowledge(e.g.psychometric
profile).

Theabsolutevalueofobservationsrelatedtoacategoryisnotameasureofimportance.Asingle
typeofobservationcanbethemostimportantinagivenexperimentasitisthemostsignificant
factorindeliveringthebenefittoauser.Collaborationhighlightsthatmultimediafeaturesaimto
benefitusersbysupportinginteraction..
Thesatisfactiongroupistypicalofany
experimentalenvironmentanditis
propaedeutictoevolvefromexperiment
toexploitation.
Contextismorecomplexasbydefinitionit
isanythingnotrelatedtoauserthatcan
influenceQualityofExperience.Analysing
theexperimentswecanestablishtwo
mainhighlevelcontextcategoriesfrom
the56contextobservations:

RealWorldContext:observations
relatedtopeopleandenvironment
conditionsassociatedwithrealworld
activities.
OnlineContext:observationsrelatedtotheperformancecharacteristicsofthesystemunder
testcoveringaspectssuchascontentqualityandinfrastructureutilisation.

Thesignificantnumberofcontextobservationsacquiredmeansthatthesurroundingenvironment
playsasignificantroleinmultimediasystems.Infact,veryoftenthebenefitdeliveredtotheuseris
thecombinationofcontextandpersonalinformation.Realworldcontextishighlydependentonthe
RealWorldActivity.WithinEXPERIMEDIAthisisdefinedbythenatureoftheliveeventsbeing
studiedatSmartVenues.Realworldcontextisdifficulttoobserveautomaticallyandinageneral
wayconsideringthespecificnatureofliveevents.EXPERIMEDIAhasfocusedonobservinguserswith
somecasesofcapturingRealWorldContextwherethisisanessentialpartoftheexperienceandthe

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |14

costisnotprohibitive.IncontrolledexperiencessuchasthoseattheCARwhereRealWorld
ActivitiesarewelldefinedandconstrainedtheRealWorldContextisknownandcanbecaptured
outofband.InmoredynamicandopensituationsatSchladmingandFHWitisnecessarytoobserve
RealWorldContexteitherdirectly(e.g.definitionofPointsofInterestwithinageographicregion,
queuewaitingtimes,etc)orindirectly(e.g.inferencesaboutgroupdynamicsfromtemporal/spatial
analysisoronlineinteraction).
MakinginferencesaboutRealWorldContextandActivitiesfromOnlineContextandActivitiesisan
essentialpartofmultimediasystemsandexperimentationespeciallyinsituationswherethecostof
directobservationisprohibitiveeitherthroughsoftwareorfeedbackfromusers.EXPERIMEDIAs
hybridmetricandprovenancemodeloffersafoundationforsuchanalytics.Thehybridapproach
providestheabilitytocollectlargequantitiesofmeasurementdata(e.g.serviceresponsetimes,
networklatency,usersatisfaction,etc)whilstallowingforexplorationofcausationbetween
observationswithinsuchdata(e.g.usersatisfactioninrelationtoserviceresponsetime).Also,itis
recognisedthatInternetofThingsdomainhasmadesignificantprogressinacquiringrealworld
contextacrossabroadrangeofdynamicsituations.Thereisanopportunitytodeliverincreased
benefitsbystrengtheningtherelationshipbetweenUserandRealWorldContextobservations.
OnlineContextisofsignificantinteresttoserviceproviderswhousethisinformationtomanage
resourcesandoptimisethedeliveryofmultimediaservices,includingadaptionofthequalityof
content.AssuchOnlineContextisanimportantfacetofexperimentsthatfocusontherelationship
betweenQoEandQoS.Ofcoursethisdependsonthenatureofthestudybuttheadvantageofthe
EXPERIMEDIAPlatformisthatitisalreadyinstrumentedforObservationofOnlineContexttoensure
thatimportanttechnicalinformationwasavailabletoexperimenters.Typicallyexperimentshave
identifiedthesignificantOnlineContextobservationsrelatedtodeliveryofadesiredQualityof
Experience.Theseincludethequalityofcontext(e.g.accuracyofbiomechanicsdata,videoquality),
networkperformance(e.g.delay,bandwidth)andcloudperformance(e.g.CPUutilisation).

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |15

MakingaDifferenceintheRealWorld
Digitaltechnologiesaremostusefultosocietywhenusedtodeliverenhancedrealworldimpactand
benefits.Onlineinteractionalone,suchasdigitalgames,canbringenjoymentbutlongerlasting
satisfactionisachievedbyusingdigitaltechnologiesinsupportofrealworldactivities.Wefocusour
experimentsonthisareabydefining,measuringandanalysinguserexperience(UX)where
multimediasystemssupporttheinterplaybetweenrealworldLiveEventsandonlineactivities.Live
eventscreatethemaincontextforuserexperience.Wehaveexploredeventssuchasasports
training,anightoutinatown,attendanceatlargescalesportsevents,andvisitinganexhibition.
StudyingUXisacomplexendeavour.TheInternationalStandardOrganisation(ISO9241210)
definesUserExperienceasaperson'sperceptionsandresponsesthatresultfromtheuseor
anticipateduseofaproduct,systemorservice.UXincludesallusers'emotions,beliefs,
preferences,perceptions,physicalandpsychologicalresponses,behavioursandaccomplishments
thatoccurbefore,duringandaftertheuseofproduct,systemorservice.Theexperiments
themselvesfocusedondistinctUXaspectsenhancedthroughmultimediafeatures.Exploringthe
experimentsweidentifysevenhighleveluserbenefitcategoriesfrom61measurablebenefits:

Learning(22of61):acquisitionorimprovementofaskill/ability,akeygoaloftheCARandFHW.
Efficiency(11of61):supportforincreasingtheproductivityprocessesintermsoftime,effortor
costtocompletetheintendedtaskorpurpose.Efficiencyisacommonquantifiablemeasurefor
allactivitiesassociatedwithliveevents
Interaction,Influence&Control(10of61):interactingwiththesurroundingcontextfor
influenceandcontrol.(e.g.remoteaccesstotrainingsessions,orincorporationofaremote
expertinaneducationsession).
SituationalAwareness(10of61):understandingofwhen/where/whysomethingishappening,
soastomaximizetheactiveparticipationoftheuserintheexperience.Thisbenefitpertainsto
thedeliveryoftherightthing(information/support/other)exactlywhenitisneeded.
Enjoyment(5of61):theenjoymentauserhasintheperformedactivities,aprimarygoalof
SchladmingVenueasatouristdestination.
Personalization(3of61):tailoringtheinformationtomaximizeusersatisfactionincluding
expressingthemselvesinsocialnetworks.

Themajorityofbenefitsareproducedthroughprocessesthatenhancerawdatacollectedfrom
multipleinformationsources.LearningisaprimarybenefitinallCARandFHWexperimentsdueto
learningbeingakeyobjectiveofthevenues.NextGenDigitalDomesfocusedonhowaugmented
realitycanprimestudentknowledgepriortovirtualrealitypresentationswhereasREENACT
introducedaroleplayinggamethatallowedparticipantstoenactanddiscusshistoricalevents.
Situationawarenessisanothercommonuserbenefitdemonstratinghowthroughsensorsand
analyticsusersareprovidedwithbetterknowledgeofsurroundingcontext.Geospatialand
temporaldatawereessentialelementsofSmartSkiGogglesandPinpointSchladming.Influence
andcontroldemonstratestheincreasedpossibilityofcontrollingandinfluencingrealworld
situationsthroughremoteinteractionwithmultimedia(usingorbeingpartofthecontent).PlayHist,
CONFettiand3DRSBAallusenetworkedcollaborativeworkingtoenableremoteuserstointeract
andinfluencetrainingandlearningsessionswhilstmeasuringtheefficiencyorsettingup3Dcapture
equipment.Greaterenjoymentisanimportantbenefitacrossallvenuesbutwasnotexpressed
significantlyatprofessionalenvironmentssuchasCARwhereobjectiveperformancegainswerea
priority.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |16

RealtimeInteractiveandImmersiveMedia
Thegamesindustryhasasignificantimpactonbusinessandinnovationmodelsofthedigitalera.In
manyways,thegamesindustryareforerunnersofinnovativecontent,servicesandbusinessmodels
ofagrowingdigitaleconomy.Consequentlythegamesindustryispreparingthewayfortheother
sectorswherethedigitalrevolutionhasnotstartedyet.Anindustrychangingdynamicisthe
transformationofmultiplayergaming,builtonvastnetworksofplayersinterlinkedbybroadband
acrosscontinentsandgrowingfurtherstillbyleveragingsocialnetworks.Withcapabilities
strengthenedfurtherbythegenerationalleapsin3Dgraphics,gameplaymechanics,and
collaborativeplatforms,gamingispartneringwithandspurringgrowthinothermediasegments.
Gamingtechnologieshavebeena
Gamemechanicswillincreasinglyemergeasthe
sourceofinspirationinEXPERIMEDIA
foundationofnetworkedcollaborationmodels
throughtheadoptionofgameengines,
3Dsensorsandadvancedpresentation
technologiesacrossarangeofapplications.Novelalgorithmshavebeendevelopedin3DMediain
SportusingdatafromthelowcostKinectsensor,builtfortheXboxconsole.Using3Dinformation,
thealgorithmsprovideathletesandcoacheswithrealtimeperformanceinsightsinboth
weightlifting(i.e.speedandtrajectory)andcycling(i.e.aerodynamics)applications.Seriousgames
wereadoptedbyREENACTandPlayHistasawayofincreasingqualityoflearningforstudents
visitingtheFHWSmartVenueandpresentedintheimmersiveTholosDomeandonmobiledevices.
Asetofabstractgamedesignpatternsweredefinedaspartofthesecondmethodologytoprovide
constructsforcreatingeffectivegameplayindependentofspecificgametypesandtechnology
implementations.
ThemultidomaincoverageoftheEXPERIMEDIAPlatformhascreatedopportunitiesfortransferof
multimediatechnologiesdevelopedwithinthelifetimeoftheprojectacrosssectors.Technical
advancesinonesectorcanberapidlytransferredtoothersectorsviatheplatform,acceleratingthe
opportunityforinnovation.Forexample,realtime3DreconstructionofmovinghumansfromKinect
isacorecapabilityoftheEXPERIMEDIAplatform.Initiallythecapabilitywasdevelopedforhigh
performancesportstrainingthegenericcapabilityof3Dacquisitionfromvisualanddepthsensors
wasidentifiedtohavepotentialforcollaborationbetweenremoteusersindifferentsituationstobe
placedintovirtualenvironments.ThisledtouseofthetechnologyattheFHWSmartVenuefor
includingexpertactorsintoseriousgameswithinPlayHist.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |17

Whatisclearisthatnovelrealtime
interactivemediadelivery
mechanismsaretransformingsocial
interactionmodelsandimmersive
experiences.Peopleincreasingly
connecttoeachotherforworkand
leisureusingaugmentedandrealistic3Dreconstructionsoftherealworlddeliveredover
heterogeneousnetworksinrealtimetoindoorandoutdoorlocations.Thesecapabilitiesaredriving
infrastructurerequirements.A3DreconstructionofamovinghumanfromaKinectsensorproduces
100MBofdataaframe(futureHDsensorswillhavemuchhigherdatavolumes),andwith
transmissionratesof8fpswithcompressionof1:30abandwidthofatleast8Mb/secisrequired.
Qualityofexperiencerequirementsinteleimmersiveapplicationsrequiressynchronisation
precisionoflessthan100mswithafixedendtoendlatency.Datademandsaredrivingtheneedfor
experimentsexploringQoSandUXtechniquessuchasendtoendQoSoverfixedandwireless
networks,contextbasedcontent/infrastructureadaptationandsynchronisedstreamandevent
processing.

Realtime3Dinteractiveteleimmersionneeds
endtoendQoSofferedbyfuture5Gnetworking
capabilities

Weknowthatliveeventsareamajordrivingforceformassaudiences.Throughdigitalproduction,
broadcasterscannowdelivercontentmoreefficiently,flexiblyandwithgreaterscalability.However,
audiencesaredemandingenhancedrealtimeparticipationinliveeventsandthisgoesbeyondwhat
ispossiblewithcurrentmodelsofmediacreationandconsumption.Thenextlogicalstepinmedia
productionwillbethecreationofmoremeaningfulrelationshipsbetweentheplayersatliveevents,
thespectatorsandthemassiveonlinecommunitiesathomeoronthemove.Currently,broadcasters
areonlyskimmingthesurfaceofsocialinteractions:postingofviewersopinionssuchastweetsor
blogsalongsideprogrammesummaries,capturinganessenceofaudienceengagementthrough
likes,encouragingpersonalisedmediaproductionthroughusersubmittedphotosorvideos,etc.
Broadcasters,GamesProviders,Event
Managers,andtosomeextentthe
BroadcastersandGamesCompaniesmustwork
onlinecommunitiesthemselves,must
togethertounderstandthepotentialinterplays
worktogethercloselytooffermore
betweenbroadcastcontentandgamingcontentto
engagingandimmersiveuser
experienceswhichcanencompassallof
createnewimmersiveandmultimodalinteractive
thedifferentactorsacrossthevarious
experiencesfromliveevents
zonesofparticipation.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |18

EconomicandSocialViabilityofDataValueChains
Datavaluechainsareatthecoreofthefuturedigitaleconomy,bringingopportunitiesfordigital
developmentsthatbuildontheincreasingavailabilityandprocessingofalltypesofdata.Today,
datavaluechainsfocusonintelligentuseofdatatoenablethecreationofnewproducts,the
optimisationoftheproductionordeliveryprocesses,theimprovementofthemarket,new
organisationandmanagementapproaches,andthereinforcementofresearchanddevelopment
costreductionofoperations,increaseofefficiencyandbetterandmorepersonalisedservicesfor
citizens[2].However,itsclearthatalthoughbigdataandvaluechainsaredrivingthenewindustrial
revolution,withoutdesignandengagementofthecreativeindustries,suchinformationatworstis
meaninglessandatbestsuboptimal[3].

Companiescreating datavaluechainsmust

Wehavedesignedandexploredmany
considertheimplicationsofaggregatingopen,
datavaluechainsassociatedwithoutdoor
leisureactivitiesandsportsperformance.
closed,freeandpersonaldata
SmartSkiGogglesisdeliveringa
commercialservicetobelaunchedin2014/2015skiseason.Theserviceenhancesvisitorexperience
whileskiingonamountainbydeliveringrealtimeinformationandnavigationsystemusingstateof
theartdatagogglesincorporatingaheadsupdisplay.Informationaboutlifts,slopes,weather,
hospitality,socialmediaandnavigationareintegratedintoasingleapplicationallowingusersto
exploretheregionaccordingtotheirinterests.Mixeddatawereconsideredincludingacombination
ofopen,closed,freeandpersonaldata.Dataandserviceproviderswithinthelocalregionwere
engagedtoexplorecost,revenueandpricepointsforbusinessmodelssupportinglongtermviability
oftheservice.
Whatisclearfromengaginginaregionalecosystemsisthatdealingwithcloseddataisfundamental
toeconomicviability.Manybusinessmodelsofthewebarebuiltonadvertisingwheredataassets
canattractlargescaleonlinepopulations.Thisisnotthecaseforregionaldataassetsthatarehighly
localised.Forexample,PinpointSchladmingdeliveredaugmentedgeospatialopendatabutthe
limiteduserbaseinSchladmingandtheavailabilityofinformationthroughotherchannelsreduced
thepotentialvalueofgeolocationdataapplication.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |19

Anotherchallengewithdatavaluechainsbuildingonopendataisassoonasadataassetattains
value,ownerswillhaveatendencytoclosedatatoprotectvalueratherthancontributeitbackto
theopendatapool.Alsovalueisoftenrealisedduetoscarcityresultingfromproductioncosts
includingcosts(e.g.privacy,time,etc)forusersinvolvedinobservations.Forexample,liftwaiting
timewasconsideredhighvalueforskiernavigationbutthecamerainstallationandvideoanalytics
costswerehigh.Viablesolutionsrequiredcommercialagreementsbetweenliftoperators,
technologyprovidersandthemobileapplicationprovider.Pricepointsinbusinessmodelsmust
considerthatthebenefittousersmustbegreaterthanthecostofdataproduction.
Datavaluechainswerecentraltoimprovingtrainingprogrammesandathleteperformanceatthe
CARSmartVenuemakingextensiveuseofwearableandnoninvasivetechniquestocapture
biomechanicsandphysiologicalinformation.Highperformancetrainingisacomplexendeavour
requiringcontinuous
Thetransition ofhighperformancesportstechniques
supportfromspecialists
responsibleforanalysisof
fromthelaboratorytothefieldprovidesamodelforwhat
multifactordata.Coaches
willbeadoptedbywidersocietyincomingyears
anddoctorsneedaccurate
measurementsinorderto
offerthecorrectfeedbackforperformanceimprovementsandtheavoidanceofinjuries.Feedback
mustbetimelyandofteninstantaneoustoincreasetheefficiencyoftrainingsessions.3DMediain
Sportsused3DinformationfromKinectcamerasforrealtimecalculationofcyclistsaerodynamic
performanceandoptimalweightliftingspeedandtrajectory.3DAcrobaticsusedwearableinertia
sensorstocalculatedetailedbiomechanicsdatawhereasCARVIRENusedwearabledevice(WIMU)
tocollectawiderangeofathletedata.
ThesuccessofsolutionsinCARsenvironmentwerenotdrivenbyeconomicsbutthecostof
participationbyathletesintermsofergonomics,inconvenienceortime.Trainingsessionsare
carefullyscheduledandchoreographed.Wearabletechnologiesthatinhibitmovementortake
significanttimetoputonorcalibratearedeemedunacceptableunlesstheinformationcapturedhas
significantbenefits(e.g.injuryavoidance).Asaconsequence,currenttechniqueshavebeenlab
basedandnotpartofeverydaytrainingroutines.ExperimentsconductedinEXPERIMEDIA
demonstratedthepossibilityofmovingadvancedmeasurementtechniquesfromthelabtothefield
withoutintroducingsignificantcoststotheathletes.WhatweseeatCARintermsofmultifactor
measurementswillberepresentativeofwidersocietyinfutureascommunitiesrealisevisionsfor
quantifyingselfthroughwearabletechnologies.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |20

InnovationwhilstRespectingPrivacy
Multimediasystemsaredevelopedwithhumanparticipantsandinparticularrequireanincreasing
understandingofhumanbehaviourandexperiencetoprovidemeaningfulcollectiveexperiencesto
individualsandsociety.Acquisition,processingandprotectionofpersonaldataisanessential
systemfeaturewhichmustbeprovidedinthecontextofprivacylegislation.Ofcourse,theprivacy
debatehasragedinrecentyearsasUSsocialnetworkprovidersexperimentwithsocietysappetite
fordisclosingpersonalinformation.Inmanyways,Europeanserviceprovidersarenotoperatingona
levelplayingfieldbutifwebelievein
preservingandpromotingEuropean
Privacyandethicaloversightarenotabarrierto
values,legislationthatincorporates
innovationinexperimentallydrivenresearch
suchvaluesmustberespected.
Wehavesuccessfullydelivered
EuropeanproductandserviceinnovationinthecontextofEUprivacydirectivessuchasDirective
95/46/EC;ii)Directive2000/31/EC;iii)Directive2010/13/EU.Althoughcompliancewiththecorrect
ethicaloversightdirectivesisoftenperceivedasabarriertoprogress,performingexperimentationin
theirframecaninfactpreparesolutionsforEuropeanmarkets.WeuseaPrivacyImpact
Assessment(PIA)methodologytouncoverpotentialprivacyriskswithmultimediasystemsandat
thesametimeproposemitigationstrategies.EarlyanalysisofthePIAallowsforsufficienttimeto
implementthenecessaryamendmentsandsafeguardstoensurethatprivacyistakenintoaccount
bydesign,ratherthanbeingaddedattheendoftheprojectdevelopment.Withtheappropriate
safeguards,systemswereabletocollectpersonaldata,profileusersandtrackusersindoorsand
outdoors.Someofthefeaturesincludedtheuseofsecuredatastorage,encryptedtransfer,
controlledandauditableaccessfordifferentclassesofdatadistributedoverthesamechanneland
obscuring/removinguseridentitiesatsource(e.g.intheusersownsmartphoneorhomenetwork,
dependingonapplication)topreventdirectusertracing.
BLUEusedpersonaldatatocorrelatecognitiveprofileswithmovementsandpersonalpreferences,
toseeifthisknowledgecanenhanceuserexperiencesintheirvisitofmuseums.Thecognitive
profileswherecalculatedusingaFacebookgameandaresensitivepersonaldata.BLUEanalysed
privacyconsequencesbyexploringquestionssuchaswhethertheprofilewouldbepublishedonor
atleastknownbyFacebook?Whatifanemployerseesit?Whatifthecognitivestyleisidentified
wrongly?AnanalysisofFacebooksPlatformPolicyhighlightedthereisnoobligationtosendbackto
Facebooktheinterpretationorobservationsoncognitiveprofilesoftheuserderivedfrom
informationextractedfromFacebookAPIs.Ifhowever,theuserchoosestopublishtheseresultson
theirprofile,thentheywillbeavailabletotheirfriends,aswellastoFacebook.
Thisexamplehighlightsasignificantchallengeformultimediasystemsbuildingonpopularsocial
networkingsites.PinPointSchladming,DigitalSchladming,MEDIAConnect,BLUE,REENACTand
CARVIRENallbuiltontheFacebookApplicationProgrammingInterfaces(APIs).Developersare
requiredtousetheAPIinaccordancewithrulesonleveragingcontentfromtheunderlyingsocial
networksasdefinedindevelopersTerms&Conditions(T&C).Whatsclearisthatcompliancewith
theSocialNetworksT&Ccansignificantlyinfluencesystemarchitectureconsideringrulesfor
publishingcontentandtheincreasinglystringentrulesforextractingcontent.Platformproviders
monitorcloselytheapplicationecosystemanddemandthatthedeveloperscooperatewiththem,
especiallyincasetheapplicationrequiresalargeamountofAPIcalls.ThroughTermsandconditions
SocialNetworkprovidersmaintaintheirpositionofpowerwithinmultimediasystemsthatrelyon
socialmediacontent.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |21

Conclusions
Wehavepresentedthefeaturesofandopportunitiesfornetworkedmultimediasystemsbuildingon
theresultsofexperimentsconductedattheEXPERIMEDIAfacility.Wehavedemonstratedthe
benefitsoftheEXPERIMEDIAapproachfordeliveringinnovativeproductsandservicestospecific
marketsasrepresentedbySmartVenuesbyconductingexperimentsatLiveEvents.
Risksinimplementingmultimediasolutionsinalivecontextwherelotsofpeopleareinvolvedare
various.Forexample,definingtechnologysolutionswithoutabusinesscasesornotbeingableto
properlyaddressprivacyissues.Bothofthesecanbemitigated,ifnotcompletelyremoved,using
EXPERIMEDIAmethodologies.
CARshighperformancetrainingplansacrossmultiplesportshavebeenradicallychangedthrough
multifactorsensing,highdefinitionvideoandvideoconferencingtechnologies.Newknowledgehas
beengeneratedthatshowshowQualityofLearningcanbeimprovedthroughseriousgames,
personalisationandinteractivemediatechnologiesatFHW.Realtimegeospatialinformationand
socialrecommendationhaveenhancedvisitorexperienceatSchladming.
Whatisclearisthatnetworkedmultimediasystemshavehugepotentialforsocioeconomicimpact
andwillbetransformedthroughthecontinuingconvergenceofinfrastructuretechnologiesandthe
increasingavailabilityofdatafromIoTplatformsandBigDataanalytics.However,torealisethe
benefitsofthisdigitalrevolutionusersanduserbenefitmustbeatthecentreofdesignprocesses,
andcreativeexperiencedesignerswillhaveamajorroletoensurethattheexplosionofdatacanbe
turnedintoenhancedexperiencesandsustainabledatavaluechains.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |22

Acknowledgements
Theauthorswouldliketothankthefollowingexpertsforadviceandreviewduringtheproductionof
thisreport:
GillianYoungs

FacultyofArts,UniversityofBrighton

TheodoreZahariadis

Synelixis

TomGross

HumanComputerInteractionGroup,UniversityofBamberg

FedericoAlvarez

UniversidadPolitcnicadeMadrid

FreekW.Bomhof

TNO

DavidGeerts

KULeuven

TheEXPERIMEDIAprojecthasreceivedfundingfromtheEuropeanCommissionaspartoftheFP7
workprogramme.

FeaturesandOpportunitiesforNetworkedMultimediaSystems

P a g e |23

References
[1]EuropeanFutureInternetResearchandExperimentation
[2]OECD(2012)reporton'Exploringdatadriveninnovationasanewsourceofgrowth',
DSTI/ICCP(2012)9/REV1,p.13
[3]DesigningtheDigitalEconomy,EmbeddingGrowthThroughDesign,InnovationandTechnology,
http://www.policyconnect.org.uk/apdig/sites/site_apdig/files/report/463/fieldreportdownload/desi
gncommissionreportdesigningthedigitaleconomy.pdf

ThisworkislicensedunderaCreativeCommonsAttribution
NonCommercialShareAlike4.0InternationalLicense.Toviewthe
licensevisithttp://creativecommons.org/licenses/byncsa/4.0/