Manual - Streaming Service Test

A Rohde & Schwarz Company
Diversity Streaming Service Test

Manual March 2013
SwissQual License AG Allmendweg 8 CH-4528 Zuchwil Switzerland t +41 32 686 65 65 f +41 32 686 65 66 e info@swissqual.com www.swissqual.com
Part Number: 16-100-200635-2 Rev 1.2
SwissQual has made every effort to ensure that eventual instructions contained in the document are adequate and free of errors and omissions. SwissQual will, if necessary, explain issues which may not be covered by the documents. SwissQuals liability for any errors in the documents is limited to the correction of errors and the aforementioned advisory services. Copyright 2000 - 2013 SwissQual AG. All rights reserved. No part of this publication may be copied, distributed, transmitted, transcribed, stored in a retrieval system, or translated into any human or computer language without the prior written permission of SwissQual AG. Confidential materials. All information in this document is regarded as commercial valuable, protected and privileged intellectual property, and is provided under the terms of existing Non-Disclosure Agreements or as commercial-in-confidence material. When you refer to a SwissQual technology or product, you must acknowledge the respective text or logo trademark somewhere in your text. SwissQual, Seven.Five, SQuad, QualiPoc, NetQual, VQuad, Diversity as well as the following logos are registered trademarks of SwissQual AG.
Diversity Explorer, Diversity Ranger, Diversity Unattended, NiNA+, NiNA, NQAgent, NQComm, NQDI, NQTM, NQView, NQWeb, QPControl, QPView, QualiPoc Freerider, QualiPoc iQ, QualiPoc Mobile, QualiPoc Static, QualiWatch-M, QualiWatch-S, SystemInspector, TestManager, VMon, VQuad-HD are trademarks of SwissQual AG. SwissQual acknowledges the following trademarks for company names and products: Adobe, Adobe Acrobat, and Adobe Postscript are trademarks of Adobe Systems Incorporated. Apple is a trademark of Apple Computer, Inc. DIMENSION, LATITUDE, and OPTIPLEX are registered trademarks of Dell Inc. ELEKTROBIT is a registered trademark of Elektrobit Group Plc. Google is a registered trademark of Google Inc. i.Scan is a trademark of CommScope, Inc. Intel, Intel Itanium, Intel Pentium, and Intel Xeon are trademarks or registered trademarks of Intel Corporation. INTERNET EXPLORER, SMARTPHONE, TABLET are registered trademarks of Microsoft Corporation. Java is a U.S. trademark of Sun Microsystems, Inc. Linux is a registered trademark of Linus Torvalds. Microsoft, Microsoft Windows, Microsoft Windows NT, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries U.S. NOKIA is a registered trademark of Nokia Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. SAMSUNG is a registered trademark of Samsung Corporation. SIERRA WIRELESS is a registered trademark of Sierra Wireless, Inc. TRIMBLE is a registered trademark of Trimble Navigation Limited. U-BLOX is a registered trademark of u-blox Holding AG. UNIX is a registered trademark of The Open Group.
Diversity Streaming Service Test Manual

2000 - 2013 SwissQual AG
Contents
1 Introduction .......................................................................................................................................... 1 Video Streaming Service ....................................................................................................................... 1 2 Configuring a Video over IP Streaming Test .................................................................................... 3 Overview ................................................................................................................................................ 3 QuickTime and RealPlayer Settings ...................................................................................................... 4 Configuring QuickTime ..................................................................................................................... 4 Configuring RealPlayer .................................................................................................................... 6 Creating a Streaming Test ..................................................................................................................... 9 Configuring a Video No Reference Test ........................................................................................ 12 Configuring a Video Full Reference Test............................................................................................. 15 Using the Auto Downsize Option in No Ref and Full Ref Tests ..........................................................17 Configuration - Timings ....................................................................................................................... 18 3 Viewing Test Results in NQDI .......................................................................................................... 19 Data Session Results .......................................................................................................................... 20 General Video Stream Information ...................................................................................................... 22 Video Stream Test Results .................................................................................................................. 22 Main Result Section ....................................................................................................................... 23 Graphical Presentation ................................................................................................................... 24 Video and Audio Content Results ........................................................................................................ 25 4 Video Full Reference Streams .......................................................................................................... 26 Available Video Full Reference Streams ............................................................................................. 26 Available Video Full Reference Streams for Down-Sized Play Out ....................................................27 Overview of VQuad Results ................................................................................................................ 27 QuickTime Player Results .............................................................................................................. 29 RealPlayer Results ......................................................................................................................... 32 5 Streaming From YouTube ................................................................................................................. 35 Source Signals ..................................................................................................................................... 35 YouTube Test Emulating PC ............................................................................................................... 35 YouTube Test Emulating a Mobile Phone ........................................................................................... 40 Uploading Video Material on YouTube ................................................................................................ 41 YouTube Test Result Presentation...................................................................................................... 42
Figures
Figure 1-1 Overview of Video Streaming test .................................................................................................... 2 Figure 2-1 Video Streaming - Layer model........................................................................................................ 3 Figure 2-2 QuickTime Audio settings ................................................................................................................ 4
Contents | CONFIDENTIAL MATERIALS ii

Figure 2-3 QuickTime Streaming settings ......................................................................................................... 5 Figure 2-4 QuickTime Advanced settings .......................................................................................................... 6 Figure 2-5 RealPlayer Content / clip cache settings ......................................................................................... 7 Figure 2-6 RealPlayer Hardware settings / Video Card compatibility ............................................................... 7 Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility .............................................................. 8 Figure 2-8 RealPlayer AutoUpdate setting ........................................................................................................ 8 Figure 2-9 RealPlayer Message Center Preferences ........................................................................................ 9 Figure 2-10 Video Streaming Job definition ................................................................................................. 10 Figure 2-11 Video Streaming Job definition ................................................................................................. 10 Figure 2-12 Video Streaming - Test definition ................................................................................................. 11 Figure 2-13 Select "Video No Reference" Streaming test type ....................................................................... 12 Figure 2-14 YouTube No Reference (Smartphone) test ................................................................................. 12 Figure 2-15 Configuration of a Video No Reference test ................................................................................ 13 Figure 2-16 Settings on the General tab of a Video Full Reference test ......................................................... 15 Figure 2-17 Schematic timing overview Full Reference ............................................................................... 17 Figure 2-18 shows the required timings for the video streaming test .............................................................. 18 Figure 3-1 Importing a measurement file into NQDI ........................................................................................ 19 Figure 3-2 Data selection in NQDI ................................................................................................................... 20 Figure 3-3 Session, Service and Test Layer in NQDI ..................................................................................... 20 Figure 3-4 Protocol information in NQDI ......................................................................................................... 21 Figure 3-5 Media information in NQDI ............................................................................................................. 21 Figure 3-6 Frame information in the media info tab-sheet .............................................................................. 22 Figure 3-7 Video Stream general information at the NQDI overview .............................................................. 22 Figure 3-8 Video Stream tab sheet in NQDI .................................................................................................... 23 Figure 3-9 Video Stream main results in NQDI ............................................................................................... 24 Figure 3-10 Graphical presentation of a Video Stream in NQDI ..................................................................... 24 Figure 3-11 Graphical presentation of a Video Stream with Freezing in NQDI ............................................... 25 Figure 3-12 Data analysis in NQDI .................................................................................................................. 25 Figure 5-1 Selection of Quality in YouTube ................................................................................................... 36 Figure 5-2 Principle of transcoding and re-sizing in YouTube......................................................................... 36 Figure 5-3 Selection of Video quality in YouTube tests ................................................................................. 37 Figure 5-4 Selection of Video quality in YouTube tests ................................................................................. 37 Figure 5-5 Restricted selection of Video Quality and Image Resolution in case of QVGA video clips ........... 38 Figure 5-6 Parameter definition for No Reference YouTube tests .................................................................. 40 Figure 5-7 YouTube test emulating a mobile phone ....................................................................................... 41
Tables
Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test .............. 13 Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test ............. 16 Table 4-1 QuickTime player results from the john clip in QCIF resolution for different coding settings ....... 29
Contents | CONFIDENTIAL MATERIALS iii

Table 4-2 QuickTime player results from the laugh clip in QCIF resolution for different coding settings ..... 29 Table 4-3 QuickTime player results from the solothurn clip in QCIF resolution for different coding settings 30 Table 4-4 QuickTime player results from the john clip in QVGA resolution for different coding settings ..... 30 Table 4-5 QuickTime player results from the laugh clip in QVGA resolution for different coding settings ... 31 Table 4-6 QuickTime player results from the solothurn clip in QVGA resolution for different coding settings ......................................................................................................................................................................... 31 Table 4-7 RealPlayer results from the john clip in QCIF resolution for different coding settings .................. 32 Table 4-8 RealPlayer results from the laugh clip in QCIF resolution for different coding settings................ 32 Table 4-9 RealPlayer results from the solothurn clip in QCIF resolution for different coding settings ......... 33 Table 4-10 RealPlayer results from the john clip in QVGA resolution for different coding settings .............. 33 Table 4-11 RealPlayer results from the laugh clip in QVGA resolution for different coding settings ............ 33 Table 4-12 RealPlayer results from the solothurn clip in QVGA resolution for different coding settings ...... 34
Contents | CONFIDENTIAL MATERIALS
iv

Introduction
The Diversity Streaming Service Test Manual describes how the SwissQual Diversity software evaluates the quality of video streaming services as well as how to configure streaming service tests, that is, Diversity Streaming PC tests. Note: As of Diversity 10.6, SwissQual renamed the former Job-Type Streaming PC to Video over IP that matches to the extended capabilities of this set of tests. Diversity 10.6 enables for first time the video streaming from YouTube using a FlashPlayer client. This emulates a PC user interface. Streaming from YouTube using a mobile phone ID allows the use of RealPlayer and QuickTime as in a phone as user interface.
Video Streaming Service

Streaming refers to the ability of an application to play synchronized media streams, such as audio and video streams, in a continuous manner while the streams are being transmitted to the client over a data network. The software client plays the incoming multimedia stream in real time as the data is received. Typical video streaming services include live and pre-recorded TV channels, informational content, such as traffic and weather, as well as surveillance and entertainment portals. Video streaming is already present in 2G and 2.5G networks but more and more used in 3G and 3.5G and in the future in 4G Packet Switched Services (PSS). Common video streaming scenarios are on-demand and live information delivery applications. Examples of the first scenario are music and on-demand news applications, while live delivery of radio and television transmissions is an example of the second category. Objective video streaming tests require a streaming server, a video client, and an objective algorithm for video quality assessment. SwissQual uses a Darwin Streaming Server to host and stream the video clips for a test and the QuickTime and RealPlayer software on the client computer to play the streams. The client uses the RTSP or HTTP protocol to send a streaming connection request to the Darwin server. After the connection is established, the video player starts to buffer, to decode the video and to display. The SwissQual analysis software starts the video capture, that is the grabbing of each individual frame of the decoded and displayed video and runs its evaluation.
Chapter 1 | Introduction CONFIDENTIAL MATERIALS

Starting and initializing video player
Time to connect Failed to connect, stream not found or connection time out
Connection to server and request video
Time to start buffering
Buffering video
Buffering time out Stream lost Video MOS Video MOS Video MOS Video MOS
Video quality evaluation Video quality evaluation Video quality evaluation Video quality evaluation
Time to display
Displaying
Video Server
Stopping playing or end of stream
Service completed
Tear down connection to server
Figure 1-1 Overview of Video Streaming test
The video becomes analysed in several ways. At first the entire stream is evaluated regarding its temporal behaviour. Here so-called Freezings (pausing display) are detected and reported as well as other parameters as frame rate and content based trigger points as time to first picture. In a lower analysis layer, short sequences of the video are analysed for compression and transmission artefacts leading to a quality prediction finally. In addition to the content based KPIs the IP information as well as the players trace become analysed for problems and status messages. In a full reference approach, the VQuad algorithm evaluates the difference between the output video sequence and the corresponding source video reference sequence on a psycho-visual scale to, predict the video quality. In a no reference approach, the VMon algorithm only evaluates the output video sequence. The reference video signal can be unknown. An additional package enables the video streaming from YouTube either emulating a mobile phone or a PC as user interface. In case of mobile phone emulation running RealPlayer, the protocol is RTP over UDP, while for PC emulation (using FlashPlayer) the protocol is HTTP over TCP. In general, HTTP over TCP is a save protocol and FlashPlayer buffers and pre-stores a significant amount of the video before and while playing. Here only severe transmission problems will lead to a temporal or full abort of the stream. Most useful results are the status information of the stream describing the availability and stability of the service. In contrast, the emulation of the phone enables streaming via RTP over UDP, here packets can get lost or distorted, along with a short buffering time, and quality problems become visible in the video as pausing displaying and destroyed image information.
Chapter 1 | Introduction CONFIDENTIAL MATERIALS

Configuring a Video over IP Streaming Test
This chapter describes how to configure Video No Reference and Video Full Reference Video over IP tests in Diversity TestManager as included in NQView. These tests are designed to evaluate a subscribers experience with a video streaming service.
Overview
To set up and to maintain a video stream for a specific amount of time, for example, 2 minutes, a Video over IP test requires a data connection. During the streaming process, the test performs a series of evaluation, once at the stream as such but also on short cut-outs of the grabbed video images. These measurements provide a signal quality profile for the duration of the streaming activity. The test uses a mobile phone or a data-card as a dial-up device and a software video player that is QuickTime, RealPlayer or FlashPlayer, on a computer to play back the video stream. This configuration enables full access to the entire IP layer, the players trace log as well as the entire stream and the media layer parameters. The test frame work, that is always identical and independent from the player and the underlying quality predictor, supports two objective quality predictors. The SwissQual VQuad algorithm objectively evaluates the quality of a full reference video capture whereas the VMon algorithm evaluates the quality of a no reference video capture. The tests capture the video from the display of a mobile phone. Video over IP tests use the following layers to provide a set of corresponding results: Data Session layer: Contains the dial-up and the IP layers Service test layer: Contains the media stream Media layer: Contains the content-based quality analysis DialUp Connect VQ Data Session Streaming Test VQ VQ VQ Connect VQ Streaming Test VQ VQ VQ
Figure 2-1 Video Streaming - Layer model
A Video over IP test analyses the data session for dial-up times, throughput on IP protocol information, and the success state of the dial up. The data session is a lower layer than the stream, one data session can containing more than streaming tests. Note: You can also use a permanent IP connection during a test instead of a dial-up device. The quality evaluation of the streaming service is based on connection and buffering times as well as the success state. On stream level also a global freezing analysis over the entire stream is performed. All these results are belonging to the video stream as a service test. A content analysis sub-divides the stream into short video sequences and provides detailed information about the perceived quality and the main signal degradations. Note: Starting with Release 11.1 of Diversity, the video stream itself is considered as test. The detailed analysis of the video clips is an underlying analysis that is still available as in prior Releases. Before you can create a Streaming PC test, you need to verify the QuickTime and RealPlayer settings. For the FlashPlayer there is no special setting required.
Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

QuickTime and RealPlayer Settings

This section describes the QuickTime player and RealPlayer settings that Diversity requires to produce comparable measurement results. Important: You need to reconfigure these settings whenever you update, repair, or reinstall QuickTime or RealPlayer. If you use Diversity in Optimizer mode, you also need to verify these settings before you start each measurement as some settings can change during a measurement.
Configuring QuickTime
This section describes how to configure QuickTime for use with Diversity.
To configure QuickTime for Diversity

1. Open the QuickTime Player and on the Edit menu, point to Preferences, and then click QuickTime Preferences. 2. On the Audio tab, specify the following audio settings. a In the Devices area, specify the following options and values: b c Select the Safe mode (waveOut only) option. In the Audio Output Device Latency (msec) box, enter 33.000.
In the Sound Out area, specify the following values: In the Rate list, click 48 kHz. In the Size list, click 16 bit. In the Channels list, click Mono.
In the Default Music Synthesizer list, click QuickTime Music Synthesizer.
Figure 2-2 QuickTime Audio settings

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS 4

3. On the Streaming tab, click Automatic in the Streaming Speed list and clear the following check boxes: Override speed for downloads Enable Instant-On
Figure 2-3 QuickTime Streaming settings
4. On the Advanced tab, click Automatic in the Transport Setup list, and then specify the following options: Clear the RTSP Proxy Server check box. Drag the Download Cache slider to 0. In the Video area, click Safe mode (GDI only).
Note: DirectX strongly affects the appearance of the video image, which in turn has a negative impact on the measurement results. Clear the Enable encoding using legacy codecs and Install QuickTime icon in system tray check boxes.

Figure 2-4 QuickTime Advanced settings
Configuring RealPlayer
This section describes how to configure RealPlayer for use with Diversity.
To configure RealPlayer for Diversity

1. In RealPlayer, click Preferences on the Tools menu. 2. Disable the temporary clip cache. a b In the Category tree, click Content. In the Cache settings area, click Settings, clear the Enable caching check box, and click OK.

Figure 2-5 RealPlayer Content / clip cache settings
3. Specify the video card compatibility. a b In the Category tree, click Hardware. In the Video Card Compatibility area, drag the slider to Most reliable, and then click OK.
Figure 2-6 RealPlayer Hardware settings / Video Card compatibility
4. Specify the playback sound quality. a In the Category tree, click Hardware.
7

b c
In the Sound Card Compatibility area, click Settings. Drag the Playback Quality slider to Best quality, and then click OK.
Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility
5. Disable the automatic update feature. a b In the Category tree, expand Automatic Services, and then click AutoUpdate. Clear the Automatically download and install important updates check box.
Figure 2-8 RealPlayer AutoUpdate setting
6. Disable the RealPlayer Message Center. a In the Category tree, expand Automatic Services, and then click Configure Message Center.
8

In the Message Center window, clear the check boxes, and then click OK.
Figure 2-9 RealPlayer Message Center Preferences
7. In the Preferences window click OK.
Creating a Streaming Test

A streaming service test requires either a permanent IP access or defined access points in the subscriber and provider configuration for dial-up connections.
To create a Video over IP test in Diversity NQView / Control Option

1. In the Control console of NQView, click Tasks and Jobs on the navigation panel. 2. Expand Job definitions, right-click Video over IP, and then click Add job.

Figure 2-10 Video Streaming Job definition
3. In the Job window, type a Name for the job, click Apply, and then click OK.
Figure 2-11 Video Streaming Job definition
4. Specify the settings for the test, click Apply, and then click OK.
10

Figure 2-12 Video Streaming - Test definition
Note: For more information, see the "Configuring a Video No Reference Test" and "Configuring a Video Full Reference Test" sections on pages 12 and 15 respectively. You can use the settings on the Video over IP tests tab to define one of the following test types: Video No Reference: Video test for unknown content, such as public streaming servers or live TV
Important: This test does not evaluate the audio signal. Video Full Reference: Video test that requires a known reference video signal. You can only use this test to evaluate video signals from a streaming server that you control.
Important: In case you have enabled he YouTube test option you can select up to four further tests YouTube Full Reference (emulate PC): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer. YouTube No Reference (emulate PC): Video test that streams a video signal from YouTube at a given URL. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer. YouTube Full Reference (emulate Phone): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers. YouTube No Reference (emulate Phone): Video test that streams a video signal from YouTube at a given URL. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers.
11

Configuring a Video No Reference Test

This section describes the settings that you can configure for a Video No Reference test on the Streaming PC tests tab for a job. The Video No Reference test streams a video from any server you define, for example, from a YouTube server onto an Android smartphone, and enables the VMon algorithm to analyse the video quality in short sections. However, since this test uses unknown content, VMon cannot evaluate the audio component of the stream. On a global view, this test provides all temporal information on a per-stream basis. The entire stream is evaluated at once, for example, freezing, Jerkiness and frame-rate.
Figure 2-13 Select "Video No Reference" Streaming test type
Figure 2-14 YouTube No Reference (Smartphone) test
For the URL of a YouTube video stream, you can use the youtube.com or youtu.be formats. Important: Use the Video No Reference test type for unknown content, such as public streaming servers. Use the Video Full Reference test type when you evaluate a known reference video stream that originates on a server that you control. VMon does not evaluate each frame in the spatial domain during image analysis and assumes that the skipped frames contain similar image artifacts as the evaluated frames. Although Video No Reference tests can use video streams in VGA and WVGA resolution or above, the recording and evaluation of the video is limited by Diversity processing power restrictions. Diversity records and calculates session statistics, session-related KPIs as well as the IP throughput and Media Info of each measurement (see also separate section for Auto Downsize).
Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS 12

Video over IP Test Settings
Figure 2-15 Configuration of a Video No Reference test
The following table describes the settings that you can configure on the general configuration window for a Video No Reference streaming test.
Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test
Setting
Stream duration
Description
Specifies the duration in seconds of the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED. Specifies the length of the sample in seconds to capture for the quality analysis. For best results, enter a value in the range of 6 to 15 seconds. The VMon algorithm uses the capture time to calculate a set of integrative quality values, for example, visual quality and blockiness, from the video capture sample. VMon seamlessly analyzes each portion of the complete capture.
Capture time
Connection timeout Stream Lost timeout URL Stream
Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming test is set FAILED / Connection TimeOut in this case. Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming test is set DROPPED / TimeOut in this case. Enter the URL for a network video stream. This option is only for Video Full Reference tests.
13

Setting
Player
Description
Specifies which video player to use for the test, that is, QuickTime or RealPlayer. Important: Only QuickTime 7.1 and higher supports H.264 video streams.
Image Resolution
Select Auto if you want the video stream to play at the native resolution, which is defined in the session description of the stream. If the player cannot interpret this description, the video plays at a resolution of 176x144 pixels, that is, QCIF. If you want to force the player to use a specific resolution, click a resolution in the list, for example, QCIF(176x144) or QVGA(320x240). Note: If the forced size does not match the native image resolution of the stream, the internal re-scaling functions of the player are applied to achieve the forced size, which can result in artifacts. For streaming videos in resolution VGA and WVGA or even above, please use the option Auto Downsize (see separate section).
Save A/V file
Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files. The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time. Note: For best results in Diversity Q, click None.
Dynamic URL Run Speech Evaluation Video Recording Quality
Keyword sequence for navigation on a pre-entered HTTP site. See separate section. This option is only for Video Full Reference tests. The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video.
Dynamic URL Text Window

In the Dynamic URL box, enter a series of keywords for dynamic link navigation. Add each new keyword to a new line. Note: You can use an asterisk (*) as a wildcard to replace an undefined number of characters in a keyword. The test searches the Web site that you specified in the URL box on the General tab, see Table 2-1 on page 13, for the first keyword in the list. If the keyword is found, the test opens page that contains the keyword and searches for the second keyword, and so on. The test only searches for the first occurrence of a keyword on a page. Important: NQView cannot retrieve a list of keywords from an HTTP page. Instead, you need to use external HTTP source interpreter to recognize the keywords, which you can then copy into the Dynamic URL box. Keywords are often hidden behind images.
14

Configuring a Video Full Reference Test

This section describes the settings that you can configure for a Video Full Reference test on the Video over IP tests tab for a job. The stream is also evaluated at once regarding temporal characteristics such as freezing, Jerkiness and frame-rate. For a detailed video analysis of short sections of the video stream, the Video Full Reference test uses the VQuad algorithm to evaluate the quality of a video stream. In addition SQuad-LQ can be used to evaluate the quality of the audio voice stream. The voice signal is evaluated in wideband mode, since video streaming is usually not restricted to telephony bandwidth. A Video Full Reference test compares the received video with the original high quality reference video clip. Differences caused by compression and transmission problems are calculated and weighted. A Video Full Reference test requires a video stream that is on a server, which you control. This stream is built on a reference video you have. For a description on how to set up the server, see the Installation RedHat Mediaserver.pdf document. For information on the available video streams, see the "Video Full Reference Streams" chapter on page 26. There is a set of test streams prepared and stored on the server. The streams consist on a repetition of a video clip separated by short blue sequences. The provided streams cover a very wide range of formats and bit-rates. In principle the entire range from 40kbps up to 5500kbps are covered. A blue sequence separates each content clip in the stream so that you can recognize the start and the end of a clip. These are start and stop markers for the underlying VQuad analysis only. Important: Video Full Reference clips in VGA or WVGA resolution are only supported with the Auto Downsize option as described in a separate section.
Figure 2-16 Settings on the General tab of a Video Full Reference test
15

Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test
Setting
Stream duration
Description
Specifies the duration in seconds for the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED. Automatically displays the duration in seconds for the streaming sequence. The capture time is included in the video stream. The capture starts with a blue synchronization sequence and ends with the detection of the next blue marker. These sequences are between 0.5 and 1.0 seconds in length and use a sequence marker in the bottom lines to identify the next streaming sequence. Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming session is set FAILED / Connection TimeOut in this case. Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming session is set DROPPED / TimeOut in this case. Address of the streaming server Select the reference video sample that you want to use for the test. If you select a QCIF sample, a warning dialog box opens to inform you that this format has been deprecated. Enter the URL that points to the server as well as the reference video sample that you want to stream. Displays the quality of the reference sample The list only contains the resolutions that VQuad supports. Required if the display resolution is greater than or equal to VGA/WVGA Specifies which video player to use for the test, that is, QuickTime or RealPlayer. Important: Only QuickTime 7.1 and higher supports H.264 video streams.
Capture time
Connection timeout
Stream lost timeout
Server Stream
URL Video quality Display resolution Run video MOS evaluation Player
Save audio/visual file
Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files. The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time. Note: For best results in Diversity Q, click None.
Run speech evaluation
Evaluates the voice signal in a video stream with SQuad-LQ in wideband mode. The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video.
16
Video recording quality

The concept of the full reference video evaluation is slightly different from No Reference using VMon, where only the received video stream is analysed for artefacts and degradations. In a Full Reference test, the received video is compared to the high quality original video sample. Due to the comparison to the reference signal there is synchronization between the two signals required. For that reason the video sequence is interlaced with short blue sequences for synch-up. In addition, the evaluation of the video sequence requires a complete capturing and cant be done in real-time as for VMon in the Video No Reference test. Since, the full reference video evaluation is very resource consuming; a CPU with high performance is highly recommended for evaluation of larger resolutions.
Capture Time Capture Time
Blue synch
Accumulating video frames Start VQuad
Accumulating video frames
Report VQuad
Report VQuad
Fast evaluation (low resolution, low frame rate, high performance CPU)
Figure 2-17 Schematic timing overview Full Reference
The computation time depends on the CPU performance but also on image resolution and frame rate (the more frames and the bigger the frames are, the higher is the computation effort). The computation can take between a few seconds until several 10s for higher resolutions and frame rates. The results will not immediately available after the recording is finished rather calculated sequentially one after another.
Using the Auto Downsize Option in No Ref and Full Ref Tests
The real-time display and capture of videos on a Diversity PCM requires a resolution of 240p (QVGA) or 360p (HVGA). Due to CPU overload, a larger resolution video cannot be played smoothly during screen capture. The limiting process in analysing native size videos, that is, VGA and WVGA (480p), is the capture of the bitmap frames of the video. To address this limitation, Diversity offers the Auto Downsize option to display a high resolution video stream at a smaller screen resolution but at the same bit-rate. If you use this option, a temporal analysis for freezing and jerkiness is performed based on the reduced resolution. Freezing and Jerkiness can be measured with the same confidence on a smaller display size. If you only need a temporal analysis, you can combine QVGA streams with the Auto Downsize option. This combination saves CPU power by playing the QVGA stream in a 160x120 QQVGA format only. Important: The minimum requirement for the fluid display of VGA and WVGA, even in downsized Player windows, is a dedicated PCM with a dual core CPU, that is, the PCM cannot run other measurements in parallel. A fluid display cannot be guaranteed for higher resolutions, such as SDTV and HD720p.
17

Configuration - Timings
Figure 2-18 shows the required timings for the video streaming test
18

Viewing Test Results in NQDI
This chapter describes how to view the results of a video streaming test in the NQDI post-processing software.
To import and view Streaming PC test data in NQDI

1. In NQDI, click the Data Management tab. 2. Click the Add files to the import list icon and browse to the folder that contains the measurement file. 3. Click Add and click Close. 4. On the toolbar, click the Start import process icon.
Figure 3-1 Importing a measurement file into NQDI
5. Select the data that you want to analyze. a b Click the Data Selection tab and click the Refresh icon on the Selection Criterion toolbar.
In the Test List of Current Section area, double-click a test block to view the stream and session results.
Note: For more information, see the following section.
Chapter 3 | Viewing Test Results in NQDI CONFIDENTIAL MATERIALS
19

Figure 3-2 Data selection in NQDI
Data Session Results

You can analyse each individual clip on the Analysis tab as well as view the Data Session, Service, and Test layers. The red elements in Figure 3-3 represent the clip, or rather, block of tests that you selected on the Data Selection tab, Stream-1 represents a full reference clip evaluation, and Stream-2 represents no reference clip evaluations.
Figure 3-3 Session, Service and Test Layer in NQDI
The figure also shows the related radio and transport layer information for a radio connection or, more specifically, HSDPA over UMTS 2100. On the Protocol info tab of the Analysis tab, you can view the IP log and a graphical representation of the IP traffic. Both tabs are related to the entire data session and show information from the IP layer or the decoded packets.
20

Figure 3-4 Protocol information in NQDI
The Media info tab of the Analysis tab contains partially decoded MPEG4 part 2, H.263, and H.264 elementary streams. Note: This media information, including the video codec in NQDI, is only decoded for RTSP streams.
Figure 3-5 Media information in NQDI
In Figure 3-5 and Figure 3-6, each bar of the top graph represents an individual frame, each colour represents a frame type, and the height of each bar represents the frame size in bytes. See an enlarged view in. The lower graph in Figure 3-5 is a copy of the IP-throughput graph and is included for direct comparison.
21 CONFIDENTIAL MATERIALS
Chapter 3 | Viewing Test Results in NQDI

Figure 3-6 Frame information in the media info tab-sheet
General Video Stream Information

The Video over IP test is giving test results for the video stream as such. There is a Data Session established and within this data session one or more streaming tests can be conducted. The data session can be a permanent data link or a dial-up connection especially for the purpose of video streaming. One layer below the data session the video stream (as the test layer) is presented. In this tab sheet only little general information is shown (Figure 3-7). In the area below the general information is a tab sheet Video Status Trace tab that provides an overview of protocol-related negotiations and player information.
Figure 3-7 Video Stream general information at the NQDI overview
Video Stream Test Results

The main information about the video stream service test can be obtained from the Video tab sheet. There is an area with information related to the entire stream.
Chapter 3 | Viewing Test Results in NQDI CONFIDENTIAL MATERIALS 22

Figure 3-8 Video Stream tab sheet in NQDI
In the upper area general information is given characterizing the video stream at once. The graphical display shows all evaluated frames of the whole video stream in upper graph. In case there are quality predictions with VMon or VQuad, these MOS scores are presented in the lower graph. Note: The quality prediction of the video by VMon or VQuad is only available when the check box Evaluate Video MOS is enabled. Otherwise the results remain restricted to the global per-stream results as Stream Status, Time to First Picture, Freezing and Frame Rate.
Main Result Section

In the upper main section are general indicators as time to first picture and frame rate but also information about the protocol used, the URL and the player.
23

Figure 3-9 Video Stream main results in NQDI
Very important is the Status information for the video stream. Ideally it is COMPLETED, that means the video was displayed completely for the desired duration. Another status is FAILED, where the server cannot be reached or the video never starts to play. DROPPED is given for videos who getting lost and stay streamed incompletely. DROPPED is also given in case a video stops (freezes) for more than a pre-defined time (see parameter stream lost time-out in the configuration sections). Main measured parameter is the Freezing parameter. It gives indication about detected frozen sections in the stream. This is given as accumulated duration as well as a percentage value relatively to the stream duration. The stream evaluation regarding status, Freezing and Frame Rate is independent from a VMon or VQuad video quality test in the sub-layer of video testing. In case the Video over IP test is conducted with VMon (No Reference) or VQuad (Full Reference) the quality predictions are also shown in the Video tab sheet as average MOS as well as its maximum reached in the stream and the deviation of the individual values.
Graphical Presentation
The graphical presentation in the Video tab sheet presents the entire stream. The example below shows a stream as used for Full Reference tests with VQuad (Figure 3-10). The green bars are showing the interframe difference of the regularly played frames. High bars (peaks) indicate a high inter-frame difference as for example, a scene cut. The blue areas are a speciality for Full Reference tests. These are the short blue sequences used for synchronization and separation of the individual video samples to be evaluated by VQuad. Blue sequences are individually shown as blue-shadowed area. Black sequences (or more accurate: unicolor sequences) are black (grey-) shadowed. Black sequences are sometimes introduced in videos with intention to separate scenes or for scene fading. They are the same as the blue sequences not considered in the MOS prediction rather counted and reported separately.
Figure 3-10 Graphical presentation of a Video Stream in NQDI
One of the most interesting degradation in video streaming and almost the only one in safe protocols as TCP is Freezing. That means the video stops playing for a while and starts (with or without skipping the content in between). Freezings (means displaying a frame / image for more than 300ms) are visually indicated by a red-shadowed area in the graphic (Figure 3-11). Freezing is reported as an accumulated number as well as a percentage relatively to the stream duration.
24

Figure 3-11 Graphical presentation of a Video Stream with Freezing in NQDI
Video and Audio Content Results

For a more detailed analysis you can view the video and audio measurement results of the individual sections of the video in the Visual Quality and Listening Quality tabs in the Analysis window. For example, Figure 3-12 shows the VMon results for a clip.
Figure 3-12 Data analysis in NQDI
For a description and interpretation of the individual results, see the following documents: SQuad-LQ: Manual - SQuad Voice Test Result Description.pdf VMon and VQuad: Manual - VMon and VQuad Results Description.pdf
In a Video Full Reference test, the audio signal component of the AV signal is evaluated by SQuad-LQ with respect to the clean reference audio signal. In a Video No Reference test, the audio signal is not analysed regarding quality since the audio signal cannot be evaluated with respect to a pre-defined audio signal.
25

Video Full Reference Streams
This chapter describes the pre-encoded video streams that are available for Video Full Reference tests. The available streams consist of 3 different video sequences that have been encoded with different settings. Each stream is 5 minutes long and contains multiple repetitions of an 8 second long video clip, which are separated by one second blue sequences.
Available Video Full Reference Streams

Diversity has 8 sets of video streams for use with Video Full Reference tests. The audio tracks of the streams do match the content of the video component. Instead, each audio track is a highly qualitative wideband reference speech clip in American English that has been encoded with different settings. In the following lists, each asterisk (*) is a placeholder for a short name that describes the content of the clip. Furthermore, the name john refers to a typical head-and-shoulder image, Laugh refers to a clip that has moderate movement, and solothurn refers to a clip that contains the most movement. The bit rates are approximate average content bit rates. However, the peak bit rates can be much higher. Due to the protocol overhead, the average bandwidth of the clips is higher than the sum of the codec bit rates. The size of the IP packets depends strongly on the bit rate and is usually in the range of 300 to 1400 bytes. The protocol overhead is around 40 bytes for each packet, which results in an overhead of 3 to 13 %. Set 1: 40kbps MPEG4/H.264 video, 8.333 / 12.5 fps, 8kbps AMR audio o o o 5min_*_QCIF_8.333fps_40k-mpg4_8k-amr_08.3gp 5min_*_QCIF_12.5fps_40k-mpg4_8k-amr_08.3gp 5min_*_QCIF_12.5fps_40k-h264_8k-amr_08.3gp
Set 2: 72kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio o o 5min_*_QCIF_12.5fps_72k-mpg4_24k-aac_08.3gp 5min_*_QCIF_12.5fps_72k-h264_24k-aac_08.3gp
Set 3: 112kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio o o o o 5min_*_QCIF_12.5fps_112k-mpg4_24k-aac_08.3gp 5min_*_QCIF_12.5fps_112k-h264_24k-aac_08.3gp 5min_*_QVGA_12.5fps_112k-mpg4_24k-aac_08.3gp 5min_*_QVGA_12.5fps_112k-h264_24k-aac_08.3gp
Set 4: 172kbps MPEG4/H.264 video, 25 fps, 32kbps AAC audio o o o o 5min_*_QCIF_25fps_172k-mpg4_32k-aac_08.3gp 5min_*_QCIF_25fps_172k-h264_32k-aac_08.3gp 5min_*_QVGA_25fps_172k-mpg4_32k-aac_08.3gp 5min_*_QVGA_25fps_172k-h264_32k-aac_08.3gp
Set 5: 320kbps MPEG4/H.264 video, 25 fps, 48kbps AAC audio o o 5min_*_QVGA_25fps_320k-mpg4_48k-aac_08.3gp 5min_*_QVGA_25fps_320k-h264_48k-aac_08.3gp
Set 6: 700kbps H.264 video, 25 fps, 64kbps AAC audio o 5min_*_QVGA_25fps_700k-h264_64k-aac_08.3gp
Set 7: 1200kbps H.264 video, 25 fps, 128kbps AAC audio o 5min_*_QVGA_25fps_1200k-h264_128k-aac_08.3gp

Chapter 4 | Video Full Reference Streams

Available Video Full Reference Streams for Down-Sized Play Out

Video streams with higher resolution than the multi-media formats are supported. These are clips in VGA (640x480) and WVGA (852x480) corresponding to NTSC standard TV in aspect ratio 4:3 and 16:9. The display size is reduced by a factor 4:1 to reduce the resource consumption during video grabbing and analysis. These clips are based on a different content named station and cover a bit-rate from 700kbs to 5500kbps. Note: The reception and decoding of video streams with higher bitrates requires a considerable amount of resources. Video streams of above 1200kbps quality might not be played out fluently; however, the main analysis results as Status, Time-to-First-Picture, Freezing and Frame rate are made available and are valid. Set 8: High resolution video, VGA, 25 fps, AAC audio o o o 5min_Station_VGA_25fps_700k-h264_64k-aac_08.3gp 5min_Station_VGA_25fps_1200k-h264_128k-aac_08.3gp 5min_Station_VGA_25fps_3100k-h264_256k-aac_08.3gp
Set 9: High resolution video, WVGA, 25 fps, AAC audio o o o o o o o o 5min_Station_WVGA_25fps_700k-h264_64k-aac_08.3gp 5min_Station_WVGA_25fps_700k-MPG4_64k-aac_08.3gp 5min_Station_WVGA_25fps_1200k-h264_128k-aac_08.3gp 5min_Station_WVGA_25fps_1200k-MPG4_128k-aac_08.3gp 5min_Station_WVGA_25fps_3100k-h264_256k-aac_08.3gp 5min_Station_WVGA_25fps_3100k-MPG4_256k-aac_08.3gp 5min_Station_WVGA_25fps_5500k-h264_256k-aac_08.3gp 5min_Station_WVGA_25fps_5500k-MPG4_256k-aac_08.3gp
Set 10: Clips available in different resolutions o o o o Station 4:3 QVGA (320, 700, 1200 kbps, H.264 and MPEG4) Station 4:3 HVGA (320, 700, 1200 kbps, H.264 and MPEG4) Station 16:9 HVGA (320, 700, 1200 kbps, H.264 and MPEG4) john / laugh / solothurn 4:3 VGA (320, 700, 1200 kbps, MPEG4)
Overview of VQuad Results

This section provides an overview of the QuickTime Player and RealPlayer Mean Opinion Scores (MOS) values and measurement results. The following types of artefacts result in a reduction of the VQuad and SQuad MOS values: Encoding artefacts in the video full reference stream on the server Transmission artefacts that the stream produces
The tables in this section present VQuad results for lossless transmission. These values represent the expectation that one has for the different individual results in perfect transmission conditions. The tests were conducted on a Diversity system and involved streaming each video clip 10 to 20 times through a LAN connection. The tables also contain the 90th percentile values of the VQuad and SQuad results for each clip as well as the 10th percentile values of the individual degradations that were calculated by VQuad. The MOS values have an uncertainty of approximately +/- 0.05 MOS, while the other values have an uncertainty of approximately +/- 5 %.
Chapter 4 | Video Full Reference Streams CONFIDENTIAL MATERIALS 27

Important: To obtain comparable results, you need to use the same QuickTime player and RealPlayer settings as described in "QuickTime and RealPlayer Settings" on page 4.
Chapter 4 | Video Full Reference Streams CONFIDENTIAL MATERIALS
28

QuickTime Player Results

Video Clip john, Resolution QCIF, Player QuickTime
Table 4-1 QuickTime player results from the john clip in QCIF resolution for different coding settings
Encoding Settings
VQuad Blockiness Blurring Jerkiness SQuad MOS [%]

81 80 2 62 3 46 4 34 5
[%]
25 29 27 17 12 13 7 11 6
[%]
9 3 3 3 2 3 2 0 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 40kbps H.264 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 72kbps H.264 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC
2.12 2.03 2.91 2.98 3.75 3.57 4.06 3.85 4.19
Video Clip laugh, Resolution QCIF, Player QuickTime

Table 4-2 QuickTime player results from the laugh clip in QCIF resolution for different coding settings
Encoding Settings

88 89 8 82 15 66 12 53 10
[%]
24 28 33 20 17 13 10 12 7
[%]
13 5 5 5 4 5 5 0 0
MOS
1.74 1.69 2.25 2.27 3.06 2.92 3.55 3.3 3.97
29

Video Clip solothurn, Resolution QCIF, Player QuickTime

Table 4-3 QuickTime player results from the solothurn clip in QCIF resolution for different coding settings
Encoding Settings

86 88 10 82 9 59 9 58 7
[%]
28 32 33 25 19 18 10 18 10
[%]
15 6 6 6 6 6 6 0 0
MOS
1.67 1.66 2.26 2.14 3.08 2.86 3.60 3.05 3.92
Video Clip john, Resolution QVGA, Player QuickTime

Table 4-4 QuickTime player results from the john clip in QVGA resolution for different coding settings
Encoding Settings

65 0 52 0 37 5 10 5
[%]
28 21 27 17 15 7 3 1
[%]
3 2 0 0 0 0 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC 320kbps H.264 video, 25 fps, 48kbps AAC 700kbps H.264 video, 25 fps, 64kbps AAC 1200kbps H.264 video, 25 fps, 128kbps AAC
2.13 2.84 2.40 3.25 3.28 3.83 4.11 4.24
30

Video Clip laugh, Resolution QVGA, Player QuickTime

Table 4-5 QuickTime player results from the laugh clip in QVGA resolution for different coding settings
Encoding Settings

84 6 82 5 61 9 9 7
[%]
40 34 37 29 20 13 3 2
[%]
5 5 0 0 0 0 0 0
MOS
1.49 1.89 1.63 2.22 2.50 3.16 3.92 4.09
Video Clip solothurn, Resolution QVGA, Player QuickTime

Table 4-6 QuickTime player results from the solothurn clip in QVGA resolution for different coding settings
Encoding Settings
VQuad MOS
Blockines s [%]
Blurring Jerkiness SQuad [%] [%] MOS
1.38 1.76 1.47 1.94 2.01 2.85 3.76 4.03
86 6 82 7 76 11 10 7
42 30 41 32 27 16 8 5
6 6 0 0 0 1 1 2
31

RealPlayer Results
Video Clip john, Resolution QCIF, Player RealPlayer
Table 4-7 RealPlayer results from the john clip in QCIF resolution for different coding settings
Encoding Settings

3 0 0 0 0
[%]
29 33 18 11 9
[%]
8 3 3 2 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC
2.85 2.81 3.60 3.94 4.08
Video Clip laugh, Resolution QCIF, Player RealPlayer

Table 4-8 RealPlayer results from the laugh clip in QCIF resolution for different coding settings
Encoding Settings

6 5 6 4 6
[%]
29 34 21 14 12
[%]
12 4 5 5 0
MOS
2.36 2.34 3.02 3.53 3.74
32

Video Clip solothurn, Resolution QCIF, Player RealPlayer

Table 4-9 RealPlayer results from the solothurn clip in QCIF resolution for different coding settings
Encoding Settings

0 0 0 0 1
[%]
31 37 26 17 16
[%]
14 6 6 6 0
MOS
2.33 2.36 2.96 3.48 3.67
Video Clip john, Resolution QVGA, Player RealPlayer

Table 4-10 RealPlayer results from the john clip in QVGA resolution for different coding settings
Encoding Settings

0 0 0
[%]
29 28 14
[%]
2 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC
2.64 2.82 3.55
Video Clip laugh, Resolution QVGA, Player RealPlayer

Table 4-11 RealPlayer results from the laugh clip in QVGA resolution for different coding settings
Encoding Settings

2 1 6
[%]
40 36 18
[%]
4 0 0
MOS
1.92 2.14 3.02
33

Video Clip solothurn, Resolution QVGA, Player RealPlayer

Table 4-12 RealPlayer results from the solothurn clip in QVGA resolution for different coding settings
Encoding Settings

1 1 3
[%]
39 37 22
[%]
6 0 0
MOS
1.75 1.88 2.58
34

Streaming From YouTube
As of Diversity Release 10.6, a YouTube streaming option is also available. This option is an extension of the regular Diversity video streaming test; however, unlike the regular test, the video streams from the YouTube.com server and not from a video server that you control, for example, a Helix or Darwin server. The design and definition of a YouTube test is almost identical to the regular video streaming approach since the technical process of streaming, capturing and evaluation is essentially the same for both tests. One difference between the tests is that the YouTube video streams are not on a single server or restricted to a single URL. YouTube provides uploaded video content in different resolutions and bitrates for different user clients and qualities. When you upload a video clip to the site, YouTube transcodes the clip automatically, an action that becomes part of the processing chain for SwissQual to measure. The dynamic nature of the device and user dependent quality of the uploaded clips results in Diversity YouTube tests that are close to the real user experience. Along with Diversity Release 12.1 the YouTube test control has changed slightly to come closer to the end users abilities to control their YouTube experience. It is mainly the separate adjustment of requested video resolution and displaying size. Nothing has changed from the technical point of streaming, capturing and evaluation. The YouTube test is an application of the SwissQual IP video streaming concept in that the KPIs and the results that VQuad and VMon produce are identical to the regular IP streaming tests.
Source Signals
Firstly, SwissQual provides a set of video clips on YouTube that have the resolutions and qualities of video content that people typically access through mobile phones or data devices. More specifically, the contents of the clips are the same as the clips that Diversity uses for the regular Video over IP tests on Darwin and Helix media servers, that is, John, Laugh and Solothurn. These clips contain low motion head and shoulder, median motion, and high motion content. Each of these contents is available with an aspect ratio of 4:3 in two small resolutions, that is, QVGA (320x240, corresponds to YouTube 240p), and ~HVGA (480x360, corresponds to YouTube 360p). Along with Diversity Release 11.1 SwissQual introduced a clip Station that represents a good average video content and should replace the traditional videos only available in smaller formats. This clip is available in HD 720p in 4:3 and 16:9 aspect ratios. SwissQual provides all video clips in high quality without visible encoding distortions for upload to the YouTube Web site for customers supporting their own YouTube account.
YouTube Test Emulating PC

If YouTube recognizes that the user client is a PC, which is visible by the operating system information that is exchanged during the HTTP setup, YouTube redirects the request to the regular YouTube Web site. YouTube converts the video clip to a special format and assigns the clip an HTTP URL. Diversity YouTube tests emulate a PC to access this type of URL and video format. The video is displayed and captured by a completely controlled Flash Player implementation. As a result, you can only select the Flash Player for the YouTube No Reference emulating PC and YouTube Full Reference emulating PC. The video player here is always Flash player.
Control Parameter Video Quality

A new control parameter in the YouTube test control is Video quality. This value corresponds to the parameter Quality of YouTube in an HTML browser and is transmitted to the server when the video clip is requested. Based on this value, the delivered video quality is determined (Figure 5-1).
Chapter 5 | Streaming From YouTube CONFIDENTIAL MATERIALS
35

Figure 5-1 Selection of Quality in YouTube
When a video clip was uploaded in HD 720p for example, this clip can be requested in all supported Video Qualities of YouTube those are 720p, 480p, 360p and 240p. YouTube transcodes the uploaded video clip into those resolutions, store and provide them on request. The chosen quality determines mainly the resolution but the compression as well. It means, when a clip in 240p is requested, YouTube provides the content in this resolution independent from a higher resolution uploaded originally. It means, a video clip that is uploaded in HD 720p can be requested in 720p but also in 480p, in 360p or even 240p (see principle in Figure 5-2).
YouTube
Video upload (e.g.HD720)
User client
720p 480p 360p 240p
Downstream (e.g.360)
Transcoding
Player
Display
Figure 5-2 Principle of transcoding and re-sizing in YouTube
The requested video quality provides re-sized and re-compressed samples. In fact, the compression is made to achieve a certain target bitrate that is linked to the requested quality, e.g. a 240p clip is usually streamed at ~270kbps, while a clip streamed at HD720p takes ~1000kbps. This control parameter Quality is now a parameter that can be set in the test definition of YouTube tests in Diversity as well (Figure 5-3).
36

Figure 5-3 Selection of Video quality in YouTube tests
Control Parameter Image Resolution

The Image resolution only defines the display size of the player. It can be chosen between QVGA (320x240) ~HVGA (480x360) and VGA (640x480). This emulates the control of YouTube as done by a user by making the display area of the player larger or smaller. While the Video Quality value determines the compression quality and therefore the size of the bit-stream, the Image size just defines the video player output window. For CPU load reasons, the largest supported Image resolution is 640x480 (VGA), an HD Image resolution is not supported.
Figure 5-4 Selection of Video quality in YouTube tests

Chapter 5 | Streaming From YouTube

The parameter Image resolution only defines the display size of the player, it is independent from the requested Video quality (resolution). Normally, the display size to be chosen corresponds to the requested resolution, it means, a clip requested in 480 is displayed at 640x480 too. However, in principle each streamed Video quality can be displayed in at any display size. For example, a stream delivered at 240 (QVGA) can be displayed at 640x480 (VGA) but it will look somewhat blurry due to the up-sizing. Vice versa, a clip streamed at 480 can be displayed at 320x240 (QVGA) and usually looks very sharp here. This up- or downsizing is required in some cases (see Note below). Note: To compare video quality of different requested Video qualities, this comparison has to be done by using the same image resolution. It means for comparing a 240p stream and a 480p stream, the same image display size have to be chosen. For those inter-resolution comparisons and tests, a display size of 480x360 is recommended in case of 4:3 aspect ratio, or 640x360 for 16:9 aspect ratio. This should be used independent from the requested Video quality. Those are a good compromise between resolution and computation effort. The selectable Image resolutions depend from the chosen stream and the selected Video quality too. Not for all video clips all resolutions are supported.
Supported video clips for Full Reference Tests

The new clip Station is uploaded in HD quality and can be streamed in any Video Quality from HD720 down to 240 when the aspect ratio is 4:3. For this clip the full range of Video Quality and Image Resolution is supported (see Figure 5-3 and Figure 5-4). This clip is provided in different length from 60s up to 300s. This pre-defined length can be used to control the test length in case the stream should be streamed completely until its end. The chosen Stream duration should then be set to a value that exceeds the stream length considerably. The traditional video contents (John, Laugh, Solothurn) those are uploaded to YouTube in QVGA (320x240) and HVGA (480x360) can be requested and displayed in their native (uploaded) resolution only. It means the QVGA clip can be requested in Video quality 240p and displayed at 320x240 pixels only. The same for HVGA that is restricted to be streamed in 360p and displayed at 480x360 only.
Figure 5-5 Restricted selection of Video Quality and Image Resolution in case of QVGA video clips
38

For demonstration purposes only, video examples are stored in a SwissQual test account on YouTube. These examples are stored in semi-private mode and cannot be found by searching YouTube. For the station clip it is e.g. http://www.youtu.be/4oyLtpXmajw For the older contents the following clips the URLs are as follow: http://www.youtube.com/watch?v=0PaGUwhwK-0 http://www.youtube.com/watch?v=kxqvGy0-d_c http://www.youtube.com/watch?v=V-A_UfaC2gg http://www.youtube.com/watch?v=UmBNNZXUNqA http://www.youtube.com/watch?v=5HbMtM1CGVA http://www.youtube.com/watch?v=teNsDytzN1M // John QVGA (240p in 4:3) // John HVGA (360p in 4:3) // Laugh QVGA (240p in 4:3) // Laugh HVGA (360p in 4:3) // Solothurn QVGA (240p in 4:3) // Solothurn HVGA (360p in 4:3) // Station 720p in 4:3 in 120s
The full set of clips and URLs are installed along with TestManager setup and can be selected directly in the GUI.
Note: For Streams uploaded at QVGA or HVGA only one Video quality can be chosen that corresponds to the native resolution of the uploaded stream. The selectable Display resolutions are also restricted to the native size.
Note: The Station clip is also available in aspect ratio 16:9. Here not all Video Qualities and Image resolutions are supported. The Station clip in 16:9 can only be requested in HD720 and 360 and displayed only using 360 lines of pixels. Important: These links are only for YouTube tests that emulate PCs with a Flash Player and not for YouTube tests that emulate mobile phones.
YouTube With No Reference Tests

YouTube No Reference tests uses the same settings as the Full Reference YouTube tests as explained above, except the support to get the URL. In Full Reference tests, the stream names, the URL and the characteristics of the video clip are stored in the TestManager database. For No Reference tests, no pre-defined video clips are used an thus the URL to the desired video clip has to be typed directly in the corresponding edit field.
39

Figure 5-6 Parameter definition for No Reference YouTube tests
Since, the video clip is unknown to the TestManager, additional the aspect ratio of this clip has to be selected. At best the aspect ratio is observed by a prior watching the video at the YouTube site in a common internet browser. All other control parameters are the same as for the Full Reference YouTube test case.
Run Video MOS Evaluation

Along with Release 12.1 of Diversity there is a new check box available that allows to enable or to disable the video MOS evaluation. In case the parameter is disabled, only the freezing analysis over the entire stream is conducted (see Note on page 23). In case of YouTube the analysis of Freezing in a clip gives a good impression about problems while transmitting the video, while the Video MOS scores gives in addition information about the compression quality and artefacts.
YouTube Test Emulating a Mobile Phone

For a YouTube test that emulates a mobile phone user client by including the operating system information during the HTTP setup, the request is directed to a special YouTube site that is tailored to typical mobile phone display resolutions. You can view a similar site at: http://www.m.youtube.com. Important: Depending on rules set for your country, the m.youtube site might not be reachable via LAN provided by your local ISP. Try to use a mobile dial-up connection instead. The YouTube Test emulating PC uses HTTP protocol and the Flash Player to access and play the Flash
Chapter 5 | Streaming From YouTube CONFIDENTIAL MATERIALS 40

video clips on the YouTube site. The mobile phone emulation tests uses RTSP protocol and RealPlayer or QuickTime to access and play the 3GP files on the YouTube site. This test emulates only mobile phones requiring RTSP streams and using e.g. RealPlayer as e.g. Nokia phones running under Symbian OS. Figure 5-7 shows an example of a YouTube No Reference test that emulates a mobile phone.
Figure 5-7 YouTube test emulating a mobile phone
The settings for this test are the close as for common video tests streaming from Darwin server. The following YouTube URLs are only for demonstration purposes of the mobile phone emulation mode. These video clip examples are saved in a SwissQual test account in semi-private mode and cannot be found by searching on the YouTube site.
rtsp://v5.cache2.c.youtube.com/CjYLENy73wIaLQntK3AIU4b20BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYLnqoJDxy5vOTAw =/0/0/0/video.3gp // John QVGA (240p in 4:3) rtsp://v2.cache6.c.youtube.com/CjYLENy73wIaLQkI2oL2UT_gVxMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYJu9s23rJvOTAw=/0/0/0/video.3gp // Laugh QVGA (240p in 4:3) rtsp://v7.cache7.c.youtube.com/CjYLENy73wIaLQlQGULNtMx25BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYPrukd3RgJvOTAw =/0/0/0/video.3gp // Solothurn QVGA (240p in 4:3)
Note: The supported resolution of the SwissQual clips on the YouTube mobile site is QVGA. Do not use higher resolutions for the mobile phone emulation mode.
Uploading Video Material on YouTube

For best results, create your own YouTube account to act as a repository for the SwissQual video clips. This approach lets you have complete control over the account, including the desired privacy settings and the number of parallel approaches to the links in the account. Full Reference tests require a special pre-processing of the video clips, such as a separation of chunks by blue sequences, as well as adding sequence marker patterns in the lowest lines of the image. Pre-processed reference clips as the Station clip that contain these sequences can be requested from SwissQual.
Chapter 5 | Streaming From YouTube CONFIDENTIAL MATERIALS 41

For No Reference tests, you can upload video clips of your own choice. In case own video clips are used for YouTube testing it is recommended to upload high quality video clips in at least at HD 720p. However, since YouTube does not accept uncompressed RAW AVI files; you need to encode the files beforehand. To avoid poor quality issues, encode the video with the H.264 codec at a high bitrate, that is, >1Mbps for QVGA. When you upload the video, YouTube trans-codes the clip into the target bitrates for the different quality ranges.
YouTube Test Result Presentation
42

Manual - Streaming Service Test

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Manual - Streaming Service Test

Transféré par

Droits d'auteur :

Formats disponibles

A Rohde & Schwarz Company

Diversity Streaming Service Test

Part Number: 16-100-200635-2 Rev 1.2

Diversity Streaming Service Test Manual

Diversity Streaming Service Test Manual

Diversity Streaming Service Test Manual

Contents | CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Video Streaming Service

Chapter 1 | Introduction CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Starting and initializing video player

Connection to server and request video

Time to start buffering

Stopping playing or end of stream

Tear down connection to server

Figure 1-1 Overview of Video Streaming test

Chapter 1 | Introduction CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Configuring a Video over IP Streaming Test

Figure 2-1 Video Streaming - Layer model

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

QuickTime and RealPlayer Settings

To configure QuickTime for Diversity

In the Default Music Synthesizer list, click QuickTime Music Synthesizer.

Figure 2-2 QuickTime Audio settings

Diversity Streaming Service Test Manual

Figure 2-3 QuickTime Streaming settings

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-4 QuickTime Advanced settings

To configure RealPlayer for Diversity

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-5 RealPlayer Content / clip cache settings

Figure 2-6 RealPlayer Hardware settings / Video Card compatibility

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility

Figure 2-8 RealPlayer AutoUpdate setting

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-9 RealPlayer Message Center Preferences

7. In the Preferences window click OK.

Creating a Streaming Test

To create a Video over IP test in Diversity NQView / Control Option

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-10 Video Streaming Job definition

Figure 2-11 Video Streaming Job definition

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Figure 2-12 Video Streaming - Test definition

Chapter 2 | Configuring a Video over IP Streaming Test CONFIDENTIAL MATERIALS

Diversity Streaming Service Test Manual

Configuring a Video No Reference Test

Figure 2-13 Select "Video No Reference" Streaming test type

Figure 2-14 YouTube No Reference (Smartphone) test

Diversity Streaming Service Test Manual

Video over IP Test Settings

Figure 2-15 Configuration of a Video No Reference test

Connection timeout Stream Lost timeout URL Stream