Quality of Experience

Using FBM to Benchmark Quality of Experience (QoE)

The perception and experience of users of video delivery systems depends on various factors. Due to the limited transmission bandwidth and terminal capacity, video content is encoded (compressed) with many advanced features for the packet network delivery. These advanced en/decoding, quantification techniques on the one hand magnify the data loss impact (e.g. spatial and temporal loss propagation), on the other hand complement the delivery impairments (e.g. error concealment techniques). The analysis of individual packet loss impact on video stream is critical in some circumstances. Most of the current studies aimed to find the direction correlation between packet loss and user experience, which make the results considerably straightforward but only suitable for certain sets of configurations on codec, bit-rate or application type.

Figure 1

In order to overcome the disadvantages of previous studies, research of video assessment is suitable to be divided into two phases. The first phase investigates the methodology of predicting the artefacts in the video frame by analyzing packet delivery information from either payload content (after decoder rendering) or packet header info (before decoder rendering). In the second phase characteristics and visibility of artefacts are studied with user tests experiments to study the correlation between artefacts and user perception. Figure 1 shows the two phase research as two System Under Test (SUT). Video steaming is the first SUT. During the video content distribution, network impairments such as delay, jitter and packet loss will have impact on video content and leave both spatial and temporal artefacts on the video frames. The environment parameters which decide the level of impact from network impairments on video content are codec, source quality, etc. The artifacts from certain packet loss can actually be predicted from the standardization of the codec. For example, the quality of a B-frame in a MPEG encoded video stream can be predicted by the number, pattern and other characteristics of packet losses within this frame and the quality of its depended frames (I-frame or P-frame), which are also predicted with the prediction mechanism. The second SUT is the end user, user tests are set in this phase to collect the user’s opinion on different artefacts so the visibility of each type of artefacts and their characteristics can be investigated.

The two-phase study is still codec dependent but highly scalable. When a new codec is introduced, researchers only need to build the first objective mapping by studying the standardization; no more costly and time-consuming subjective tests are required to build the model from sketch.

First system under test (Case 1)

Figure 2

Second system under test (Case 2)

Figure 5 Second system under test

Pure technical (quantitative) metrics for perceived video quality are not sufficient indicators of the performance of considered service (SUT). Quantitative metrics like PSNR (Peak Signal to Noise Ratio) or QoS parameters have to be expressed in qualitative scale. For this reason subjective experiments have to be performed and a model (i.e. mapping function) has to be developed. General provision for the subjective experiments' methodology is presented in [15]. The process of model creation is quite complex and based on some statistical techniques. In fact, the model is an empirical function that allows the mapping between multiple quantitative input parameters resulting in one quantitative output score.

References

[1]Z. Wang, H. R. Sheikh, and A. C. Bovik, "No-Reference Perceptual Quality Assessment of JPEG Compressed Images," in IEEE International Conference on Image Processing, 2002.

[2]T. Szigeti and C. Hattingh, End-to-End QoS Network Design: Quality of Service in LANs, WANs, and VPNs: Cisco Press, 2004.

[3]D. Miras, "A Survey of Network QoS Needs of Advanced Internet Application," Internet2 QoS Working Group, November 2002.

[4]T. Rahrer, R. Fiandra, and S. Wright, "Triple-play Services Quality of Experience (QoE) Requirements and Mechanisms - For Architecture & Transport," DSL Forum, 2006.

[5]J. Welch and J. Clark, "RFC 4445: A Proposed Media Delivery Index (MDI)," 2006.

[6]S.-k. Kwon, A. Tamhankar, and K. R. Rao, "Overview of H.264/MPEG-4 part 10," Journal of Visual Communication and Image Representation, vol. 17, 2006.

[7]T. Stockhammer, M. M. Hannuksela, and T. Wiegand, "H.264/AVC in Wireless Environments," IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, 2003.

[8]M. Flierl and B. Girod, "Generalized B pictures and the draft H.264/AVC video-compression standard," IEEE Transactions on Circuits and Systems for Video Technology, , vol. 13, pp. 587-597, July 2003.

[9]M. M. Hannuksela, "Non-normative error concealment algorithms," ITU-T VCEG-N62, 2001.

[10]W. Ye-Kui, M. Hannuksela M., V. Varsa, A. Hourunranta, and M. Gabbouj, "The Error Concealment Feature In The H.26L Test Model " in Proc. ICIP, 2002.

[11]B. Girod, "The information theoretical significance of spatial and temporal masking in video signals," SPIE Human Vision, Visual Processing, and Digital Display, vol. 1077, pp. 178-187, 1989.

[12]M. Yuen and H. R. Wu, "A survey of hybrid MC/DPCM/DCT video coding distortions.," Signal Processing, vol. 70, pp. 247-278, November 1998.

[13]M. Kaneko, Y. Hatori, and A. Koike, "Improvements of Transform Coding Algorithm for Motion-Compensated Interframe Prediction Errors -- DCT/SQ Coding " IEEE J. Sel. Areas Commun. , pp. 1068-1078, 1987.

[14]A. N. Netravali and B. G. Haskell, Digital pictures : representation, compression, and standards, 2nd ed. New York: Plenum Press, 1995.

[15]"Methodology for the subjective assessment of the quality of television pictures," ITU Recommendation BT.500-11.

[16]"Subjective video quality assessment methods for multimedia applications," ITU-T Recommendation P.910.

[17]"The E-model, a computational model for use in transmission planning," ITU Recommendation G.107, 1998.

[18]"Methods for Subjective Determination of Transmission Quality," ITU Recommendation P.800, 1996.