N a t i o n a l S c i e n c e F o u n d a t i o n E n g i n e e r i n g R e s e a r c h C e n t e r

Roger ZimmermannAlexander A. Sawchuk, Cyrus Shahabi

Ulrich Neumann, Chris KyriakakisTom Holman, Christos Papadopoulos

Integrated Media Systems CenterUniversity of Southern California

Los Angeles, CA 90089

http://dmrl.usc.edu and http://imsc.usc.edu

RMI: RMI: Remote Media Remote Media ImmersionImmersion

Integrated Media Systems Center University of Southern California, Los Angeles

OutlineOutline

• IMSC Introduction• RMI Goals and Challenges• System Components• Experiments• Streaming Media Architecture: Yima• Research Challenges• Future Possibilities

Integrated Media Systems Center University of Southern California, Los Angeles

IMSC ERC Research StructureIMSC ERC Research Structure

Integrated Media Systems Center University of Southern California, Los Angeles

Integrated Media SystemsIntegrated Media Systems

Education

Entertainment

Media Communications

Information Management

SensoryInterfaces

Application Research

Communication

3 Vision AreasCharter: Immersipresence

User Centered Sciences

Media Immersion Environment

6 Research Areas

Integrated Media Systems Center University of Southern California, Los Angeles

What is the RMI?What is the RMI?

““The goal of the Remote Media Immersion The goal of the Remote Media Immersion system is to build a testbed for the creation system is to build a testbed for the creation

of immersive applications.”of immersive applications.”

Immersive application aspects:1. Multi-model environment (aural, visual, haptic,

…)2. Shared space with virtual and real elements3. High fidelity4. Geographically distributed5. Interactive

Integrated Media Systems Center University of Southern California, Los Angeles

Remote Media Immersion GoalsRemote Media Immersion Goals• Reproduce the complete

audio and video ambience placing people in a virtual space Experience events occurring at

remote site(s) Natural communication,

interaction and collaboration

• Application scenarios: Unidirectional off-line

acquisition, processing and storage of immersidata and synchronized display (rendering)

Real-time, two-way version Stereoscopic visual display

Immersed in a college football game

Doctors assisting in a remote procedure

Business people negotiating like they are in the same room

Students visiting an aquarium a thousand miles away

Integrated Media Systems Center University of Southern California, Los Angeles

RMI ChallengesRMI Challenges

Immersive, high-quality video acquisition and rendering High Definition video 1080i and

720p (40 Mb/s)

Immersive, high-quality audio acquisition and rendering 10.2 channels of uncompressed

audio (12 Mb/s)

Storage and transmission of media streams across networks

Synchronization between streams (A/V, A/A, V/V)!

Integrated Media Systems Center University of Southern California, Los Angeles

RMI ArchitectureRMI Architecture

Integrated Media Systems Center University of Southern California, Los Angeles

Remote Media Immersion ClientRemote Media Immersion Client

...

HDVideo

10.2 Ch.Audio

HD Video Uncompressed

Linear Time Code

Word Clock

10.2 Ch. Audio (Digital)

HD Video Rendering

10.2 Ch. Audio Rendering

Network

Network

YimaClientSW

Integrated Media Systems Center University of Southern California, Los Angeles

Experimental SetupExperimental SetupSynchronized combinations of

• Immersive audio and HDTV streamed playback from Yima Streaming of 16 channels of immersive audio,

uncompressed at 12 Mb/s Streaming of 1920x1080i HDTV content, MPEG-2

compressed at 40 Mb/s

IMSCIMSCISI EastISI East

Integrated Media Systems Center University of Southern California, Los Angeles

Internet2 Fall ‘02Internet2 Fall ‘02Member MeetingMember Meeting

Video: HDTV 1280x720p

Audio: 10.2 channel,immersive soundsystem

New World Symphony, Miami, FL

Integrated Media Systems Center University of Southern California, Los Angeles

Internet2 DemonstrationInternet2 Demonstration

Integrated Media Systems Center University of Southern California, Los Angeles

Storage, Streaming & RenderingStorage, Streaming & Rendering

Server Storage Scheduling Scalability

Clients Multi-

stream Synchro-nized playback

Transmission Robust VBR

Flow control

Requirements:

A streaming platform that can scale and handle synchronized, high-bandwidth streams.

Focus:

End-to-end streaming architecture.

Integrated Media Systems Center University of Southern California, Los Angeles

Yima ArchitectureYima Architecture Multi-node,

multi-disk architecture Scalable

Industry-standard network protocols: RTP, RTSP

Robust media transmission: Adaptive flow

control Selective

retransmission Clients:

Multi-stream synch.

Integrated Media Systems Center University of Southern California, Los Angeles

Research Focus and Unique Research Focus and Unique ApproachesApproaches

• Scalability (enable large scale systems) Multi-node architecture with distributed scheduler and

distributed file system on commodity PCs[Computer ‘02]

Incremental system growth: SCADDAR – an efficient randomized technique to reorganize continuous media blocks[ICDE 2002]

• Robust stream delivery Multi-threshold flow control between clients and server:

avoids data starvation and overflow, supports variable bit rate media[MTAP 2003?]

Selective packet retransmission protocol[NOSSDAV’96, MMCN’03]

Integrated Media Systems Center University of Southern California, Los Angeles

Challenge: Real-Time MediaChallenge: Real-Time Media

010

2030

405060

708090

100

Mb/s

Bandwidth requirements for different media types:

1 Mb/s

4-6 Mb/s

31 Mb/s

50 Mb/s

20 Mb/s

100 Mb/s

Integrated Media Systems Center University of Southern California, Los Angeles

Yima Server S/W ArchitectureYima Server S/W Architecture

Integrated Media Systems Center University of Southern California, Los Angeles

Challenge: ScalabilityChallenge: Scalability As continuous media (CM)

repositories increase, the need for larger storage capacity arises

1. Multi-node, multi-disk support

2. Disk scaling (adding new and/or removing old disks) SCADDAR

Quick access to data Online 24/7 operation (i.e. no

downtime) Fault-tolerance Load balancing of the data

before/after scaling to ensure maximum utilization of disk I/O and capacity

Integrated Media Systems Center University of Southern California, Los Angeles

Scalability: Multi-Node, Multi-Scalability: Multi-Node, Multi-DiskDisk

Yima-1 Yima-2

Data and control network traffic can be routed with different logical topologies Yima-1: single data path

(high inter-node traffic) Yima-2: multiple data paths

(low inter-node traffic)

Integrated Media Systems Center University of Southern California, Los Angeles

SCADDARSCADDAR Disk scaling (adding new and/or removing old disks) Example of adding one disk to an existing 4-disk storage

system:

Integrated Media Systems Center University of Southern California, Los Angeles

Challenge: Robust Stream Challenge: Robust Stream DeliveryDelivery

Variable bit rate (VBR) media encoders allocate more bits to complex scenes and less bits to simple ones

Smoothing of VBR media traffic has the following quality benefits: Better resource utilization (less bursty) More streams with the same network capacity

Multi-Threshold Flow Control(MTFC) algorithm objectives: Online operation Content independence Minimizing feedback

control signaling Rate smoothing

Motivation & ObjectivesMotivation & Objectives

Integrated Media Systems Center University of Southern California, Los Angeles

MTFC Buffer ManagementMTFC Buffer Management

• Multiple Thresholds: goal is middle of buffer• Send rate adjust command to server whenever

threshold is crossed

Integrated Media Systems Center University of Southern California, Los Angeles

Multi-Threshold Flow ControlMulti-Threshold Flow ControlResultsResults

Integrated Media Systems Center University of Southern California, Los Angeles

– IP networks are based on “best effort” delivery– Client at USC, LA, server at ISI East, Arlington, VA– One aspect: high-bandwidth video and audio transmissions

• HDTV @ 40-45 Mb/s• 16-channels of uncompressed PCM audio @ 11-22 Mb/s

– RTP/UDP is industry standard, but UDP loss creates problems• Tests on high-performance network: Internet2 and DARPA NGI SuperNet

(WAN) & Gigabit Ethernet (LAN)• In the order of 10 packets lost every 1 million (10-5)• Such low loss is still visible/audible!• Loss may result in synchronization problems

Challenge: Packet Loss!Challenge: Packet Loss!

Integrated Media Systems Center University of Southern California, Los Angeles

Solution Space Solution Space

– No silver bullet (“one size fits all”)• We chose selective retransmissions

FECFast

Sacrifices BW for reliabilityVulnerable to burst loss

FECFast

Sacrifices BW for reliabilityVulnerable to burst loss

RetransmissionOne RTT to recoverOptimal use of BW

RetransmissionOne RTT to recoverOptimal use of BW

ConcealmentFast

Optimal use of BWQuality may sufferMedia dependent

ConcealmentFast

Optimal use of BWQuality may sufferMedia dependent

TradeoffsReliability vs. BW

Reliability vs. Latency

TradeoffsReliability vs. BW

Reliability vs. Latency

Integrated Media Systems Center University of Southern California, Los Angeles

Selective RetransmissionsSelective Retransmissions

– Originally proposed in [NOSSDAV’96] by Papadopolous and Parulkar

– Need: sender retransmission buffer and receiver playout buffer

Receiver-driven operation: Ask for retransmissions

only missing data will be consumed after estimated RTT

Receiver-driven operation: Ask for retransmissions

only missing data will be consumed after estimated RTT

Integrated Media Systems Center University of Southern California, Los Angeles

Fast Error RecoveryFast Error Recovery

Senderread frame every T secs

111101Play-outbuffer

send new frame every T secs

Receiver

discard

kji

ijk

Retransmitbuffer

110111

101011

New frame

NAK

Integrated Media Systems Center University of Southern California, Los Angeles

Multi-node ServerMulti-node Server

CentralizedDesign

BipartiteDesign

– Originally Data and control network traffic can be routed with different logical topologies

• Centralized: single data path (high inter-node traffic)• Bipartite: multiple data paths (low inter-node traffic)

Integrated Media Systems Center University of Southern California, Los Angeles

Multi-node Server: RBECMulti-node Server: RBEC– To improve scalability and online data

reorganization data blocks are randomly assigned to server nodes.

Challenge:If a packet does not arrive at the client side, how does the client know which node attempted to send it?

Possible solutions:1. Broadcast retransmission requests2. Compute which node should have the data3. Introduce node-specific local sequence numbers (LSN) in addition to a global

sequence number (GSN)

Integrated Media Systems Center University of Southern California, Los Angeles

Yima ApproachYima Approach

3 3GSNLSNPayload

2 2GSNLSNPayload

1 1GSNLSNPayload

3 6GSNLSNPayload

2 5GSNLSNPayload

1 4GSNLSNPayload

6 9GSNLSNPayload

5 8GSNLSNPayload

4 7GSNLSNPayload

Server Node 1:

Server Node 2:

Server Node 1:

Assumption: 3 packets per storage block

Integrated Media Systems Center University of Southern California, Los Angeles

LSN Retransmission OperationLSN Retransmission Operation

Integrated Media Systems Center University of Southern California, Los Angeles

ExperimentsExperiments

– Multi-node server with 1, 2 and 4 nodes– LAN and WAN: DARPA NGI SuperNet, cross-continental link

(4000km)– Gilbert loss model: and

with p = 0.0192 and q = 0.8454, therefore Ploss is approx. 2.2%

– Media file “Twister” (MPEG-2)• avg. BW of 698 kB/s• length 25 minutes• throughput std. dev. 308283

qp

qarrivalP

qp

plossP

Integrated Media Systems Center University of Southern California, Los Angeles

Results for WANResults for WAN

4%

2%

0%

3%

1%

4%

2%

0%

3%

1%

4%

2%

0%

3%

1%

4%

2%

0%

3%

1%

25 Minutes 25 Minutes

Raw Loss 1 Node

2 Nodes 4 NodesNatural Losses

Integrated Media Systems Center University of Southern California, Los Angeles

Yima Client FeaturesYima Client Features

• Synchronization between multiple clients Coarse-grained via flow & rate control Fine-grained via hardware support (30 fps & 48,000 s/sec) Media streams can come from different physical locations

Integrated Media Systems Center University of Southern California, Los Angeles

RMI & Yima AccomplishmentsRMI & Yima Accomplishments

• Publications: SCADDAR ICDE, March 2002 IEEE Computer, June 2002 GMeN IEEE TPDS, June 2002

• RMI Transcontinential Tests: Server at ISI East, Arlington, VA Internet2 Fall’02 Meeting

• RMI Press Coverage: New York Times, May 9, 2002 NBC-4, May 9, 2002 KTLA-5, May 9, 2002

COMPUTER

Integrated Media Systems Center University of Southern California, Los Angeles

Future PossibilitiesFuture Possibilities

• Distributed virtual social events • Immersive gaming

• Large screen displays• Multiple cameras and microphones; 3-D scene

description• Speech and gesture extraction• Face and body tracking• Wireless glasses or head-mounted displays• Stereo display without glasses (autostereoscopic)

Integrated Media Systems Center University of Southern California, Los Angeles

Distributed Immersive Distributed Immersive PerformancePerformance

• Outgrowth of Remote Media Immersion (RMI)– Create seamless immersive environment for

distributed musicians, conductor (active) and audience (passive)

– Compelling relevance for any human interaction scenario: education, journalism, communications

• Scenario:– Orchestra not available in town– Famous soloist cannot fit travel into schedule– Multiple soloists in different places

Integrated Media Systems Center University of Southern California, Los Angeles

Integrated Media Systems Center University of Southern California, Los Angeles

Integrated Media Systems Center University of Southern California, Los Angeles

Integrated Media Systems Center University of Southern California, Los Angeles

Integrated Media Systems Center University of Southern California, Los Angeles

30 ms

20 ms

30 ms

10 ms

40 ms

60 ms

Challenge: network latency

Integrated Media Systems Center University of Southern California, Los Angeles

Technical ChallengesTechnical Challenges

processing power

latency(delay)

compression

data rates, error characteristics

multi-stream synchronization

Integrated Media Systems Center University of Southern California, Los Angeles

Yima Ongoing WorkYima Ongoing Work• Real-time recording of multiple streams

– For example, from a panoramic camera with 5 individual camera heads:

– Streams need to recorded in sync and played back in sync

• Statistical admission control algorithm for better utilization of the storage system

• Interactive, live streaming

Integrated Media Systems Center University of Southern California, Los Angeles

Thank You! Questions?Thank You! Questions?

• More info at:– Data Management Research Lab

• http://dmrl.usc.edu

– Integrated Media Systems Center• http://imsc.usc.edu

• Acknowledgments:– Kun Fu, Didi Shu-Yuen Yao, Beomjoo Seo, Shihua Liu, Mehrdad

Jahangiri, Farnoush Banaei-Kashani, Nitin Nahata, Sahitya Gupta, Vasan N. Sundar, Rishi Sinha, Hong Zhu