UniGrids Streaming Framework: EnablingStreaming for the New Generation of Grids

Krzysztof Benedyczak1, Aleksander Nowinski2,Krzysztof Nowinski2, and Piotr Ba�la1,2

1 Faculty of Mathematics and Computer Science,Nicolaus Copernicus University,

Chopina 12/18, PL-87-100 Torun, Polandgolbi@mat.uni.torun.pl

2 Interdisciplinary Center for Mathematical and Computational Modelling,Warsaw University,

Pawinskiego 5a, PL-02-106 Warsaw, Poland

Abstract. We present a new infrastructure for high performancestreaming in OGSA/WSRF compliant grid. The UniGrids StreamingFramework (UGSF) works with UnicoreGS as WSRF hosting environ-ment. The paper discusses the advantages of mixed SOAP based controlwith highly efficient streaming. The UGSF components, streaming serverand WSRF web service are described along with a detailed performanceanalysis including comparison to standard solutions. Some applicationsbased on the UGSF are also presented.

1 Introduction

Current trends in grid technology is clearly focused on OGSA (Open Grid Ser-vices Architecture) [1] which implies usage of the web services. The detailedguidelines on how to build grid services are given by the WSRF specifications[2]. The consensus about the importance of such approach was motivated bymany reasons. Here we can point to an interoperability as the most significantone. The WSRF as well as other specifications allow developers to easily creategrid software compatible with other WSRF implementations. Moreover, as webservices technology is widely adopted in B2B applications, one can make use ofexisting experiences and adopt available solutions. A good example is the BPEL[3] specification defined for business processes, which is now being used as thetool to define grid workflows. These reasons form a solid base for OGSA whichplays a vital role in grids nowadays. But we can not forget that web services alsohave disadvantages. Here we will focus on two of them that are crucial for datastreaming.

The first and the most important drawback of web services is efficiency. Webservices technology is based on the SOAP protocol. This results in extensiveusage of XML. The obvious consequence is a large overhead for even a simple op-eration: the SOAP engine has to perform a lot of XML parsing or encoding. More-over, XML data encoding is very verbose, thus ineffective. In the most streaming

B. Kagstrom et al. (Eds.): PARA 2006, LNCS 4699, pp. 809–818, 2007.c© Springer-Verlag Berlin Heidelberg 2007

810 K. Benedyczak et al.

applications such data overhead is undesirable. The other problem is data stream-ing: SOAP is message driven and XML to be parsed must be fully read.1

Presented disadvantages cause that web services technology can not be seenas suitable for any interactive, real-time application. It is hard to imagine a sci-entist steering an interactive device with latency of every operation measured inseconds. Of course, various XML technologies like binary encodings, aforemen-tioned streaming processing of XML, MTOM [6] or TCP bindings can be usedto boost the performance of web services. We are sure that in some cases it ispossible to build streaming technology on top of such optimised web services.An obvious advantage of such approach, is better integration with existing webservices agents (like new UniGrids gateway) and not much more. It is also clearthat such solutions can be useful only for less demanding streaming applications.

To solve the problem we have developed a hybrid system which is a platformto build any type of streaming services managed in WSRF compliant way on.The solution is highly responsive and efficient.

2 System Design

The UGSF system is based on the WSRF’s compliant version of the well rec-ognized UNICORE middleware [8]. The UnicoreGS [7] is used as the WSRFhosting environment.

The aim of the UniGrids Streaming Framework (UGSF) is to provide directdata streaming and steering for applications. The main part of UGSF is UGSFcore which is a middleware that allows developers to create dedicated streamingsolutions.2 Substantial effort was made to prepare a system where the creationof a specialized solution is as easy and quick as possible. Every system based onthe UGSF will use the core together with some application dependent code. TheUGSF core provides basic functionality common for all streaming applications.This includes creation or shut-down of a connection. The system is designedin such a way that a group of versatile software pieces can be reused. A goodexample is the component to locate UnicoreGS job’s working directory and accessit’s contents.

The UGSF core consists of a UGSF Web Service part, a Streaming Server partand a library to create clients. The usage of the last component is optional. TheUGSF Web Service takes advantage of WSRF capabilities. It is used to controla set of available stream types, to create new streams and to manage alreadycreated ones. The Streaming Server part is managed by a UGSF Web Serviceand performs streaming. The cClient library is used to simplify the creation ofthe client-side software. Overall architecture is shown in Figure 1.

1 It is worth to note that currently there are intensive efforts to eliminate this issue andhopefully new generations of SOAP engines (as AXIS 2 [4] or XFire [5] with supportfor StaX) will solve it.

2 We will use this therm whenever we will refer to basic framework, without actualstream implementations which are also included in UGSF distribution.

UGSF: Enabling Streaming for the New Generation of Grids 811

Fig. 1. The general architecture of UGSF

The UGSF core is complemented with stream implementations. These consistsof two parts: the streaming server and the web service modules. The web servicemodule implements control operations specific to the stream implementation.The streaming server module deals with a wire streaming protocol and dataconsumption/acquisition.

The general pattern of UGSF usage is as follows:

– The UGSF installation is configured by the system administrator, who de-fines so called stream types. Every stream type is one stream implementationwith some configuration parameters (which can’t be modified by users).

– The user chooses the stream type and creates its instance. If the implemen-tation stipulates that some user’s parameters are needed, then the user mustsupply them. As a result a reference to the newly created stream manage-ment WS-Resource is returned.

– The user can invoke any common (provided by UGSF core) or special (streamimplementation defined) operation on the WS-Resource assigned to the cre-ated stream. The resource properties contain (among others) informationabout how to connect to the UGSF streaming server to start streaming.

– The user connects to the UGSF streaming server and starts the data transfer.It is possible to control the connection via the web service interface.

The Streaming Server and the clients built for UGSF are grid-enabled. There-fore, the UGSF can be used to let legacy applications benefit from the gridtechnology (e. g. grid authorization), using already developed stream implemen-tations.

To accomplish general overview of the UGSF we present details about theunderlying transport level protocol. In principle, the UGSF is highly flexible and

812 K. Benedyczak et al.

can be used for any application level protocol. Currently there is no possibility touse any protocol than TCP. This decision was motivated by multiple factors. Ourfirst aim was to support tunneling of streamed data with the UNICORE gateway,which can operate only on TCP connections. Another reason is that the usualuse of grid middleware requires high security and reliability of connections (e.g.scientific applications which stream video must not loose any frames contraryto typical multimedia situations when such loss is acceptable). This is mucheasier to implement in general framework based on TCP/TLS. Nevertheless, inthe future versions of the UGSF the UDP entry points can be added. This willinvolve some redesign of the UGSF Streaming Server.

2.1 UGSF Web Service

The UGSF Web Service component consists of two kinds of web services. Abase one (called StreamingFrameworkService) is responsible for the connectionauthorisation, the creation of stream and its setup. During this process the newWS-Resource (called StreamManagementService) is created with a dedicatedweb service interface. This WS-Resource acts as a controller of an active stream-ing connection.

The StreamingFrameworkService is a WS-Resource which maintains lists ofStreamManagementServices. It can be argued that this is a perfect case for theWSRF Service Group which is a federation of WS-Resources. Unfortunately, theService Group can’t be used here due to the security restrictions. The WSRFspecification doesn’t permit filtering the Service Group’s content. As a resultevery user would have the possibility to see other users streams.

The StreamingFrameworkService allows users to get a list of available stream-ing services and setup a connection to the specific streaming service. The list ofboth owned and accessible streams is available (see section 2.3 for details). Inaddition, the StreamingFrameworkService has an administrative interface, whichempowers a system administrator to enable and disable particular stream typeson the fly. The service reconfiguration such as addition or removal of streamtypes is also possible.

For each created stream, an instance of the StreamManagementService allowsthe user to perform universal operations for all streams. This includes shut-ting the stream down (by means of WS-Lifetime interface), getting status andstatistics of the connection, as well as pausing or resuming streaming. This func-tionality can be easily enriched by the developer. He can extend StreamMan-agementService with additional operations. The enriched implementations arefree to consume any special XML configuration supplied to the StreamingFrame-workService and required for service setup and creation.

We have also developed an additional service called StreamingFrameworkFac-tory, which allows site administrators to create base UGSF services and configurethem initially. The developed service follows the pattern of the UniGrids atomicservices [7].

UGSF: Enabling Streaming for the New Generation of Grids 813

Fig. 2. Services and modules of UGSF components

2.2 UGSF Streaming Server

The UGSF Streaming Server is a stand-alone, modular application, which per-forms streaming to and from the target system. The server is tightly connectedto the UGSF Web Service which maintains stream definitions. The communi-cation is done with Java RMI. The server is modular, and highly configurable.The dedicated modules were created to access the actual streaming data source.Such a module also gives access to the running job’s outcome. It can also pro-vide it with input, if required. On the other hand there are stream modules thatdon’t need any job to cooperate with. A module which gives access to physicalresources like a video camera is a good example. Another one is a module whichenables grid usage of the legacy TCP or UDP servers. There is also a whole classof auxiliary modules which acts without any external resources. These modules,for example, convert input data from one format to another.

For a particular site, there can be more than one Streaming Server, oper-ated by only one UGSF Web Service. Each server is able to handle multiplestream modules. There is also the possibility to configure another setup: onesingle Streaming Server can be managed by more than one UGSF Web Service.However, such a scenario is of little practical value.

The access to the UGSF Streaming Server is accomplished with a specialprotocol. Currently the protocol is trivial but it may be developed to a morecomplicated one when new features are needed. The access to the StreamingServer is done by means of exchangeable entry modules. More than one entrymodule can be turned on simultaneously. Every stream can be accessed by manyentry protocols and the application can choose the one it prefers or understands.Currently there are available HTTP and HTTPS entry points with simple POSTbased protocols (in fact there is one entry point which can be configured to useor not to use TLS). The system is ready to use the other protocols as well.

814 K. Benedyczak et al.

2.3 Advanced Features and Security

In addition to the basic infrastructure for streaming connection creation, theUGSF provides a set of advanced stream related operations. These operationsfocus on a sophisticated data flow creation. By the term data flow we mean thehere composition of one or more streams between servers and/or clients createdfor one application.

Every stream implementation can contain more than one flow. Flow is a syn-onym to a connection, e.g. if one stream maintains three flows then it is possibleto open three concurrent connections to this stream. This provides an opportu-nity to create more advanced streams with a clear separation of logical ”flows“of data, including separation of input and output.

The UGSF streams have metadata attached. For every flow there are defined,among others, supported formats. It is possible to specify more than one formatfor a single flow, as well as express the only supported format combinations.Any flow can have two-way traffic, but it is suggested that a flow should onlyuse input or output whenever possible (so to be one directional). When two-waytraffic is required, two flows are preferred. Streams designed this way are muchmore effortlessly integrated into data flows.

In order to enable composition of other than trivial data flows (i.e. client ↔server), UGSF offers a connect operation. It instructs an already created flow ofone stream to exit its passive state and to actively initiate connection to anotherflow.

There is also a possibility to create a flow with ”cloning“ ability. Such a flowcan be used to dynamically create new flows in a stream implementation. Agood example of the cloning feature is a multiplexer, which basically managestwo flows, the input and output. The output flow has cloning ability and theuser can clone the output flow multiple times. As a result he can fork the inputinto arbitrary number of outputs.

Up to now, we haven’t covered one significant aspect of the UGSF system:security. The main question here is: What are the requirements to open connec-tion to the Streaming Server? The simplest approach is to enable access to thestream only to its owner. Unfortunately, such a method is not sufficient for morecomplicated scenarios, such as server ↔ server connections. To give an example;Let’s consider a data flow where server A is the source of data. This data shouldbe processed by a server B and finally the output should be received by the userU . If U creates appropriate stream instances on A and B, B will not be allowedto access A’s stream - only U will be.

To solve this problem, every flow is assigned a token, which is the identity ofits owner and its access policy. The token is a large unique number. The accesspolicy is defined by the creator of the stream and describes who is authorisedto contact the flow. The default policy is ”owner-only“. In this case only userswith a certificate matching the flow owner’s certificate can open a connection.He has also to present the flow token for identification purposes. Please note thatthe token value is not sensitive as it is valid only after a connection is established

UGSF: Enabling Streaming for the New Generation of Grids 815

using a valid certificate. Policy can allow public (non restricted) access and alsoan explicitly specified entity to access the stream.

In the matter of UGSF security we still have some work to do. We would liketo provide XACML [9] support for policy description. Also some trust delega-tion should be supported to achieve better integration with standard grid trustdelegation (but this is a matter of better system cohesion).

3 Applications

The UGSF system includes several basic stream implementations.The first one is TCPStream, which can be seen as a grid version of the SSH

tunnel. It has a similar functionality to such a tunnel and an obvious advantage isthat users don’t need shell access to the grid site. Moreover, they are authorisedin the same way as for any other UniGrid services.

Using available client software for creating TCP tunnels in the UGSF pack-age, we have used UGSF to steer an Advantech’s ADAM/5000TCP device withan existing application. The ADAM/5000TCP is a Modbus [10] Ethernet de-vice. The UGSF is very useful because Modbus Ethernet devices are in generalinsecure and must be protected by firewalls. This example shows that by usingUGSF, the whole range of Ethernet devices can be secured and grid enabledonly by putting in a few lines in the UGSF configuration file.

The TCPStream is accompanied by UDPStream which does a task not widelyavailable by any other software. It tunnels UDP datagrams over the TCP pro-tocol, maintaining UDP ”sessions“ in a manner similar to that used by firewalls(packets are scanned for changes of destination ports).

In the UGSF there is also a FileStream implemented. It serves as a streamingversion of a file access service. The FileStream has the ability to detect filegrowth, and allows to stream file content as new data is put in. Clearly, thissolution is targeted to receive results of arbitrary simulation in real time. Tomonitor grid job results, there is another stream called IVisStream which is asimple extension of FileStream. It supports, in addition to FileStream features,location of files outputted by a given grid job.

Currently we are working on more universal stream types, which will supportdata flow creation, (as e.g., the Multiplexer stream for splitting arbitrary flowinto multiple ones) and to add generic support for video streaming which is anecessity for many streaming applications. We chose Theora [12] as our ”native“codec. Streams to compress raw video (and decompress it) will be availableshortly.

4 Performance

During the design of the UGSF, our aim was not to introduce any penalty onthroughput (except enforcement of TCP and use of SSL in most cases). It wasachieved, as after stream setup, the developer can use an arbitrary protocolon open socket connection. The UGSF core does not add any extra data to the

816 K. Benedyczak et al.

Fig. 3. The performance of plain netcat stream versus netcat over UGSF TCP tunnel(over the 100Mbit network)

opened stream. To check the performance and see how tunneling over a particularstream will impact it, we have run performance tests on the TCPStream. For thetests we have used netcat TCP session. We have compared a direct connection toone tunneled via TCPStream. Two machines running Fedora Core 4 operatingsystem (kernel version 2.6.13-1.1532 FC4) were used. The systems were inter-connected with 100Mbit Ethernet. Server machine (B) was equipped with IntelXeon 2,4GHz CPU and the client was AMD Athlon 64 3400+ CPU (A).

As it is shown in the figure 3, the plain UGSF tunnel performs nearly the sameas the plain netcat connection. The SSL version is, of course, slightly slower, butstill the difference is tiny and acceptable in most usage scenarios.

We have also looked at the CPU usage reported by the Streaming Server. Theserver consumed less than 15% of the CPU time at A and about 2–3% more CPU

Table 1. Performance comparison of the UGSF and web service based implementation.The data was sent in small chunks in two directions. RQ stands for ”request“ and REfor ”response“. The third column contains the total number of full message exchanges(i.e. sending request and receiving response) per second.

Messages Implementation (RQ + RS) Relativesizes per second speedupRQ 16B/RS 10kB web service/UnicoreGS 1.4.1 0.96 1RQ 10kB/RS 10kB web service/UnicoreGS 1.4.1 0.48 1RQ 16B/RS 10kB UGSF/Java DataStream 23.38 24RQ 10kB/RS 10kB UGSF/Java DataStream 20.76 43

UGSF: Enabling Streaming for the New Generation of Grids 817

time at B machine. When SSL is turned on, the CPU intensive encryption causedthe increase of the systems utilization to about 50 and 55 percents respectively.

To summarize, the results are promising: practically there is nothing to opti-mize. The CPU operations on recent hardware does not impact throughput of100MBit streams and there is still some CPU power left. Moreover, the CPU in-tensive operations are mostly those coming from the SSL sockets implementationof the Java toolkit.

The most interesting topic is the comparison of operations invoked by meansof standard web service calls, with an analogous system based on the UGSF. Theweb services operate exclusively on messages. The AXIS 1.x environment usedin the UniGrids project limits it to complete message exchanges instead of realstreaming. It can be theoretically proven that one-way web services can resolvethis issue. However, it is problematic from the server side when a client is behindthe firewall/NAT. Some progress can be made by using HTTP 1.1 persistentconnections [11] but currently this (along with other needed functionality) isnot available in AXIS 1.x.

In order to run comparison tests we have developed a trivial UnicoreGS ser-vice with only one operation, which consumes and returns a configurable amountof raw data (Base64 encoded). The results of running series of operations on thisservice are given in the first two rows of tab. 1. Also small client-server applica-tion was prepared to test UGSF version. It was used through UGSF TCPStream,and as internal protocol we used Java DataStream. As it can be noted from thelast column of table 1, the speed up is more than 20. In fact, this is the minimalperformance gain. In reality UGSF can be used to operate much more effec-tively: by implementing specialised UGSF stream type, the two extra data hopsintroduced by generic TCPStream and it’s client can be eliminated. Moreover,in many cases streaming applications can benefit from parallel streaming, whilein the tests we were using synchronised message exchanges. Test results wereobtained on the same machines as above.

We would like to mention that there is a lot to improve in the web serviceversion too. A Better SOAP engine (e.g. AXIS 2), and usage of its features,can give a substantial performance boost. Also UnicoreGS currently is still indevelopment and there were no optimisations made.

5 Summary

The presented development is focused on various applications where UGSF willhave a possibility to prove its value. We consider a device access and remotesteering, video transmission and scientific image processing. Of course, visual-isation and real-time monitoring of computation are also of interest, as it hasalready been presented for the UNICORE middleware [13]. The UGSF includestwo stream implementations that allow for tunneling connections to both TCPand UDP legacy servers on the grid site. Services to stream changing content ofgrid jobs in real-time are also ready to be used. Support for data flow creationsencourages to use UGSF in a component driven way, where already created

818 K. Benedyczak et al.

stream implementations are reused in larger applications. In general, the devel-oped infrastructure opens a field to numerous applications, which require on-linedata streaming and steering.

This work was supported by European Commission under IST grant UniGrids(No. 004279).

References

1. Foster, I., Kesselman, C., Nick, J., Tuecke, S.: The Physiology of the Grid: An OpenGrid Services Architecture for Distributed Systems Integration. Globus Project(2002), http://www.globus.org/alliance/publications/papers/ogsa.pdf

2. Czajkowski, K., Ferguson, F.D., Foster, I., Frey, J., Graham, S., Sedukhin, I.,Snelling, D., Tuecke, S., Vambenepe, W.: The WS-Resource Framework, Version1.2. OASIS (Organization for the Advancement of Structured Information Stan-dards) (April 2006),http://docs.oasis-open.org/wsrf/wsrf-ws resource-1.2-spec-os.pdf

3. Andrews, T., Curbera, F., Dholakia, H., Goland, Y., Klein, J., Leymann, F., Liu,K., Roller, D., Smith, D., Thatte, S. (eds.): Trickovic, I., Weerawarana, S. BusinessProcess Execution Language for Web Services, Version 1.1. OASIS (2003)

4. Axis 2 project (August 2006), http://ws.apache.org/axis25. XFire project (August 2006), http://xfire.codehaus.org6. Gudgin, M., Mendelsohn, N., Nottingham, M., Ruellan, H.: SOAP Message Trans-

mission Optimization Mechanism, W3C Recommendation (January 25, 2005),http://www.w3.org/TR/2005/REC-soap12-mtom-20050125

7. UniGrids project website (August 2006), http://www.unigrids.org8. UNICORE at SorceForge (August 2006),

http://sourceforge.net/projects/unicore9. Godik, S., Moses, T.: eXtensible Access Control Markup Language, Version 1.1.

OASIS Committee Specification (August 07, 2003),http://www.oasis-open.org/committees/xacml/repository/cs-xacml-specification-1.1.pdf

10. Modbus Organization, Inc. and Schneider Automation Inc.: MODBUS ApplicationProtocol Specification, vol. 1.1 (August 2006), http://www.modbus.org/specs.php

11. Fielding, R., et al.: RFC 2616 - Hypertext Transfer Protocol – HTTP/1.1, Section8.1: Persistent Connections. The Internet Society (1999),http://www.ietf.org/rfc/rfc2616.txt

12. Theora I Specification. Xiph.org Foundation (March 7, 2006),http://www.theora.org/doc/Theora I spec.pdf

13. Ba�la, P., Benedyczak, K., Nowinski, A., Nowinski, K.S.: Real-time visualisation forUnicore middleware. In: Wyrzykowski, R., Dongarra, J.J., Meyer, N., Wasniewski,J. (eds.) PPAM 2005. LNCS, vol. 3911, pp. 608–615. Springer, Heidelberg (2006)