Gadgeteer

Traditionally, muti-screen immersive systems have relied upon dedicated high-end shared memory graphics workstations or supercomputers to generate interactive virtual environments. These multi-screen immersive systems typically require one or two video outputs for each screen and simultaneously utilize several interaction devices. In recent years this trend of almost exclusively using high-end systems has started to change as commodity hardware has become a viable alternative to high-end systems.

Current technologies have empowered PC-based systems with high-quality graphics hardware, significant amount of memory and computing power, as well as support for many external devices. Their application to virtual reality applications is motivated by the dramatic cost decrease they represent and by the wide range of options and availability. To drive a multi-screen immersive environment we need multiple commodity systems working as a single unit, that is, a tightly synchronized cluster. The challenge is that, although the base technology is standard off-the-shelf technology, there is a lack of software for weaving together the cluster into a platform that supports the creation of virtual environments. Furthermore, there is an even greater lack of software that can allow existing virtual environment designed for high-end system to transparently migrate to a cluster.

This document presents ClusterJuggler, an extension to the VR Juggler architecture which allows an application to transparently run on a cluster of comodity PC's. The main goals of ClusterJuggler are to allow the cluster software to adapt to the particular hardware configuration of the virtual reality system, to provide application portability and scalability from high-end to commodity by hiding the clustering from developers, and to allow users to customize the clustering methods being used.

ClusterJuggler is our third generation of clustering software. Based on our past experiences, we have created a system that combines recommended practices of clustering into a modular and extensible system. This clustering research contributes several innovations:

Figure 1.1. Cluster Juggler Layered Architecture

As Figure 1.1, “Cluster Juggler Layered Architecture” shows, ClusterJuggler is comprised of a set of components that are arranged into several layers. This technique was chosen to make the system as modular as possible. Each modular layer only uses the functionality provided to it by the layers below it.

Figure 1.2. ClusterManager and Plugins

Cluster Manager Layer. The Cluster Manager layer is responsible for the configuration of all layers in ClusterJuggler and managing the active plugins (Figure 1.2, “ClusterManager and Plugins”). This includes maintaining a list of all current plugins and synchronizing the calls to each plugin to perform their specialized tasks. Plugins are added to the system when a new plugin registers itself with theClusterManager.

Cluster Network Layer. The Cluster Network layer maintains an abstract representation of the system of interconnected nodes that comprise the cluster (Figure 1.2, “ClusterManager and Plugins”). This abstraction provides the rest of the system with a messaging interface for communicating with the entire cluster. Internally it maintains a list of each node in the cluster along with the network connection used to communicate with it. Because the networking layer is an abstraction, developers can provide different implementations that make use of alternative low-level networking methods. The networking implementation can then be configured by the user.

Cluster Plugins Layer. ClusterJuggler is based on a plugin architecture that allows the system to be modular and extensible. Plugins are dynamically loaded on demand at run-time. This allows users and developers to easily add new plugins to support additional cluster-based functionality. All ClusterJuggler plugins are derived from a common base interface: ClusterPlugin. To create a new plugin, a developer must implement a few of the base interface methods. In return, they get a support for network communication, synchronization, and run-time reconfiguration.

ClusterJuggler provides several commonly used plugins that serve as the base for clustering in VR Juggler and as examples for other developers to extend the system. These base plugins are the RemoteInputManager, ApplicationDataManager, and SwapLockManager. The RemoteInputManager plugin is responsible for distributing synchronized device data across the cluster. The ApplicationDataManager provides developers with a method for sharing application data within a clustered application. The SwapLockManager implements several methods for synchronizing display updates within a cluster. We plan to add additional plugins in the future to support extended clustering functionality.

The ClusterManager(Figure 2.1, “Class Diagram: ClusterManager”) is one of the only two static components of ClusterJuggler. It acts as a facade for the entire ClusterJuggler sub-system. It is responsible for maintaining a list of all current plugins, configuring all sub layers, calling the plugins to perform their specific tasks, and syncronizing the kernel loop between machines.

Dynamic Plugin Registration. When a plugin is dynamically loading it automatically registers itself with the ClusterManger where it is thrown into a list of plugins for later use. This list is the only way that ClusterJuggler knows about the plugins. This means that that there is absolutely no way to access these plugins unless you go through the ClusterManager's API and this list.

Run Time Re-Configuration. VR Juggler's architeture has the amazing ability to reconfigure itself while it is running. This is a complex task of run time reconfiguration is difficult when running on one machine let alone across a cluster. The ClusterManager is resoponsible for making this all happen behind the sceens.

Figure 2.1. Class Diagram: ClusterManager

Note

VR Juggler is continually running through a loop of method calls that receive information from external devices, calculates application data, and draws the users perspective called the kernel loop.

Figure 2.2. VR Juggler Kernel Loop

Plugin Method Calls. Within the VR Juggler kernel loop (Figure 2.2, “VR Juggler Kernel Loop”)there are many different locations that we might need to send information across the network in order to fully synchronize the cluster. For example, application data needs to be shared directly before the draw function because this data needs to be identical across the cluster right before we draw the scene. But, on the other hand input data needs to be shared at the beginning of the loop so that preFrame() can use this data to calculate the state of the scene such as the navigation matrix. In order to accommodate all possible needs of a plug in, there are calls into the ClusterManager at numerous locations during the kernel loop. The Cluster Manager is then responsible for calling any plugin that needs to use the network to transfer data.

Frame Synchronization. In order to guarantee that one node does not progress further through the kernel loop than the other nodes the ClusterManager must provide a level of synchronization on every method call from the kernel loop. This is accomplished by requiring the ClusterManager to receive all data packets. Each plugin is only responsible for sending packets to other cluster nodes. Once all plugins have sent their data, the ClusterManager sends an END_BLOCK packet telling the remote nodes that it is done sending information and then reads data from the network until it receives an END_BLOCK from all other nodes. The syncronization occurs because each node must indicate to all other nodes that it has completed the current frame(Figure 2.3, “Frame Syncronization”).

Figure 2.3. Frame Syncronization

Step 1) Create a new ClusterManager config element.

Figure 2.4. Create ClusterManager Element

Step 2) Give the new ClusterManager element a unique name.

Figure 2.5. Set Name

Step 3) Enter the name of the first plugin that your application requires.

Figure 2.6. Enter Plugin Name

Step 4) Add a new value and enter the name of each additional plugin that your application requires.

Figure 2.7. Adding more Plugins

Step 5) Select the first node that will be in your cluster.

Figure 2.8. Selecting First Node

Step 6) Add a new value for each additioanl node in your cluster.

Figure 2.9. Adding More Nodes

Step 7) Select the node for each additional value you just created.

Figure 2.10. Select Node

Step 8) Select the node that is responsible for starting the cluster at the same time.

Figure 2.11. Select Barrier Master

The Cluster Network provides each plugin with a representation of the cluster as a system of interconnected nodes and defines an interface for exchanging message packets between nodes.

The packets are structured in a way to allow the ClusterNetwork to take a given packet, serialize it to a string of characters and send the string to a remote node. When received this string is de-serialized into a packet object again. This is accomplished by having a packet header, as seen in Figure 3.1, “Packet Structure”, that begins all packets. This header contains integers representing the packet type and the packet length. In order to allow the objects to be serialized and de-serialized into different formats we took advantage of the ObjectReader and ObjectWriter abstractions within Vapor, VR Juggler's system abstraction module. This enables us to serialize the packets into many different formats such as a string of characters or even an XML tree.

Figure 3.1. Packet Structure

The process of reading the header, determining the packet type, creating the packet structure, and de-serializing the data can be a very tedious task. Because of this, we have created a packet factory, seen in Figure 3.2, “Class Diagram: PacketFactory”, that automates this process. This simplifies code and eliminates one location of possible errors.

Figure 3.2. Class Diagram: PacketFactory

As you can see from Figure 3.3, “Class Diagram: ClusterNetwork”, the ClusterNetwork has been designed to have a very intuitive API without any reference to any other sub-systems of ClusterJuggler. The ClusterNetwork provides everything that a programmer needs to send messages between nodes in a cluster. It is also because of this robust design that it is possible to develop a replacement for the ClusterNetwork that might use a different low level networking technique.

Figure 3.3. Class Diagram: ClusterNetwork

A machine specific config element contains information about a particular node in the cluster. Fields include, but are not limited to, the local display system, display windows, host name, and accepting socket port.

Step 1) Create a new Machine Specific Config element for each cluster node.

Figure 3.4. Create New Machine-Specific Element

Step 2) Enter a unique name for the new node. Cluster nodes are referenced by name throughout the configuration process, this requires that each node have a unique name that will not interfere with any other node.

Figure 3.5. Enter Unique Name

Step 3) Create an embedded display system element

Figure 3.6. Create Display System

Step 4) Create an embedded Display Window. In order to display graphics on either the desktop or a projected surface, you must create an associated display window. These windows can then either contain a simulated view or a surface view port such as the front wall in a CAVE.

Figure 3.7. Create Display Window

Step 5) Enter details about this node.

Figure 3.8. Machine Info

Our goal is to design a distributed shared memory system for VR application I/O (input/output) data. As a result of distributed I/O, application development and execution can transparently move between shared memory VR systems and PC cluster VR systems. This means the same VR applications will run on both systems, without any changes needed to the application.

VR Juggler already has an input manager that handles local input data on a single computer. ClusterJuggler extends VR Juggler by using the RemoteInputManager plugin as an extension to the existing input manager. The RemoteInputManager provides a few key functions: device location transparency, eteroerogeneous clusters, support for platform specific device drivers, and an unlimited number of devices.

Device Location Transparency. Not only does the remote input manager have useful functionality, but it also intends to avoid forcing the application programmer to worry about the location of devices. Once a cluster is set up, the programmer and application can act as if each input device is connected to every cluster node(Figure 5.1, “Virtual Devices”). The remote input manager avoids putting extra burden on application developers and hides the complications of the cluster from them.

Figure 5.1. Virtual Devices

Heterogeneous Clusters. Also by distributing the input to cluster nodes across the network, we can remove the typical cluster computing constraint of having identical computers at each node. We are able to accomplish this very easily since our ClusterNetwork layer works across multiple platforms. This allows us to construct a cluster from any combination of the platforms supported by VR Juggler(Figure 5.2, “Heterogeneous Cluster”).

Figure 5.2. Heterogeneous Cluster

Support for Platform Specific Device Drivers. Although VR Juggler supports many platforms, new devices require their own drivers for each platform. This can restrict the use of new devices on VR systems because writing device drivers is time consuming. With the help of the RemoteInputManager, devices whose drivers have been integrated into VR Juggler on one platform can be utilized on other platforms as well. By connecting to another computer through the RemoteInputManager, other platforms can retrieve device data across the network until drivers for the specific platforms are finally integrated.

Unlimited Number of Devices. Also, if a large number of input devices are to be used, our design allows the devices to be spread across the cluster nodes. This prevents problems such as multiple input devices creating a large workload on a single computer, or all input hardware not being able to attach to a single PC.

Step 1) Find the device(s) that you want to share by browsing through the list of Config Elements under the Devices heading.

Figure 5.3. Find Device

Step 2) Select the cluster node from the drop down list in the Host Node property.

Figure 5.4. Set Device Host

Application Data Manager allows the application developer to share any arbitrary type of data. For example we might have a GUI running on a hand held device that interacts with the application to provide data to control the application. Since we can not expect this GUI to connect to all nodes in the cluster, the application data manager allows the hand held device to connect to a single machine which will synchronize this data across the cluster.

Figure 6.1. User Data

As you can see from Figure 6.1, “User Data”, you can create your own data structure and use two templated mixin classes in order to gain the needed interface to allow the Application Data Manager to handle you data.

Using the application data manager to share information within your application is very easy and powerful. You only need to do the following:

Once you have completed this simple tasks you will have a custom data type that is being synchronized across all cluster nodes behind the scenes. This can make the process of sharing the transformation matrix a simple task that can be implemented in 50 lines of code.

Example 6.1. Define Your Data Structure

  1 #include <vpr/IO/SerializableObject.h>
    struct MyType 
    { 
       int something; 
  5    float otherStuff;
       char stupid;
       bool drawBool;
    }; 
    
 10 template<class MyType> 
    vpr::ReturnStatus 
    vpr::SerializableObjectMixin<MyType>::writeObject(vpr::ObjectWriter* writer)  
    {
       writer->writeUint16(something); 
 15    writer->writeBool(drawBool); 
    }
       
    template<class MyType> 
    vpr::ReturnStatus
 20 vpr::SerializableObjectMixin<MyType>::readObject(vpr::ObjectReader* reader) 
    { 
       something = reader->readUint16();
       drawBool = reader->readBool(); 
    }

	Define the data type that you want to share across the cluster.
	Implement the function that will serialize the structure.
	Implement the function that will de-serialize the structure.

Example 6.2. Declare an Instance

cluster::UserData<vpr::SerializableObjectMixin<MyType> > mMyData;

Declare an instance of your data type called mMyData

Example 6.3. Initialize the Data Structure

vpr::GUID new_guid("d6be4359-e8cf-41fc-a72b-a5b4f3f29aa2"); 
        std::string hostname = "gfxn1"; 
        mMyData.init(new_guid, hostname);

	Create a new GUID to identify the new data.
	Store the hostname of the cluster node that will be responsible for updating the value of the data structure.
	Initialize the data structure to be shared across the cluster.

Example 6.4. Calculate the Value of the Data Structure

  1 if (mMyData.isLocal()) 
    {
       if(0 != mButton0->getData()) 
       { 
  5       mMyData->drawBool = true; 
       } 
       else
       {
          mMyData->drawBool = false; 
 10    } 
    }

	Determine if the data structure is local, the value is being updated by the local machine.
	Get the current state of Button0.
	Change the value of the shared data structure to match Button0.

Example 6.5. Use the Data Structure

if (mMyData->drawBool == true) 
{
   drawNetwork(); 
}

	Test if the value of the mMyData is true.
	If true then call a function that draws the representation of the cluster network.

The SwapLockPlugin gives the application developer a set of methods which they can choose from while configuring the application.

Step 1) Create a new SwapLockPlugin Config Element.

Figure 7.1. Create SwapLockPlugin

Step 2) Give the SwapLockPlugin a unique name.

Figure 7.2. Set Unique Name

Step 3) Set the TCP/IP port to use on the Sync Server for communications.

Figure 7.3. Set Sync Server Port

Step 4) Set the method of synchronization to use.

Figure 7.4. Set Sync Method

Step 5) Select the cluster node that should be responsible for keeping the cluster synchronized.

Figure 7.5. Set Sync Server