One approach to implementing clustering for interactive graphics applications is to share all data received from input devices. This is based on an assumption that the interaction with the computer graphics will occur through input devices such as 6DOF trackers, pointing devices, etc. Using this approach, a distinct copy of the VR application is run on each cluster node, but they all see the same input data. Since changes to the scene are based on information from the input devices, all the nodes will make the same state changes each frame and therefore remain synchronized.

This capability is implemented through the Gadgeteer Remote Input Manager and the RIMPlugin used by the Cluster Manager. The basic goal of these components is to provide a distributed shared memory system for VR input device data. Through shared input data, applications can migrate transparently between shared memory VR systems and PC cluster VR systems.

Shared input data is the easiest VR Juggler clustering feature that can be utilized by application object programmers. In simple cases, nothing about an application will have to change to take advantage of shared input data because the details are hidden within the VR Juggler configuration. The Cluster Manager simply needs to be configured to load the RIMPlugin and the Start Barrier Plug-in (StartBarrierPlugin) to enable applications to take advantage of shared input data.

When input data sharing is not sufficient to enable a VR Juggler application to run on a cluster, the next option is to use application-specific shared data. Using this option, VR application developers can easily exchange any type of data across a cluster of machines. For example, we might have a GUI running on a hand-held device that interacts with the VR application to control it. We cannot expect this GUI to connect to all nodes in the cluster. Instead, the GUI connects to a single node that accepts the commands from the GUI and then relays them to the rest of the cluster nodes using application-specific shared data.

Application-specific shared data is implemented through the generic (templated) container type cluster::UserData<T> and the Application Data Manager plug-in (ApplicationDataManager in the Cluster Manager config element plug-in list). As is typically the case with generic containers, any sort of data can be stored and therefore shared. The only caveat is that the contained type must have the following two methods:

Respectively, these two methods are used for serializing and de-serializing shared data types. The simplest way to achieve this is to create a data structure that is a subclass of the abstract type vpr::SerializableObject and overriding its pure virtual writeObject() and readObject() methods.

Deriving from vpr::SerializableObject is not viable in all cases, however. If the data that must be shared is of a type defined in a third-party library, it cannot be modified to derive from vpr::SerializableObject. In that case, the type vpr::SerializableObjectMixin<T> can be used. The methods vpr::SerializableObjectMixin<T>::writeObject() and vpr::SerializableObjectMixin<T>::readObject() must be specialized for the desired type T.

In either case, the end result is a means to serialize the data to be shared across the cluster, and application object programmers will implement methods named writeObject() and readObject(). When writeObject() is invoked, it is passed a pointer to a vpr::ObjectWriter object. An object writer is a simple wrapper around an expandable block of memory. Each write operation appends some number of bytes to the memory block based on the size of the data written. The type vpr::ObjectWriter provides methods for writing all the basic C++ data types (int, float, bool, char, etc.), though they are named based on the cross-platform type identifiers provided by the VR Juggler Portable Runtime (vpr::Int32, vpr::Uint8, etc.). Byte ordering (endian) issues are handled internally by the object writer. The implementation of writeObject() for any shared data type simply copies the data members of the shared data structure into the object writer.

Inversely, the implementation of readObject() reads data from an object reader (an instance of vpr::ObjectReader) into the local copy of the data structure. The object reader contains a fixed-size block of memory and a pointer to the current location in that memory block. Each read operation moves the pointer some number of bytes in the memory block based on the size of the data read.

We now present two examples of using the serializable object concept. The first demonstrates the case when a new data structure can be created; the second is the case when a third-party type must be made serializable. When we make a new data structure, it is quite easy to enable serialization. Consider the basic type shown in Example 6.1, “Declaration of a Serializable Type”. It derives from vpr::SerializableObject and overrides writeObject() and readObject() just as it must. It has three data members of different types that define the state of an instance of our type. The serialization and de-serialization implementation, which is quite straightforward, is shown in Example 6.2, “Serializing an Application-Specific Type”.

#include <vpr/IO/SerializableObject.h>

class MyType : public vpr::SerializableObject
{
public:
   vpr::ReturnStatus writeObject(vpr::ObjectWriter* writer);
   vpr::ReturnStatus readObject(vpr::ObjectReader* reader);
   // Other public methods ...

private:
   unsigned int mIntData;
   char         mByteData;
   float        mFloatData;
};

vpr::ReturnStatus MyType::writeObject(vpr::ObjectWriter* writer)
{
   writer->writeUint32(mIntData);
   writer->writeInt8(mByteData);
   writer->writeFloat(mFloatData);
   return vpr::ReturnStatus();
}

vpr::ReturnStatus MyType::readObject(vpr::ObjectReader* reader)
{
   mIntData   = reader->readUint32();
   mByteData  = reader->readInt8();
   mFloatData = reader->readFloat();
   return vpr::ReturnStatus();
}

Now we consider the case when we need to serialize a third-party type. First, let us assume that we have a type, called SomeType, defined in a header file from a third-party C++ library. This is shown in Example 6.3, “Sample Third-Party Type”. The type has three accessor methods for reading its data and three for writing. We can then specialize the methods of vpr::SerializableObjectMixin<T> as shown in Example 6.4, “Serializing a Third-Party Type Using vpr::SerializableObjectMixin<T>”.

class SomeType
{
public:
   unsigned int getIntData();
   void setIntData(unsigned int v);

   char getByteData();
   void setByteData(char v);

   float getFloatDat();
   void setFloatData(float v);

private:
   // Private data ...
};

#include <vpr/IO/SerializableObject.h>

template<>
vpr::ReturnStatus
vpr::SerializableObjectMixin<SomeType>::
   writeObject(vpr::ObjectWriter* writer)
{
   writer->writeUint32(getIntData());
   writer->writeInt8(getByteData());
   write->writeFloat(getFloatData());
   return vpr::ReturnStatus();
}

template<>
vpr::ReturnStatus
vpr::SerializableObjectMixin<SomeType>::
   readObject(vpr::ObjectReader* reader)
{
   setIntData(reader->readUint32());
   setByteData(reader->readInt8());
   setFloatData(reader->readFloat());
   return vpr::ReturnStatus();
}

The magic of vpr::SerializableObjectMixin<T> allows the specialized methods to behave as member functions in SomeType. This means that the specialized members have easy access to all public and protected members of SomeType.

Now that we have data serialization out of the way, we can turn our attention to the use of cluster::UserData<T>, the special type that automates application-specific data sharing. For each type of shared data, the application object will have at least one instance of cluster::UserData<T>. Example instantiations of cluster::UserData<T> are shown in Example 6.5, “Declaring Instances of cluster::UserData<T>”.

#include <vrj/Draw/OGL/GlApp.h>
#include <plugins/ApplicationDataManager/UserData.h>
#include <SomeType.h>
#include "MyType.h"

class AppObject : public vrj::GlApp
{
public:
   void init();
   void preFrame();
   void latePreFrame();
   void draw();

   // Other public member functions ...

private:
   cluster::UserData<MyType> mMyTypeObj;
   cluster::UserData< vpr::SerializableObjectMixin<SomeType> > mSomeTypeObj;
};

Next, we must initialize the cluster::UserData<T> instances so that the Application Data Manager plug-in can identify the shared data types and so that the application can determine which cluster node will be allowed to write to the shared data. While there are two ways to do this, we will show only the recommended approach here. First, a globally unique identifier (GUID) must be defined for each and every shared data type instance. The command-line utility uuidgen is available on most operating systems for generating new GUIDs (also known as universally unique identifiers or UUIDs). These will be used in the application object init() method, as shown in Example 6.6, “Initializing Application-Specific Shared Data”.

void AppObject::init()
{
   vpr::GUID mytype_guid("99CFD306-32AB-11D9-A963-000D933B5E6A");
   mMyTypeObj.init(mytype_guid);

   vpr::GUID sometype_guid("A154B8E8-32AB-11D9-B4C9-000D933B5E6A");
   mSomeTypeObj.init(sometype_guid);
}

In conjunction with this, two config elements need to be created. These will be used by the Application Data Manager plug-in to identify which cluster node will be the writer node. The “guid” properties must match the string values used to initialize the vpr::GUID objects in Example 6.6, “Initializing Application-Specific Shared Data”. The “hostname” properties set the name of the cluster node that will be the shared data writer. An example of this is shown in Example 6.7, “Application-Specific Shared Data Configuration”.

<?xml version="1.0" encoding="UTF-8"?>
<?org-vrjuggler-jccl-settings configuration.version="3.0"?>
<configuration
   xmlns="http://www.vrjuggler.org/jccl/xsd/3.0/configuration"
   name="Example Shared Application Data Configuration"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.vrjuggler.org/jccl/xsd/3.0/configuration http://www.vrjuggler.org/jccl/xsd/3.0/configuration.xsd">

   <elements>
      <application_data name="MyType Shared Data" version="1">
         <guid>99CFD306-32AB-11D9-A963-000D933B5E6A</guid>
         <hostname>machine1</hostname>
      </application_data>
      <application_data name="SomeType Shared Data" version="1">
         <guid>A154B8E8-32AB-11D9-B4C9-000D933B5E6A</guid>
         <hostname>machine1</hostname>
      </application_data>
   </elements>
</configuration>

Now that the shared data is initialized and ready to use, we can write to and read from it—the Application Data Manager plug-in will take care of the rest. Only one node can be allowed to write to the data. This is determined through the use of the method cluster::UserData<T>::isLocal(). This method returns a Boolean value that indicates whether the data is “local.” The local node is the one named in the configuration element, as shown earlier. Writes to shared data should only occur in preFrame() or postFrame() after testing the result of cluster::UserData<T>::isLocal(). This is shown in Example 6.8, “Writing to Application-Specific Shared Data”.

void AppObject::preFrame()
{
   if ( mMyTypeObj.isLocal() )
   {
      // Computations ...
      mMyTypeObj->setIntData(...);
      mMyTypeObj->setByteData(...);
      mMyTypeObj->setFloatData(...);
   }

   if ( mSomeTypeObj.isLocal() )
   {
      // Computations ...
      mSomeTypeObj->setIntData(...);
      mSomeTypeObj->setByteData(...);
      mSomeTypeObj->setFloatData(...);
   }
}

After the snapshot of the application-specific shared data for the current frame has been distributed to the cluster nodes, it is time to read the shared data and set up the application state for rendering the current frame. This should be done in the application object method latePreFrame() or in draw(). This is demonstrated in Example 6.9, “Reading from Application-Specific Shared Data in latePreFrame()” and in Example 6.10, “Reading from Application-Specific Shared Data in draw()”. All the nodes in the cluster will read the results of the computations made in preFrame() and set up the application state before rendering. In general, there should be no need to use cluster::UserData<T>::isLocal() at this point.

void AppObject::latePreFrame()
{
   mStateVar1 = mMyTypeObj->getIntData();
   mStateVar2 = mMyTypeObj->getByteData();
   // And so on ...
}

void AppObject::draw()
{
   int  state_var1 = mMyTypeObj->getIntData();
   char state_var2 = mMyTypeObj->getByteData();
   // And so on ...
   // Render the scene ...
}

The choice of which method to use depends on the application type and on the data flow of the application object. Scene graph-based application objects will not have a draw() method, so latePreFrame() must be used. For application object types not based on a scene graph (currently only vrj::GlApp), there is a trade off to consider. If latePreFrame() is used, then the rendering state information must be stored in member variables of the application class. If draw() is used, then the state can be defined using stack variables within the method, but the additional function call overhead and pointer derference (resulting from the use of the Smart Pointer pattern) could impact the application frame rate. Remember that draw() is invoked for every window and every viewport within each window. The number of calls to draw() increases further when stereoscopic rendering is enabled. If latePreFrame() is used instead, the Smart Pointer overhead will only be exhibited once per frame.

With the tools for VR Juggler cluster programming in hand, we can turn our attention to specific, higher level areas that must be handled carefully when writing applications that may run on a graphics cluster.

static vpr::Interval last_frame; 
vpr::Interval current_frame = mHead->getTimeStamp();
vpr::Interval diff(current_frame - last_frame);
last_frame = current_frame; // You can get the delta in microseconds from 
                            // vpr::Uint64 delta = diff.usecs();

In this final section, we present some frequently asked questions regarding VR Juggler application programming and clustering.