Overview¶

Most features of the XR Recorder generally fall into one of two categories:

• Features for Capturing

• Features for Playback

To start though, we will go through the one feature that is used in both sections.

File Format¶

In this section we’ll go over the file format used by the XR Recorder, the XREC file format.

XREC is an extensible binary format, which can store any type that can be defined in .NET. The high level structure of the file is presented below:

The file is divided into 3 different sections:

• The start packet: Contains metadata regarding the recording

  • Version of the application

  • Compression hints

  • Version/Revision of the XREC file format.

• Multiple event packets: Contain the captured events during the recording

• The end packet: Mainly indicates that the recording finished without errors;

  • It can also contain additional metadata.

Each packet comprises of two parts:

• The Header: Contains the packet type, as well as the size (in bytes) of the data section.

• The Payload: Depends on the packet type

  • Start Packet: Contains begin timestamp the metadata

  • Event Packet: Contains event data, explained in the next section

  • End Packet: Contains end timestamp and additional metadata

Event Packet¶

An event packet encapsulates information from a section of the recording. It’s size is not based on some set temporal interval, but rather on the amount of data that can be stored in a single packet.

• Frame Begin: The earliest frame of one of the captured events in the packet

• Frame End: The latest frame of one of the captured events in the packet

• RTSS Begin: The real-time since startup of the earliest frame in the packet

• RTSS End: The real-time since startup of the latest frame in the packet

• Captured Events Array: An array of captured events that occurred between RTSS Begin and RTSS End

The reason for which there is no set frequency for the captured events is that the amount of events generated during a session can vary from second to second. One second might not have many events, but the next second (where, for example one or more actions are performed) can have a lot of events. As a result, reading a packet from disk would be unpredictable regarding the amount of data it would contain.

Instead, specific utilities have been created to parse and interpret the data in the XREC file. These utilities will be explained at the end of this manual page.

Captured Events¶

Captured events are the lifeblood of the XR Recorder. They are meant to be POD (“Plain Old Data”) containers that designate an important change in the scene. What is important to keep in mind when deciding what events need to be captured is the concept of redundancy.

Redundant events occur when we capture events that are the direct (or indirect) result of other events. For instance, capturing a button turning green, while the hand that’s pressing it is already captured, would be an arguably redundant event. As such, the XR Recorder generally aims to capture the most important events: movement, interaction, joining/leaving sessions, etc… But it does provide the necessary interfaces and classes so that users can add support for their own custom features, as seen here.

A captured event must:

1. Implement the interface ICapturedEvent

2. Be a serializable class type (i.e. [System.Serializable])

Captors¶

Captors are classes that are instantiated at the start of the recording, and are responsible for collecting events into the output file. Any given captor must:

1. Implement the interface ICaptor

2. Have a top-level constructor that requires zero arguments.

Captors can generally be polled each frame to record changes, or can be event based. Each and every captor is initialized at the start of the recording and the order in which they are initialized is not guaranteed.

Interpreters¶

Interpreters, as the name implies can interpret one or more captured events. Any interpreter must:

1. Implement the interface IInterpreter.

2. Have a top-level constructor that requires zero arguments.

3. Have a InterpretsEvent attribute with a list of captured event types.

Interpreters implement the same function as most Unity MonoBehaviours, the difference being that the interpreters, just like captors, are not located in the scene, and are instead managed by the runtime of the recorder module.

It is important to note that the function Interpret will not be called at the exact time at which the event supplied is executed, but it is usually called at most 1 second before that event happens. This is done for interpolation reasons. A function is supplied that can be used to schedule an action to happen at a specific cursor position: Hub.Instance.Get<RecorderModule>().PlayerInstance.Schedule(action, time).

Writing/Reading XREC Files¶

To work with XREC files, we provide the MAGES.Recorder.Container class. The container class is used to parse (or write to) an XREC file and is the main entry point for working with the XR Recorder on a low-level.

Open a file for reading or writing¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

// using MAGES.Recorder;
// ...

// Open a file for writing
{
    string fileName = "myfile.xrec";
    FileAccess fileAccess = FileAccess.Write;
    var container = new Container(fileName, fileAccess);
    container.Close();
}

// Open a file for reading
{
    string fileName = "myfile.xrec";
    FileAccess fileAccess = FileAccess.Read;
    var container = new Container(fileName, fileAccess);
    container.Close();
}

Write a simple XREC file¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// using MAGES.Recorder;
// ...

Container c = new Container("myFile.xrec", System.IO.FileAccess.Write);

// Write Start Packet
c.Write(new StartPacket(1, CompressionMethod.None, 0, 0.0));

// Create & Write Event Packet
var evt = new EventPacket(new ICapturedEvent[]
{
    // Equivalent to: GameObject.Find("Scene/Can").transform.position = new Vector3(0, 0, 0);
    new PositionChangeEvent("Scene/Can", new Vector3(0, 0, 0), 1, 1.0, isLocal: false),

    // Equivalent to: GameObject.Find("Scene/Can").transform.position = new Vector3(0, 2, 0);
    new PositionChangeEvent("Scene/Can", new Vector3(0, 1, 0), 2, 2.0, isLocal: false),
});
c.Write(evt);

c.Close();

Read an XREC file¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

// using MAGES.Recorder;
// ...

Container c = new Container("myFile.xrec", System.IO.FileAccess.Write);

// Read all packets
while (c.Read(out Packet packet))
{
    switch (packet.Type)
    {
        case PacketType.Event:
            var evt = packet as EventPacket;

            // Read all captured events for the event packet
            foreach (var e in evt.CapturedEvents)
            {
                var capturedEvent = e;
                if (capturedEvent is PositionChangeEvent pce)
                {
                    Debug.Log($"PositionChangeEvent: {pce.Path} {pce.Position}");
                }
            }
            break;
        case PacketType.Start:
            break;
        case PacketType.End:
            break;
    }
}

c.Close();

Build Interval Tree¶

In the sample shown above, we parse all of the events, one-by-one. But what if we wanted to query/search for events that happen during a specific timeframe? This is where the IntervalTree comes in. The IntervalTree is a data structure that allows for fast querying of events that happen during a range of keys.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// using MAGES.Recorder;
// ...

Container c = new Container("myFile.xrec", System.IO.FileAccess.Write);
IntervalTree<double, Packet> tree = new IntervalTree<double, Packet>();

while (c.Read(out Packet packet))
{
    switch (packet.Type)
    {
        case PacketType.Event:
            var eventPacket = (EventPacket)packet;
            tree.Add(eventPacket.RtssBegin, eventPacket.RtssEnd, eventPacket);
            break;
        case PacketType.Start:
            var startPacket = (StartPacket)packet;
            tree.Add(0.0, 0.0, startPacket);
            break;
        case PacketType.End:
            var endPacket = (EndPacket)packet;
            tree.Add(endPacket.Rtss, endPacket.Rtss, endPacket);
            break;
    }
}

Query Interval Tree¶

1
2
3
4
5
6
7

// using MAGES.Recorder;
// ...

IntervalTree<double, Packet> tree = new IntervalTree<double, Packet>();

// Build Interval tree...
List<Packet> packets = tree.Q(0.0, 1.0); // Returns all packets that happen between 0.0 and 1.0

Utilities¶

Game Object Path Cache¶

The GameObjectPathCache is functionality on top of GameObject.Find that can cache results and also resolve custom variables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

// using MAGES.Recorder;
// ...

// Get the currently used path cache.
var pathCache = Hub.Instance.Get<RecorderModule>().PathCache;

// Equivalent of GameObject.Find("Scene/Can")
var go = pathCache.Get("Scene/Can");

// Set a custom variable
pathCache.SetVariable("#my_id", go);

// Get a variable
Debug.Log($"{pathCache.Get("#my_id") == go}"); // True

Transform Capture Group¶

The TransformCaptureGroup is a utility class that can be used to capture the position, rotation, and scale of a Transform component. It can be configured to capture local or world space values.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

// using MAGES.Recorder;
// ...

GameObject objectToRecord = GameObject.Find("Scene/Can");
var captureGroup = new TransformCaptureGroup();
captureGroup.Add(
    objectToRecord,
    flags: TransformCaptureGroup.CaptureMethod.Transform,
    varName: "customVarName");

// ... In Update

// Poll for changes in the capture group, and send them to the recorder with a timestamp of 1.0
captureGroup.Poll(
    (ICapturedEvent evt) =>
    {
        Hub.Instance.Get<RecorderModule>().RecorderInstance.Add(evt);
    },
    0,
    1.0);

Write Packets with C# streamed I/O¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// using MAGES.Recorder;
// ...

var fileStream = new FileStream("file.xrec", FileMode.Create, FileAccess.Write);
var bufferedStream = new BufferedStream(fileStream);

// Write Start Packet
StartPacket startPacket = new StartPacket(1, CompressionMethod.None, CurrentFrame, Time.realtimeSinceStartupAsDouble);
Utilities.WritePacket(startPacket, bufferedStream);

// Write End Packet
Utilities.WritePacket(new EndPacket(12.0), bufferedStream);

// Close the streams
bufferedStream.Close();
fileStream.Close();

Playback Transformations¶

The ContinuousPlaybackObject is a utility class that can be used to playback a series of events that have a continuous value. It can be configured to use a custom interpolation function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

// using MAGES.Recorder;
// ...

// Create a new playback object with a custom interpolation function
var newPlaybackObject = new ContinuousPlaybackObject<FlaggedValue<Vector3>>(YourCustomInterpolationFunction);

// Add a sample in time
bool isLocal = false;
var position = new Vector3(1, 0, 0);
newPlaybackObject.AddSampleInTime(new FlaggedValue<Vector3>() { Flag = isLocal, Value = position }, 1.0);

// Calculate the new position at time 0.5
var newPosition = newPlaybackObject.Update(0.5);