1. Overview of hdmf-common

hdmf-common defines common data structures to be used across applications.

1.1. DynamicTable

The DynamicTable type is used to store tabular data. The tables are created in a columnar fashion with each column stored in its own VectorData object. Rows of the table are assigned unique ids with the required id column of type ElementIdentifier. The colnames attribute indicates the order of the columns.

1.2. VectorData

VectorData is the datatype used to store a column in a DynamicTable. If unpaired with a VectorIndex object the first dimension is the row dimension, which must be the same across all of the columns in that DynamicTable.

1.3. Ragged Arrays

(also known as Jagged Arrays)

Sometimes, you want to have a 2-d array where each row of the array has a different number of elements. For instance, in neuroscience, when storing the action potential times of sorted neurons, you might want to store them as a neuron x times matrix, but the problem is that each neuron will have a different number of spikes, so the second dimension will be inconsistent.

ragged array goal

There are a number of possible solutions to this problem. Some solve it by NaN-padding the array. You might want to store the spike times of each neuron in a separate dataset, but that will not scale well if you have many neurons. In HDMF, you would store this using a pair of objects a VectorData and a VectorIndex object. The VectorData array holds all of the data concatenated as a 1-d array, and it is paired with a link to a VectorIndex object that indexes the data, forming a map between the rows of the ragged array and the indices of VectorData.

ragged arrays in HDMF

These objects are generally stored inside a DynamicTable, and the elements of VectorIndex map onto the rows of the table. The VectorData object may be n-dimensional, but only the first dimension is ragged.

2. Experimental data structures

The following data structures are currently available under the HDMF-experimental schema. These are subject to change! They are not guaranteed to exist in the future nor maintain backward compatibility.

2.1. ExternalResources

The ExternalResources type is used to store references to data stored in external, web-accessible databases. This information is maintained using four row-based tables.