Project

General

Profile

ContainerFragment Design Notes

ContainerFragment is a simple Fragment where the payload is composed of packed Fragments. ContainerFragment provides vector-like accessors, implementing at() and operator[].

Old Implementation

The main feature of ContainerFragment is the metadata which provides an index of offsets to each Fragment in the Container.

struct MetadataV0
    {
        /**
         * \brief The maximum capacity of the ContainerFragment (in fragments)
         */
        static constexpr int CONTAINER_FRAGMENT_COUNT_MAX = 100;

        typedef uint8_t data_t; ///< Basic unit of data-retrieval
        typedef uint64_t count_t; ///< Size of block_count variables

        count_t block_count : 55; ///< The number of Fragment objects stored in the ContainerFragment
        count_t fragment_type : 8; ///< The Fragment::type_t of stored Fragment objects
        count_t missing_data : 1; ///< Flag if the ContainerFragment knows that it is missing data

        /// Offset of each Fragment within the ContainerFragment
        size_t index[CONTAINER_FRAGMENT_COUNT_MAX];

        /// Size of the Metadata object
        static size_t const size_words = 8ul + CONTAINER_FRAGMENT_COUNT_MAX * sizeof(size_t) / sizeof(data_t); // Units of Header::data_t
    };
    static_assert (sizeof(MetadataV0) == MetadataV0::size_words * sizeof(MetadataV0::data_t), "ContainerFragment::MetadataV0 size changed");
There are several disadvantages to this implementation:
  1. Container is fixed-size
  2. If fewer Fragments are in container, index is wasted space
  3. No indication in metadata what the maximum size was if changed from default

New Implementation

In the new implementation, the index is stored after the last Fragment, and the metadata simply has an offset to find the index:

    struct Metadata
    {
        typedef uint8_t data_t; ///< Basic unit of data-retrieval
        typedef uint64_t count_t; ///< Size of block_count variables

        count_t block_count : 16; ///< The number of Fragment objects stored in the ContainerFragment
        count_t fragment_type : 8; ///< The Fragment::type_t of stored Fragment objects
        count_t version : 4;
        count_t missing_data : 1; ///< Flag if the ContainerFragment knows that it is missing data
        count_t has_index : 1;
        count_t unused_flag1 : 1;
        count_t unused_flag2 : 1;
        count_t unused : 32;

        uint64_t index_offset;

        /// Size of the Metadata object
        static size_t const size_words = 16ul; // Units of Header::data_t
    };
    static_assert (sizeof(Metadata) == Metadata::size_words * sizeof(Metadata::data_t), "ContainerFragment::Metadata size changed");
    mutable const size_t* index_ptr_;
    mutable bool index_alloc_;
    mutable const Metadata* metadata_;
    mutable bool metadata_alloc_;

Several key members have been added to ContainerFragment to help with backwards compatibility. UpgradeMetadata takes a MetadataV0 and populates the relevant fields of a new Metadata struct, which is pointed to by metadta_. If the new metadata structure is allocated, metadata_alloc_ is set so that the ContainerFragment destructor can free it. Similarly, a pointer to the index (whether in the MetadataV0 or at the end of the Fragment data) is kept, with a flag to indicate whether this index was allocated on the heap or is part of the Fragment payload.

The index is terminated with a check word, defined as ContainerFragment::CONTAINER_MAGIC. The reset_index_pointer_ function checks for this check word, and if it is not preset, it will allocate a new index. (When reading an old ContainerFragment, reset_index_pointer_ should never be called, as the index_ptr is initalized to the index present in the metadata. The only time the check word is accessed is in reset_index_pointer_, so there is no danger of read buffer overruns.)

Several other helper functions have been added, namely get_index_, reset_index_pointer_, and create_index_, which abstract the business of managing the index. ContainerFragmentLoader does not have to concern itself with backwards compatibility, and uses these functions to ensure that the index is regenerated and put into the ContainerFragment at the appropriate location each time a Fragment is added to the Container.