Project

General

Profile

Details on lock-free buffer management

The artdaq shared memory interface uses several flags in the buffer descriptor structure to achieve self-contained, lock-free operation.

  1. std::atomic<BufferSemaphoreFlags> sem
    • The state of the buffer. One of Empty, Writing, Full, or Reading.
  2. std::atomic<int16_t> sem_id
    • The current owner of the buffer. Each instance of SharedMemoryManager (readers and writers) takes an ID number (from ShmStruct::std::atomic<int> next_id), and uses this number to indicate that it is the current owner of the buffer. Buffers without owners have sem_id set to -1.
  3. std::atomic<uint64_t> last_touch_time
    • The last time (microseconds, system clock) the buffer was "touched". Buffers should be touched whenever a modifying operation occurs, such as writes or reads, or before the buffer transitions from an inactive state (Empty, Full) to an active state (Writing, Reading).
  • The checkBuffer_ function is used to determine if the buffer's sem and sem_id are currently set to the expected values. No read or write operations should be performed without calling this function.
  • sem should only be updated by the owner of the buffer, except in the timeout case where another manager (usually ID 0) notices that the buffer has not been touched for longer than the configured timeout interval.
  • Buffer touch time should only be updated by the owner of the buffer, except the case of unowned (sem_id -1) buffers which can be updated by anyone. Buffers may be touched when in any mode. The only negative side effect of a buffer being touched out-of-turn is that it may take longer for a timeout to occur.
    • The logic for resetting buffers from an active state to a passive state is in the ResetBuffer function. Additionally, when buffer read operations do not empty the buffer, ResetBuffer ensures that old broadcasts are removed from the shared memory once they have timed out.

Operations on buffers

Acquire a buffer for reading/writing

  1. Check sem_id -1
  2. Check sem for corresponding inactive state (Empty for writing, Full for reading)
  3. Touch buffer
  4. Update sem_id
  5. Check sem_id manager_id_, if failure, retry from start (max 5 tries)
  6. Update sem
  7. Check sem
  8. Check sem_id manager_id_, if failure, retry from start (max 5 tries)
  9. Touch buffer

Read/Write from buffer

  1. Check sem_id manager_id_
  2. Check sem Reading/Writing
  3. Touch buffer
  4. Read from/Write to buffer
  5. Touch buffer
  6. Check sem_id manager_id_ and sem Reading/Writing. Return false if not the case

Check buffer for timeout in non-broadcast mode

  1. Read last_touch_time and calculate delta from current time
  2. IF delta < timeout OR sem Empty return false
  3. IF sem_id manager_id_ AND sem Writing return true <-- This should never happen!
  4. IF sem_id != manager_id_ AND sem Reading THEN
    1. Re-calculate delta
    2. IF delta < timeout return false
    3. return true
  5. return false

Check buffer for timeout in broadcast mode

  1. Read last_touch_time and calculate delta from current time
  2. IF delta < timeout OR sem Empty return false
  3. IF sem_id manager_id_ AND sem Writing return true <-- This should never happen!
  4. IF sem Full AND (manager_id_ 0 OR buffer already seen) THEN
    1. Reset buffer to Empty state
    2. return true
  5. IF sem_id != manager_id_ AND sem Reading THEN
    1. Re-calculate delta from current time and last touch time
    2. IF delta < timeout return false
    3. Reset buffer to Full state
    4. return true
  6. return false