Project

General

Profile

Details on lock-free buffer management » History » Version 11

Eric Flumerfelt, 10/17/2018 03:22 PM

1 1 Eric Flumerfelt
h1. Details on lock-free buffer management
2 1 Eric Flumerfelt
3 1 Eric Flumerfelt
The _artdaq_ shared memory interface uses several flags in the buffer descriptor structure to achieve self-contained, lock-free operation. 
4 1 Eric Flumerfelt
5 1 Eric Flumerfelt
# std::atomic<BufferSemaphoreFlags> sem
6 1 Eric Flumerfelt
** The state of the buffer. One of Empty, Writing, Full, or Reading.
7 1 Eric Flumerfelt
# std::atomic<int16_t> sem_id
8 9 Eric Flumerfelt
** The current owner of the buffer. Each instance of SharedMemoryManager (readers and writers) takes an ID number (from ShmStruct::std::atomic<int> next_id), and uses this number to indicate that it is the current owner of the buffer. Buffers without owners have sem_id set to -1.
9 1 Eric Flumerfelt
# std::atomic<uint64_t> last_touch_time
10 1 Eric Flumerfelt
** The last time (microseconds, system clock) the buffer was "touched". Buffers should be touched whenever a modifying operation occurs, such as writes or reads, or before the buffer transitions from an inactive state (Empty, Full) to an active state (Writing, Reading).
11 2 Eric Flumerfelt
12 1 Eric Flumerfelt
* The checkBuffer_ function is used to determine if the buffer's sem and sem_id are currently set to the expected values. No read or write operations should be performed without calling this function.
13 9 Eric Flumerfelt
* sem should only be updated by the owner of the buffer, except in the timeout case where another manager (usually ID 0) notices that the buffer has not been touched for longer than the configured timeout interval.
14 6 Eric Flumerfelt
* Buffer touch time should only be updated by the owner of the buffer, except the case of unowned (sem_id == -1) buffers which can be updated by anyone. Buffers may be touched when in any mode. The only negative side effect of a buffer being touched out-of-turn is that it may take longer for a timeout to occur.
15 1 Eric Flumerfelt
** The logic for resetting buffers from an active state to a passive state is in the ResetBuffer function. Additionally, when buffer read operations do not empty the buffer, ResetBuffer ensures that old broadcasts are removed from the shared memory once they have timed out.
16 6 Eric Flumerfelt
17 6 Eric Flumerfelt
h2. Operations on buffers
18 6 Eric Flumerfelt
19 6 Eric Flumerfelt
h3. Acquire a buffer for reading/writing
20 6 Eric Flumerfelt
21 7 Eric Flumerfelt
# Check @sem_id == -1@
22 6 Eric Flumerfelt
# Check sem for corresponding inactive state (Empty for writing, Full for reading)
23 6 Eric Flumerfelt
# Touch buffer
24 6 Eric Flumerfelt
# Update sem_id
25 11 Eric Flumerfelt
# Check @sem_id == manager_id_@, if failure, retry from start (max 5 tries)
26 6 Eric Flumerfelt
# Update sem
27 6 Eric Flumerfelt
# Check sem
28 11 Eric Flumerfelt
# Check @sem_id == manager_id_@, if failure, retry from start (max 5 tries)
29 6 Eric Flumerfelt
# Touch buffer
30 6 Eric Flumerfelt
31 6 Eric Flumerfelt
h3. Read/Write from buffer
32 6 Eric Flumerfelt
33 7 Eric Flumerfelt
# Check @sem_id == manager_id_@
34 7 Eric Flumerfelt
# Check @sem == Reading/Writing@
35 6 Eric Flumerfelt
# Touch buffer
36 1 Eric Flumerfelt
# Read from/Write to buffer
37 1 Eric Flumerfelt
# Touch buffer
38 1 Eric Flumerfelt
39 1 Eric Flumerfelt
h3. Check buffer for timeout in non-broadcast mode
40 1 Eric Flumerfelt
41 7 Eric Flumerfelt
# Read last_touch_time and calculate delta from current time
42 8 Eric Flumerfelt
# IF @delta < timeout@ OR @sem == Empty@ *return false*
43 8 Eric Flumerfelt
# IF @sem_id == manager_id_@ AND @sem == Writing@ *return true* <-- This should never happen!
44 7 Eric Flumerfelt
# IF @sem_id != manager_id_@ AND @sem == Reading@ THEN
45 7 Eric Flumerfelt
## Re-calculate delta
46 8 Eric Flumerfelt
## IF @delta < timeout@ *return false*
47 8 Eric Flumerfelt
## *return true*
48 8 Eric Flumerfelt
# *return false*
49 7 Eric Flumerfelt
50 1 Eric Flumerfelt
h3. Check buffer for timeout in broadcast mode
51 7 Eric Flumerfelt
52 7 Eric Flumerfelt
# Read last_touch_time and calculate delta from current time
53 8 Eric Flumerfelt
# IF @delta < timeout@ OR @sem == Empty@ *return false*
54 8 Eric Flumerfelt
# IF @sem_id == manager_id_@ AND @sem == Writing@ *return true* <-- This should never happen!
55 7 Eric Flumerfelt
# IF @sem == Full@ AND (@manager_id_ == 0@ OR buffer already seen) THEN
56 7 Eric Flumerfelt
## Reset buffer to Empty state
57 8 Eric Flumerfelt
## *return true*
58 7 Eric Flumerfelt
# IF @sem_id != manager_id_@ AND @sem == Reading@ THEN
59 7 Eric Flumerfelt
## Re-calculate delta
60 8 Eric Flumerfelt
## IF @delta < timeout@ *return false*
61 7 Eric Flumerfelt
## Reset buffer to Full state
62 8 Eric Flumerfelt
## *return true*
63 8 Eric Flumerfelt
# *return false*
64 3 Eric Flumerfelt
65 5 Eric Flumerfelt
!buffer_state_diagram.png!