Implementing custom checksum calculator¶
To implement custom checksum we will need to modify encp, EXfer, file db, pnfs layer 4 (possibly), mover.py, chimera.py, file_clerk.py.
Also other modules may need modification, which can be identified during project implementation.
Enstore database and file clerk.¶
The new parameter needs to get added to file table (record):
'crc_type'. It will define the name of the function (method) calculating checksum.
Currently we have only one type of checksum: adler32.
The possible values of "checksum_type" will be "adler32". "MD5", etc. (as needed).
The corresponding entry in the file table will be also checksum_type
The new table will be added to file db: crc_table containing crc related information:
crc_type, crc_value. This table will keep crc information (may have more than one row).
All file clerk methods obtaining / changing crc information will be changed to transparently retrieve / modify crc_table.
CRC values will be kept in the crc_table in string (or hexadecimal) format and will be converted appropriately to / from formats defined by
crc type: long for ADLER32 and string for MD5.
FTE estimate - 4 FTE weeks including testing and integration.
encp will have 2 additional command-line options: --complete-crc (this is not required by this project, but is a good helper).
--crc-type - tells what crc will be calculated / used.
EXfer.c needs to get modified to calculate crc based on checksum type. Currently we need to add MD5 CS calculator. Example code is attached.
FTE estimate - 4 FTE days
mover.py will be modified to calculate crc based on checksum type. Currently we need to add MD5 CS calculator.
FTE estimate - 3 FTE days