[Reagan Moore]

 

Papers

An essential component of trustworthiness is the ability to verify integrity of files. This includes that the requisite number of replicas has been made, and that the checksum of each file is verified periodically. The attached paper demonstrates an integrity verification rule using the iRODS rule language. The rule is analyzed to extract generic operational components that are required for production use. These include:

1. Verify all input parameters for consistency.

2. Query the iRODS metadata catalog to retrieve information about the number of files in a collection, their sizes, and the location of replicas.

3. Verify the integrity of each file in a collection by comparing the saved checksum with a new evaluation of each checksum. This requires reading each file.

4. Update all replicas to the most recent version.

5. Minimize the load on production services. The average storage system I/O rate needs to be as small as possible. We implement a deadline scheduler to ensure that the checksums are calculated at the slowest possible rate to meet the specified deadline.

6. Differentiate between the logical name for a file and the physical replica locations.

7. Identify all missing replicas and document their absence.

8. Create new replicas to replace missing replicas.

9. Implement load leveling to distribute the new replicas across the storage systems that are being used to support the collection.

10. Create a log file that records all repair operations performed upon the collection.

11. Track progress of the policy execution.

12. Initialize the rule for the first execution. This includes initializing variables, setting up a directory in which log files are stored, calculating the required I/O rate to meet the deadline, creating collection status flags for tracking progress, identifying the storage systems that are being used for the collection replicas, and verifying that the number of storage systems is greater than or equal to the number of required replicas.

13. Enable restart of the process from the last set of checked files in case of a system halt.

14. Manipulate files in batches of 256 files at a time to handle arbitrarily large collections.

15. Minimize the number of sleep periods used by the deadline scheduler. This is set by specifying a minimum amount of time to sleep when the execution rate is too fast.

16. Include the checking of new files that have been added during the execution of the policy if a restart is needed.

17. Write out statistics about the effective execution rate, and the number of files checked.

http://centosext1.irods.renci.org/irodswiki/index.php/Special:IrodsMetad…

H W Odum Institute for Research in Social Science has a policy set implemented within the LOCKSS preservation network. The SafeArchive system is used for auditing the preservation network. It checks, replication,geographic distribution,currency,replication effeciency, storage allocation, data usage etc

Jonathan Crabtree Assistant Director of Computing and Archival Research HW Odum Institute for Research in Social Science

iRODS

The attached file lists the integrity verification rule. The rule can serve as a template for evaluating other properties of a collection.

Rules: http://centosext1.irods.renci.org/irodswiki/index.php/Special:IrodsMetadataPage?filename=/renci/home/wiki/6/63/Verify-replica.r

Discussion