CEDPS Milestones/CDC1

From CEDPS

CEDPS Data Challenge #1

Challenge date: December 2009
Description: 2 sites; move 10K x 100MB =1 TB within 3 days
Comments: A first test of Globus.org scalability, focusing on number of transfers. Performance requirements are modest: 30 megabit/sec, 0.04 transfers/sec.
Globus.org version: Pre-release
Number of users: 1
Number of files: 10,000
File size: 100MB
Total data volume: 976GB
Dataset: Synthetic (files generated with the "dd" command)
Source endpoints: ALCF
Destination endpoints: NERSC
Authentication: GSI
Degree of compression: none
Data integrity: Source and destination checksums compared after each run
Run details:
CDC1R1
CDC1R2

Findings:
The CEDPS team was successful in meeting its first challenge, as a single user was able to transfer the prescribed data in less than three days. Because our initial run finished so quickly, we decided to repeat it under more challenging conditions. In the first run the transfers were allowed to proceed without interference. During the second run we induced a transient failure by making the destination directory non-writeable for a 15-minute period. In both cases the files were successfully transferred without user intervention.

We note the second run took more than twice the time to complete as the first – well beyond 15-minute induced failure. We speculate that the difference was due to variations in ALCF’s system load (the first run occurred during an ALCF maintenance period), but no objective evidence of this was collected at the time. Indeed the amount of information typically available to non-privileged users about their wide-area, cross-domain transfers is quite limited.

Two issues were filed with the Globus.org development team as a result of this challenge:
Request for additional information about failed transfer attempts
Error count does not increase when a transfer attempt fails because the destination is readonly