CEDPS Milestones/CDC2

From CEDPS

CEDPS Data Challenge #2

Challenge date: February 2010
Description: 2 sites; move 100K x 100MB within 3 days
Comments: Increased throughput requirements (0.4 transfers/second)
Globus.org version: Pre-release
Number of users: 1
Number of files: 100,000
File size: 100MB
Total data volume: 9.5TB
Dataset: Synthetic (files generated with the "dd" command)
Source endpoints: ALCF
Destination endpoints: PADS (run 1), GFDL (run 2)
Authentication: GSI
Degree of compression: none
Data integrity: Source and destination checksums compared after each run (in-progress)
Run details:
CDC2R1
CDC2R2
Findings:
The CEDPS team was successful in meeting its second data challenge, as a single user was able to transfer the prescribed data in less than three days. We conducted two runs, one from ALCF to PADS and another from ALCF to GFDL. In both cases the files were successfully transferred without user intervention.

Five issues were filed with the Globus.org development team as a result of this challenge:
summary command takes 5 minutes to describe group containing 100k requests
bad xfer_times returned by status command
Status command on group containing 11K requests took 73 minutes
1hr40mins to submit 100K requests
Database Contention from Lock held by Idle Transaction