Snippets
From CEDPS
Contents |
Speeding File Transfers
December 2007
Many applications work with thousands of small (less than 100 MB) files. A team of scientists funded by the SciDAC CEDPS center and led by Raj Kettimuthu and John Bresnahan, both of Argonne National Laboratory, have been addressing the problem of speeding the transfer of such small files. Previously, GridFTP provided very fast performance for large files, but performance dropped precipitously for files under 500 MB.
One reason is that GridFTP is a command response protocol.That is, a client sends one command and then waits for a "Finished response" before sending another. Adding this overhead on a per- file basis for a large dataset partitioned into many small files makes the performance suffer. The SciDAC researchers have now pipelining. This approach allows the client to have many outstanding, unacknowledged transfer commands at once. Instead of being forced to wait for the "Finished response" message, the client is free to send transfer commands at any time. The approach all but eliminates the file transfer channel startup overhead and thereby enables bundles of small files to transfer as fast as single large files. The technique, which has been implemented in the Globus Toolkit 4.1.2 release, already has proven useful to researchers in Argonne's ADvanced Photon Source for transferring beamline users data in near-real time to a compute-intensive cluster at the TeraGrid at the University of Chicago/Argonne Computation Institute.
Running Remote Applications Easier
December 2007
The CEDPS-funded Remote Application Virtualization Infrastructure (RAVi) project, lead by Argonne researcher R. Madduri, is working with the APS to allow them to run applications over the TeraGrid resources without the end user having to write a single line of code. RAVi provides GUI- based tools to guide the user through the process of identifying an application, mapping from strongly typed Web Services operations to application arguments, defining authentication and authorization requirements, and deploying a service onto an execution site. It uses Globus job submissions tools, including GRAM, to remotely execute an application.
Making Small File Transfers Faster
October 2007
A team of scientists funded by the SciDAC CEDPS center and led by Raj Kettimuthu and John Bresnahan, both of Argonne National Laboratory, have been working to address the problem of making smaller file transfers go fast. Many applications work with thousands of small (less than 100 MB) files. Previously, GridFTP provided very fast performance for large files, but performance dropped precipitously for files under 500 MB. The SciDAC researchers have now improved the performance by streamlining multiple small files into an open channel. The new approach all but eliminates the file transfer channel startup overhead and thereby enables bundles of small files to transfer as fast as single large files. This technique has proven extremely useful for transferring data in near-real time to enable distributed processing capabilities needed by topographic experiments in Argonne's Advanced Photon Source. The technique is available in the Globus Toolkit 4.1.2 development release.
The First One-Click STAR Production Cluster
September 2007
The STAR community successfully completed its first production-size deployment of a VM-based virtual cluster managed by the workspace service and backed by EC2 resources. The 100 node cluster was composed of a head node and worker nodes based on the OSG 0.6.0 grid middleware stack and Torque. Its deployment-time configuration was securely coordinated by the new workspace contextualization technology. Additional information can be found at http://workspace.globus.org/news.html
