Portable File Staging Tools

During the development phase, the user normally uses a small-sized cluster, e.g., 128 or so CPU cores, for testing. Once the program’s functionality and scalability are tested, it is supposed to run on 1000 or more CPU cores. Contrary to the user’s expectations, however, execution of the program may not scale up properly due to various scalability issues. One of the bottlenecks is global file system performance. When an application runs on a large-scale cluster as shown in Figure 1, a large amount of simultaneous requests to the server causes network congestion and an overloaded situation at the server. One approach to overcoming this issue is to provide a file staging mechanism (Figure 2).

Figure 1. File I/O Issues

STG

STG is a new file staging tool whereby files needed in the application are moved to local disks from an external system before execution, and files created on local disks during execution are stored on the external system. STG has the following three features:

1) Modularity of external file access protocols

There are many file access methods used to transfer files between an external system and the cluster, such as ftp, scp, and http protocols. And new protocols will surely be available in the future. Thus, STG even specifies an API for adding a new file access method protocol that may be defined in the future. Currently, STG provides the scp and http protocols.

2) Delegation nodes

If all nodes access an external system, the file transfer performance may decrease due to congestion of both the network and the external file system. Instead of that, in STG, some nodes, called delegation nodes, access the external system to transfer files, and the delegation nodes and other nodes exchange files using the MPI communication library.

3) Definition of communication topology

STG enables users to customize the communication topology used for transferring files between delegation and other nodes, such as ring and tree topologies. Based on the communication topology specified, files are transferred between delegation and other nodes.

Figure 2. File Staging