A hypergraph partitioning based approach for scheduling of tasks with batch-shared I/O
- Resource Type
- Conference
- Authors
- Gaurav Khanna; Nagavijayalakshmi Vydyanathan; Kurc, T.; Catalyurek, U.; Wyckoff, P.; Saltz, J.; Sadayappan, P.
- Source
- CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005. Cluster Computing and the Grid Cluster Computing and the Grid, 2005. CCGrid 2005. IEEE International Symposium on. 2:792-799 Vol. 2 2005
- Subject
- Computing and Processing
Communication, Networking and Broadcast Technologies
Processor scheduling
Data analysis
Pipelines
Biomedical engineering
Data engineering
Image analysis
Biomedical imaging
Subcontracting
Data processing
Computational modeling
- Language
This paper proposes a novel, hypergraph partitioning based strategy to schedule multiple data analysis tasks with batch-shared I/O behavior. This strategy formulates the sharing of files among tasks as a hypergraph to minimize the I/O overheads due to transferring of the same set of files multiple times and employs a dynamic scheme for file transfers to reduce contention on the storage system. We experimentally evaluate the proposed approach using application emulators from two application domains; analysis of remotely-sensed data and biomedical imaging.