An integrated framework for parameter-based optimization of scientific workflows

Vijay S. Kumar; P. Sadayappan; Gaurang Mehta; Karan Vahi; Ewa Deelman; Varun Ratnakar; Jihie Kim; Yolanda Gil; Mary Hall; Tahsin Kurc; Joel Saltz

doi:10.1145/1551609.1551638

An integrated framework for parameter-based optimization of scientific workflows

Vijay S. Kumar, P. Sadayappan, Gaurang Mehta, Karan Vahi, Ewa Deelman, Varun Ratnakar, Jihie Kim, Yolanda Gil, Mary Hall, Tahsin Kurc, Joel Saltz

Department of Computer Science and Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

19 Scopus citations

Abstract

Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not affect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.

Original language	English
Title of host publication	Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09
Pages	177-186
Number of pages	10
DOIs	https://doi.org/10.1145/1551609.1551638
State	Published - 2009
Event	18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conference, HPDC'09 - Garching, Germany Duration: 11 Jun 2009 → 13 Jun 2009

Publication series

Name	Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09

Conference

Conference	18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conference, HPDC'09
Country/Territory	Germany
City	Garching
Period	11/06/09 → 13/06/09

Keywords

Application QoS
Grid
Performance parameters
Scientific workflow
Semantic representations

Access to Document

10.1145/1551609.1551638

Cite this

Kumar, V. S., Sadayappan, P., Mehta, G., Vahi, K., Deelman, E., Ratnakar, V., Kim, J., Gil, Y., Hall, M., Kurc, T., & Saltz, J. (2009). An integrated framework for parameter-based optimization of scientific workflows. In Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09 (pp. 177-186). Article 1551638 (Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09). https://doi.org/10.1145/1551609.1551638

Kumar, Vijay S. ; Sadayappan, P. ; Mehta, Gaurang et al. / An integrated framework for parameter-based optimization of scientific workflows. Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09. 2009. pp. 177-186 (Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09).

@inproceedings{ecad2f10f6bc4396ac8e2b2fcaa36b6a,

title = "An integrated framework for parameter-based optimization of scientific workflows",

abstract = "Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not affect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.",

keywords = "Application QoS, Grid, Performance parameters, Scientific workflow, Semantic representations",

author = "Kumar, {Vijay S.} and P. Sadayappan and Gaurang Mehta and Karan Vahi and Ewa Deelman and Varun Ratnakar and Jihie Kim and Yolanda Gil and Mary Hall and Tahsin Kurc and Joel Saltz",

year = "2009",

doi = "10.1145/1551609.1551638",

language = "English",

isbn = "9781605585871",

series = "Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09",

pages = "177--186",

booktitle = "Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09",

note = "18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conference, HPDC'09 ; Conference date: 11-06-2009 Through 13-06-2009",

}

Kumar, VS, Sadayappan, P, Mehta, G, Vahi, K, Deelman, E, Ratnakar, V, Kim, J, Gil, Y, Hall, M, Kurc, T & Saltz, J 2009, An integrated framework for parameter-based optimization of scientific workflows. in Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09., 1551638, Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09, pp. 177-186, 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conference, HPDC'09, Garching, Germany, 11/06/09. https://doi.org/10.1145/1551609.1551638

An integrated framework for parameter-based optimization of scientific workflows. / Kumar, Vijay S.; Sadayappan, P.; Mehta, Gaurang et al.
Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09. 2009. p. 177-186 1551638 (Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An integrated framework for parameter-based optimization of scientific workflows

AU - Kumar, Vijay S.

AU - Sadayappan, P.

AU - Mehta, Gaurang

AU - Vahi, Karan

AU - Deelman, Ewa

AU - Ratnakar, Varun

AU - Kim, Jihie

AU - Gil, Yolanda

AU - Hall, Mary

AU - Kurc, Tahsin

AU - Saltz, Joel

PY - 2009

Y1 - 2009

N2 - Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not affect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.

AB - Data analysis processes in scientific applications can be expressed as coarse-grain workflows of complex data processing operations with data flow dependencies between them. Performance optimization of these workflows can be viewed as a search for a set of optimal values in a multi-dimensional parameter space. While some performance parameters such as grouping of workflow components and their mapping to machines do not affect the accuracy of the output, others may dictate trading the output quality of individual components (and of the whole workflow) for performance. This paper describes an integrated framework which is capable of supporting performance optimizations along multiple dimensions of the parameter space. Using two real-world applications in the spatial data analysis domain, we present an experimental evaluation of the proposed framework.

KW - Application QoS

KW - Grid

KW - Performance parameters

KW - Scientific workflow

KW - Semantic representations

UR - http://www.scopus.com/inward/record.url?scp=70449633075&partnerID=8YFLogxK

U2 - 10.1145/1551609.1551638

DO - 10.1145/1551609.1551638

M3 - Conference contribution

AN - SCOPUS:70449633075

SN - 9781605585871

T3 - Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09

SP - 177

EP - 186

BT - Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09

T2 - 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conference, HPDC'09

Y2 - 11 June 2009 through 13 June 2009

ER -

Kumar VS, Sadayappan P, Mehta G, Vahi K, Deelman E, Ratnakar V et al. An integrated framework for parameter-based optimization of scientific workflows. In Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09. 2009. p. 177-186. 1551638. (Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09). doi: 10.1145/1551609.1551638

An integrated framework for parameter-based optimization of scientific workflows

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this