DataConnector: A Data processing framework integrating hadoop and a grid middleware OGSA-DAI for cloud environment

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Recent mobile internet services make use of computing resources provided in forms of Cloud computing. However, conventional data management framework faces performance problems when importing external heterogeneous data and processing the vast amount of data with Cloud computing technology. In this paper, we propose a data processing framework for cloud applications based on OGSA-DAI (Open Grid Service Architecture-Data Access & Integration) for heterogeneous external data importing and MapReduce for big data processing. We designed and implemented a framework called DataConnector extending OGSA-DAI middleware which can access and integrate distributed data in a heterogeneous environment, and we deployed DataConnector into a Cloud environment. We conducted various experiments for evaluation and showed that our approach can be used for fast heterogeneous external data access and efficient large data processing with negligible or no system overhead.

Original languageEnglish
Pages (from-to)801-806
Number of pages6
JournalInformation (Japan)
Volume16
Issue number1 B
StatePublished - Jan 2013

Keywords

  • Cloud computing
  • Distributed computing
  • Grid computing
  • MapReduce
  • OGSA-DAI

Fingerprint

Dive into the research topics of 'DataConnector: A Data processing framework integrating hadoop and a grid middleware OGSA-DAI for cloud environment'. Together they form a unique fingerprint.

Cite this