Record Details

Benchmarking and Scheduling Strategies for Distributed Stream Processing

Electronic Theses of Indian Institute of Science

View Archive Info
 
 
Field Value
 
Title Benchmarking and Scheduling Strategies for Distributed Stream Processing
 
Creator Shukla, Anshu
 
Subject Distributed Stream Processing
Distributed Programming
Apache Storm Dataflows
Stream Processing Benchmark
Distributed Stream Processing Systems (DSPS)
IoT Applications
Streaming Dataflows
Cloud Virtual Machines (VMs)
Model Based Allocation (MBA)
Slot Aware Mapping (SAM)
Computer Science
 
Description The velocity dimension of Big Data refers to the need to rapidly process data that arrives continuously as streams of messages or events. Distributed Stream Processing Systems (DSPS) refer to distributed programming and runtime platforms that allow users to define a composition of dataflow logic that are executed on distributed resources over streams of incoming messages.
A DSPS uses commodity clusters and Cloud Virtual Machines (VMs) for its execution. In order to meet the required performance for these applications, the DSPS needs to schedule these dataßows efficiently over the resources. Despite their growing use, resource scheduling for DSPSÕs tends to be done in an ad hoc manner, favoring empirical and reactive approaches, rather than a model-driven and analytical approach. Such empirical strategies may arrive at an approximate schedule for the dataflow that needs further tuning to meet the quality of service.
We propose a model-based scheduling approach that makes use of performance profiles and benchmarks developed for tasks in the dataßow to plan both the resource allocation and the resource mapping that together form the schedule planning process. We propose the Model Based Allocation (MBA) and the Slot Aware Mapping (SAM) approaches that efectively utilize knowledge of the performance model of logic tasks to provide an efficient and predictable scheduling behavior. We implemented and validate these algorithms using the popular open source Apache Storm DSPS for several micro and application dataflows. The results show that our model-driven approach is able to reduce the amount of required resources (VMs) by 30% − 50% relative to existing techniques. Also we see that our strategies o↵er a predictable behavior that ensures that the expected and actual rates supported and resources used match closely. This can enable deterministic schedule planning even under dynamic conditions.
Besides this static scheduling, we also examine the ability to dynamically consolidate tasks onto fewer VMs when the load on the dataßow decreases or the VMs get fragmented. We propose reliable task migration models for Apache Storm dataßows that are able to rapidly move the task assignment in the cluster, and resume the dataflow execution without any message loss.
 
Contributor Simmhan, Yogesh
 
Date 2018-08-20T14:04:08Z
2018-08-20T14:04:08Z
2018-08-20
2017
 
Type Thesis
 
Identifier http://etd.iisc.ernet.in/2005/3984
http://etd.iisc.ernet.in/abstracts/4872/G28612-Abs.pdf
 
Language en_US
 
Relation G28612