Near Data Filtering for Distributed Database Systems
- Resource Type
- Conference
- Authors
- Zhou, Zimeng; Sun, Xuan; Yu, Jinghuan; Nutanong, Sarana; Xue, Chun Jason
- Source
- 2018 Ninth International Green and Sustainable Computing Conference (IGSC) Green and Sustainable Computing Conference (IGSC), 2018 Ninth International. :1-8 Oct, 2018
- Subject
- Communication, Networking and Broadcast Technologies
Components, Circuits, Devices and Systems
Computing and Processing
Power, Energy and Industry Applications
Distributed databases
Data communication
Data processing
Computer architecture
Throughput
Task analysis
Data compression
distributed systems
separation between storage and computation
data movement
near data filtering
- Language
Over the past decade, data movement costs dominate the execution time of data-intensive applications for distributed systems and they are expected to be even more important in the future. Near data processing is a straightforward solution to reduce data movement which brings compute resources closer to the data source. This paper explores near data processing in a generic distributed system to improve the performance by reducing data movement. An efficient near data filtering solution is designed and implemented by introducing a filter layer which performs tuple-level near data filtering. In order to reduce idle time of processing nodes and improve data transmission throughput the proposed solution is extended to support block-level near data filtering by creating index for each data block. Furthermore, to answer the question when and how to perform near data filtering this paper proposes an adaptive near data filtering solution to balance the computation and data transmission throughput. Experimental results show that the proposed solutions are superior to the best existing method for most cases. The adaptive near data filtering solution achieves an average speedup factor of 4:59 for queries with low selectivity.