![](/images/graphics-bg.png)
Query Execution Optimization in Spark SQL
Joint Authors
Zhao, Maoxian
Ji, Xuechun
Zhai, Mingyu
Wu, Qingxi
Source
Issue
Vol. 2020, Issue 2020 (31 Dec. 2020), pp.1-12, 12 p.
Publisher
Hindawi Publishing Corporation
Publication Date
2020-02-07
Country of Publication
Egypt
No. of Pages
12
Main Subjects
Abstract EN
Spark SQL is a big data processing tool for structured data query and analysis.
However, due to the execution of Spark SQL, there are multiple times to write intermediate data to the disk, which reduces the execution efficiency of Spark SQL.
Targeting on the existing issues, we design and implement an intermediate data cache layer between the underlying file system and the upper Spark core to reduce the cost of random disk I/O.
By using the query pre-analysis module, we can dynamically adjust the capacity of cache layer for different queries.
And the allocation module can allocate proper memory for each node in cluster.
According to the sharing of the intermediate data in the Spark SQL workflow, this paper proposes a cost-based correlation merging algorithm, which can effectively reduce the cost of reading and writing redundant data.
This paper develops the SSO (Spark SQL Optimizer) module and integrates it into the original Spark system to achieve the above functions.
This paper compares the query performance with the existing Spark SQL by experiment data generated by TPC-H tool.
The experimental results show that the SSO module can effectively improve the query efficiency, reduce the disk I/O cost and make full use of the cluster memory resources.
American Psychological Association (APA)
Ji, Xuechun& Zhao, Maoxian& Zhai, Mingyu& Wu, Qingxi. 2020. Query Execution Optimization in Spark SQL. Scientific Programming،Vol. 2020, no. 2020, pp.1-12.
https://search.emarefa.net/detail/BIM-1209060
Modern Language Association (MLA)
Ji, Xuechun…[et al.]. Query Execution Optimization in Spark SQL. Scientific Programming No. 2020 (2020), pp.1-12.
https://search.emarefa.net/detail/BIM-1209060
American Medical Association (AMA)
Ji, Xuechun& Zhao, Maoxian& Zhai, Mingyu& Wu, Qingxi. Query Execution Optimization in Spark SQL. Scientific Programming. 2020. Vol. 2020, no. 2020, pp.1-12.
https://search.emarefa.net/detail/BIM-1209060
Data Type
Journal Articles
Language
English
Notes
Includes bibliographical references
Record ID
BIM-1209060