[SPARK-27779] Regression when explode on map in Generate

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 20/May/19 15:36
• Updated: 16/Mar/20 22:52

Description

When I ran MiscBenchmark for SPARK-27707, I found a regression regarding explode map:

OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate explode map:                     Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
generate explode map wholestage off               50273          51458        1676          0.3        2996.5       1.0X
generate explode map wholestage on                43113          43351         278          0.4        2569.8       1.2X

I got this benchmark result by running MiscBenchmark on current master. All benchmark result is at https://gist.github.com/viirya/c850b49e364e8e6aaa6271ae9e3343fc

The last committed benchmark number in MiscBenchmark-results.txt:

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate explode map:                    Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
generate explode map wholestage off         22128 / 22578          0.8        1318.9       1.0X
generate explode map wholestage on          16421 / 16520          1.0         978.8       1.3X

It looks significant regression, hopefully I don't miss something...

Maybe others can also help verify if there is really a regression.