[SPARK-21720] Filter predicate with many conditions throw stackoverflow error

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: SQL
• External issue URL: https://issues.apache.org/jira/browse/SPARK-19372
• Assignee: Kazuaki Ishizaki
• Reporter: srinivasan
• Shepherd: Kazuaki Ishizaki
• Created: 14/Aug/17 02:10
• Updated: 24/May/24 11:27
• Resolved: 12/Nov/17 22:15

Description

When trying to filter on dataset with many predicate conditions on both spark sql and dataset filter transformation as described below, spark throws a stackoverflow exception

Case 1: Filter Transformation on Data

Dataset<Row> filter = sourceDataset.filter(String.format("not(%s)", buildQuery()));
filter.show();

where buildQuery() returns

Field1 = "" and Field2 = "" and Field3 = "" and Field4 = "" and Field5 = "" and BLANK_5 = "" and Field7 = "" and Field8 = "" and Field9 = "" and Field10 = "" and Field11 = "" and Field12 = "" and Field13 = "" and Field14 = "" and Field15 = "" and Field16 = "" and Field17 = "" and Field18 = "" and Field19 = "" and Field20 = "" and Field21 = "" and Field22 = "" and Field23 = "" and Field24 = "" and Field25 = "" and Field26 = "" and Field27 = "" and Field28 = "" and Field29 = "" and Field30 = "" and Field31 = "" and Field32 = "" and Field33 = "" and Field34 = "" and Field35 = "" and Field36 = "" and Field37 = "" and Field38 = "" and Field39 = "" and Field40 = "" and Field41 = "" and Field42 = "" and Field43 = "" and Field44 = "" and Field45 = "" and Field46 = "" and Field47 = "" and Field48 = "" and Field49 = "" and Field50 = "" and Field51 = "" and Field52 = "" and Field53 = "" and Field54 = "" and Field55 = "" and Field56 = "" and Field57 = "" and Field58 = "" and Field59 = "" and Field60 = "" and Field61 = "" and Field62 = "" and Field63 = "" and Field64 = "" and Field65 = "" and Field66 = "" and Field67 = "" and Field68 = "" and Field69 = "" and Field70 = "" and Field71 = "" and Field72 = "" and Field73 = "" and Field74 = "" and Field75 = "" and Field76 = "" and Field77 = "" and Field78 = "" and Field79 = "" and Field80 = "" and Field81 = "" and Field82 = "" and Field83 = "" and Field84 = "" and Field85 = "" and Field86 = "" and Field87 = "" and Field88 = "" and Field89 = "" and Field90 = "" and Field91 = "" and Field92 = "" and Field93 = "" and Field94 = "" and Field95 = "" and Field96 = "" and Field97 = "" and Field98 = "" and Field99 = "" and Field100 = "" and Field101 = "" and Field102 = "" and Field103 = "" and Field104 = "" and Field105 = "" and Field106 = "" and Field107 = "" and Field108 = "" and Field109 = "" and Field110 = "" and Field111 = "" and Field112 = "" and Field113 = "" and Field114 = "" and Field115 = "" and Field116 = "" and Field117 = "" and Field118 = "" and Field119 = "" and Field120 = "" and Field121 = "" and Field122 = "" and Field123 = "" and Field124 = "" and Field125 = "" and Field126 = "" and Field127 = "" and Field128 = "" and Field129 = "" and Field130 = "" and Field131 = "" and Field132 = "" and Field133 = "" and Field134 = "" and Field135 = "" and Field136 = "" and Field137 = "" and Field138 = "" and Field139 = "" and Field140 = "" and Field141 = "" and Field142 = "" and Field143 = "" and Field144 = "" and Field145 = "" and Field146 = "" and Field147 = "" and Field148 = "" and Field149 = "" and Field150 = "" and Field151 = "" and Field152 = "" and Field153 = "" and Field154 = "" and Field155 = "" and Field156 = "" and Field157 = "" and Field158 = "" and Field159 = "" and Field160 = "" and Field161 = "" and Field162 = "" and Field163 = "" and Field164 = "" and Field165 = "" and Field166 = "" and Field167 = "" and Field168 = "" and Field169 = "" and Field170 = "" and Field171 = "" and Field172 = "" and Field173 = "" and Field174 = "" and Field175 = "" and Field176 = "" and Field177 = "" and Field178 = "" and Field179 = "" and Field180 = "" and Field181 = "" and Field182 = "" and Field183 = "" and Field184 = "" and Field185 = "" and Field186 = "" and Field187 = "" and Field188 = "" and Field189 = "" and Field190 = "" and Field191 = "" and Field192 = "" and Field193 = "" and Field194 = "" and Field195 = "" and Field196 = "" and Field197 = "" and Field198 = "" and Field199 = "" and Field200 = "" and Field201 = "" and Field202 = "" and Field203 = "" and Field204 = "" and Field205 = "" and Field206 = "" and Field207 = "" and Field208 = "" and Field209 = "" and Field210 = "" and Field211 = "" and Field212 = "" and Field213 = "" and Field214 = "" and Field215 = "" and Field216 = "" and Field217 = "" and Field218 = "" and Field219 = "" and Field220 = "" and Field221 = "" and Field222 = "" and Field223 = "" and Field224 = "" and Field225 = "" and Field226 = "" and Field227 = "" and Field228 = "" and Field229 = "" and Field230 = "" and Field231 = "" and Field232 = "" and Field233 = "" and Field234 = "" and Field235 = "" and Field236 = "" and Field237 = "" and Field238 = "" and Field239 = "" and Field240 = "" and Field241 = "" and Field242 = "" and Field243 = "" and Field244 = "" and Field245 = "" and Field246 = "" and Field247 = "" and Field248 = "" and Field249 = "" and Field250 = "" and Field251 = "" and Field252 = "" and Field253 = "" and Field254 = "" and Field255 = "" and Field256 = "" and Field257 = "" and Field258 = "" and Field259 = "" and Field260 = "" and Field261 = "" and Field262 = "" and Field263 = "" and Field264 = "" and Field265 = "" and Field266 = "" and Field267 = "" and Field268 = "" and Field269 = "" and Field270 = "" and Field271 = "" and Field272 = "" and Field273 = "" and Field274 = "" and Field275 = "" and Field276 = "" and Field277 = "" and Field278 = "" and Field279 = "" and Field280 = "" and Field281 = "" and Field282 = "" and Field283 = "" and Field284 = "" and Field285 = "" and Field286 = "" and Field287 = "" and Field288 = "" and Field289 = "" and Field290 = "" and Field291 = "" and Field292 = "" and Field293 = "" and Field294 = "" and Field295 = "" and Field296 = "" and Field297 = "" and Field298 = "" and Field299 = "" and Field300 = "" and Field301 = "" and Field302 = "" and Field303 = "" and Field304 = "" and Field305 = "" and Field306 = "" and Field307 = "" and Field308 = "" and Field309 = "" and Field310 = "" and Field311 = "" and Field312 = "" and Field313 = "" and Field314 = "" and Field315 = "" and Field316 = "" and Field317 = "" and Field318 = "" and Field319 = "" and Field320 = "" and Field321 = "" and Field322 = "" and Field323 = "" and Field324 = "" and Field325 = "" and Field326 = "" and Field327 = "" and Field328 = "" and Field329 = "" and Field330 = "" and Field331 = "" and Field332 = "" and Field333 = "" and Field334 = ""

Case 2 : Using Spark SQL on temp view

select * from temp where not( Field1 = '' and Field2 = '' and Field3 = '' and Field4 = '' and Field5 = '' and BLANK_5 = '' and Field7 = '' and Field8 = '' and Field9 = '' and Field10 = '' and Field11 = '' and Field12 = '' and Field13 = '' and Field14 = '' and Field15 = '' and Field16 = '' and Field17 = '' and Field18 = '' and Field19 = '' and Field20 = '' and Field21 = '' and Field22 = '' and Field23 = '' and Field24 = '' and Field25 = '' and Field26 = '' and Field27 = '' and Field28 = '' and Field29 = '' and Field30 = '' and Field31 = '' and Field32 = '' and Field33 = '' and Field34 = '' and Field35 = '' and Field36 = '' and Field37 = '' and Field38 = '' and Field39 = '' and Field40 = '' and Field41 = '' and Field42 = '' and Field43 = '' and Field44 = '' and Field45 = '' and Field46 = '' and Field47 = '' and Field48 = '' and Field49 = '' and Field50 = '' and Field51 = '' and Field52 = '' and Field53 = '' and Field54 = '' and Field55 = '' and Field56 = '' and Field57 = '' and Field58 = '' and Field59 = '' and Field60 = '' and Field61 = '' and Field62 = '' and Field63 = '' and Field64 = '' and Field65 = '' and Field66 = '' and Field67 = '' and Field68 = '' and Field69 = '' and Field70 = '' and Field71 = '' and Field72 = '' and Field73 = '' and Field74 = '' and Field75 = '' and Field76 = '' and Field77 = '' and Field78 = '' and Field79 = '' and Field80 = '' and Field81 = '' and Field82 = '' and Field83 = '' and Field84 = '' and Field85 = '' and Field86 = '' and Field87 = '' and Field88 = '' and Field89 = '' and Field90 = '' and Field91 = '' and Field92 = '' and Field93 = '' and Field94 = '' and Field95 = '' and Field96 = '' and Field97 = '' and Field98 = '' and Field99 = '' and Field100 = '' and Field101 = '' and Field102 = '' and Field103 = '' and Field104 = '' and Field105 = '' and Field106 = '' and Field107 = '' and Field108 = '' and Field109 = '' and Field110 = '' and Field111 = '' and Field112 = '' and Field113 = '' and Field114 = '' and Field115 = '' and Field116 = '' and Field117 = '' and Field118 = '' and Field119 = '' and Field120 = '' and Field121 = '' and Field122 = '' and Field123 = '' and Field124 = '' and Field125 = '' and Field126 = '' and Field127 = '' and Field128 = '' and Field129 = '' and Field130 = '' and Field131 = '' and Field132 = '' and Field133 = '' and Field134 = '' and Field135 = '' and Field136 = '' and Field137 = '' and Field138 = '' and Field139 = '' and Field140 = '' and Field141 = '' and Field142 = '' and Field143 = '' and Field144 = '' and Field145 = '' and Field146 = '' and Field147 = '' and Field148 = '' and Field149 = '' and Field150 = '' and Field151 = '' and Field152 = '' and Field153 = '' and Field154 = '' and Field155 = '' and Field156 = '' and Field157 = '' and Field158 = '' and Field159 = '' and Field160 = '' and Field161 = '' and Field162 = '' and Field163 = '' and Field164 = '' and Field165 = '' and Field166 = '' and Field167 = '' and Field168 = '' and Field169 = '' and Field170 = '' and Field171 = '' and Field172 = '' and Field173 = '' and Field174 = '' and Field175 = '' and Field176 = '' and Field177 = '' and Field178 = '' and Field179 = '' and Field180 = '' and Field181 = '' and Field182 = '' and Field183 = '' and Field184 = '' and Field185 = '' and Field186 = '' and Field187 = '' and Field188 = '' and Field189 = '' and Field190 = '' and Field191 = '' and Field192 = '' and Field193 = '' and Field194 = '' and Field195 = '' and Field196 = '' and Field197 = '' and Field198 = '' and Field199 = '' and Field200 = '' and Field201 = '' and Field202 = '' and Field203 = '' and Field204 = '' and Field205 = '' and Field206 = '' and Field207 = '' and Field208 = '' and Field209 = '' and Field210 = '' and Field211 = '' and Field212 = '' and Field213 = '' and Field214 = '' and Field215 = '' and Field216 = '' and Field217 = '' and Field218 = '' and Field219 = '' and Field220 = '' and Field221 = '' and Field222 = '' and Field223 = '' and Field224 = '' and Field225 = '' and Field226 = '' and Field227 = '' and Field228 = '' and Field229 = '' and Field230 = '' and Field231 = '' and Field232 = '' and Field233 = '' and Field234 = '' and Field235 = '' and Field236 = '' and Field237 = '' and Field238 = '' and Field239 = '' and Field240 = '' and Field241 = '' and Field242 = '' and Field243 = '' and Field244 = '' and Field245 = '' and Field246 = '' and Field247 = '' and Field248 = '' and Field249 = '' and Field250 = '' and Field251 = '' and Field252 = '' and Field253 = '' and Field254 = '' and Field255 = '' and Field256 = '' and Field257 = '' and Field258 = '' and Field259 = '' and Field260 = '' and Field261 = '' and Field262 = '' and Field263 = '' and Field264 = '' and Field265 = '' and Field266 = '' and Field267 = '' and Field268 = '' and Field269 = '' and Field270 = '' and Field271 = '' and Field272 = '' and Field273 = '' and Field274 = '' and Field275 = '' and Field276 = '' and Field277 = '' and Field278 = '' and Field279 = '' and Field280 = '' and Field281 = '' and Field282 = '' and Field283 = '' and Field284 = '' and Field285 = '' and Field286 = '' and Field287 = '' and Field288 = '' and Field289 = '' and Field290 = '' and Field291 = '' and Field292 = '' and Field293 = '' and Field294 = '' and Field295 = '' and Field296 = '' and Field297 = '' and Field298 = '' and Field299 = '' and Field300 = '' and Field301 = '' and Field302 = '' and Field303 = '' and Field304 = '' and Field305 = '' and Field306 = '' and Field307 = '' and Field308 = '' and Field309 = '' and Field310 = '' and Field311 = '' and Field312 = '' and Field313 = '' and Field314 = '' and Field315 = '' and Field316 = '' and Field317 = '' and Field318 = '' and Field319 = '' and Field320 = '' and Field321 = '' and Field322 = '' and Field323 = '' and Field324 = '' and Field325 = '' and Field326 = '' and Field327 = '' and Field328 = '' and Field329 = '' and Field330 = '' and Field331 = '' and Field332 = '' and Field333 = '' and Field334 = '')

in both cases, the error thrown is

java.lang.StackOverflowError
at org.codehaus.janino.CodeContext.extract16BitValue(CodeContext.java:679)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:545)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)

......

Thanks

Issue Links

• is duplicated by: SPARK-22523 Janino throws StackOverflowError on nested structs with many fields
• links to: [Github] Pull Request #18972 (kiszk)