Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-35437

BlockStatementGrouper uses lots of memory

    XMLWordPrintableJSON

Details

    Description

      For deeply nested if else statements BlockStatementGrouper uses loads of memory and fails with OOM quickly.

      When running JMs with around 400mb a query like:

      select case when orderid = 0 then 1 when orderid = 1 then 2 when orderid
          = 2 then 3 when orderid = 3 then 4 when orderid = 4 then 5 when orderid = 5 then
          6 when orderid = 6 then 7 when orderid = 7 then 8 when orderid = 8 then 9 when
          orderid = 9 then 10 when orderid = 10 then 11 when orderid = 11 then 12 when orderid
          = 12 then 13 when orderid = 13 then 14 when orderid = 14 then 15 when orderid
          = 15 then 16 when orderid = 16 then 17 when orderid = 17 then 18 when orderid
          = 18 then 19 when orderid = 19 then 20 when orderid = 20 then 21 when orderid
          = 21 then 22 when orderid = 22 then 23 when orderid = 23 then 24 when orderid
          = 24 then 25 when orderid = 25 then 26 when orderid = 26 then 27 when orderid
          = 27 then 28 when orderid = 28 then 29 when orderid = 29 then 30 when orderid
          = 30 then 31 when orderid = 31 then 32 when orderid = 32 then 33 when orderid
          = 33 then 34 when orderid = 34 then 35 when orderid = 35 then 36 when orderid
          = 36 then 37 when orderid = 37 then 38 when orderid = 38 then 39 when orderid
          = 39 then 40 when orderid = 40 then 41 when orderid = 41 then 42 when orderid
          = 42 then 43 when orderid = 43 then 44 when orderid = 44 then 45 when orderid
          = 45 then 46 when orderid = 46 then 47 when orderid = 47 then 48 when orderid
          = 48 then 49 when orderid = 49 then 50 when orderid = 50 then 51 when orderid
          = 51 then 52 when orderid = 52 then 53 when orderid = 53 then 54 when orderid
          = 54 then 55 when orderid = 55 then 56 when orderid = 56 then 57 when orderid
          = 57 then 58 when orderid = 58 then 59 when orderid = 59 then 60 when orderid
          = 60 then 61 when orderid = 61 then 62 when orderid = 62 then 63 when orderid
          = 63 then 64 when orderid = 64 then 65 when orderid = 65 then 66 when orderid
          = 66 then 67 when orderid = 67 then 68 when orderid = 68 then 69 when orderid
          = 69 then 70 when orderid = 70 then 71 when orderid = 71 then 72 when orderid
          = 72 then 73 when orderid = 73 then 74 when orderid = 74 then 75 when orderid
          = 75 then 76 when orderid = 76 then 77 when orderid = 77 then 78 when orderid
          = 78 then 79 when orderid = 79 then 80 when orderid = 80 then 81 when orderid
          = 81 then 82 when orderid = 82 then 83 when orderid = 83 then 84 when orderid
          = 84 then 85 when orderid = 85 then 86 when orderid = 86 then 87 when orderid
          = 87 then 88 when orderid = 88 then 89 when orderid = 89 then 90 when orderid
          = 90 then 91 when orderid = 91 then 92 when orderid = 92 then 93 when orderid
          = 93 then 94 when orderid = 94 then 95 when orderid = 95 then 96 when orderid
          = 96 then 97 when orderid = 97 then 98 when orderid = 98 then 99 when orderid
          = 99 then 100 when orderid = 100 then 101 when orderid = 101 then 102 when orderid
          = 102 then 103 when orderid = 103 then 104 when orderid = 104 then 105 when orderid
          = 105 then 106 when orderid = 106 then 107 when orderid = 107 then 108 when orderid
          = 108 then 109 when orderid = 109 then 110 when orderid = 110 then 111 when orderid
          = 111 then 112 when orderid = 112 then 113 when orderid = 113 then 114 when orderid
          = 114 then 115 when orderid = 115 then 116 when orderid = 116 then 117 when orderid
          = 117 then 118 when orderid = 118 then 119 when orderid = 119 then 120 when orderid
          = 120 then 121 when orderid = 121 then 122 when orderid = 122 then 123 when orderid
          = 123 then 124 when orderid = 124 then 125 when orderid = 125 then 126 when orderid
          = 126 then 127 when orderid = 127 then 128 when orderid = 128 then 129 when orderid
          = 129 then 130 when orderid = 130 then 131 when orderid = 131 then 132 when orderid
          = 132 then 133 when orderid = 133 then 134 when orderid = 134 then 135 when orderid
          = 135 then 136 when orderid = 136 then 137 when orderid = 137 then 138 when orderid
          = 138 then 139 when orderid = 139 then 140 when orderid = 140 then 141 when orderid
          = 141 then 142 when orderid = 142 then 143 when orderid = 143 then 144 when orderid
          = 144 then 145 when orderid = 145 then 146 when orderid = 146 then 147 when orderid
          = 147 then 148 when orderid = 148 then 149 when orderid = 149 then 150 when orderid
          = 150 then 151 when orderid = 151 then 152 when orderid = 152 then 153 when orderid
          = 153 then 154 when orderid = 154 then 155 when orderid = 155 then 156 when orderid
          = 156 then 157 when orderid = 157 then 158 when orderid = 158 then 159 when orderid
          = 159 then 160 when orderid = 160 then 161 when orderid = 161 then 162 when orderid
          = 162 then 163 when orderid = 163 then 164 when orderid = 164 then 165 when orderid
          = 165 then 166 when orderid = 166 then 167 when orderid = 167 then 168 when orderid
          = 168 then 169 when orderid = 169 then 170 when orderid = 170 then 171 when orderid
          = 171 then 172 when orderid = 172 then 173 when orderid = 173 then 174 when orderid
          = 174 then 175 when orderid = 175 then 176 when orderid = 176 then 177 when orderid
          = 177 then 178 when orderid = 178 then 179 when orderid = 179 then 180 when orderid
          = 180 then 181 when orderid = 181 then 182 when orderid = 182 then 183 when orderid
          = 183 then 184 when orderid = 184 then 185 when orderid = 185 then 186 when orderid
          = 186 then 187 when orderid = 187 then 188 when orderid = 188 then 189 when orderid
          = 189 then 190 when orderid = 190 then 191 when orderid = 191 then 192 when orderid
          = 192 then 193 when orderid = 193 then 194 when orderid = 194 then 195 when orderid
          = 195 then 196 when orderid = 196 then 197 when orderid = 197 then 198 when orderid
          = 198 then 199 when orderid = 199 then 200 when orderid = 200 then 201 when orderid
          = 201 then 202 when orderid = 202 then 203 when orderid = 203 then 204 when orderid
          = 204 then 205 when orderid = 205 then 206 when orderid = 206 then 207 when orderid
          = 207 then 208 when orderid = 208 then 209 when orderid = 209 then 210 when orderid
          = 210 then 211 when orderid = 211 then 212 when orderid = 212 then 213 when orderid
          = 213 then 214 when orderid = 214 then 215 when orderid = 215 then 216 when orderid
          = 216 then 217 when orderid = 217 then 218 when orderid = 218 then 219 when orderid
          = 219 then 220 when orderid = 220 then 221 when orderid = 221 then 222 when orderid
          = 222 then 223 when orderid = 223 then 224 when orderid = 224 then 225 when orderid
          = 225 then 226 when orderid = 226 then 227 when orderid = 227 then 228 when orderid
          = 228 then 229 when orderid = 229 then 230 when orderid = 230 then 231 when orderid
          = 231 then 232 when orderid = 232 then 233 when orderid = 233 then 234 when orderid
          = 234 then 235 when orderid = 235 then 236 when orderid = 236 then 237 when orderid
          = 237 then 238 when orderid = 238 then 239 when orderid = 239 then 240 when orderid
          = 240 then 241 when orderid = 241 then 242 when orderid = 242 then 243 when orderid
          = 243 then 244 when orderid = 244 then 245 when orderid = 245 then 246 when orderid
          = 246 then 247 when orderid = 247 then 248 when orderid = 248 then 249 when orderid
          = 249 then 250 else 9999 end case_when_col from sample_data_1;
      

      fails with an OOM. (Yes, I know the query can be simplified, but it shows the case).

      Attachments

        Issue Links

          Activity

            People

              dwysakowicz Dawid Wysakowicz
              dwysakowicz Dawid Wysakowicz
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: