Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4762

RECOVER PARTITIONS should send new partitions in small batches to HMS

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Catalog
    • Labels:

      Description

      HMS cannot handle more than 32k partitions in one call. When adding large amount of partitions, Impala should send the new partitions in smaller batches. otherwise HMS could go OOM.

        Issue Links

          Activity

          Hide
          zamsden Zach Amsden added a comment -

          IMPALA-4762: RECOVER PARTITIONS should batch partition updates

          Batch updates when doing a RECOVER PARTITIONS on over 500
          partitions at a time to avoid HMS timeouts, possible OOM.

          Testing: Expanded test coverage with a new python test
          for this case. Test takes ~18s to run.

          Change-Id: I7f9334051b11ba8fa16159b7ca67ddc7f2392733
          Reviewed-on: http://gerrit.cloudera.org:8080/6275
          Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
          Tested-by: Impala Public Jenkins
          Author
          Zach Amsden <zamsden@cloudera.com>
          Mar 6, 2017 3:43 PM
          Committer
          Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
          Mar 13, 2017 6:38 PM
          Commit
          eec8d6fd15dc9e914a773aab1390f95d97f515ae
          Parent(s)
          d9602df71b6d686a8e8268d0e18e99f2a51d4e78
          Change-Id
          I7f9334051b11ba8fa16159b7ca67ddc7f2392733

          Show
          zamsden Zach Amsden added a comment - IMPALA-4762 : RECOVER PARTITIONS should batch partition updates Batch updates when doing a RECOVER PARTITIONS on over 500 partitions at a time to avoid HMS timeouts, possible OOM. Testing: Expanded test coverage with a new python test for this case. Test takes ~18s to run. Change-Id: I7f9334051b11ba8fa16159b7ca67ddc7f2392733 Reviewed-on: http://gerrit.cloudera.org:8080/6275 Reviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com> Tested-by: Impala Public Jenkins Author Zach Amsden <zamsden@cloudera.com> Mar 6, 2017 3:43 PM Committer Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org> Mar 13, 2017 6:38 PM Commit eec8d6fd15dc9e914a773aab1390f95d97f515ae Parent(s) d9602df71b6d686a8e8268d0e18e99f2a51d4e78 Change-Id I7f9334051b11ba8fa16159b7ca67ddc7f2392733

            People

            • Assignee:
              zamsden Zach Amsden
              Reporter:
              jyu@cloudera.com Juan Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development