Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3983

Clean up UDF jars after reloading functions

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:

      Description

      As per IMPALA-2843, we copy the UDF jars to a local temp directory. Upon reloading the functions (e.g. catalog restart) we should clean up the old files.

        Issue Links

          Activity

          Hide
          bharathv bharath v added a comment -

          IMPALA-3983/IMPALA-3974: Delete function jar resources after load

          The Catalog copies the UDF jar files to the local file system to
          load the Java UDF classes for validation purposes. However we
          do not clean them up after the UDF load and hence on a deployment
          with large number of functions registered, these jar can accumulate
          over a period of time and can fill up the tmp space. We fix it by
          deleting the jar resource once the function is loaded.

          Also, this patch switches to --local_library_dir for copying these
          temporary jars instead of using the path from java.io.tmpdir.

          Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199
          Reviewed-on: http://gerrit.cloudera.org:8080/4617
          Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
          Tested-by: Internal Jenkins

          Commit: 9f5c6a5b6230ee7c565eade38a75912edb07dd7d

          Show
          bharathv bharath v added a comment - IMPALA-3983 / IMPALA-3974 : Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Reviewed-on: http://gerrit.cloudera.org:8080/4617 Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com> Tested-by: Internal Jenkins Commit: 9f5c6a5b6230ee7c565eade38a75912edb07dd7d

            People

            • Assignee:
              bharathv bharath v
              Reporter:
              balazsj_impala_220b Balazs Jeszenszky
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development