Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3983

Clean up UDF jars after reloading functions

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: Impala 2.5.0
        • Fix Version/s: Impala 2.8.0
        • Component/s: Catalog
        • Labels:

          Description

          As per IMPALA-2843, we copy the UDF jars to a local temp directory. Upon reloading the functions (e.g. catalog restart) we should clean up the old files.

            Issue Links

            duplicates

            Bug - A problem which impairs or prevents the functions of the product. IMPALA-3974 IMPALA-2843 should consider using local_library_dir instead of java.io.tmpdir

            • Major - Major loss of function.
            • Resolved

              Activity

              Hide
              Permalink
              bharathv bharath v added a comment -

              IMPALA-3983/IMPALA-3974: Delete function jar resources after load

              The Catalog copies the UDF jar files to the local file system to
              load the Java UDF classes for validation purposes. However we
              do not clean them up after the UDF load and hence on a deployment
              with large number of functions registered, these jar can accumulate
              over a period of time and can fill up the tmp space. We fix it by
              deleting the jar resource once the function is loaded.

              Also, this patch switches to --local_library_dir for copying these
              temporary jars instead of using the path from java.io.tmpdir.

              Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199
              Reviewed-on: http://gerrit.cloudera.org:8080/4617
              Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
              Tested-by: Internal Jenkins

              Commit: 9f5c6a5b6230ee7c565eade38a75912edb07dd7d

              Show
              bharathv bharath v added a comment - IMPALA-3983 / IMPALA-3974 : Delete function jar resources after load The Catalog copies the UDF jar files to the local file system to load the Java UDF classes for validation purposes. However we do not clean them up after the UDF load and hence on a deployment with large number of functions registered, these jar can accumulate over a period of time and can fill up the tmp space. We fix it by deleting the jar resource once the function is loaded. Also, this patch switches to --local_library_dir for copying these temporary jars instead of using the path from java.io.tmpdir. Change-Id: I5f9dedb5b342415380c83e61a72eb497371a8199 Reviewed-on: http://gerrit.cloudera.org:8080/4617 Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com> Tested-by: Internal Jenkins Commit: 9f5c6a5b6230ee7c565eade38a75912edb07dd7d

                People

                • Assignee:
                  bharathv bharath v
                  Reporter:
                  balazsj_impala_220b Balazs Jeszenszky
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development