HBase
  1. HBase
  2. HBASE-9046

Coprocessors can't be upgraded in service reliably

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.94.8, 0.96.0
    • Fix Version/s: None
    • Component/s: Coprocessors
    • Labels:
      None
    • Environment:

      Description

      My team and another user from the mailing list have run into an issue where replacing the coprocessor jar in HDFS and reloading the table does not load the latest jar. It may load the latest version on some percentage of RS but not all of them.

      This may be a config oversight or a lack of understanding of a caching mechanism that has a purge capability, but I thought I would log it here for confirmation.

      Workaround is to name the coprocessor JAR uniquely, place in HDFS, and re-enable the table using the new jar's name.

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          Unlikely this will make it into 0.98 as there is not a patch available.

          Show
          Andrew Purtell added a comment - Unlikely this will make it into 0.98 as there is not a patch available.
          Hide
          Benoit Sigoure added a comment -

          I can further confirm this because in my current environment I use a single coprocessor, so I devised a workaround for this bug: my coprocessor class has a static int I use as a reference count: every time my coprocessor's start is called, I increment it, and in stop I decrement it. In stop, when the count drops down to 0, I call CoprocessorClassLoader.clearCache(). This fixes the problem for me. This trick doesn't work for multiple co-processors, because clearCache() would clear everything.

          Also note that clearCache() is only exposed for testing purposes so it's technically not part of the public API.

          Another workaround I can think of (but haven't tried) would be to use reflection to access the underlying map and clear out the entry.

          I think the right way to fix this bug is to maintain the reference count manually by doing the increment/decrement from the startup() and shutdown() methods of CoprocessorHost$Environment.

          Show
          Benoit Sigoure added a comment - I can further confirm this because in my current environment I use a single coprocessor, so I devised a workaround for this bug: my coprocessor class has a static int I use as a reference count: every time my coprocessor's start is called, I increment it, and in stop I decrement it. In stop , when the count drops down to 0, I call CoprocessorClassLoader.clearCache() . This fixes the problem for me. This trick doesn't work for multiple co-processors, because clearCache() would clear everything. Also note that clearCache() is only exposed for testing purposes so it's technically not part of the public API. Another workaround I can think of (but haven't tried) would be to use reflection to access the underlying map and clear out the entry. I think the right way to fix this bug is to maintain the reference count manually by doing the increment/decrement from the startup() and shutdown() methods of CoprocessorHost$Environment .
          Hide
          Benoit Sigoure added a comment -

          I think the problem is that CoprocessorClassLoader.classLoadersCache retains the previous cache loader in its cache. This is a cache that maps the path of the .jar file to its corresponding CoprocessorClassLoader. The values in the cache are weak references, but that doesn't guarantee that they will go away in a timely fashion. Therefore if you edit the schema of your table to unset the coprocessor and re-set it, most of the time you will get the same CoprocessorClassLoader as before and the new jar won't be loaded. I can reproduce this trivially and consistently on a single-node non-distributed HBase instance.

          Show
          Benoit Sigoure added a comment - I think the problem is that CoprocessorClassLoader.classLoadersCache retains the previous cache loader in its cache. This is a cache that maps the path of the .jar file to its corresponding CoprocessorClassLoader . The values in the cache are weak references, but that doesn't guarantee that they will go away in a timely fashion. Therefore if you edit the schema of your table to unset the coprocessor and re-set it, most of the time you will get the same CoprocessorClassLoader as before and the new jar won't be loaded. I can reproduce this trivially and consistently on a single-node non-distributed HBase instance.

            People

            • Assignee:
              Unassigned
              Reporter:
              iain wright
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development