HBase
  1. HBase
  2. HBASE-8607

Allow custom filters and coprocessors to be updated for a region server without requiring a restart

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: regionserver
    • Labels:
      None
    • Tags:
      Phoenix

      Description

      One solution to allowing custom filters and coprocessors to be updated for a region server without requiring a restart might be to run the HBase server in an OSGi container (maybe there are other approaches as well?). Typically, applications that use coprocessors and custom filters also have shared classes underneath, so putting the burden on the user to include some kind of version name in the class is not adequate. Including the version name in the package might work in some cases (at least until dependent jars start to change as well), but is cumbersome and overburdens the app developer.

      Regardless of what approach is taken, we'd need to define the life cycle of the coprocessors and custom filters when a new version is loaded. For example, in-flight invocations could continue to use the old version while new invocations would use the new ones. Once the in-flight invocations are complete, the old code/jar could be unloaded.

        Activity

        Hide
        James Taylor added a comment -

        My thinking is that an OSGi container would allow a new version of a coprocessor (and/or custom filter) jar to be loaded. Class conflicts between the old jar and the new jar would no longer be a problem - you'd never need to unload the old jar. Instead, future HBase operations that invoke the coprocessor would cause the newly loaded jar to be used instead of the older one. I'm not sure if this is possible or not. The whole idea would be to prevent a rolling restart or region close/reopen.

        Show
        James Taylor added a comment - My thinking is that an OSGi container would allow a new version of a coprocessor (and/or custom filter) jar to be loaded. Class conflicts between the old jar and the new jar would no longer be a problem - you'd never need to unload the old jar. Instead, future HBase operations that invoke the coprocessor would cause the newly loaded jar to be used instead of the older one. I'm not sure if this is possible or not. The whole idea would be to prevent a rolling restart or region close/reopen.
        Hide
        Andrew Purtell added a comment -

        As far as I know, OSGi does not support hot class replacement, but you can shut down, reload, and restart bundles/subsystems. This will only work as far as the interfaces and objects exchanged between the bundle and everything else doesn't change. For this to be possible with coprocessors, we would need to I guess refactor the server into a bundle then have coprocessors and filters also be structured and loaded as a bundle. As an alternative to hosting HBase itself in an OSGi runtime, we might embed the runtime as the coprocessor host and refactor only coprocessors and filters as bundles. So that would imply some kind of filter host environment too or a merging of filters and coprocessors.

        That is not a problem per se but an incompatible change and major surgery. We should prototype this if serious to see if an OSGi runtime (such as Apache Felix) can actually reliably do this. I look at Eclipse as an example of OSGi in action and wonder. When designing coprocessors we felt it easier and more reliable to require a process reload - guaranteed to work under every circumstance.

        As to whether or not we should do a quick reload, which might possibly be minimized to a close then reopen of a region when a coprocessor is reloaded, consider the circumstances of hot reload of a coprocessor in a regionserver with many ops in flight. What will the internal state of the coprocessor look like? Will there be cross-op dependencies? Currently at the hook points we enumerate a CopyOnWriteList to find installed coprocessors. The swap of the old coprocessor instance for the new will be fast and lockless but will happen at an arbitrary point in the op stream and for some brief period of time old and new instances will be simultaneously active. Replace this with OSGi particulars and the coprocessor side issues don't change. For this reason I think it best to have the coprocessor lifecycle tied to the region or regionserver lifecycle anyway.

        Show
        Andrew Purtell added a comment - As far as I know, OSGi does not support hot class replacement, but you can shut down, reload, and restart bundles/subsystems. This will only work as far as the interfaces and objects exchanged between the bundle and everything else doesn't change. For this to be possible with coprocessors, we would need to I guess refactor the server into a bundle then have coprocessors and filters also be structured and loaded as a bundle. As an alternative to hosting HBase itself in an OSGi runtime, we might embed the runtime as the coprocessor host and refactor only coprocessors and filters as bundles. So that would imply some kind of filter host environment too or a merging of filters and coprocessors. That is not a problem per se but an incompatible change and major surgery. We should prototype this if serious to see if an OSGi runtime (such as Apache Felix) can actually reliably do this. I look at Eclipse as an example of OSGi in action and wonder. When designing coprocessors we felt it easier and more reliable to require a process reload - guaranteed to work under every circumstance. As to whether or not we should do a quick reload, which might possibly be minimized to a close then reopen of a region when a coprocessor is reloaded, consider the circumstances of hot reload of a coprocessor in a regionserver with many ops in flight. What will the internal state of the coprocessor look like? Will there be cross-op dependencies? Currently at the hook points we enumerate a CopyOnWriteList to find installed coprocessors. The swap of the old coprocessor instance for the new will be fast and lockless but will happen at an arbitrary point in the op stream and for some brief period of time old and new instances will be simultaneously active. Replace this with OSGi particulars and the coprocessor side issues don't change. For this reason I think it best to have the coprocessor lifecycle tied to the region or regionserver lifecycle anyway.
        Hide
        Andrew Purtell added a comment - - edited

        Another option is to consider external coprocessor hosts. Then a reload is a process restart but involving only the coprocessor / child JVM. That is HBASE-4047.

        Show
        Andrew Purtell added a comment - - edited Another option is to consider external coprocessor hosts. Then a reload is a process restart but involving only the coprocessor / child JVM. That is HBASE-4047 .
        Hide
        stack added a comment -

        If a CP were in an external process, the Interface would have to be more coarse than it is currently, right? CPs would not be able to be as intimate as they are now if their invocation requires a trip to an external process?

        Show
        stack added a comment - If a CP were in an external process, the Interface would have to be more coarse than it is currently, right? CPs would not be able to be as intimate as they are now if their invocation requires a trip to an external process?
        Hide
        Andrew Purtell added a comment -

        External coprocessors wouldn't have arbitrary access to regionserver internals. There might be a way to keep them inline with RS processing as observers as we have now though, maybe using a shared memory channel. I'd not want to say without trying something out and measuring what we can get. It would be safe to assume OSGi/in process as always having more access and bandwidth.

        Show
        Andrew Purtell added a comment - External coprocessors wouldn't have arbitrary access to regionserver internals. There might be a way to keep them inline with RS processing as observers as we have now though, maybe using a shared memory channel. I'd not want to say without trying something out and measuring what we can get. It would be safe to assume OSGi/in process as always having more access and bandwidth.
        Hide
        Andrew Purtell added a comment -

        Something might worth trying out is an OSGi socket as coprocessor. Like with HBASE-4047 where one side of the umbilical is a "normal" coprocessor, here that is instead a shim for another piece of code that is (re)loaded as a bundle. Would give us a sense of how workable embedding an OSGi runtime and reloading a coprocessor might be.

        Show
        Andrew Purtell added a comment - Something might worth trying out is an OSGi socket as coprocessor. Like with HBASE-4047 where one side of the umbilical is a "normal" coprocessor, here that is instead a shim for another piece of code that is (re)loaded as a bundle. Would give us a sense of how workable embedding an OSGi runtime and reloading a coprocessor might be.
        Hide
        stack added a comment -

        Andrew Purtell I like the direction you suggest; would proof out two concepts in the one go (osgi and moving cp hosting out of the regionserver).

        Show
        stack added a comment - Andrew Purtell I like the direction you suggest; would proof out two concepts in the one go (osgi and moving cp hosting out of the regionserver).
        Hide
        Nicolas Liochon added a comment -

        Running the coprocessor within the region server removes the problem of managing the life cycle of two different processes, with all the questions of "the coprocessor takes time, is it dead? Is it gc-ing?" (with the same questions on the coprocessor side: "I'm not called, why?").

        btw, do we manage today the fact that we can have conflicts between hbase third parties and the coprocessor ones?

        While OSGi is nice (and allows more features on the long term), I also think that the rolling restart should work well enough (or should be made working well enough) for most cases, as it shares a lot of code with stuff like balance or recovery.

        Show
        Nicolas Liochon added a comment - Running the coprocessor within the region server removes the problem of managing the life cycle of two different processes, with all the questions of "the coprocessor takes time, is it dead? Is it gc-ing?" (with the same questions on the coprocessor side: "I'm not called, why?"). btw, do we manage today the fact that we can have conflicts between hbase third parties and the coprocessor ones? While OSGi is nice (and allows more features on the long term), I also think that the rolling restart should work well enough (or should be made working well enough) for most cases, as it shares a lot of code with stuff like balance or recovery.
        Hide
        Andrew Purtell added a comment -

        I like the direction you suggest; would proof out two concepts in the one go (osgi and moving cp hosting out of the regionserver).

        stack It would demonstrate if an OSGi runtime can be embedded and if CPs hosted by it can be unloaded or reloaded, but everything would all still be in the same process.

        Show
        Andrew Purtell added a comment - I like the direction you suggest; would proof out two concepts in the one go (osgi and moving cp hosting out of the regionserver). stack It would demonstrate if an OSGi runtime can be embedded and if CPs hosted by it can be unloaded or reloaded, but everything would all still be in the same process.
        Hide
        James Taylor added a comment -

        To a large degree, Phoenix gets it's performance from being able to run inside the JVM of the region server, so if I understand this idea, we'd lose that with this approach.

        I'm no OSGi expert, but I talked with some folks here, and it seems that we could do what we want through OSGi and swap in new jars without requiring a rolling restart.

        Might be worth a visit here to explore? I'm happy to set something up.

        Show
        James Taylor added a comment - To a large degree, Phoenix gets it's performance from being able to run inside the JVM of the region server, so if I understand this idea, we'd lose that with this approach. I'm no OSGi expert, but I talked with some folks here, and it seems that we could do what we want through OSGi and swap in new jars without requiring a rolling restart. Might be worth a visit here to explore? I'm happy to set something up.
        Hide
        Nick Dimiduk added a comment -

        Running the coprocessor within the region server removes the problem of managing the life cycle of two different processes, with all the questions of "the coprocessor takes time, is it dead? Is it gc-ing?" (with the same questions on the coprocessor side: "I'm not called, why?").

        Coprocessors remain a reasonably advanced feature. I think the benefits to a novice user of running coprocessors as a managed child process out-weight the pains (which are easily worked around) for an advanced user.

        There might be a way to keep them inline with RS processing as observers as we have now though, maybe using a shared memory channel.

        This post [0] is an inspiring read for anyone considering an external process implementation. We would require some work to achieve the word alignment that's critical to the performance numbers posted here, but I think it's worth a shot.

        [0]: http://psy-lob-saw.blogspot.co.uk/2013/04/lock-free-ipc-queue.html

        Show
        Nick Dimiduk added a comment - Running the coprocessor within the region server removes the problem of managing the life cycle of two different processes, with all the questions of "the coprocessor takes time, is it dead? Is it gc-ing?" (with the same questions on the coprocessor side: "I'm not called, why?"). Coprocessors remain a reasonably advanced feature. I think the benefits to a novice user of running coprocessors as a managed child process out-weight the pains (which are easily worked around) for an advanced user. There might be a way to keep them inline with RS processing as observers as we have now though, maybe using a shared memory channel. This post [0] is an inspiring read for anyone considering an external process implementation. We would require some work to achieve the word alignment that's critical to the performance numbers posted here, but I think it's worth a shot. [0] : http://psy-lob-saw.blogspot.co.uk/2013/04/lock-free-ipc-queue.html
        Hide
        Andrew Purtell added a comment - - edited

        James, I think you may have missed some discussion further up on this issue. Specifically:

        Something might worth trying out is an OSGi socket as coprocessor. Like with HBASE-4047 where one side of the umbilical is a "normal" coprocessor, here that is instead a shim for another piece of code that is (re)loaded as a bundle. Would give us a sense of how workable embedding an OSGi runtime and reloading a coprocessor might be.

        Stated a different way, we create a coprocessor that embeds an OSGi runtime, perhaps Apache Felix. This coprocessor is just a shell. It is configured to load an OSGi bundle, from another path. Handlers for all upcalls are delegated to the loaded bundle. We can expose an Endpoint for triggering bundle reload. So, sure, we can play around with this and see if it works, without changing anything core, since the OSGi glue would be a coprocessor itself. What do you think? Meet up about this some time around HBaseCon?

        I'm no OSGi expert, but I talked with some folks here, and it seems that we could do what we want through OSGi and swap in new jars without requiring a rolling restart.

        Can you (or one of your folks) address this:

        As to whether or not we should do a quick reload, which might possibly be minimized to a close then reopen of a region when a coprocessor is reloaded, consider the circumstances of hot reload of a coprocessor in a regionserver with many ops in flight. What will the internal state of the coprocessor look like? Will there be cross-op dependencies? Currently at the hook points we enumerate a CopyOnWriteList to find installed coprocessors. The swap of the old coprocessor instance for the new will be fast and lockless but will happen at an arbitrary point in the op stream and for some brief period of time old and new instances will be simultaneously active. Replace this with OSGi particulars and the coprocessor side issues don't change. For this reason I think it best to have the coprocessor lifecycle tied to the region or regionserver lifecycle anyway.

        What happens if a scan is in progress and Phoenix is upgraded? The new instance would not have been around when the scanner was opened, yet it will start getting upcalls from scanner.next(). Do we need any special considerations for this? Or can we say that Phoenix, or any other coprocessor reloaded this way, must handle this somehow through state sharing between instances?

        Show
        Andrew Purtell added a comment - - edited James, I think you may have missed some discussion further up on this issue. Specifically: Something might worth trying out is an OSGi socket as coprocessor. Like with HBASE-4047 where one side of the umbilical is a "normal" coprocessor, here that is instead a shim for another piece of code that is (re)loaded as a bundle. Would give us a sense of how workable embedding an OSGi runtime and reloading a coprocessor might be. Stated a different way, we create a coprocessor that embeds an OSGi runtime, perhaps Apache Felix. This coprocessor is just a shell. It is configured to load an OSGi bundle, from another path. Handlers for all upcalls are delegated to the loaded bundle. We can expose an Endpoint for triggering bundle reload. So, sure, we can play around with this and see if it works, without changing anything core, since the OSGi glue would be a coprocessor itself. What do you think? Meet up about this some time around HBaseCon? I'm no OSGi expert, but I talked with some folks here, and it seems that we could do what we want through OSGi and swap in new jars without requiring a rolling restart. Can you (or one of your folks) address this: As to whether or not we should do a quick reload, which might possibly be minimized to a close then reopen of a region when a coprocessor is reloaded, consider the circumstances of hot reload of a coprocessor in a regionserver with many ops in flight. What will the internal state of the coprocessor look like? Will there be cross-op dependencies? Currently at the hook points we enumerate a CopyOnWriteList to find installed coprocessors. The swap of the old coprocessor instance for the new will be fast and lockless but will happen at an arbitrary point in the op stream and for some brief period of time old and new instances will be simultaneously active. Replace this with OSGi particulars and the coprocessor side issues don't change. For this reason I think it best to have the coprocessor lifecycle tied to the region or regionserver lifecycle anyway. What happens if a scan is in progress and Phoenix is upgraded? The new instance would not have been around when the scanner was opened, yet it will start getting upcalls from scanner.next(). Do we need any special considerations for this? Or can we say that Phoenix, or any other coprocessor reloaded this way, must handle this somehow through state sharing between instances?

          People

          • Assignee:
            Unassigned
            Reporter:
            James Taylor
          • Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:

              Development