Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None

      Description

      This patch should allow reloading classes at runtime. The software will maintain a set of extra jars to load classes from. The interface for adding and removing from this set will be done using the refresh protocol and a RefreshHandler.

      Considerations include:

      • Which classes are eligible for reloading?
      • Does reloading change existing classes?
      • Does reloading affect classes created from existing Configuration objects?

      Details will be added in comments for each patch preview.

        Activity

        Chris Li created issue -
        Chris Li made changes -
        Field Original Value New Value
        Component/s conf [ 12310711 ]
        Chris Li made changes -
        Description This patch should allow reloading classes at runtime. The software will maintain a set of extra jars to load classes from. The interface for adding and removing from this set will be done using the refresh protocol and a RefreshHandler.

        Considerations include:

        - Which classes are eligible for reloading?
        - Does reloading change existing classes?
        - Does reloading affect classes created from existing Configuration objects?

        Details will be added in comments for each patch preview.
        Hide
        Chris Li added a comment -

        First a disclaimer: not knowledgable enough in java metaprogramming to know if I’ve done something horribly wrong, this is an attempt at adding this feature.

        This patch adds runtime class reloading. The user submits a file URL (haven’t worked in support for network URLs yet) as an argument in a refresh request:

        hadoop dfsadmin -refresh nn ClassLoadPathManager.addPath file:/usr/lib/jars/new-jar-dir

        This adds the given URL to a set of URLs and causes all new Configuration objects to use a new URLClassLoader with the paths in the set of URLs the manager maintains.

        This works in my unit tests and testing on my VM cluster.

        ------------

        We instantiate a new SelectiveClassLoader each time to get around the limitation of classloaders caching their classes (and to bust the class cache in the ClassCreator)

        We have our subclass only load classes it knows in advance are reloadable, so that normal classes can be loaded by parent class loaders. This is a limitation in this way of doing things that I'm not yet sure how to work around short of reimplementing a ton of stuff in the native class loaders.

        For this patch preview, I've hardcoded some reloadable classes (but this would be a configurable property).

        Looking for feedback on this approach; thanks.

        Show
        Chris Li added a comment - First a disclaimer: not knowledgable enough in java metaprogramming to know if I’ve done something horribly wrong, this is an attempt at adding this feature. This patch adds runtime class reloading. The user submits a file URL (haven’t worked in support for network URLs yet) as an argument in a refresh request: hadoop dfsadmin -refresh nn ClassLoadPathManager.addPath file:/usr/lib/jars/new-jar-dir This adds the given URL to a set of URLs and causes all new Configuration objects to use a new URLClassLoader with the paths in the set of URLs the manager maintains. This works in my unit tests and testing on my VM cluster. ------------ We instantiate a new SelectiveClassLoader each time to get around the limitation of classloaders caching their classes (and to bust the class cache in the ClassCreator) We have our subclass only load classes it knows in advance are reloadable, so that normal classes can be loaded by parent class loaders. This is a limitation in this way of doing things that I'm not yet sure how to work around short of reimplementing a ton of stuff in the native class loaders. For this patch preview, I've hardcoded some reloadable classes (but this would be a configurable property). Looking for feedback on this approach; thanks.
        Chris Li made changes -
        Attachment HADOOP-10811-10828-preview.patch [ 12655744 ]

          People

          • Assignee:
            Chris Li
            Reporter:
            Chris Li
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development