Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I have been running some tests getting hadoop to run within an osgi environment (specifically the Newton framework) and this has uncovered a number of minor bugs when mapred classes are instantiated from a different start point than their main methods.

      I have created a number of patches which I'll attach which solve these issues. It's possible these patches could be dealt with as separate issues but all are required to resolve the osgi issue. Happy to split up if easier to manage though.

      classpath.patch: this rearranges the classloader hierarchies for Task objects such that a Task is able to resolve api classes in the case where the api classes are no longer loaded from the system classloader.

      tasklog.patch: this ensures the log files are able to be resolved in the case where the child process is launched from a different directory to the parent process

      taskrunner.patch: this enables the TaskRunner to find a log dir in the case where the parent jvm is not launched by the hadoop scripts, also allows for a client to specify a substitute main class (which delegates to the TaskTracker$Child) in this case for purposes of resolving osgi classpaths but could be more general? Finally adds some extra logging in case where things go wrong.

      tasktracker.patch: allow parent to pass through configuration to child taskrunner (specifically in this case for purposes of passing classpath and laucher to taskrunner)

      1. tasktracker.patch
        0.9 kB
        David Savage
      2. taskrunner.patch
        2 kB
        David Savage
      3. tasklog.patch
        0.8 kB
        David Savage
      4. classpath.patch
        11 kB
        David Savage

        Issue Links

          Activity

          David Savage created issue -
          David Savage made changes -
          Field Original Value New Value
          Attachment tasktracker.patch [ 12365268 ]
          David Savage made changes -
          Attachment classpath.patch [ 12365269 ]
          David Savage made changes -
          Attachment tasklog.patch [ 12365270 ]
          David Savage made changes -
          Attachment taskrunner.patch [ 12365271 ]
          Christophe Taton made changes -
          Assignee Christophe Taton [ kryzthov ]
          Christophe Taton made changes -
          Link This issue is blocked by HADOOP-1650 [ HADOOP-1650 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-1844 MAPREDUCE-243
          Component/s mapred [ 12310690 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-7977 [ HADOOP-7977 ]
          Jean-Baptiste Onofré made changes -
          Assignee Christophe Taton [ kryzthov ] Jean-Baptiste Onofré [ jbonofre ]
          Gavin made changes -
          Link This issue blocks HADOOP-7977 [ HADOOP-7977 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-7977 [ HADOOP-7977 ]

            People

            • Assignee:
              Jean-Baptiste Onofré
              Reporter:
              David Savage
            • Votes:
              3 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:

                Development