Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-223

JobClient should work with -1/+1 version of JobTracker

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

      Description

      Currently there is version check on the RPC calls that enforces the same Hadoop version on the client and the server.

      To enable phased upgrades of systems using Hadoop and Hadoop itself the JobClient should be able to interact with a JobTracker of the previous and the next version of Hadoop (or with a range).

        Activity

        Hide
        Allen Wittenauer added a comment -

        I need a state between stale and fixed.

        Show
        Allen Wittenauer added a comment - I need a state between stale and fixed.
        Hide
        Alejandro Abdelnur added a comment -

        Wasn't the idea that Avro would help fixing this?

        Yes, doing things over HTTP (assuming you take care of not breaking things a payload level) works.

        Still Hadoop does not support HTTP natively for client side calls, so this is not option without add-on protocol adapter systems fronting JT and NN/DNs. In other words, a JT proxy and a HDFS proxy.

        FYI, Oozie is planning to provide JT proxy capabilities.

        Show
        Alejandro Abdelnur added a comment - Wasn't the idea that Avro would help fixing this? Yes, doing things over HTTP (assuming you take care of not breaking things a payload level) works. Still Hadoop does not support HTTP natively for client side calls, so this is not option without add-on protocol adapter systems fronting JT and NN/DNs. In other words, a JT proxy and a HDFS proxy. FYI, Oozie is planning to provide JT proxy capabilities.
        Hide
        Krishna Ramachandran added a comment -

        it has been sitting for over 2 years and I do not believe anything has changed. hdfs I believe provide read only interface for listing/retrieving data over http.

        Is this still critical - to have similar interface to embedded JT http server on top of what the web interface already provides (for accessing task or job logs?)

        Show
        Krishna Ramachandran added a comment - it has been sitting for over 2 years and I do not believe anything has changed. hdfs I believe provide read only interface for listing/retrieving data over http. Is this still critical - to have similar interface to embedded JT http server on top of what the web interface already provides (for accessing task or job logs?)
        Hide
        Alejandro Abdelnur added a comment -

        And doing it as my second option? over HTTP to isolate rpc versions as HDFS over HTTP

        Show
        Alejandro Abdelnur added a comment - And doing it as my second option? over HTTP to isolate rpc versions as HDFS over HTTP
        Hide
        Owen O'Malley added a comment -

        This requires more version tolerant rpc. The current version of rpc can't do it and so this is a big project.

        Show
        Owen O'Malley added a comment - This requires more version tolerant rpc. The current version of rpc can't do it and so this is a big project.
        Hide
        Alejandro Abdelnur added a comment -

        This could be done by either allowing a range version of clients on the JobTracker, or having an HTTP transport with potentially reduced functionality (like the HDFS over HTTP).

        Show
        Alejandro Abdelnur added a comment - This could be done by either allowing a range version of clients on the JobTracker , or having an HTTP transport with potentially reduced functionality (like the HDFS over HTTP).

          People

          • Assignee:
            Unassigned
            Reporter:
            Alejandro Abdelnur
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development