Hadoop Common
  1. Hadoop Common
  2. HADOOP-7738

Document incompatible API changes between 0.20.20x and 0.23.0 release

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      0.20.20x to 0.23.0 will be a common upgrade path, so we should document any incompatible API changes that will affect users.

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          Here's a preliminary report showing the incompatible changes between 0.20.204.0 and trunk (I'll run it against 0.23 after HADOOP-7642 is in the 23 branch, but there aren't significant API differences between the two). There are a number of false positives here, but I'd like to go through the rest to see if they are significant.

          I used https://github.com/tomwhite/hadoop-compatibility-tools to generate the report. See also HADOOP-7035.

          Show
          Tom White added a comment - Here's a preliminary report showing the incompatible changes between 0.20.204.0 and trunk (I'll run it against 0.23 after HADOOP-7642 is in the 23 branch, but there aren't significant API differences between the two). There are a number of false positives here, but I'd like to go through the rest to see if they are significant. I used https://github.com/tomwhite/hadoop-compatibility-tools to generate the report. See also HADOOP-7035 .
          Hide
          Mahadev konar added a comment -

          Nice work Tom. As you stated, most of them look like false positives. How do you intend to document the incompatible changes? Just checking in the SigTest output?

          Show
          Mahadev konar added a comment - Nice work Tom. As you stated, most of them look like false positives. How do you intend to document the incompatible changes? Just checking in the SigTest output?
          Hide
          Tom White added a comment -

          I would like to eliminate the false positives (e.g. by excluding them from SigTest), and (time permitting) go through the remaining ones so they can either be fixed if possible, or documented in release notes.

          Show
          Tom White added a comment - I would like to eliminate the false positives (e.g. by excluding them from SigTest), and (time permitting) go through the remaining ones so they can either be fixed if possible, or documented in release notes.
          Hide
          Eli Collins added a comment -

          Should do this in 23, doesn't have to be .1 though.

          Show
          Eli Collins added a comment - Should do this in 23, doesn't have to be .1 though.
          Hide
          Tom White added a comment -

          Here are some notes on the differences I have found so far between the 0.20.x release series and 0.23.0 which break compatibility in some way.

          • MAPREDUCE-954 changed some context classes to interfaces (e.g. JobContext, MapContext, TaskAttemptContext, TaskInputOutputContext). This change should not impact user code (since such code doesn't implement these interfaces) although it does mean that user code (including libraries like Pig) will need to be recompiled. See note (1) at http://wiki.eclipse.org/Evolving_Java-based_APIs_2#Evolving_API_packages
          • MAPREDUCE-901 changes Counter from a class to an interface. Clients need to recompile.
          • HADOOP-6201 changed FileSystem#listStatus to throw FileNotFoundException when the file is not found, rather than returning null. Clients need to review usage of this method and update their code to handle this case.
          Show
          Tom White added a comment - Here are some notes on the differences I have found so far between the 0.20.x release series and 0.23.0 which break compatibility in some way. MAPREDUCE-954 changed some context classes to interfaces (e.g. JobContext, MapContext, TaskAttemptContext, TaskInputOutputContext). This change should not impact user code (since such code doesn't implement these interfaces) although it does mean that user code (including libraries like Pig) will need to be recompiled. See note (1) at http://wiki.eclipse.org/Evolving_Java-based_APIs_2#Evolving_API_packages MAPREDUCE-901 changes Counter from a class to an interface. Clients need to recompile. HADOOP-6201 changed FileSystem#listStatus to throw FileNotFoundException when the file is not found, rather than returning null. Clients need to review usage of this method and update their code to handle this case.
          Hide
          Thomas Weise added a comment -

          For HCatalog, following are the differences beyond recompile:

          diff shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java shims/src/20S/java/org/apache/hcatalog/shims/HCatHadoopShims20S.java
          25,26d24
          < import org.apache.hadoop.mapreduce.task.JobContextImpl;
          < import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
          28c26
          < public class HCatHadoopShims23 implements HCatHadoopShims {
          ---
          > public class HCatHadoopShims20S implements HCatHadoopShims {
          33,34c31,32
          <         return new TaskAttemptContextImpl(conf, taskId);
          < 	}
          ---
          >         return new TaskAttemptContext(conf, taskId);
          >     }
          36c34
          < 	@Override
          ---
          >     @Override
          39,40c37
          <         JobContext newContext = new JobContextImpl(conf, jobId);
          <         return newContext;
          ---
          >         return new JobContext(conf, jobId);
          42d38
          < 
          
          Show
          Thomas Weise added a comment - For HCatalog, following are the differences beyond recompile: diff shims/src/23/java/org/apache/hcatalog/shims/HCatHadoopShims23.java shims/src/20S/java/org/apache/hcatalog/shims/HCatHadoopShims20S.java 25,26d24 < import org.apache.hadoop.mapreduce.task.JobContextImpl; < import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl; 28c26 < public class HCatHadoopShims23 implements HCatHadoopShims { --- > public class HCatHadoopShims20S implements HCatHadoopShims { 33,34c31,32 < return new TaskAttemptContextImpl(conf, taskId); < } --- > return new TaskAttemptContext(conf, taskId); > } 36c34 < @Override --- > @Override 39,40c37 < JobContext newContext = new JobContextImpl(conf, jobId); < return newContext; --- > return new JobContext(conf, jobId); 42d38 <
          Hide
          Harsh J added a comment -

          Remarking target versions, since this seems to be open as a blocker.

          Show
          Harsh J added a comment - Remarking target versions, since this seems to be open as a blocker.
          Hide
          Jonathan Eagles added a comment -

          Thoughts on what to do with this ticket. 0.23.x is in maintenance mode and add the docs to 0.23.x at this point may be too late. At this point a much more likely upgrade target will be from 1.x to 2.x.

          Show
          Jonathan Eagles added a comment - Thoughts on what to do with this ticket. 0.23.x is in maintenance mode and add the docs to 0.23.x at this point may be too late. At this point a much more likely upgrade target will be from 1.x to 2.x.
          Hide
          Jonathan Eagles added a comment -

          I've retargeted this for 2.5.0 since this is still an issue.

          Show
          Jonathan Eagles added a comment - I've retargeted this for 2.5.0 since this is still an issue.
          Hide
          Karthik Kambatla added a comment -

          We should probably change the title to reflect that this is repurposed to capture 1.x -> 2.x. That said, I believe the status of API compatibility between 1.x and 2.x is reasonably well know at this point. I wonder if there is much to gain from this documentation at this point.

          In any case, do we want this to block 2.5?

          Show
          Karthik Kambatla added a comment - We should probably change the title to reflect that this is repurposed to capture 1.x -> 2.x. That said, I believe the status of API compatibility between 1.x and 2.x is reasonably well know at this point. I wonder if there is much to gain from this documentation at this point. In any case, do we want this to block 2.5?
          Hide
          Karthik Kambatla added a comment -

          Moving to 2.6 and demoted to a Critical issue. Please update if you strongly feel differently.

          Show
          Karthik Kambatla added a comment - Moving to 2.6 and demoted to a Critical issue. Please update if you strongly feel differently.

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:

                Development