Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10396

Using -diff option with DistCp may get "Comparison method violates its general contract" exception

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Using -diff option get the following exception due to a bug in the comparison operator:

      16/04/21 14:34:18 WARN tools.DistCp: Failed to use snapshot diff for distcp
      java.lang.IllegalArgumentException: Comparison method violates its general contract!
      	at java.util.TimSort.mergeHi(TimSort.java:868)
      	at java.util.TimSort.mergeAt(TimSort.java:485)
      	at java.util.TimSort.mergeForceCollapse(TimSort.java:426)
      	at java.util.TimSort.sort(TimSort.java:223)
      	at java.util.TimSort.sort(TimSort.java:173)
      	at java.util.Arrays.sort(Arrays.java:659)
      	at org.apache.hadoop.tools.DistCpSync.moveToTarget(DistCpSync.java:293)
      	at org.apache.hadoop.tools.DistCpSync.syncDiff(DistCpSync.java:261)
      	at org.apache.hadoop.tools.DistCpSync.sync(DistCpSync.java:131)
      	at org.apache.hadoop.tools.DistCp.execute(DistCp.java:163)
      	at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
      16/04/21 14:34:18 ERROR tools.DistCp: Exception encountered 
      
      

        Attachments

          Activity

            People

            • Assignee:
              yzhangal Yongjun Zhang
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: