Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-2377

RandomWriter ends up using TextOutputFormat instead of SequenceFileOutputFormat

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar randomwriter "-Dmapreduce.randomwriter.totalbytes=10737418" /tmp/test1
      

      This ends up generating TextOutputFormat.

      yarn jar ./dist/tez/tez-tests-0.7.0-SNAPSHOT.jar sort  
      "-Dmapreduce.framework.name=yarn-tez" -r 5 /tmp/test1 /tmp/test_sorted
      

      This ends up throwing error

      Failure while running task:java.io.IOException: hdfs://tez-vm:56565/tmp/test1/part-00000 not a SequenceFile
              at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1851)
              at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1811)
              at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1760)
              at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1774)
              at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
              at org.apache.tez.mapreduce.lib.MRReaderMapReduce.setupNewRecordReader(MRReaderMapReduce.java:149)
              at org.apache.tez.mapreduce.lib.MRReaderMapReduce.<init>(MRReaderMapReduce.java:78)
              at org.apache.tez.mapreduce.input.MRInput.initializeInternal(MRInput.java:475)
              at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:108)
      

        Attachments

        1. TEZ-2377.1.patch
          0.9 kB
          Rajesh Balamohan

          Activity

            People

            • Assignee:
              rajesh.balamohan Rajesh Balamohan
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: