Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-999

SequenceFile key class need to be compatible.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: Storage
    • Labels:
      None

      Description

      Currently, tajo uses ByteWritable class for key class in SequenceFile. If we used ByteWritable class for key class, it is no problem. But if we use another class for key class, we must find IOException. For example, When I used LongWritable class for key class in TextSerde, I found error as follows:

      ERROR org.apache.tajo.worker.Task: wrong key class: org.apache.hadoop.io.BytesWritable is not class org.apache.hadoop.io.LongWritable
      java.io.IOException: wrong key class: org.apache.hadoop.io.BytesWritable is not class org.apache.hadoop.io.LongWritable
              at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2251)
              at org.apache.tajo.storage.sequencefile.SequenceFileScanner.next(SequenceFileScanner.java:147)
      

      Thus, we need to improve SequenceFileScanner and SequenceFileScannerAppender compatible for all writable classes.

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/110

        TAJO-999: SequenceFile key class need to be compatible.

        I uploaded the patch and I finished to test it successfully on TPC-H benchmarking cluster.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-999

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/110.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #110


        commit 6932366c854c62caf1be7c442bf438774c5933e1
        Author: Jaehwa Jung <blrunner@apache.org>
        Date: 2014-08-07T16:28:59Z

        TAJO-999: SequenceFile key class need to be compatible.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/110 TAJO-999 : SequenceFile key class need to be compatible. I uploaded the patch and I finished to test it successfully on TPC-H benchmarking cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-999 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/110.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #110 commit 6932366c854c62caf1be7c442bf438774c5933e1 Author: Jaehwa Jung <blrunner@apache.org> Date: 2014-08-07T16:28:59Z TAJO-999 : SequenceFile key class need to be compatible.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/110#issuecomment-51735770

        +1

        The change looks straightforward, and the unit test looks good to me.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/110#issuecomment-51735770 +1 The change looks straightforward, and the unit test looks good to me.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/110

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/110
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/110#issuecomment-51737133

        Thanks @hyunsik.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/110#issuecomment-51737133 Thanks @hyunsik.
        Hide
        blrunner Jaehwa Jung added a comment -

        I've just committed it to the master branch.

        Show
        blrunner Jaehwa Jung added a comment - I've just committed it to the master branch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #329 (See https://builds.apache.org/job/Tajo-master-build/329/)
        TAJO-999: SequenceFile key class need to be compatible. (jaehwa) (blrunner: rev 0603b49dddb5d2b3c6f6447fa672e0ea03eae09e)

        • CHANGES
        • tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java
        • tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java
        • tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #329 (See https://builds.apache.org/job/Tajo-master-build/329/ ) TAJO-999 : SequenceFile key class need to be compatible. (jaehwa) (blrunner: rev 0603b49dddb5d2b3c6f6447fa672e0ea03eae09e) CHANGES tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development