Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.2-incubating
    • Fix Version/s: 0.8.0
    • Component/s: Storage
    • Labels:
      None

      Description

      If tail of row remained in buffer, scanner sometimes are terminated at the end of the block.
      In addition, I'll improve writer performance in CSVFile.

      1. TAJO-332_2.patch
        52 kB
        Jinho Kim
      2. TAJO-332.patch
        54 kB
        Jinho Kim

        Activity

        Hide
        jhkim Jinho Kim added a comment -

        I've attached the patch. please review this patch.

        Show
        jhkim Jinho Kim added a comment - I've attached the patch. please review this patch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        The patch looks good for me. Bytes::writeVLongToByteArray() method and Bytes::writeVLong seem strange. It is not a static method and unused in the project.

        Show
        hyunsik Hyunsik Choi added a comment - The patch looks good for me. Bytes::writeVLongToByteArray() method and Bytes::writeVLong seem strange. It is not a static method and unused in the project.
        Hide
        jhkim Jinho Kim added a comment -

        Thank you for the review. I've uploaded the second patch.

        Show
        jhkim Jinho Kim added a comment - Thank you for the review. I've uploaded the second patch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1

        I verified 'mvn clean install'. The patch looks good for me.

        Show
        hyunsik Hyunsik Choi added a comment - +1 I verified 'mvn clean install'. The patch looks good for me.
        Hide
        jhkim Jinho Kim added a comment -

        I've just committed it.
        Thanks.

        Show
        jhkim Jinho Kim added a comment - I've just committed it. Thanks.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-trunk-postcommit #568 (See https://builds.apache.org/job/Tajo-trunk-postcommit/568/)
        TAJO-332: Invalid row count of CSVScanner. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=b8435e7c2dd9999023cca6040b33427dd886c2ae)

        • tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/LazyTuple.java
        • tajo-common/src/main/java/org/apache/tajo/util/Bytes.java
        • tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
        • tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/LineReader.java
        • tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/TextSerializeDeserialize.java
        • CHANGES.txt
        • tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-trunk-postcommit #568 (See https://builds.apache.org/job/Tajo-trunk-postcommit/568/ ) TAJO-332 : Invalid row count of CSVScanner. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=b8435e7c2dd9999023cca6040b33427dd886c2ae ) tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/LazyTuple.java tajo-common/src/main/java/org/apache/tajo/util/Bytes.java tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/LineReader.java tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/TextSerializeDeserialize.java CHANGES.txt tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development