Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1534

DelimitedTextFile return null instead of a NullDatum

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0, 0.10.1
    • Component/s: Storage
    • Labels:
      None

      Description

      If a text row is less than table schema size, vtuple returns null

      1. TAJO-1534_2.patch
        6 kB
        Jinho Kim
      2. TAJO-1534.patch
        6 kB
        Jinho Kim

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #693 (See https://builds.apache.org/job/Tajo-master-build/693/)
        TAJO-1534: DelimitedTextFile return null instead of a NullDatum. (jinho) (jhkim: rev b6b9d46318d0aaa0346578b5ecf11b9d9ba74b0c)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/text/CSVLineDeserializer.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        • CHANGES
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #693 (See https://builds.apache.org/job/Tajo-master-build/693/ ) TAJO-1534 : DelimitedTextFile return null instead of a NullDatum. (jinho) (jhkim: rev b6b9d46318d0aaa0346578b5ecf11b9d9ba74b0c) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/text/CSVLineDeserializer.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java CHANGES
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-master-CODEGEN-build #332 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/332/)
        TAJO-1534: DelimitedTextFile return null instead of a NullDatum. (jinho) (jhkim: rev b6b9d46318d0aaa0346578b5ecf11b9d9ba74b0c)

        • CHANGES
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/text/CSVLineDeserializer.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #332 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/332/ ) TAJO-1534 : DelimitedTextFile return null instead of a NullDatum. (jinho) (jhkim: rev b6b9d46318d0aaa0346578b5ecf11b9d9ba74b0c) CHANGES tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestStorages.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/text/CSVLineDeserializer.java
        Hide
        jhkim Jinho Kim added a comment -

        committed it

        Show
        jhkim Jinho Kim added a comment - committed it
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/522

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/522
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/522#issuecomment-99299127

        @blrunner
        Thank you for your review. I will commit soon

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/522#issuecomment-99299127 @blrunner Thank you for your review. I will commit soon
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/522#issuecomment-98460589

        +1

        It looks good to me and all unit test cases finished successfully.
        And if possible, could you add 0.11.0 to the fix version of JIRA issue?

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/522#issuecomment-98460589 +1 It looks good to me and all unit test cases finished successfully. And if possible, could you add 0.11.0 to the fix version of JIRA issue?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/522#discussion_r28577893

        — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java —
        @@ -1255,16 +1255,21 @@ public void init() throws IOException {

        for (int i = 0; i < targetColumnIndexes.length; i++) {
        int tid = targetColumnIndexes[i];
        + SelectedColumn col = new SelectedColumn();
        if (tid < columnNumber)

        { skippedColIDs[tid] = false; - SelectedColumn col = new SelectedColumn(); col.colIndex = tid; col.runLength = 0; col.prvLength = -1; col.rowReadIndex = 0; selectedColumns[i] = col; colValLenBufferReadIn[i] = new NonSyncDataInputBuffer(); + }

        else {
        — End diff –

        Thank you for nice finding.
        I've remove the duplicated codes

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/522#discussion_r28577893 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java — @@ -1255,16 +1255,21 @@ public void init() throws IOException { for (int i = 0; i < targetColumnIndexes.length; i++) { int tid = targetColumnIndexes [i] ; + SelectedColumn col = new SelectedColumn(); if (tid < columnNumber) { skippedColIDs[tid] = false; - SelectedColumn col = new SelectedColumn(); col.colIndex = tid; col.runLength = 0; col.prvLength = -1; col.rowReadIndex = 0; selectedColumns[i] = col; colValLenBufferReadIn[i] = new NonSyncDataInputBuffer(); + } else { — End diff – Thank you for nice finding. I've remove the duplicated codes
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/522#discussion_r28572801

        — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java —
        @@ -1255,16 +1255,21 @@ public void init() throws IOException {

        for (int i = 0; i < targetColumnIndexes.length; i++) {
        int tid = targetColumnIndexes[i];
        + SelectedColumn col = new SelectedColumn();
        if (tid < columnNumber)

        { skippedColIDs[tid] = false; - SelectedColumn col = new SelectedColumn(); col.colIndex = tid; col.runLength = 0; col.prvLength = -1; col.rowReadIndex = 0; selectedColumns[i] = col; colValLenBufferReadIn[i] = new NonSyncDataInputBuffer(); + }

        else {
        — End diff –

        The newly added code is the mostly same as those of existing codes.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/522#discussion_r28572801 — Diff: tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java — @@ -1255,16 +1255,21 @@ public void init() throws IOException { for (int i = 0; i < targetColumnIndexes.length; i++) { int tid = targetColumnIndexes [i] ; + SelectedColumn col = new SelectedColumn(); if (tid < columnNumber) { skippedColIDs[tid] = false; - SelectedColumn col = new SelectedColumn(); col.colIndex = tid; col.runLength = 0; col.prvLength = -1; col.rowReadIndex = 0; selectedColumns[i] = col; colValLenBufferReadIn[i] = new NonSyncDataInputBuffer(); + } else { — End diff – The newly added code is the mostly same as those of existing codes.
        Hide
        tajoqa Tajo QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12723601/TAJO-1534.patch
        against master revision release-0.9.0-rc0-244-gd160f6e.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-storage/tajo-storage-hdfs.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/722//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/722//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12723601/TAJO-1534.patch against master revision release-0.9.0-rc0-244-gd160f6e. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-storage/tajo-storage-hdfs. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/722//testReport/ Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/722//console This message is automatically generated.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/522

        TAJO-1534: DelimitedTextFile should be returned NullDatum instead of null

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1534

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/522.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #522


        commit 3e734d96a9e7d76e4efc2acdc848873eae365227
        Author: Jinho Kim <jhkim@apache.org>
        Date: 2015-04-07T10:17:42Z

        TAJO-1534: DelimitedTextFile should be returned NullDatum instead of null.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/522 TAJO-1534 : DelimitedTextFile should be returned NullDatum instead of null You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1534 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/522.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #522 commit 3e734d96a9e7d76e4efc2acdc848873eae365227 Author: Jinho Kim <jhkim@apache.org> Date: 2015-04-07T10:17:42Z TAJO-1534 : DelimitedTextFile should be returned NullDatum instead of null.

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development