Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1242

Json scanner can not read some case of truncated text

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.10.0
    • Component/s: Storage
    • Labels:
      None

      Description

      This error caused by complex type of truncated json.

      sample test : {"id":[{"text":"json test
      
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 25
      	at net.minidev.json.parser.JSONParserByteArray.indexOf(JSONParserByteArray.java:78)
      	at net.minidev.json.parser.JSONParserMemory.readString(JSONParserMemory.java:120)
      	at net.minidev.json.parser.JSONParserBase.readMain(JSONParserBase.java:394)
      	at net.minidev.json.parser.JSONParserBase.readObject(JSONParserBase.java:530)
      	at net.minidev.json.parser.JSONParserBase.readMain(JSONParserBase.java:387)
      	at net.minidev.json.parser.JSONParserBase.readArray(JSONParserBase.java:257)
      	at net.minidev.json.parser.JSONParserBase.readMain(JSONParserBase.java:390)
      	at net.minidev.json.parser.JSONParserBase.readObject(JSONParserBase.java:530)
      	at net.minidev.json.parser.JSONParserBase.readFirst(JSONParserBase.java:285)
      	at net.minidev.json.parser.JSONParserBase.parse(JSONParserBase.java:150)
      	at net.minidev.json.parser.JSONParserByteArray.parse(JSONParserByteArray.java:57)
      	at net.minidev.json.parser.JSONParserByteArray.parse(JSONParserByteArray.java:39)
      	at net.minidev.json.parser.JSONParser.parse(JSONParser.java:173)
      	at org.apache.tajo.storage.json.JsonLineDeserializer.deserialize(JsonLineDeserializer.java:66)
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/296

        TAJO-1242: Json scanner can not read some case of trucated text.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1242

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/296.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #296


        commit f65eb0f4b27c3b8e13e0a90744cc2ff4501b2cde
        Author: jhkim <jhkim@apache.org>
        Date: 2014-12-11T05:11:24Z

        TAJO-1242: Json scanner can not read some case of trucated text.

        commit 95ee7b09dc1b1be0f200a1d68428902891fc9479
        Author: jhkim <jhkim@apache.org>
        Date: 2014-12-11T05:14:26Z

        fix the typo


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/296 TAJO-1242 : Json scanner can not read some case of trucated text. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1242 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/296.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #296 commit f65eb0f4b27c3b8e13e0a90744cc2ff4501b2cde Author: jhkim <jhkim@apache.org> Date: 2014-12-11T05:11:24Z TAJO-1242 : Json scanner can not read some case of trucated text. commit 95ee7b09dc1b1be0f200a1d68428902891fc9479 Author: jhkim <jhkim@apache.org> Date: 2014-12-11T05:14:26Z fix the typo
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/296#issuecomment-66618208

        +1
        The fix looks straightforward. The unit test is enough to reproduce the problematic case and prove the bug fix.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/296#issuecomment-66618208 +1 The fix looks straightforward. The unit test is enough to reproduce the problematic case and prove the bug fix.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/296

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/296
        Hide
        jhkim Jinho Kim added a comment -

        committed it!
        Thanks.

        Show
        jhkim Jinho Kim added a comment - committed it! Thanks.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #144 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/144/)
        TAJO-1242: Json scanner can not read some case of trucated text. (jinho) (jhkim: rev c665ae1f6fc1e35e6a743e7e4e377c7885686b32)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/json/JsonLineDeserializer.java
        • tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #144 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/144/ ) TAJO-1242 : Json scanner can not read some case of trucated text. (jinho) (jhkim: rev c665ae1f6fc1e35e6a743e7e4e377c7885686b32) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/json/JsonLineDeserializer.java tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json CHANGES
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #503 (See https://builds.apache.org/job/Tajo-master-build/503/)
        TAJO-1242: Json scanner can not read some case of trucated text. (jinho) (jhkim: rev c665ae1f6fc1e35e6a743e7e4e377c7885686b32)

        • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/json/JsonLineDeserializer.java
        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        • tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #503 (See https://builds.apache.org/job/Tajo-master-build/503/ ) TAJO-1242 : Json scanner can not read some case of trucated text. (jinho) (jhkim: rev c665ae1f6fc1e35e6a743e7e4e377c7885686b32) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/json/JsonLineDeserializer.java tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json CHANGES
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #145 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/145/)
        Fix the broken unit test of TAJO-1242 (jhkim: rev bee0fd86e07a77237c0644f708c220d7ed7fafe8)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #145 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/145/ ) Fix the broken unit test of TAJO-1242 (jhkim: rev bee0fd86e07a77237c0644f708c220d7ed7fafe8) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-build #504 (See https://builds.apache.org/job/Tajo-master-build/504/)
        Fix the broken unit test of TAJO-1242 (jhkim: rev bee0fd86e07a77237c0644f708c220d7ed7fafe8)

        • tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #504 (See https://builds.apache.org/job/Tajo-master-build/504/ ) Fix the broken unit test of TAJO-1242 (jhkim: rev bee0fd86e07a77237c0644f708c220d7ed7fafe8) tajo-storage/tajo-storage-hdfs/src/test/java/org/apache/tajo/storage/TestDelimitedTextFile.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #146 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/146/)
        Fix the rat checking of TAJO-1242 (jhkim: rev 5e043bd87bc6f84848d0745f435706421932cf60)

        • tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #146 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/146/ ) Fix the rat checking of TAJO-1242 (jhkim: rev 5e043bd87bc6f84848d0745f435706421932cf60) tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-master-build #505 (See https://builds.apache.org/job/Tajo-master-build/505/)
        Fix the rat checking of TAJO-1242 (jhkim: rev 5e043bd87bc6f84848d0745f435706421932cf60)

        • tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #505 (See https://builds.apache.org/job/Tajo-master-build/505/ ) Fix the rat checking of TAJO-1242 (jhkim: rev 5e043bd87bc6f84848d0745f435706421932cf60) tajo-storage/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance3.json

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development