Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Storage
    • Labels:
      None

      Description

      We should add ByteBuffer-based De/Serializer instead of byte[]

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/228

        TAJO-1151: Implement the ByteBuffer-based De/Serializer

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1151

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/228.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #228


        commit aed406357e016e602b11dbaed3d20994dd651a17
        Author: jhkim <jhkim@apache.org>
        Date: 2014-10-31T12:10:08Z

        TAJO-1149: Implement direct read of TextFile scanner

        commit f95d8dc0ef04dfb82290e23d5430eefc70f2552d
        Author: jhkim <jhkim@apache.org>
        Date: 2014-10-31T13:42:47Z

        remove the past buffer reading

        commit f87e56f942b523f2bd2675d2c75484d3fea6023e
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-01T14:47:56Z

        refactor the compact buffer

        commit bd15ea07a0187ff1d76bb0ba887bf899a281aa07
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-03T08:03:30Z

        rename LineDelimitedReader to DelimitedLineReader

        commit 166e913be69d8ee7944d67c3fd82f015762d82b1
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-03T09:36:38Z

        rename csvfile.delimiter, csvfile.null to text.delimiter, text.null

        commit 06362436c25db3a3332e8dc1eb398e4cb55c6413
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-03T14:59:49Z

        cleanup the stats

        commit d84715c789a1c240862eb0368f73a99232c916b1
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-03T16:02:26Z

        fixed missing sample data

        commit 36e55a6a1402b1b59d6815fffd503747d08a3154
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-04T07:57:16Z

        change the default text format to textfile

        commit a9f8c183f66856ad9ca6a6f14b9269cfa9cb0c1e
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-04T08:05:38Z

        Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1149

        commit 6d3ab4f97739ddd3d6128a075eb76239ab9ddc82
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-04T08:53:18Z

        Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1149

        commit 6fedc5d1362a35b9643eb551164dcc148d59f090
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-04T09:03:23Z

        fixed compile error

        commit e06acf9f523815423b67a373da90f2d561285e18
        Author: jhkim <jhkim@apache.org>
        Date: 2014-11-05T11:36:01Z

        TAJO-1151: Implement the ByteBuffer-based De/Serializer


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/228 TAJO-1151 : Implement the ByteBuffer-based De/Serializer You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1151 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/228.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #228 commit aed406357e016e602b11dbaed3d20994dd651a17 Author: jhkim <jhkim@apache.org> Date: 2014-10-31T12:10:08Z TAJO-1149 : Implement direct read of TextFile scanner commit f95d8dc0ef04dfb82290e23d5430eefc70f2552d Author: jhkim <jhkim@apache.org> Date: 2014-10-31T13:42:47Z remove the past buffer reading commit f87e56f942b523f2bd2675d2c75484d3fea6023e Author: jhkim <jhkim@apache.org> Date: 2014-11-01T14:47:56Z refactor the compact buffer commit bd15ea07a0187ff1d76bb0ba887bf899a281aa07 Author: jhkim <jhkim@apache.org> Date: 2014-11-03T08:03:30Z rename LineDelimitedReader to DelimitedLineReader commit 166e913be69d8ee7944d67c3fd82f015762d82b1 Author: jhkim <jhkim@apache.org> Date: 2014-11-03T09:36:38Z rename csvfile.delimiter, csvfile.null to text.delimiter, text.null commit 06362436c25db3a3332e8dc1eb398e4cb55c6413 Author: jhkim <jhkim@apache.org> Date: 2014-11-03T14:59:49Z cleanup the stats commit d84715c789a1c240862eb0368f73a99232c916b1 Author: jhkim <jhkim@apache.org> Date: 2014-11-03T16:02:26Z fixed missing sample data commit 36e55a6a1402b1b59d6815fffd503747d08a3154 Author: jhkim <jhkim@apache.org> Date: 2014-11-04T07:57:16Z change the default text format to textfile commit a9f8c183f66856ad9ca6a6f14b9269cfa9cb0c1e Author: jhkim <jhkim@apache.org> Date: 2014-11-04T08:05:38Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1149 commit 6d3ab4f97739ddd3d6128a075eb76239ab9ddc82 Author: jhkim <jhkim@apache.org> Date: 2014-11-04T08:53:18Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1149 commit 6fedc5d1362a35b9643eb551164dcc148d59f090 Author: jhkim <jhkim@apache.org> Date: 2014-11-04T09:03:23Z fixed compile error commit e06acf9f523815423b67a373da90f2d561285e18 Author: jhkim <jhkim@apache.org> Date: 2014-11-05T11:36:01Z TAJO-1151 : Implement the ByteBuffer-based De/Serializer
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/228#discussion_r20152412

        — Diff: tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java —
        @@ -136,9 +135,10 @@ public void init() throws IOException {

        try {
        // we need to discuss the De/Serializer interface. so custom serde is to disable

        • String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE,
          + /*String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE,
            • End diff –

        If we don't need this code line, please remove it.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/228#discussion_r20152412 — Diff: tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java — @@ -136,9 +135,10 @@ public void init() throws IOException { try { // we need to discuss the De/Serializer interface. so custom serde is to disable String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE, + /*String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE, End diff – If we don't need this code line, please remove it.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/228#discussion_r20152417

        — Diff: tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java —
        @@ -316,9 +316,10 @@ public void init() throws IOException {

        try {
        // we need to discuss the De/Serializer interface. so custom serde is to disable

        • String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE,
          + /*String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE,
            • End diff –

        If we don't need this code line, please remove it.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/228#discussion_r20152417 — Diff: tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java — @@ -316,9 +316,10 @@ public void init() throws IOException { try { // we need to discuss the De/Serializer interface. so custom serde is to disable String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE, + /*String serdeClass = this.meta.getOption(StorageConstants.TEXTFILE_SERDE, End diff – If we don't need this code line, please remove it.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/228#issuecomment-62554776

        The patch looks good to me, but needs to be updated against the master branch.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/228#issuecomment-62554776 The patch looks good to me, but needs to be updated against the master branch.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/228#issuecomment-62661286

        @jihoonson
        I’ve remove the commented out line
        Thank you for your review.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/228#issuecomment-62661286 @jihoonson I’ve remove the commented out line Thank you for your review.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/228#issuecomment-62782583

        +1

        The patch looks great to me. This work will remove the unnecessary overheads of byte-array allocation, and it will be very helpful for zero-copy tuple in on-going ```block_iteration``` works.

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/228#issuecomment-62782583 +1 The patch looks great to me. This work will remove the unnecessary overheads of byte-array allocation, and it will be very helpful for zero-copy tuple in on-going ```block_iteration``` works.
        Hide
        jihoonson Jihoon Son added a comment -

        I cannot access to GitHub currently, so left my +1 here.
        Great work. Ship it!

        Show
        jihoonson Jihoon Son added a comment - I cannot access to GitHub currently, so left my +1 here. Great work. Ship it!
        Hide
        jhkim Jinho Kim added a comment -

        Committed it
        Thanks!

        Show
        jhkim Jinho Kim added a comment - Committed it Thanks!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #447 (See https://builds.apache.org/job/Tajo-master-build/447/)
        TAJO-1151: Implement the ByteBuffer-based De/Serializer. (jinho) (jhkim: rev c193cfaa8eb50a83bbbf46624bcccb81c0cc7445)

        • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java
        • CHANGES
        • tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java
        • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
        • tajo-common/pom.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #447 (See https://builds.apache.org/job/Tajo-master-build/447/ ) TAJO-1151 : Implement the ByteBuffer-based De/Serializer. (jinho) (jhkim: rev c193cfaa8eb50a83bbbf46624bcccb81c0cc7445) tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java CHANGES tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-common/pom.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #89 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/89/)
        TAJO-1151: Implement the ByteBuffer-based De/Serializer. (jinho) (jhkim: rev c193cfaa8eb50a83bbbf46624bcccb81c0cc7445)

        • tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java
        • tajo-common/pom.xml
        • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
        • CHANGES
        • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #89 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/89/ ) TAJO-1151 : Implement the ByteBuffer-based De/Serializer. (jinho) (jhkim: rev c193cfaa8eb50a83bbbf46624bcccb81c0cc7445) tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java tajo-common/pom.xml tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java CHANGES tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/228

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/228

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development