Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1921

Hbase Storage can cause NPE when the hbase cluster is restarted

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0, 0.11.1
    • Component/s: HBase Storage
    • Labels:
      None

      Description

      See the title. The stacktrace is as follow:

      2015-10-10 01:16:58,577 ERROR org.apache.tajo.querymaster.Stage: Stage (eb_1444397501528_0096_000001) ERROR:
      java.lang.NullPointerException
             at org.apache.tajo.storage.hbase.HBaseTablespace.getSplits(HBaseTablespace.java:529)
             at org.apache.tajo.querymaster.Stage$InitAndRequestContainer.scheduleFragmentsForLeafQuery(Stage.java:1147)
             at org.apache.tajo.querymaster.Stage$InitAndRequestContainer.schedule(Stage.java:1039)
             at org.apache.tajo.querymaster.Stage$InitAndRequestContainer.access$800(Stage.java:821)
             at org.apache.tajo.querymaster.Stage$InitAndRequestContainer$1.run(Stage.java:847)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
             at java.lang.Thread.run(Thread.java:745)
      

        Issue Links

          Activity

          Hide
          combine Byunghwa Yun added a comment -

          Hyunsik Choi, Could I get to develop this issue?
          It is often occur in the my cluster even region rebalance.
          Thank you.

          Show
          combine Byunghwa Yun added a comment - Hyunsik Choi , Could I get to develop this issue? It is often occur in the my cluster even region rebalance. Thank you.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Byunghwa Yun,

          Sure, thank you for taking this issue. I've just assigned it to you.

          Show
          hyunsik Hyunsik Choi added a comment - Hi Byunghwa Yun , Sure, thank you for taking this issue. I've just assigned it to you.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user combineads opened a pull request:

          https://github.com/apache/tajo/pull/918

          TAJO-1921: Hbase Storage can cause NPE when the hbase cluster is restarted

          It was similarly modified by referring to TableInputFormatBase#getSplits(JobContext). Cause as much as possible to follow a policy of HBase.
          And then the size of memstore will not have a big impact for bigdata analysis. So it was removed.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/combineads/tajo TAJO-1921

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/918.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #918


          commit d3ffcabb44539427532da20678c788e867098800
          Author: Byunghwa Yun <combine@combineads.co.kr>
          Date: 2015-12-16T05:26:27Z

          Remove unused import class

          commit 31637c41cdcebbccc5854de8a131e5cd86f1c4a3
          Author: combineads <combine@combineads.co.kr>
          Date: 2015-12-16T05:57:01Z

          Merge remote-tracking branch 'upstream/master'

          commit 463dd93c0dcc0b5f022308ad2adc3da7068c4441
          Author: combineads <combine@combineads.co.kr>
          Date: 2015-12-18T02:35:32Z

          Merge remote-tracking branch 'upstream/master'

          commit f51da8eb507c3ca5745f3afd73c496a28dd7b107
          Author: Byunghwa Yun <combine@combineads.co.kr>
          Date: 2015-12-22T03:49:09Z

          TAJO-1921: Hbase Storage can cause NPE when the hbase cluster is
          restarted


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user combineads opened a pull request: https://github.com/apache/tajo/pull/918 TAJO-1921 : Hbase Storage can cause NPE when the hbase cluster is restarted It was similarly modified by referring to TableInputFormatBase#getSplits(JobContext). Cause as much as possible to follow a policy of HBase. And then the size of memstore will not have a big impact for bigdata analysis. So it was removed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/combineads/tajo TAJO-1921 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/918.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #918 commit d3ffcabb44539427532da20678c788e867098800 Author: Byunghwa Yun <combine@combineads.co.kr> Date: 2015-12-16T05:26:27Z Remove unused import class commit 31637c41cdcebbccc5854de8a131e5cd86f1c4a3 Author: combineads <combine@combineads.co.kr> Date: 2015-12-16T05:57:01Z Merge remote-tracking branch 'upstream/master' commit 463dd93c0dcc0b5f022308ad2adc3da7068c4441 Author: combineads <combine@combineads.co.kr> Date: 2015-12-18T02:35:32Z Merge remote-tracking branch 'upstream/master' commit f51da8eb507c3ca5745f3afd73c496a28dd7b107 Author: Byunghwa Yun <combine@combineads.co.kr> Date: 2015-12-22T03:49:09Z TAJO-1921 : Hbase Storage can cause NPE when the hbase cluster is restarted
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user combineads commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-166500787

          @hyunsik I think TAJO-1921 and TAJO-1940 seems to be a conflict. If TAJO-1940 is deployed, I can also be modified again.
          Thanks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-166500787 @hyunsik I think TAJO-1921 and TAJO-1940 seems to be a conflict. If TAJO-1940 is deployed, I can also be modified again. Thanks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-169565259

          I'm reviewing the patch. Your suggestion sounds good to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-169565259 I'm reviewing the patch. Your suggestion sounds good to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-169914203

          The patch looks good to me. Here are some comments.

          Our coding standard uses 2 space instead of tab. Some lines do not follow the indent of our coding standard. Please refer to https://cwiki.apache.org/confluence/display/TAJO/Code+Style.

          Other parts looks good to me. Here is my +1. Could you correct line indents? Then, I'll commit it soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-169914203 The patch looks good to me. Here are some comments. Our coding standard uses 2 space instead of tab. Some lines do not follow the indent of our coding standard. Please refer to https://cwiki.apache.org/confluence/display/TAJO/Code+Style . Other parts looks good to me. Here is my +1. Could you correct line indents? Then, I'll commit it soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user combineads commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-169935838

          I had to modify the source.
          Thank you for comment.

          Show
          githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-169935838 I had to modify the source. Thank you for comment.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user combineads commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-171588421

          @hyunsik Anything else to commit this PR?

          Show
          githubbot ASF GitHub Bot added a comment - Github user combineads commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-171588421 @hyunsik Anything else to commit this PR?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/918#issuecomment-172994141

          +1 the patch looks good to me. I'm sorry for late.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/918#issuecomment-172994141 +1 the patch looks good to me. I'm sorry for late.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/918

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/918
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #658 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/658/)
          TAJO-1921: Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 72ebc43d6e685dad58b2bf238fae12e51c8dec96)

          • CHANGES
          • tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java
          • tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #658 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/658/ ) TAJO-1921 : Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 72ebc43d6e685dad58b2bf238fae12e51c8dec96) CHANGES tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #1057 (See https://builds.apache.org/job/Tajo-master-build/1057/)
          TAJO-1921: Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 72ebc43d6e685dad58b2bf238fae12e51c8dec96)

          • tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result
          • tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java
          • CHANGES
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #1057 (See https://builds.apache.org/job/Tajo-master-build/1057/ ) TAJO-1921 : Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 72ebc43d6e685dad58b2bf238fae12e51c8dec96) tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java CHANGES tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-0.11.1-build #159 (See https://builds.apache.org/job/Tajo-0.11.1-build/159/)
          TAJO-1921: Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 0e8c82a3286d61807fd962ffed5ecbd4f9ac806b)

          • tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result
          • tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java
          • tajo-storage/tajo-storage-pgsql/src/test/java/org/apache/tajo/storage/pgsql/TestPgSQLQueryTests.java
          • tajo-storage/tajo-storage-pgsql/src/test/resources/results/TestPgSQLQueryTests/testTPCH_Q5.result
          • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java
          • tajo-storage/tajo-storage-pgsql/src/test/resources/results/TestPgSQLQueryTests/testTPCH_Q3.result
          • CHANGES
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.1-build #159 (See https://builds.apache.org/job/Tajo-0.11.1-build/159/ ) TAJO-1921 : Hbase Storage can cause NPE when the hbase cluster is (hyunsik: rev 0e8c82a3286d61807fd962ffed5ecbd4f9ac806b) tajo-core-tests/src/test/resources/results/TestHBaseTable/testGetSplitsWhenRestartHBase.result tajo-storage/tajo-storage-hbase/src/main/java/org/apache/tajo/storage/hbase/HBaseTablespace.java tajo-storage/tajo-storage-pgsql/src/test/java/org/apache/tajo/storage/pgsql/TestPgSQLQueryTests.java tajo-storage/tajo-storage-pgsql/src/test/resources/results/TestPgSQLQueryTests/testTPCH_Q5.result tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestHBaseTable.java tajo-storage/tajo-storage-pgsql/src/test/resources/results/TestPgSQLQueryTests/testTPCH_Q3.result CHANGES

            People

            • Assignee:
              combine Byunghwa Yun
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development