Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0, 0.10.0
    • Fix Version/s: 0.10.0, 0.11.0
    • Component/s: QueryMaster, Worker
    • Labels:
      None

      Description

      Currently, QueryMaster and Worker is store the histories in memory and file. and these will be removed. but memory cleaner incorrectly work. We should fix it

        Issue Links

          Activity

          Hide
          jhkim Jinho Kim added a comment -

          I will remove the possible OOM in this issue but we should improve it in TAJO-1326

          Show
          jhkim Jinho Kim added a comment - I will remove the possible OOM in this issue but we should improve it in TAJO-1326
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jinossy opened a pull request:

          https://github.com/apache/tajo/pull/371

          TAJO-1325: Invalid history cleaner timeout

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jinossy/tajo TAJO-1325

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/371.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #371


          commit 400c31bfeb1719da4a6b901f4acf65c744a43c3a
          Author: jhkim <jhkim@apache.org>
          Date: 2015-02-03T05:08:30Z

          TAJO-1325: Invalid history cleaner timeout


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/371 TAJO-1325 : Invalid history cleaner timeout You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1325 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/371.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #371 commit 400c31bfeb1719da4a6b901f4acf65c744a43c3a Author: jhkim <jhkim@apache.org> Date: 2015-02-03T05:08:30Z TAJO-1325 : Invalid history cleaner timeout
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/371#discussion_r23987652

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java —
          @@ -525,7 +524,13 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) {
          synchronized(finishedQueryMasterTasks) {
          List<QueryId> expiredQueryIds = new ArrayList<QueryId>();
          for(Map.Entry<QueryId, QueryMasterTask> entry: finishedQueryMasterTasks.entrySet()) {

          • if(entry.getValue().getStartTime() < expireTime) {
            + long finishedTime = entry.getValue().getStartTime();
              • End diff –

          Would you leave some comments?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/371#discussion_r23987652 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java — @@ -525,7 +524,13 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) { synchronized(finishedQueryMasterTasks) { List<QueryId> expiredQueryIds = new ArrayList<QueryId>(); for(Map.Entry<QueryId, QueryMasterTask> entry: finishedQueryMasterTasks.entrySet()) { if(entry.getValue().getStartTime() < expireTime) { + long finishedTime = entry.getValue().getStartTime(); End diff – Would you leave some comments?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/371#discussion_r23987664

          — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java —
          @@ -240,7 +240,9 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) {
          synchronized(taskRunnerHistoryMap) {
          List<String> expiredIds = new ArrayList<String>();
          for(Map.Entry<String, TaskRunnerHistory> entry: taskRunnerHistoryMap.entrySet()) {

          • if(entry.getValue().getStartTime() > expireTime) {
            +
            + long finishedTime = Math.max(entry.getValue().getStartTime(), entry.getValue().getFinishTime());
              • End diff –

          Would you leave some comments?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/371#discussion_r23987664 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java — @@ -240,7 +240,9 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) { synchronized(taskRunnerHistoryMap) { List<String> expiredIds = new ArrayList<String>(); for(Map.Entry<String, TaskRunnerHistory> entry: taskRunnerHistoryMap.entrySet()) { if(entry.getValue().getStartTime() > expireTime) { + + long finishedTime = Math.max(entry.getValue().getStartTime(), entry.getValue().getFinishTime()); End diff – Would you leave some comments?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/371#issuecomment-72608899

          +1 LGTM.
          Please add some explanations before you commit.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/371#issuecomment-72608899 +1 LGTM. Please add some explanations before you commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/371#discussion_r23987763

          — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java —
          @@ -240,7 +240,9 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) {
          synchronized(taskRunnerHistoryMap) {
          List<String> expiredIds = new ArrayList<String>();
          for(Map.Entry<String, TaskRunnerHistory> entry: taskRunnerHistoryMap.entrySet()) {

          • if(entry.getValue().getStartTime() > expireTime) {
            +
            + long finishedTime = Math.max(entry.getValue().getStartTime(), entry.getValue().getFinishTime());
              • End diff –

          Sure!

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/371#discussion_r23987763 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java — @@ -240,7 +240,9 @@ private void cleanExpiredFinishedQueryMasterTask(long expireTime) { synchronized(taskRunnerHistoryMap) { List<String> expiredIds = new ArrayList<String>(); for(Map.Entry<String, TaskRunnerHistory> entry: taskRunnerHistoryMap.entrySet()) { if(entry.getValue().getStartTime() > expireTime) { + + long finishedTime = Math.max(entry.getValue().getStartTime(), entry.getValue().getFinishTime()); End diff – Sure!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/371

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/371
          Hide
          jhkim Jinho Kim added a comment -

          committed it.
          Thank you for the review!

          Show
          jhkim Jinho Kim added a comment - committed it. Thank you for the review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #577 (See https://builds.apache.org/job/Tajo-master-build/577/)
          TAJO-1325: Invalid history cleaner timeout. (jinho) (jhkim: rev 8ac868352a2a3e5c26cf438e81ef4e23af88dc2b)

          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #577 (See https://builds.apache.org/job/Tajo-master-build/577/ ) TAJO-1325 : Invalid history cleaner timeout. (jinho) (jhkim: rev 8ac868352a2a3e5c26cf438e81ef4e23af88dc2b) tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java CHANGES tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #216 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/216/)
          TAJO-1325: Invalid history cleaner timeout. (jinho) (jhkim: rev 8ac868352a2a3e5c26cf438e81ef4e23af88dc2b)

          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #216 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/216/ ) TAJO-1325 : Invalid history cleaner timeout. (jinho) (jhkim: rev 8ac868352a2a3e5c26cf438e81ef4e23af88dc2b) CHANGES tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development