Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-846

Clean up the task history in woker

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: Worker
    • Labels:
      None

      Description

      The task history currently store the thread in memory. it can’t store to other storage(hdfs, local file ..). It would be nice if task history separate from taskrunner

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/20

        TAJO-846: Clean up the task history in woker

        The task history currently store the thread in memory. it can’t store to other storage(hdfs, local file ..). It would be nice if task history separate from taskrunner

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo history

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/20.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #20


        commit bd983dced741c2c9660cabb0d45d8e2ad14152fe
        Author: jinossy <jinossy@gmail.com>
        Date: 2014-05-26T05:15:28Z

        history


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/20 TAJO-846 : Clean up the task history in woker The task history currently store the thread in memory. it can’t store to other storage(hdfs, local file ..). It would be nice if task history separate from taskrunner You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo history Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/20.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20 commit bd983dced741c2c9660cabb0d45d8e2ad14152fe Author: jinossy <jinossy@gmail.com> Date: 2014-05-26T05:15:28Z history
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13138838

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockHistory.java —
        @@ -0,0 +1,147 @@
        +/**
        + * Licensed to the Apache Software Foundation (ASF) under one
        + * or more contributor license agreements. See the NOTICE file
        + * distributed with this work for additional information
        + * regarding copyright ownership. The ASF licenses this file
        + * to you under the Apache License, Version 2.0 (the
        + * "License"); you may not use this file except in compliance
        + * with the License. You may obtain a copy of the License at
        + *
        + * http://www.apache.org/licenses/LICENSE-2.0
        + *
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.tajo.worker;
        +
        +import com.google.common.base.Objects;
        +import com.google.common.collect.Maps;
        +import org.apache.hadoop.service.Service;
        +import org.apache.hadoop.yarn.api.records.ContainerId;
        +import org.apache.hadoop.yarn.util.ConverterUtils;
        +import org.apache.tajo.ExecutionBlockId;
        +import org.apache.tajo.QueryUnitAttemptId;
        +import org.apache.tajo.common.ProtoObject;
        +
        +import java.util.Collections;
        +import java.util.Map;
        +
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.ExecutionBlockHistoryProto;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto;
        +
        +/**
        + * The history class for ExecutionBlock processing.
        + */
        +public class ExecutionBlockHistory implements ProtoObject<ExecutionBlockHistoryProto> {
        — End diff –

        As you know, an ExecutionBlock is a logical unit to represent distributed query execution stage.
        However, this class seems to contain the information of each TaskRunner instead of an ExecutionBlock.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13138838 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockHistory.java — @@ -0,0 +1,147 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.worker; + +import com.google.common.base.Objects; +import com.google.common.collect.Maps; +import org.apache.hadoop.service.Service; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.util.ConverterUtils; +import org.apache.tajo.ExecutionBlockId; +import org.apache.tajo.QueryUnitAttemptId; +import org.apache.tajo.common.ProtoObject; + +import java.util.Collections; +import java.util.Map; + +import static org.apache.tajo.ipc.TajoWorkerProtocol.ExecutionBlockHistoryProto; +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto; + +/** + * The history class for ExecutionBlock processing. + */ +public class ExecutionBlockHistory implements ProtoObject<ExecutionBlockHistoryProto> { — End diff – As you know, an ExecutionBlock is a logical unit to represent distributed query execution stage. However, this class seems to contain the information of each TaskRunner instead of an ExecutionBlock.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13139322

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java —
        @@ -1,198 +0,0 @@
        -/**

        • * Licensed to the Apache Software Foundation (ASF) under one
        • * or more contributor license agreements. See the NOTICE file
        • * distributed with this work for additional information
        • * regarding copyright ownership. The ASF licenses this file
        • * to you under the Apache License, Version 2.0 (the
        • * "License"); you may not use this file except in compliance
        • * with the License. You may obtain a copy of the License at
        • *
        • * http://www.apache.org/licenses/LICENSE-2.0
        • *
        • * Unless required by applicable law or agreed to in writing, software
        • * distributed under the License is distributed on an "AS IS" BASIS,
        • * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        • * See the License for the specific language governing permissions and
        • * limitations under the License.
        • */
          -
          -package org.apache.tajo.worker;
          -
          -import org.apache.tajo.catalog.statistics.TableStats;
          -import org.apache.tajo.util.FileUtil;
          -
          -import java.net.URI;
          -import java.util.Collection;
          -import java.util.Map;
          -
          -public class TaskHistory {
            • End diff –

        Nowadays, I'm investigating the shuffle cost of Tajo, and TaskHistory and FetcherHistory provide very useful information to me. In addition, they will be useful for the detailed performance analysis.
        Would you leave these classes as they are?

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13139322 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java — @@ -1,198 +0,0 @@ -/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ - -package org.apache.tajo.worker; - -import org.apache.tajo.catalog.statistics.TableStats; -import org.apache.tajo.util.FileUtil; - -import java.net.URI; -import java.util.Collection; -import java.util.Map; - -public class TaskHistory { End diff – Nowadays, I'm investigating the shuffle cost of Tajo, and TaskHistory and FetcherHistory provide very useful information to me. In addition, they will be useful for the detailed performance analysis. Would you leave these classes as they are?
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/20#issuecomment-44428613

        Hi Jinho, thanks for your contribution.
        The patch looks good to me, but I left a minor comment. Also, I asked you a favor as above. If you can, please leave them.
        Thanks.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/20#issuecomment-44428613 Hi Jinho, thanks for your contribution. The patch looks good to me, but I left a minor comment. Also, I asked you a favor as above. If you can, please leave them. Thanks.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13165437

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockHistory.java —
        @@ -0,0 +1,147 @@
        +/**
        + * Licensed to the Apache Software Foundation (ASF) under one
        + * or more contributor license agreements. See the NOTICE file
        + * distributed with this work for additional information
        + * regarding copyright ownership. The ASF licenses this file
        + * to you under the Apache License, Version 2.0 (the
        + * "License"); you may not use this file except in compliance
        + * with the License. You may obtain a copy of the License at
        + *
        + * http://www.apache.org/licenses/LICENSE-2.0
        + *
        + * Unless required by applicable law or agreed to in writing, software
        + * distributed under the License is distributed on an "AS IS" BASIS,
        + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        + * See the License for the specific language governing permissions and
        + * limitations under the License.
        + */
        +
        +package org.apache.tajo.worker;
        +
        +import com.google.common.base.Objects;
        +import com.google.common.collect.Maps;
        +import org.apache.hadoop.service.Service;
        +import org.apache.hadoop.yarn.api.records.ContainerId;
        +import org.apache.hadoop.yarn.util.ConverterUtils;
        +import org.apache.tajo.ExecutionBlockId;
        +import org.apache.tajo.QueryUnitAttemptId;
        +import org.apache.tajo.common.ProtoObject;
        +
        +import java.util.Collections;
        +import java.util.Map;
        +
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.ExecutionBlockHistoryProto;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto;
        +
        +/**
        + * The history class for ExecutionBlock processing.
        + */
        +public class ExecutionBlockHistory implements ProtoObject<ExecutionBlockHistoryProto> {
        — End diff –

        I'll change name to TaskRunnerHistory

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13165437 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockHistory.java — @@ -0,0 +1,147 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.worker; + +import com.google.common.base.Objects; +import com.google.common.collect.Maps; +import org.apache.hadoop.service.Service; +import org.apache.hadoop.yarn.api.records.ContainerId; +import org.apache.hadoop.yarn.util.ConverterUtils; +import org.apache.tajo.ExecutionBlockId; +import org.apache.tajo.QueryUnitAttemptId; +import org.apache.tajo.common.ProtoObject; + +import java.util.Collections; +import java.util.Map; + +import static org.apache.tajo.ipc.TajoWorkerProtocol.ExecutionBlockHistoryProto; +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto; + +/** + * The history class for ExecutionBlock processing. + */ +public class ExecutionBlockHistory implements ProtoObject<ExecutionBlockHistoryProto> { — End diff – I'll change name to TaskRunnerHistory
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13165439

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java —
        @@ -1,198 +0,0 @@
        -/**

        • * Licensed to the Apache Software Foundation (ASF) under one
        • * or more contributor license agreements. See the NOTICE file
        • * distributed with this work for additional information
        • * regarding copyright ownership. The ASF licenses this file
        • * to you under the Apache License, Version 2.0 (the
        • * "License"); you may not use this file except in compliance
        • * with the License. You may obtain a copy of the License at
        • *
        • * http://www.apache.org/licenses/LICENSE-2.0
        • *
        • * Unless required by applicable law or agreed to in writing, software
        • * distributed under the License is distributed on an "AS IS" BASIS,
        • * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        • * See the License for the specific language governing permissions and
        • * limitations under the License.
        • */
          -
          -package org.apache.tajo.worker;
          -
          -import org.apache.tajo.catalog.statistics.TableStats;
          -import org.apache.tajo.util.FileUtil;
          -
          -import java.net.URI;
          -import java.util.Collection;
          -import java.util.Map;
          -
          -public class TaskHistory {
            • End diff –

        FetcherHistories use the large memory, it will be OOM
        If we add the history storage, it will be good informations
        In my opinion, If you need fetcherHistory in this time, I add the debug option

        ex) tajo.debug = true, added the fetchhistories in TaskHistory

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13165439 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java — @@ -1,198 +0,0 @@ -/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ - -package org.apache.tajo.worker; - -import org.apache.tajo.catalog.statistics.TableStats; -import org.apache.tajo.util.FileUtil; - -import java.net.URI; -import java.util.Collection; -import java.util.Map; - -public class TaskHistory { End diff – FetcherHistories use the large memory, it will be OOM If we add the history storage, it will be good informations In my opinion, If you need fetcherHistory in this time, I add the debug option ex) tajo.debug = true, added the fetchhistories in TaskHistory
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/20#issuecomment-44485450

        Thanks @jihoonson for the review!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/20#issuecomment-44485450 Thanks @jihoonson for the review!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy closed the pull request at:

        https://github.com/apache/tajo/pull/20

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy closed the pull request at: https://github.com/apache/tajo/pull/20
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/20#issuecomment-44485839

        Oops, I've just reopen

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/20#issuecomment-44485839 Oops, I've just reopen
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy reopened a pull request:

        https://github.com/apache/tajo/pull/20

        TAJO-846: Clean up the task history in woker

        The task history currently store the thread in memory. it can’t store to other storage(hdfs, local file ..). It would be nice if task history separate from taskrunner

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo history

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/20.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #20


        commit bd983dced741c2c9660cabb0d45d8e2ad14152fe
        Author: jinossy <jinossy@gmail.com>
        Date: 2014-05-26T05:15:28Z

        history


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy reopened a pull request: https://github.com/apache/tajo/pull/20 TAJO-846 : Clean up the task history in woker The task history currently store the thread in memory. it can’t store to other storage(hdfs, local file ..). It would be nice if task history separate from taskrunner You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo history Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/20.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20 commit bd983dced741c2c9660cabb0d45d8e2ad14152fe Author: jinossy <jinossy@gmail.com> Date: 2014-05-26T05:15:28Z history
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13169692

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java —
        @@ -1,198 +0,0 @@
        -/**

        • * Licensed to the Apache Software Foundation (ASF) under one
        • * or more contributor license agreements. See the NOTICE file
        • * distributed with this work for additional information
        • * regarding copyright ownership. The ASF licenses this file
        • * to you under the Apache License, Version 2.0 (the
        • * "License"); you may not use this file except in compliance
        • * with the License. You may obtain a copy of the License at
        • *
        • * http://www.apache.org/licenses/LICENSE-2.0
        • *
        • * Unless required by applicable law or agreed to in writing, software
        • * distributed under the License is distributed on an "AS IS" BASIS,
        • * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        • * See the License for the specific language governing permissions and
        • * limitations under the License.
        • */
          -
          -package org.apache.tajo.worker;
          -
          -import org.apache.tajo.catalog.statistics.TableStats;
          -import org.apache.tajo.util.FileUtil;
          -
          -import java.net.URI;
          -import java.util.Collection;
          -import java.util.Map;
          -
          -public class TaskHistory {
            • End diff –

        That's a good idea!
        As you said, FetcherHistory requires a lot of memory space. The debug option will be good solution in this time.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13169692 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java — @@ -1,198 +0,0 @@ -/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ - -package org.apache.tajo.worker; - -import org.apache.tajo.catalog.statistics.TableStats; -import org.apache.tajo.util.FileUtil; - -import java.net.URI; -import java.util.Collection; -import java.util.Map; - -public class TaskHistory { End diff – That's a good idea! As you said, FetcherHistory requires a lot of memory space. The debug option will be good solution in this time.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13179545

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java —
        @@ -18,107 +18,160 @@

        package org.apache.tajo.worker;

        -import org.apache.tajo.catalog.statistics.TableStats;
        -import org.apache.tajo.util.FileUtil;
        +import com.google.common.base.Objects;
        +import com.google.common.collect.Lists;
        +import org.apache.tajo.QueryUnitAttemptId;
        +import org.apache.tajo.catalog.proto.CatalogProtos;
        +import org.apache.tajo.common.ProtoObject;

        -import java.net.URI;
        -import java.util.Collection;
        -import java.util.Map;
        +import java.util.Collections;
        +import java.util.List;

        -public class TaskHistory {
        +import static org.apache.tajo.TajoProtos.TaskAttemptState;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.FetcherHistoryProto;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto;
        +
        +/**
        + * The history class for Task processing.
        + */
        +public class TaskHistory implements ProtoObject<TaskHistoryProto> {
        +
        + private QueryUnitAttemptId queryUnitAttemptId;
        + private TaskAttemptState state;
        + private float progress;
        private long startTime;
        private long finishTime;
        -

        • private String status;
          + private CatalogProtos.TableStatsProto inputStats;
          + private CatalogProtos.TableStatsProto outputStats;
          private String outputPath;
          private String workingPath;
        • private float progress;
          -
        • private TableStats inputStats;
        • private TableStats outputStats;
        • Map<URI, FetcherHistory> fetchers;
          + //TAJO-846
            • End diff –

        Please remove this line.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13179545 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java — @@ -18,107 +18,160 @@ package org.apache.tajo.worker; -import org.apache.tajo.catalog.statistics.TableStats; -import org.apache.tajo.util.FileUtil; +import com.google.common.base.Objects; +import com.google.common.collect.Lists; +import org.apache.tajo.QueryUnitAttemptId; +import org.apache.tajo.catalog.proto.CatalogProtos; +import org.apache.tajo.common.ProtoObject; -import java.net.URI; -import java.util.Collection; -import java.util.Map; +import java.util.Collections; +import java.util.List; -public class TaskHistory { +import static org.apache.tajo.TajoProtos.TaskAttemptState; +import static org.apache.tajo.ipc.TajoWorkerProtocol.FetcherHistoryProto; +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto; + +/** + * The history class for Task processing. + */ +public class TaskHistory implements ProtoObject<TaskHistoryProto> { + + private QueryUnitAttemptId queryUnitAttemptId; + private TaskAttemptState state; + private float progress; private long startTime; private long finishTime; - private String status; + private CatalogProtos.TableStatsProto inputStats; + private CatalogProtos.TableStatsProto outputStats; private String outputPath; private String workingPath; private float progress; - private TableStats inputStats; private TableStats outputStats; Map<URI, FetcherHistory> fetchers; + // TAJO-846 End diff – Please remove this line.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on a diff in the pull request:

        https://github.com/apache/tajo/pull/20#discussion_r13213693

        — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java —
        @@ -18,107 +18,160 @@

        package org.apache.tajo.worker;

        -import org.apache.tajo.catalog.statistics.TableStats;
        -import org.apache.tajo.util.FileUtil;
        +import com.google.common.base.Objects;
        +import com.google.common.collect.Lists;
        +import org.apache.tajo.QueryUnitAttemptId;
        +import org.apache.tajo.catalog.proto.CatalogProtos;
        +import org.apache.tajo.common.ProtoObject;

        -import java.net.URI;
        -import java.util.Collection;
        -import java.util.Map;
        +import java.util.Collections;
        +import java.util.List;

        -public class TaskHistory {
        +import static org.apache.tajo.TajoProtos.TaskAttemptState;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.FetcherHistoryProto;
        +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto;
        +
        +/**
        + * The history class for Task processing.
        + */
        +public class TaskHistory implements ProtoObject<TaskHistoryProto> {
        +
        + private QueryUnitAttemptId queryUnitAttemptId;
        + private TaskAttemptState state;
        + private float progress;
        private long startTime;
        private long finishTime;
        -

        • private String status;
          + private CatalogProtos.TableStatsProto inputStats;
          + private CatalogProtos.TableStatsProto outputStats;
          private String outputPath;
          private String workingPath;
        • private float progress;
          -
        • private TableStats inputStats;
        • private TableStats outputStats;
        • Map<URI, FetcherHistory> fetchers;
          + //TAJO-846
            • End diff –

        done

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/20#discussion_r13213693 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java — @@ -18,107 +18,160 @@ package org.apache.tajo.worker; -import org.apache.tajo.catalog.statistics.TableStats; -import org.apache.tajo.util.FileUtil; +import com.google.common.base.Objects; +import com.google.common.collect.Lists; +import org.apache.tajo.QueryUnitAttemptId; +import org.apache.tajo.catalog.proto.CatalogProtos; +import org.apache.tajo.common.ProtoObject; -import java.net.URI; -import java.util.Collection; -import java.util.Map; +import java.util.Collections; +import java.util.List; -public class TaskHistory { +import static org.apache.tajo.TajoProtos.TaskAttemptState; +import static org.apache.tajo.ipc.TajoWorkerProtocol.FetcherHistoryProto; +import static org.apache.tajo.ipc.TajoWorkerProtocol.TaskHistoryProto; + +/** + * The history class for Task processing. + */ +public class TaskHistory implements ProtoObject<TaskHistoryProto> { + + private QueryUnitAttemptId queryUnitAttemptId; + private TaskAttemptState state; + private float progress; private long startTime; private long finishTime; - private String status; + private CatalogProtos.TableStatsProto inputStats; + private CatalogProtos.TableStatsProto outputStats; private String outputPath; private String workingPath; private float progress; - private TableStats inputStats; private TableStats outputStats; Map<URI, FetcherHistory> fetchers; + // TAJO-846 End diff – done
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/20#issuecomment-44609635

        +1
        Thanks Jinho!!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/20#issuecomment-44609635 +1 Thanks Jinho!!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/20

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/20
        Hide
        jhkim Jinho Kim added a comment -

        I've just committed it.

        Show
        jhkim Jinho Kim added a comment - I've just committed it.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #231 (See https://builds.apache.org/job/Tajo-master-build/231/)
        TAJO-846: Clean up the task history in woker. (jinho) (jinossy: rev 8e6502231c79de900bf2dc5c4f37cab521e50c2b)

        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-common/src/main/java/org/apache/tajo/ExecutionBlockId.java
        • tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java
        • tajo-core/src/main/resources/webapps/worker/querytasks.jsp
        • tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java
        • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
        • tajo-core/src/main/resources/webapps/worker/querydetail.jsp
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java
        • tajo-core/src/main/resources/webapps/worker/queryunit.jsp
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
        • tajo-core/src/main/resources/webapps/worker/tasks.jsp
        • tajo-common/src/main/java/org/apache/tajo/QueryId.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
        • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
        • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java
        • tajo-common/src/main/java/org/apache/tajo/QueryUnitId.java
        • tajo-core/src/main/proto/TajoWorkerProtocol.proto
        • tajo-storage/src/main/java/org/apache/tajo/storage/MergeScanner.java
        • tajo-common/src/main/java/org/apache/tajo/QueryUnitAttemptId.java
        • tajo-core/src/main/resources/webapps/worker/header.jsp
        • tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
        • tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp
        • tajo-core/src/main/resources/webapps/worker/taskdetail.jsp
        • tajo-common/src/main/proto/tajo_protos.proto
        • tajo-core/src/main/resources/webapps/worker/index.jsp
        • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #231 (See https://builds.apache.org/job/Tajo-master-build/231/ ) TAJO-846 : Clean up the task history in woker. (jinho) (jinossy: rev 8e6502231c79de900bf2dc5c4f37cab521e50c2b) tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-common/src/main/java/org/apache/tajo/ExecutionBlockId.java tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java tajo-core/src/main/resources/webapps/worker/querytasks.jsp tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-core/src/main/resources/webapps/worker/querydetail.jsp tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryJobManager.java tajo-core/src/main/resources/webapps/worker/queryunit.jsp CHANGES tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/src/main/resources/webapps/worker/tasks.jsp tajo-common/src/main/java/org/apache/tajo/QueryId.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java tajo-common/src/main/java/org/apache/tajo/QueryUnitId.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-storage/src/main/java/org/apache/tajo/storage/MergeScanner.java tajo-common/src/main/java/org/apache/tajo/QueryUnitAttemptId.java tajo-core/src/main/resources/webapps/worker/header.jsp tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp tajo-core/src/main/resources/webapps/worker/taskdetail.jsp tajo-common/src/main/proto/tajo_protos.proto tajo-core/src/main/resources/webapps/worker/index.jsp tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development