Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1427

Improve "unknown disk id" warning messages

    Details

      Description

      The warning that users see when volume mapping information is not available is not very clear, and can be repeated many, many times:

      "Unknown disk id. This will negatively affect performance. "
      "Check your hdfs settings to enable block location metadata.

      We should improve this to a) not print out so frequently b) point more directly to the setting to enable block data c) more clearly explain the problem.

        Activity

        Hide
        mariusvniekerk Marius Van Niekerk added a comment -

        I am seeing these issues in my install, but i have block tracking enabled on all the nodes as per the documentation.

        Are there other causes that could make this happen?

        The statestore logs

        W1217 08:20:01.989208 11291 HdfsTable.java:310] Attempted to get block locations but the call returned nulls
        

        very frequently.

        Show
        mariusvniekerk Marius Van Niekerk added a comment - I am seeing these issues in my install, but i have block tracking enabled on all the nodes as per the documentation. Are there other causes that could make this happen? The statestore logs W1217 08:20:01.989208 11291 HdfsTable.java:310] Attempted to get block locations but the call returned nulls very frequently.
        Hide
        ippokratis Ippokratis Pandis added a comment -

        Martin, I think we resolved this issue. Can you please confirm (and close the issue if we indeed resolved it)?

        Show
        ippokratis Ippokratis Pandis added a comment - Martin, I think we resolved this issue. Can you please confirm (and close the issue if we indeed resolved it)?
        Hide
        dtsirogiannis Dimitris Tsirogiannis added a comment -

        We should try to reproduce this issue and mark it as resolved if that is not the case.

        Show
        dtsirogiannis Dimitris Tsirogiannis added a comment - We should try to reproduce this issue and mark it as resolved if that is not the case.
        Hide
        bharathv bharath v added a comment -

        MPALA-1427: Improvements to "Unknown disk-ID" warning

        • Removes the runtime unknown disk ID reporting and instead moves
          it to the explain plan as a counter that prints the number of
          scan ranges missing disk IDs in the corresponding HDFS scan nodes.
        • Adds a warning to the header of query profile/explain plan with a
          list of tables missing disk ids.
        • Removes reference to enabling dfs block metadata configuration,
          since it doesn't apply anymore.
        • Removes VolumeId terminology from the runtime profile.

        Change-Id: Iddb132ff7ad66f3291b93bf9d8061bd0525ef1b2
        Reviewed-on: http://gerrit.cloudera.org:8080/5828
        Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
        Tested-by: Impala Public Jenkins

        M be/src/exec/hdfs-scan-node-base.cc
        M be/src/exec/hdfs-scan-node-base.h
        M be/src/service/query-exec-state.cc
        M common/thrift/ImpalaInternalService.thrift
        M common/thrift/generate_error_codes.py
        M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
        M fe/src/main/java/org/apache/impala/planner/Planner.java
        M fe/src/main/java/org/apache/impala/service/Frontend.java
        8 files changed, 166 insertions, 102 deletions

        Show
        bharathv bharath v added a comment - MPALA-1427: Improvements to "Unknown disk-ID" warning Removes the runtime unknown disk ID reporting and instead moves it to the explain plan as a counter that prints the number of scan ranges missing disk IDs in the corresponding HDFS scan nodes. Adds a warning to the header of query profile/explain plan with a list of tables missing disk ids. Removes reference to enabling dfs block metadata configuration, since it doesn't apply anymore. Removes VolumeId terminology from the runtime profile. Change-Id: Iddb132ff7ad66f3291b93bf9d8061bd0525ef1b2 Reviewed-on: http://gerrit.cloudera.org:8080/5828 Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com> Tested-by: Impala Public Jenkins — M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/service/query-exec-state.cc M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/service/Frontend.java 8 files changed, 166 insertions , 102 deletions
        Hide
        bharathv bharath v added a comment -

        Re-opening this. We can add a more meaningful message instead of just mentioning "Unknown disk ID count" for supportability.

        Show
        bharathv bharath v added a comment - Re-opening this. We can add a more meaningful message instead of just mentioning "Unknown disk ID count" for supportability.

          People

          • Assignee:
            bharathv bharath v
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development