Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3972

Trash emptier fails in secure HA cluster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.1-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

      WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
      is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.

      The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

      1. hdfs-3972.txt
        5 kB
        Todd Lipcon

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Eli Collins made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Target Version/s 3.0.0, 2.0.2-alpha [ 12320356, 12322472 ]
        Fix Version/s 2.0.2-alpha [ 12322472 ]
        Resolution Fixed [ 1 ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Todd Lipcon made changes -
        Attachment hdfs-3972.txt [ 12546574 ]
        Todd Lipcon made changes -
        Field Original Value New Value
        Summary Trash emptier fails in secure cluster Trash emptier fails in secure HA cluster
        Description In a secure cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

        WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
        is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.

        The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
        In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

        WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
        is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.

        The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
        Todd Lipcon created issue -

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development