Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3486

offlineimageviewer can't read fsimage files that contain persistent delegation tokens

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: security, tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      OfflineImageViewer (oiv) crashes when trying to read fsimage files that contain persistent delegation tokens.

      Example stack trace:

      Caused by: java.lang.IndexOutOfBoundsException
              at java.io.DataInputStream.readFully(DataInputStream.java:175)
              at org.apache.hadoop.io.Text.readFields(Text.java:284)
              at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier.readFields(AbstractDelegationTokenIdentifier.java:178)
              at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.processDelegationTokens(ImageLoaderCurrent.java:222)
              at org.apache.hadoop.hdfs.tools.offlineImageViewer.ImageLoaderCurrent.loadImage(ImageLoaderCurrent.java:186)
              at org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewer.go(OfflineImageViewer.java:129)
      

      The oiv and loadFSImage code paths are separate. The issue here seems to be that the loadFSImage code path has diverged from the oiv code path.

      On the loadFSImage code path (from FSImageFormat#loadCurrentTokens):

        /**
         * Private helper methods to load Delegation tokens from fsimage
         */
        private synchronized void loadCurrentTokens(DataInputStream in)
            throws IOException {
          int numberOfTokens = in.readInt();
          for (int i = 0; i < numberOfTokens; i++) {
            DelegationTokenIdentifier id = new DelegationTokenIdentifier();
            id.readFields(in);
            long expiryTime = in.readLong();
            addPersistedDelegationToken(id, expiryTime);
          }
        }
      

      Notice how it loads a 4-byte int after every DelegationTokenIdentifier.

      On the oiv code path (from ImageLoaderCurrent#processDelegationTokens):

          int numDTokens = in.readInt();
          v.visitEnclosingElement(ImageElement.DELEGATION_TOKENS,
              ImageElement.NUM_DELEGATION_TOKENS, numDTokens);
          for(int i=0; i<numDTokens; i++){
            DelegationTokenIdentifier id = new  DelegationTokenIdentifier();
            id.readFields(in);
            v.visit(ImageElement.DELEGATION_TOKEN_IDENTIFIER, id.toString());
          }
      

      Notice how it does not load a 4-byte int after every DelegationTokenIdentifier.

      This bug seems to have been introduced by change 916534, the same change which introduced persistent delegation tokens. So I don't think oiv was ever able to decode them in the past.

        Attachments

        1. HDFS-3486.001.patch
          15 kB
          Colin P. McCabe
        2. HDFS-3486.002.patch
          15 kB
          Colin P. McCabe

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                cmccabe Colin P. McCabe
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: