Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8807

dfs.datanode.data.dir does not handle spaces between storageType and URI correctly

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.8.0, 3.0.0-alpha1
    • datanode
    • None
    • Reviewed

    Description

      if you add a space between the storage type and file URI then datanodes fail during startup.
      Here is an example of "mis-configration" that leads to datanode failure.

      <property>
          <name>dfs.datanode.data.dir</name>
          <value>
            [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
          </value>
        </property>
      

      Here is the "fixed" version. Please note the lack of space between [DISK] and file URI

      <property>
          <name>dfs.datanode.data.dir</name>
          <value>
            [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
          </value>
        </property>
      

      we fail with a parsing error, here is the info from the datanode logs.

      2015-07-22 13:34:36,812 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
      java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0:  file://tmp/hadoop-aengineer/disk1/dfs/data
              at org.apache.hadoop.fs.Path.initialize(Path.java:204)
              at org.apache.hadoop.fs.Path.<init>(Path.java:170)
              at org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
              at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
      Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 0:  file://tmp/hadoop-aengineer/disk1/dfs/data
              at java.net.URI$Parser.fail(URI.java:2829)
              at java.net.URI$Parser.checkChars(URI.java:3002)
              at java.net.URI$Parser.checkChar(URI.java:3012)
              at java.net.URI$Parser.parse(URI.java:3028)
              at java.net.URI.<init>(URI.java:753)
              at org.apache.hadoop.fs.Path.initialize(Path.java:201)
              ... 7 more
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            aengineer Anu Engineer Assign to me
            aengineer Anu Engineer
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment