Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9683

HistoryServer should honor fs.default-scheme

    XMLWordPrintableJSON

    Details

      Description

      I am using release-1.4.2,

       

      With fs.default-scheme and  fs.hdfs.hadoopconf set correctly, 

      when setting 

      historyserver.archive.fs.dir: /tmp/flink/cluster-name/jmarchive
      

      I am seeing

      2018-06-27 18:51:12,692 WARN org.apache.flink.runtime.webmonitor.history.HistoryServer - Failed to create Path or FileSystem for directory '/tmp/flink/cluster-name/jmarchive'. Directory will not be monitored.
      java.lang.IllegalArgumentException: The scheme (hdfs://, file://, etc) is null. Please specify the file system scheme explicitly in the URI.
      at org.apache.flink.runtime.webmonitor.WebMonitorUtils.validateAndNormalizeUri(WebMonitorUtils.java:300)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:168)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:113)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer$1.call(HistoryServer.java:110)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
      at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer.main(HistoryServer.java:110)
      

       

      And then if I set 

      historyserver.archive.fs.dir: hdfs:///tmp/flink/cluster-name/jmarchive

      I am seeing:

       

      java.io.IOException: The given file system URI (hdfs:///tmp/flink/cluster-name/jmarchive) did not describe the authority (like for example HDFS NameNode address/port or S3 host). The attempt to use a configured default authority failed: Hadoop configuration for default file system ('fs.default.name' or 'fs.defaultFS') contains no valid authority component (like hdfs namenode, S3 host, etc)
      at org.apache.flink.runtime.fs.hdfs.HadoopFsFactory.create(HadoopFsFactory.java:149)
      at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:401)
      at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:320)
      at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:169)
      at org.apache.flink.runtime.webmonitor.history.HistoryServer.<init>(HistoryServer.java:132)
      

      The only way it works is to provide the full path of hdfs like:

      #historyserver.archive.fs.dir: hdfs://<hdfs-cluster>/tmp/flink/cluster-name/jmarchive
      

       

      Above situations are because there are two parts of code treating "scheme" differently. 

      https://github.com/apache/flink/blob/release-1.4.2/flink-runtime/src/main/java/org/apache/flink/runtime/webmonitor/WebMonitorUtils.java#L299-L302

      https://github.com/apache/flink/blob/release-1.4.2/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L335-L338

      I believe the first case should be supported if users have set fs.default-scheme 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Jiang Xin Jiang Xin
                Reporter:
                ethanli Ethan Li
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: