Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8304

Bulkload fails to remove files if fs.default.name / fs.defaultFS is configured without default port

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      When fs.default.name or fs.defaultFS in hadoop core-site.xml is configured as hdfs://ip, and hbase.rootdir is configured as hdfs://ip:port/hbaserootdir where port is the hdfs namenode's default port. the bulkload operation will not remove the file in bulk output dir. Store::bulkLoadHfile will think hdfs:://ip and hdfs:://ip:port as different filesystem and go with copy approaching instead of rename.

      The root cause is that hbase master will rewrite fs.default.name/fs.defaultFS according to hbase.rootdir when regionserver started, thus, dest fs uri from the hregion will not matching src fs uri passed from client.

      any suggestion what is the best approaching to fix this issue?

      I kind of think that we could check for default port if src uri come without port info.

        Attachments

        1. 0.94-8304.patch
          9 kB
          haosdent
        2. 0.96-8304.patch
          9 kB
          haosdent
        3. 8304-v4.patch
          8 kB
          Ted Yu
        4. HBASE-8304.patch
          8 kB
          haosdent
        5. HBASE-8304-v2.patch
          9 kB
          haosdent
        6. HBASE-8304-v3.patch
          9 kB
          haosdent

          Issue Links

            Activity

              People

              • Assignee:
                haosdent@gmail.com haosdent
                Reporter:
                colorant Raymond Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: