Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12177 [Umbrella] Update and extend filesystem specification
  3. HADOOP-11601

Enhance FS spec & tests to mandate FileStatus.getBlocksize() >0 for non-empty files

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.6.0
    • 2.8.0, 3.0.0-alpha2
    • fs, test
    • None
    • Reviewed

    Description

      HADOOP-11584 has shown that the contract tests are not validating that FileStatus.getBlocksize() must be >0 for any analytics jobs to partition workload correctly.

      Clarify in text and add test to do this. Test MUST be designed to work against eventually consistent filesystems where getFileStatus() may not be immediately visible, by retrying operation if FS declares it is an object store.

      Attachments

        1. HADOOP-11601-007.patch
          9 kB
          Mingliang Liu
        2. HADOOP-11601-006.patch
          9 kB
          Mingliang Liu
        3. HADOOP-11601-005.patch
          9 kB
          Steve Loughran
        4. HADOOP-11601-004.patch
          9 kB
          Steve Loughran
        5. HADOOP-11601-003.patch
          9 kB
          Steve Loughran
        6. HADOOP-11601-002.patch
          8 kB
          Steve Loughran
        7. HADOOP-11601-001.patch
          11 kB
          Steve Loughran

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified