Hadoop Common
  1. Hadoop Common
  2. HADOOP-9361

Strictly define the expected behavior of filesystem APIs and write tests to verify compliance

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0, 2.2.0, 2.4.0
    • Fix Version/s: None
    • Component/s: fs, test
    • Labels:
      None
    • Target Version/s:

      Description

      FileSystem and FileContract aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't.

      The only tests that are common are those of FileSystemContractTestBase, which HADOOP-9258 shows is incomplete.

      I propose

      1. writing more tests which clarify expected behavior
      2. testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite.
      3. Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing.
      1. HADOOP-9361-011.patch
        311 kB
        Steve Loughran
      2. HADOOP-9361-009.patch
        327 kB
        Steve Loughran
      3. HADOOP-9361-008.patch
        323 kB
        Steve Loughran
      4. HADOOP-9361-007.patch
        316 kB
        Steve Loughran
      5. HADOOP-9361-006.patch
        295 kB
        Steve Loughran
      6. HADOOP-9361-005.patch
        276 kB
        Steve Loughran
      7. HADOOP-9361-004.patch
        248 kB
        Steve Loughran
      8. HADOOP-9361-003.patch
        225 kB
        Steve Loughran
      9. HADOOP-9361-002.patch
        146 kB
        Steve Loughran
      10. HADOOP-9361-001.patch
        65 kB
        Steve Loughran

        Issue Links

          Activity

          Steve Loughran created issue -
          Steve Loughran made changes -
          Field Original Value New Value
          Assignee Steve Loughran [ stevel@apache.org ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-9355 [ HADOOP-9355 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-9355 [ HADOOP-9355 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-9432 [ HADOOP-9432 ]
          Steve Loughran made changes -
          Summary Write a set of tests to more strictly define the expected behavior of filesystems Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
          Gavin made changes -
          Link This issue depends on HADOOP-9432 [ HADOOP-9432 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-9432 [ HADOOP-9432 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-8545 [ HADOOP-8545 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-9438 [ HADOOP-9438 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-9597 [ HADOOP-9597 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7363 [ HADOOP-7363 ]
          Steve Loughran made changes -
          Link This issue relates to HDFS-303 [ HDFS-303 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7352 [ HADOOP-7352 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-4872 [ HDFS-4872 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-8722 [ HBASE-8722 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-001.patch [ 12590695 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 2.1.0-beta [ 12324030 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-9702 [ HADOOP-9702 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-002.patch [ 12591079 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-353 [ HDFS-353 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-003.patch [ 12591494 ]
          Steve Loughran made changes -
          Link This issue depends upon HADOOP-6240 [ HADOOP-6240 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591660 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591660 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591663 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-005.patch [ 12605475 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 2.2.0 [ 12325048 ]
          Affects Version/s 2.1.0-beta [ 12324030 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-006.patch [ 12623691 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.4.0 [ 12324587 ]
          Steve Loughran made changes -
          Link This issue is related to MAPREDUCE-5264 [ MAPREDUCE-5264 ]
          Steve Loughran made changes -
          Link This issue is depended upon by HADOOP-10400 [ HADOOP-10400 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s
          Steve Loughran made changes -
          Attachment HADOOP-9361-007.patch [ 12636110 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-10419 [ HADOOP-10419 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-008.patch [ 12636263 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.4.0 [ 12326144 ]
          Target Version/s 3.0.0 [ 12320357 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10440 [ HADOOP-10440 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-009.patch [ 12638079 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10457 [ HADOOP-10457 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10458 [ HADOOP-10458 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-011.patch [ 12638246 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-6193 [ HDFS-6193 ]

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development