Hadoop Common
  1. Hadoop Common
  2. HADOOP-9361

Strictly define the expected behavior of filesystem APIs and write tests to verify compliance

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: fs, test
    • Labels:
      None

      Description

      FileSystem and FileContract aren't tested rigorously enough -while HDFS gets tested downstream, other filesystems, such as blobstore bindings, don't.

      The only tests that are common are those of FileSystemContractTestBase, which HADOOP-9258 shows is incomplete.

      I propose

      1. writing more tests which clarify expected behavior
      2. testing operations in the interface being in their own JUnit4 test classes, instead of one big test suite.
      3. Having each FS declare via a properties file what behaviors they offer, such as atomic-rename, atomic-delete, umask, immediate-consistency -test methods can downgrade to skipped test cases if a feature is missing.
      1. HADOOP-9361.awang-addendum.patch
        96 kB
        Andrew Wang
      2. HADOOP-9361-001.patch
        65 kB
        Steve Loughran
      3. HADOOP-9361-002.patch
        146 kB
        Steve Loughran
      4. HADOOP-9361-003.patch
        225 kB
        Steve Loughran
      5. HADOOP-9361-004.patch
        248 kB
        Steve Loughran
      6. HADOOP-9361-005.patch
        276 kB
        Steve Loughran
      7. HADOOP-9361-006.patch
        295 kB
        Steve Loughran
      8. HADOOP-9361-007.patch
        316 kB
        Steve Loughran
      9. HADOOP-9361-008.patch
        323 kB
        Steve Loughran
      10. HADOOP-9361-009.patch
        327 kB
        Steve Loughran
      11. HADOOP-9361-011.patch
        311 kB
        Steve Loughran
      12. HADOOP-9361-012.patch
        304 kB
        Steve Loughran
      13. HADOOP-9361-013.patch
        377 kB
        Steve Loughran
      14. HADOOP-9361-014.patch
        378 kB
        Steve Loughran
      15. HADOOP-9361-015.patch
        380 kB
        Steve Loughran
      16. HADOOP-9361-016.patch
        380 kB
        Steve Loughran
      17. HADOOP-9361-017.patch
        383 kB
        Steve Loughran
      18. HADOOP-9361-018.patch
        383 kB
        Steve Loughran

        Issue Links

          Activity

          Steve Loughran created issue -
          Steve Loughran made changes -
          Field Original Value New Value
          Assignee Steve Loughran [ stevel@apache.org ]
          Steve Loughran made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-9355 [ HADOOP-9355 ]
          Steve Loughran made changes -
          Link This issue blocks HADOOP-9355 [ HADOOP-9355 ]
          Steve Loughran made changes -
          Link This issue depends on HADOOP-9432 [ HADOOP-9432 ]
          Steve Loughran made changes -
          Summary Write a set of tests to more strictly define the expected behavior of filesystems Strictly define the expected behavior of filesystem APIs and write tests to verify compliance
          Gavin made changes -
          Link This issue depends on HADOOP-9432 [ HADOOP-9432 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-9432 [ HADOOP-9432 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-8545 [ HADOOP-8545 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-9438 [ HADOOP-9438 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-9597 [ HADOOP-9597 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7363 [ HADOOP-7363 ]
          Steve Loughran made changes -
          Link This issue relates to HDFS-303 [ HDFS-303 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-7352 [ HADOOP-7352 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-4872 [ HDFS-4872 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-8722 [ HBASE-8722 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-001.patch [ 12590695 ]
          Steve Loughran made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 2.1.0-beta [ 12324030 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-9702 [ HADOOP-9702 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-002.patch [ 12591079 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-353 [ HDFS-353 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-003.patch [ 12591494 ]
          Steve Loughran made changes -
          Link This issue depends upon HADOOP-6240 [ HADOOP-6240 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591660 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591660 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-004.patch [ 12591663 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-005.patch [ 12605475 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 2.2.0 [ 12325048 ]
          Affects Version/s 2.1.0-beta [ 12324030 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-006.patch [ 12623691 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.4.0 [ 12324587 ]
          Steve Loughran made changes -
          Link This issue is related to MAPREDUCE-5264 [ MAPREDUCE-5264 ]
          Steve Loughran made changes -
          Link This issue is depended upon by HADOOP-10400 [ HADOOP-10400 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s
          Steve Loughran made changes -
          Attachment HADOOP-9361-007.patch [ 12636110 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-10419 [ HADOOP-10419 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-008.patch [ 12636263 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.4.0 [ 12326144 ]
          Target Version/s 3.0.0 [ 12320357 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10440 [ HADOOP-10440 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-009.patch [ 12638079 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10457 [ HADOOP-10457 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10458 [ HADOOP-10458 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-011.patch [ 12638246 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-6193 [ HDFS-6193 ]
          Steve Loughran made changes -
          Link This issue depends upon HDFS-6262 [ HDFS-6262 ]
          Steve Loughran made changes -
          Link This issue depends upon HDFS-4258 [ HDFS-4258 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-012.patch [ 12641004 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 3.0.0 [ 12320357 ] 3.0.0, 2.5.0 [ 12320357, 12326263 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-10533 [ HADOOP-10533 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-013.patch [ 12647652 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.2.0 [ 12325048 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-014.patch [ 12647777 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Andrew Wang made changes -
          Attachment HADOOP-9361.awang-addendum.patch [ 12648399 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-015.patch [ 12648875 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-016.patch [ 12650971 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-017.patch [ 12653489 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Steve Loughran made changes -
          Attachment HADOOP-9361-018.patch [ 12653552 ]
          Steve Loughran made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-9495 [ HADOOP-9495 ]
          Steve Loughran made changes -
          Link This issue depends upon HDFS-6262 [ HDFS-6262 ]
          Steve Loughran made changes -
          Link This issue incorporates HADOOP-10458 [ HADOOP-10458 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-10458 [ HADOOP-10458 ]
          Steve Loughran made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.5.0 [ 12326263 ]
          Resolution Fixed [ 1 ]
          Akira AJISAKA made changes -
          Link This issue relates to HADOOP-10801 [ HADOOP-10801 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-6803 [ HDFS-6803 ]
          Steve Loughran made changes -
          Link This issue is related to HDFS-6735 [ HDFS-6735 ]
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-9565 [ HADOOP-9565 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-11417 [ HADOOP-11417 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-3257 [ HADOOP-3257 ]
          Steve Loughran made changes -
          Link This issue contains HADOOP-10589 [ HADOOP-10589 ]
          Steve Loughran made changes -
          Link This issue is related to HADOOP-11708 [ HADOOP-11708 ]

            People

            • Assignee:
              Steve Loughran
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development