Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduce write support for Fuse; requires Linux kernel 2.6.15 or better.

      Description

      1. dfs_write should return the #of bytes written and not 0
      2. implement dfs_flush
      3. uncomment/fix dfs_create
      4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
      5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test

      1. TEST-TestFuseDFS.txt
        17 kB
        Pete Wyckoff
      2. patch4.txt
        25 kB
        Pete Wyckoff
      3. patch3.txt
        24 kB
        Pete Wyckoff
      4. patch2.txt
        28 kB
        Pete Wyckoff
      5. patch1.txt
        2 kB
        Pete Wyckoff

        Issue Links

          Activity

          Pete Wyckoff created issue -
          Pete Wyckoff made changes -
          Field Original Value New Value
          Attachment patch1.txt [ 12384875 ]
          Pete Wyckoff made changes -
          Attachment patch1.txt [ 12384875 ]
          Pete Wyckoff made changes -
          Attachment patch1.txt [ 12384876 ]
          Pete Wyckoff made changes -
          Attachment patch2.txt [ 12385690 ]
          Pete Wyckoff made changes -
          Assignee Pete Wyckoff [ wyckoff ]
          Release Note Write support added, but performance of writes not looked at yet.
          I also changed the unit tests to be callable from the top level hadoop build file
          Status Open [ 1 ] Patch Available [ 10002 ]
          Pete Wyckoff made changes -
          Release Note Write support added, but performance of writes not looked at yet.
          I also changed the unit tests to be callable from the top level hadoop build file
          Write support added, but performance of writes not looked at yet.
          I also changed the unit tests to be callable from the top level hadoop build file
          Note that on Linux, writes require kernel 2.6.15 or above
          Pete Wyckoff made changes -
          Attachment patch3.txt [ 12385996 ]
          Pete Wyckoff made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Pete Wyckoff made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Pete Wyckoff made changes -
          Summary implement writes fix writes
          Description Doesn't support writes because fuse protocol first creates the file then closes it and re-opens it to start writing to it. So, (until hadoop-1700), we need a work around.

          One way to do this is to open the file with overwrite flag on the second open. For security, would only want to do this for zero length files (could even check the creation ts too, but because of clock skew, that may be harder).

          Doug, Craig, Nicholas - Comments?

          -- pete

          ps since mostly already implemented, this should be a very quick patch
          1. dfs_write should return the #of bytes written and not 0
          2. implement dfs_flush
          3. uncomment/fix dfs_create
          4. fix the flags argument passed to libhdfs openFile to get around the bug in hadoop-3723
          5. Since I am adding a write unit test, I noticed the unit tests are in the wrong directory - should be in contrib/fuse-dfs/src/test and not contrib/fuse-dfs/test
          Pete Wyckoff made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Pete Wyckoff made changes -
          Attachment patch4.txt [ 12386360 ]
          Pete Wyckoff made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          dhruba borthakur made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Pete Wyckoff made changes -
          Attachment TEST-TestFuseDFS.txt [ 12386444 ]
          dhruba borthakur made changes -
          Affects Version/s 0.19.0 [ 12313211 ]
          Doug Cutting made changes -
          Fix Version/s 0.19.0 [ 12313211 ]
          Robert Chansler made changes -
          Release Note Write support added, but performance of writes not looked at yet.
          I also changed the unit tests to be callable from the top level hadoop build file
          Note that on Linux, writes require kernel 2.6.15 or above
          Introduce write support for Fuse.
          Robert Chansler made changes -
          Release Note Introduce write support for Fuse. Introduce write support for Fuse; requires Linux kernel 2.6.15 or better.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s contrib/fuse-dfs [ 12312376 ]
          Lam Thien Son made changes -
          Link This issue is blocked by LOGGING-160 [ LOGGING-160 ]

            People

            • Assignee:
              Pete Wyckoff
              Reporter:
              Pete Wyckoff
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development