Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2755

dfs fsck extremely slow, dfs ls times out

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Jan 28 nightly build

      With patches 2095, 2119, and 2723

      Description

      I upgraded a Hadoop installation to the Jan 28 nightly build.
      DFS contains 2.4+ M files.
      Upgrade finished but not finalized.

      Before finalizing I wanted to run fsck on the DFS. It hardly progressed after 6 hours (not finished yet). With the '-files' option turned on, it lists about 300 entries in 10 minutes.

      And when I tried to list a subdirectory with 100,000 files, it repeatedly (about 20 attempts) timed out.
      Changing timeout from 1 to 10 minutes did not help.

        Attachments

        1. 2755_20080131.patch
          5 kB
          Tsz-wo Sze
        2. 2755_20080131b.patch
          5 kB
          Tsz-wo Sze

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                ckunz Christian Kunz
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: