Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-708

A stress-test tool for HDFS.

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • test, tools
    • None
    • Reviewed
    • Hide
      Does not currently provide anything but uniform distribution.
      Uses some older depreciated class interfaces (for mapper and reducer)
      This was tested on 0.20 and 0.22 (locally) so it should be fairly backwards compatible.
      Show
      Does not currently provide anything but uniform distribution. Uses some older depreciated class interfaces (for mapper and reducer) This was tested on 0.20 and 0.22 (locally) so it should be fairly backwards compatible.

    Description

      It would be good to have a tool for automatic stress testing HDFS, which would provide IO-intensive load on HDFS cluster.
      The idea is to start the tool, let it run overnight, and then be able to analyze possible failures.

      Attachments

        1. SLiveTest.pdf
          41 kB
          Konstantin Shvachko
        2. SLiveTest.pdf
          41 kB
          Konstantin Shvachko
        3. slive.patch.1
          216 kB
          Joshua Harlow
        4. slive.patch
          229 kB
          Joshua Harlow
        5. SLiveTest.pdf
          42 kB
          Konstantin Shvachko

        Issue Links

          is duplicated by

          Sub-task - The sub-task of the issue HDFS-522 Create stress tests for append feature

          • Major - Major loss of function.
          • Resolved
          is part of

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-1804 Stress-test tool for HDFS introduced in HDFS-708

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2984 S-live: Rate operation count for delete is worse than 0.20.204 by 28.8%

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-6332 Large-scale Automated Test Framework

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1893 Multiple reducers for Slive

          • Major - Major loss of function.
          • Closed

          Test - A new unit, integration or system test. HDFS-523 Create performance/scalability tests for append feature

          • Major - Major loss of function.
          • Resolved

          Test - A new unit, integration or system test. HADOOP-193 DFS i/o benchmark.

          • Major - Major loss of function.
          • Closed

          Activity

            People

              harlowja Joshua Harlow
              shv Konstantin Shvachko
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: