Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      herriot

      Description

      Hadoop would benefit from having a large-scale, automated, test-framework. This jira is meant to be a master-jira to track relevant work.


      The proposal is a junit-based, large-scale test framework which would run against real clusters.

      There are several pieces we need to achieve this goal:

      1. A set of utilities we can use in junit-based tests to work with real, large-scale hadoop clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers, datanodes, entire racks of both etc.
      2. Enhanced control-ability and inspect-ability of the various components in the system e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation etc. Tests would be much more relevant if we could for e.g. query for specific states of the jobtracker, scheduler etc. Clearly these apis should not be part of the production clusters - hence the proposal is to use aspectj to weave these new apis to debug-deployments.

      Related note: we should break up our tests into at least 3 categories:

      1. src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
      2. src/test/integration -> Current junit tests with Mini* clusters etc.
      3. src/test/system -> HADOOP-6332 and it's children

        Attachments

        1. 6332_v1.patch
          20 kB
          Sharad Agarwal
        2. 6332_v2.patch
          24 kB
          Sharad Agarwal
        3. 6332.patch
          183 kB
          Konstantin Boudnik
        4. 6332.patch
          183 kB
          Konstantin Boudnik
        5. 6332.patch
          86 kB
          Sharad Agarwal
        6. 6332-phase2.fix1.patch
          2 kB
          Konstantin Boudnik
        7. 6332-phase2.fix2.patch
          1 kB
          Konstantin Boudnik
        8. 6332-phase2.patch
          358 kB
          Konstantin Boudnik
        9. HADOOP-6332.0.22.patch
          267 kB
          Konstantin Boudnik
        10. HADOOP-6332.0.22.patch
          267 kB
          Konstantin Boudnik
        11. HADOOP-6332.0.22.patch
          249 kB
          Konstantin Boudnik
        12. HADOOP-6332.0.22.patch
          254 kB
          Konstantin Boudnik
        13. HADOOP-6332.0.22.patch
          253 kB
          Konstantin Boudnik
        14. HADOOP-6332.0.22.patch
          253 kB
          Konstantin Boudnik
        15. HADOOP-6332.0.22.patch
          253 kB
          Konstantin Boudnik
        16. HADOOP-6332.0.22.patch
          256 kB
          Konstantin Boudnik
        17. HADOOP-6332.patch
          13 kB
          Konstantin Boudnik
        18. HADOOP-6332.patch
          13 kB
          Konstantin Boudnik
        19. HADOOP-6332-MR.patch
          13 kB
          Konstantin Boudnik
        20. HADOOP-6332-MR.patch
          11 kB
          Konstantin Boudnik

          Issue Links

            Activity

              People

              • Assignee:
                cos Konstantin Boudnik
                Reporter:
                acmurthy Arun C Murthy
              • Votes:
                0 Vote for this issue
                Watchers:
                34 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: