Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-8084 Sundry mapreduce improvements
  3. HBASE-7938

Add integration test for ImportTsv/LoadIncrementalHFiles workflow

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.98.0, 0.95.0
    • mapreduce
    • None
    • Reviewed
    • Hide
      The new IntegrationTestImportTsv is written in the style of the
      recently added IntegrationTestLoadAndVerify. It is designed to be run
      from maven or directly against a distributed cluster. It shares core
      test logic with TestImportTsv.

      To run it with maven, first refresh your install jars, and the run the
      test:

        $ mvn clean package install -DskipTests
        $ cd hbase-it
        $ mvn failsafe:integration-test -Dit.test=IntegrationTestImportTsv

      NOTE: Running from maven fails because HFileOutputFormat is creating
      the partition file on localFs, not HDFS.

      Or run it stand-alone:

        $ HBASE_CLASSPATH="hbase-it/target/hbase-it-<version>-tests.jar" \
          ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv

      When run stand-alone, it supports all the usual Hadoop Tool arguments,
      particularly -conf.

      ImportTsv was refactored to make code sharing easier. It is upgraded
      from MediumTest to LargeTest because it take 180+ seconds to run on my
      machine. Tests for the TsvParser are separated into their own
      SmallTest.
      Show
      The new IntegrationTestImportTsv is written in the style of the recently added IntegrationTestLoadAndVerify. It is designed to be run from maven or directly against a distributed cluster. It shares core test logic with TestImportTsv. To run it with maven, first refresh your install jars, and the run the test:   $ mvn clean package install -DskipTests   $ cd hbase-it   $ mvn failsafe:integration-test -Dit.test=IntegrationTestImportTsv NOTE: Running from maven fails because HFileOutputFormat is creating the partition file on localFs, not HDFS. Or run it stand-alone:   $ HBASE_CLASSPATH="hbase-it/target/hbase-it-<version>-tests.jar" \     ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv When run stand-alone, it supports all the usual Hadoop Tool arguments, particularly -conf. ImportTsv was refactored to make code sharing easier. It is upgraded from MediumTest to LargeTest because it take 180+ seconds to run on my machine. Tests for the TsvParser are separated into their own SmallTest.

    Description

      We have existing unit tests for smoke-testing the packaged MR jobs, however they do not create a runtime environment that is true to running on a real MR cluster. This is particularly true in regard to classpaths (HBASE-7934) but also other static state (HBASE-4802). An integration test that can be pointed to run on a pseudo-distributed Hadoop deployed on localhost would find these kinds of problems.

      Attachments

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HBASE-8011 Refactor ImportTsv

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is depended upon by

          Sub-task - The sub-task of the issue HBASE-4285 partitions file created in user's home directory by importtsv

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue HBASE-7934 TableMapReduceUtil doesn't include all dependency jars in new modular build

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-7644 Port HBASE-4802 'Disable show table metrics in bulk loader' to 0.94

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HBASE-4802 Disable show table metrics in bulk loader

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          Activity

            People

              ndimiduk Nick Dimiduk
              ndimiduk Nick Dimiduk
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: