Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.5
    • 1.4.7
    • None
    • None

    Description

      There is no way where user can define the upper limit of volume that each mapper can handle. Current Sqoop does the calculation based on mapper by Switch -m and --split-by but this does not give control user to specify the upper limit of volume handle by the mapper .

      if we can add such functionality in the Sqoop that would help us to load the bigger data set in case we don't have continuous key data available and there is a huge gap in maximum and minimum data set value.

      Attachments

        1. SQOOP-2334_1.patch
          19 kB
          Rakesh Sharma
        2. SQOOP-2334_2.patch
          19 kB
          Rakesh Sharma
        3. SQOOP-2334.patch
          19 kB
          Rakesh Sharma

        Issue Links

          Activity

            raksharma Rakesh Sharma added a comment -

            This is the patch created using git diff and contains all the changes required for Volume per mapper changes.

            raksharma Rakesh Sharma added a comment - This is the patch created using git diff and contains all the changes required for Volume per mapper changes.
            raksharma Rakesh Sharma added a comment -

            The new review request is created : https://reviews.apache.org/r/34536/
            The patch is also uploaded there for review

            raksharma Rakesh Sharma added a comment - The new review request is created : https://reviews.apache.org/r/34536/ The patch is also uploaded there for review
            raksharma Rakesh Sharma added a comment -

            Attaching a new patch based on review comments from Venkat

            raksharma Rakesh Sharma added a comment - Attaching a new patch based on review comments from Venkat
            raksharma Rakesh Sharma added a comment -

            Final with new changes is also attached and older patches removed

            raksharma Rakesh Sharma added a comment - Final with new changes is also attached and older patches removed
            raksharma Rakesh Sharma added a comment -

            Changed the patch after merging with latest code from trunk. Please find updated patch.

            raksharma Rakesh Sharma added a comment - Changed the patch after merging with latest code from trunk. Please find updated patch.

            Updated JIRA with link to RB. It is easier to navigate to RB link with a web link place instead of in comments. Please do so in future.

            venkatnrangan Venkat Ranganathan added a comment - Updated JIRA with link to RB. It is easier to navigate to RB link with a web link place instead of in comments. Please do so in future.
            raksharma Rakesh Sharma added a comment -

            Fixed Checktyle issues suggested by Venkat

            raksharma Rakesh Sharma added a comment - Fixed Checktyle issues suggested by Venkat

            Commit e21529ac6aad03bdcb572c61420e258be2d823fe in sqoop's branch refs/heads/trunk from venkatnrangan
            [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=e21529a ]

            SQOOP-2334: Sqoop Volume Per Mapper
            (Rakesh Sharma via Venkat Ranganathan)

            jira-bot ASF subversion and git services added a comment - Commit e21529ac6aad03bdcb572c61420e258be2d823fe in sqoop's branch refs/heads/trunk from venkatnrangan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=e21529a ] SQOOP-2334 : Sqoop Volume Per Mapper (Rakesh Sharma via Venkat Ranganathan)

            Thanks for your contribution raksharma

            venkatnrangan Venkat Ranganathan added a comment - Thanks for your contribution raksharma
            hudson Hudson added a comment -

            FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #1004 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/1004/)
            SQOOP-2334: Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe)

            • src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java
            • src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java
            • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
            • src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
            • src/java/org/apache/sqoop/SqoopOptions.java
            • src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java
            • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
            • src/java/org/apache/sqoop/config/ConfigurationHelper.java
            • src/docs/user/import.txt
            • src/java/org/apache/sqoop/config/ConfigurationConstants.java
            • src/java/org/apache/sqoop/tool/ImportTool.java
            • src/docs/man/import-args.txt
            hudson Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #1004 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/1004/ ) SQOOP-2334 : Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe ) src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/config/ConfigurationHelper.java src/docs/user/import.txt src/java/org/apache/sqoop/config/ConfigurationConstants.java src/java/org/apache/sqoop/tool/ImportTool.java src/docs/man/import-args.txt
            hudson Hudson added a comment -

            FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #1011 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/1011/)
            SQOOP-2334: Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe)

            • src/java/org/apache/sqoop/config/ConfigurationConstants.java
            • src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java
            • src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java
            • src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
            • src/docs/user/import.txt
            • src/java/org/apache/sqoop/tool/ImportTool.java
            • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
            • src/java/org/apache/sqoop/config/ConfigurationHelper.java
            • src/docs/man/import-args.txt
            • src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java
            • src/java/org/apache/sqoop/SqoopOptions.java
            • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
            hudson Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #1011 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/1011/ ) SQOOP-2334 : Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe ) src/java/org/apache/sqoop/config/ConfigurationConstants.java src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java src/docs/user/import.txt src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/java/org/apache/sqoop/config/ConfigurationHelper.java src/docs/man/import-args.txt src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1207 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1207/)
            SQOOP-2334: Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe)

            • src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java
            • src/java/org/apache/sqoop/SqoopOptions.java
            • src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
            • src/java/org/apache/sqoop/config/ConfigurationHelper.java
            • src/java/org/apache/sqoop/config/ConfigurationConstants.java
            • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
            • src/docs/user/import.txt
            • src/docs/man/import-args.txt
            • src/java/org/apache/sqoop/tool/ImportTool.java
            • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
            • src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java
            • src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java
            hudson Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1207 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1207/ ) SQOOP-2334 : Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe ) src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java src/java/org/apache/sqoop/SqoopOptions.java src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java src/java/org/apache/sqoop/config/ConfigurationHelper.java src/java/org/apache/sqoop/config/ConfigurationConstants.java src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/docs/user/import.txt src/docs/man/import-args.txt src/java/org/apache/sqoop/tool/ImportTool.java src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java
            hudson Hudson added a comment -

            FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #971 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/971/)
            SQOOP-2334: Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe)

            • src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java
            • src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java
            • src/java/org/apache/sqoop/tool/ImportTool.java
            • src/docs/man/import-args.txt
            • src/java/org/apache/sqoop/tool/BaseSqoopTool.java
            • src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java
            • src/java/org/apache/sqoop/config/ConfigurationConstants.java
            • src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java
            • src/java/org/apache/sqoop/SqoopOptions.java
            • src/java/org/apache/sqoop/config/ConfigurationHelper.java
            • src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java
            • src/docs/user/import.txt
            hudson Hudson added a comment - FAILURE: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #971 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/971/ ) SQOOP-2334 : Sqoop Volume Per Mapper (venkat: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=e21529ac6aad03bdcb572c61420e258be2d823fe ) src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java src/test/org/apache/sqoop/mapreduce/db/TestIntegerSplitter.java src/java/org/apache/sqoop/tool/ImportTool.java src/docs/man/import-args.txt src/java/org/apache/sqoop/tool/BaseSqoopTool.java src/java/org/apache/sqoop/mapreduce/db/DateSplitter.java src/java/org/apache/sqoop/config/ConfigurationConstants.java src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java src/java/org/apache/sqoop/SqoopOptions.java src/java/org/apache/sqoop/config/ConfigurationHelper.java src/java/org/apache/sqoop/mapreduce/db/IntegerSplitter.java src/docs/user/import.txt

            People

              raksharma Rakesh Sharma
              atugupta Atul Gupta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: