Sqoop
  1. Sqoop
  2. SQOOP-1282

Consider avro files even if they carry no extension

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.4
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None

      Description

      The Export jobs in Sqoop have a mechanism that checks file contents to determine if its of Avro type or etc. regardless of extension, so it makes less sense to then also go forward and silently ignore all files in the export-dir that carry no .avro extension.

      Kinda similar to AVRO-1234, except we do not check for file validity in the Avro project's input format or driver levels as we already do in Sqoop.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #842 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/842/)
          SQOOP-1282: Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26)

          • src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          • src/test/com/cloudera/sqoop/TestAvroExport.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #842 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/842/ ) SQOOP-1282 : Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26 ) src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java src/test/com/cloudera/sqoop/TestAvroExport.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #877 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/877/)
          SQOOP-1282: Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26)

          • src/test/com/cloudera/sqoop/TestAvroExport.java
          • src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #877 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/877/ ) SQOOP-1282 : Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26 ) src/test/com/cloudera/sqoop/TestAvroExport.java src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1079 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1079/)
          SQOOP-1282: Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26)

          • src/test/com/cloudera/sqoop/TestAvroExport.java
          • src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1079 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1079/ ) SQOOP-1282 : Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26 ) src/test/com/cloudera/sqoop/TestAvroExport.java src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #883 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/883/)
          SQOOP-1282: Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26)

          • src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java
          • src/test/com/cloudera/sqoop/TestAvroExport.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #883 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/883/ ) SQOOP-1282 : Consider avro files even if they carry no extension (jarcec: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=d37bd1168c81e72f71bbdb7eedcf643f366e5b26 ) src/java/org/apache/sqoop/mapreduce/AvroInputFormat.java src/test/com/cloudera/sqoop/TestAvroExport.java
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for your contribution Harsh J!

          Show
          Jarek Jarcec Cecho added a comment - Thank you for your contribution Harsh J !
          Hide
          ASF subversion and git services added a comment -

          Commit d37bd1168c81e72f71bbdb7eedcf643f366e5b26 in branch refs/heads/trunk from Jarek Jarcec Cecho
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=d37bd11 ]

          SQOOP-1282: Consider avro files even if they carry no extension

          (Harsh J via Jarek Jarcec Cecho)

          Show
          ASF subversion and git services added a comment - Commit d37bd1168c81e72f71bbdb7eedcf643f366e5b26 in branch refs/heads/trunk from Jarek Jarcec Cecho [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=d37bd11 ] SQOOP-1282 : Consider avro files even if they carry no extension (Harsh J via Jarek Jarcec Cecho)
          Hide
          Jarek Jarcec Cecho added a comment -

          +1

          Show
          Jarek Jarcec Cecho added a comment - +1
          Hide
          Hari Sekhon added a comment - - edited

          Great work as usual, thanks Harsh

          Additional information: the full java exception that resulted is in the ticket I raised just after this one, now marked as duplicate:

          https://issues.apache.org/jira/browse/SQOOP-1283

          Thanks

          Hari Sekhon
          http://www.linkedin.com/in/harisekhon

          Show
          Hari Sekhon added a comment - - edited Great work as usual, thanks Harsh Additional information: the full java exception that resulted is in the ticket I raised just after this one, now marked as duplicate: https://issues.apache.org/jira/browse/SQOOP-1283 Thanks Hari Sekhon http://www.linkedin.com/in/harisekhon
          Hide
          Harsh J added a comment -

          Patch was compiled and tested with ant clean test -Dtestcase=TestAvroExport.

          Show
          Harsh J added a comment - Patch was compiled and tested with ant clean test -Dtestcase=TestAvroExport .

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development