Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-577

NumberFormatException when parsing dfs.block.size

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.8.1, 0.9.0, 0.8.2, 0.10.0, 0.8.3, 0.8.4, 0.11.0, 0.12.0
    • Fix Version/s: 0.14.0
    • Component/s: IO
    • Labels:

      Description

      When using file size abbreviations (like "128m") for the HDFS configuration property "dfs.block.size" the Crunch job crashes with a NumberFormatException. According to the Hadoop documentation (https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml), this style of abbreviations should be supported.

      The problem occurs at line 38 in CrunchCombineFileInputFormat.java when the configuration property is parsed using the getLong() method instead of getLongBytes() method. Furthermore, obsolete configuration key "dfs.block.size" is used instead of "dfs.blocksize" (see https://issues.apache.org/jira/browse/HDFS-631) which leads to a warning message being emitted when starting a MR pipeline.

      The proposed solution discussed on the crunch-users mailing list (http://mail-archives.apache.org/mod_mbox/crunch-user/201511.mbox/browser) is to use the getLongBytes() method and the new config key "dfs.blocksize".

        Attachments

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              tcechal Tomáš Čechal
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified