Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11379

DFSInputStream may infinite loop requesting block locations

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha4
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      DFSInputStream creation caches file size and initial range of locations. If the file is truncated (or replaced) and the client attempts to read outside the initial range, the client goes into a tight infinite looping requesting locations for the nonexistent range.

      1. HDFS-11379.branch-2.7.patch
        7 kB
        Xiao Chen
      2. HDFS-11379.branch-2.patch
        7 kB
        Daryn Sharp
      3. HDFS-11379.trunk.patch
        7 kB
        Daryn Sharp

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        It seems from other jiras a backport doesn't need a +1, so with pre-commit not complaining, I have committed this to branch-2.7.

        Show
        xiaochen Xiao Chen added a comment - It seems from other jiras a backport doesn't need a +1, so with pre-commit not complaining, I have committed this to branch-2.7.
        Hide
        xiaochen Xiao Chen added a comment -

        Ping, does the 2.7 backport look good?
        (Also not sure if backports require a +1 to commit. Conflicts are minor and please let me know if +1 isn't required)

        Show
        xiaochen Xiao Chen added a comment - Ping, does the 2.7 backport look good? (Also not sure if backports require a +1 to commit. Conflicts are minor and please let me know if +1 isn't required)
        Hide
        xiaochen Xiao Chen added a comment -

        All the red and yellow are not introduced by the patch, and the failed tests passed locally.

        Kihwal and Daryn, could you please review the backport?

        Show
        xiaochen Xiao Chen added a comment - All the red and yellow are not introduced by the patch, and the failed tests passed locally. Kihwal and Daryn, could you please review the backport?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 16m 2s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 2s branch-2.7 passed
        +1 compile 0m 57s branch-2.7 passed with JDK v1.8.0_121
        +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_121
        +1 checkstyle 0m 23s branch-2.7 passed
        +1 mvnsite 0m 57s branch-2.7 passed
        +1 mvneclipse 0m 14s branch-2.7 passed
        +1 findbugs 2m 56s branch-2.7 passed
        +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_121
        +1 javadoc 1m 40s branch-2.7 passed with JDK v1.7.0_121
        +1 mvninstall 0m 52s the patch passed
        +1 compile 0m 55s the patch passed with JDK v1.8.0_121
        +1 javac 0m 55s the patch passed
        +1 compile 1m 0s the patch passed with JDK v1.7.0_121
        +1 javac 1m 0s the patch passed
        -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 88 unchanged - 4 fixed = 89 total (was 92)
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 1704 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 43s The patch 75 line(s) with tabs.
        +1 findbugs 3m 7s the patch passed
        +1 javadoc 0m 54s the patch passed with JDK v1.8.0_121
        +1 javadoc 1m 37s the patch passed with JDK v1.7.0_121
        -1 unit 59m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
        -1 asflicense 0m 20s The patch generated 3 ASF License warnings.
        163m 16s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
        JDK v1.7.0_121 Failed junit tests hadoop.hdfs.TestLeaseRecovery2
          hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.TestFileTruncate



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue HDFS-11379
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852611/HDFS-11379.branch-2.7.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux da42c553f643 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / 18dfff8
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18373/testReport/
        asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18373/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 16m 2s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 2s branch-2.7 passed +1 compile 0m 57s branch-2.7 passed with JDK v1.8.0_121 +1 compile 1m 2s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 23s branch-2.7 passed +1 mvnsite 0m 57s branch-2.7 passed +1 mvneclipse 0m 14s branch-2.7 passed +1 findbugs 2m 56s branch-2.7 passed +1 javadoc 0m 56s branch-2.7 passed with JDK v1.8.0_121 +1 javadoc 1m 40s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 52s the patch passed +1 compile 0m 55s the patch passed with JDK v1.8.0_121 +1 javac 0m 55s the patch passed +1 compile 1m 0s the patch passed with JDK v1.7.0_121 +1 javac 1m 0s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 88 unchanged - 4 fixed = 89 total (was 92) +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 1704 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 43s The patch 75 line(s) with tabs. +1 findbugs 3m 7s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 37s the patch passed with JDK v1.7.0_121 -1 unit 59m 41s hadoop-hdfs in the patch failed with JDK v1.7.0_121. -1 asflicense 0m 20s The patch generated 3 ASF License warnings. 163m 16s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_121 Failed junit tests hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue HDFS-11379 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852611/HDFS-11379.branch-2.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux da42c553f643 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 18dfff8 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18373/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/18373/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18373/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        2.7 patch has some trivial conflicts: fetchBlockAt signature, imports in the test class, and the renamed DFSConfigKeys.DFS_CLIENT_READ_PREFETCH_SIZE_KEY

        Show
        xiaochen Xiao Chen added a comment - 2.7 patch has some trivial conflicts: fetchBlockAt signature, imports in the test class, and the renamed DFSConfigKeys.DFS_CLIENT_READ_PREFETCH_SIZE_KEY
        Hide
        xiaochen Xiao Chen added a comment -

        Sorry for reopening this. Attaching a branch-2.7 patch.

        2.6 seems will be EOL'ed soon so maybe we can skip (and more conflicts seemingly due to the lack of HDFS-7495). thoughts?

        Show
        xiaochen Xiao Chen added a comment - Sorry for reopening this. Attaching a branch-2.7 patch. 2.6 seems will be EOL'ed soon so maybe we can skip (and more conflicts seemingly due to the lack of HDFS-7495 ). thoughts?
        Hide
        busbey Sean Busbey added a comment -

        could we get this into a 2.7 release? (maybe a 2.6 if 2.6 is similarly impacted?)

        Show
        busbey Sean Busbey added a comment - could we get this into a 2.7 release? (maybe a 2.6 if 2.6 is similarly impacted?)
        Hide
        kihwal Kihwal Lee added a comment -

        The impact of this bug can vary depending on how wide the job is, but it essentially becomes DDOS using getBlockLocations(). In one incident, the NN could deal with the rpc load, but it ran out of space for logging.

        Show
        kihwal Kihwal Lee added a comment - The impact of this bug can vary depending on how wide the job is, but it essentially becomes DDOS using getBlockLocations() . In one incident, the NN could deal with the rpc load, but it ran out of space for logging.
        Hide
        kihwal Kihwal Lee added a comment -

        Thanks, I just cherry-picked it to branch-2.8.0.

        Show
        kihwal Kihwal Lee added a comment - Thanks, I just cherry-picked it to branch-2.8.0.
        Hide
        djp Junping Du added a comment -

        Junping Du, this looks to be important enough to put in 2.8.0. Is it okay to commit this to the release branch?

        Hi Kihwal Lee, there is another blocker for 2.8.0 - YARN-6147 to fix an incompatible issues. So I am ok to put this patch to 2.8.0 if this is really important.

        Show
        djp Junping Du added a comment - Junping Du, this looks to be important enough to put in 2.8.0. Is it okay to commit this to the release branch? Hi Kihwal Lee , there is another blocker for 2.8.0 - YARN-6147 to fix an incompatible issues. So I am ok to put this patch to 2.8.0 if this is really important.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11233 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11233/)
        HDFS-11379. DFSInputStream may infinite loop requesting block locations. (kihwal: rev 07a5184f74fdeffc42cdaec42ad4378c0e41c541)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11233 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11233/ ) HDFS-11379 . DFSInputStream may infinite loop requesting block locations. (kihwal: rev 07a5184f74fdeffc42cdaec42ad4378c0e41c541) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java
        Hide
        kihwal Kihwal Lee added a comment -

        I've committed this to trunk, branch-2 and branch-2.8.

        Show
        kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.8.
        Hide
        kihwal Kihwal Lee added a comment -

        Nicely done. Although the retry in readWithStrategy() makes it try one more time before throwing an exception back to the reader, it is far better than going into an infinite loop.

        +1

        Junping Du, this looks to be important enough to put in 2.8.0. Is it okay to commit this to the release branch?

        Show
        kihwal Kihwal Lee added a comment - Nicely done. Although the retry in readWithStrategy() makes it try one more time before throwing an exception back to the reader, it is far better than going into an infinite loop. +1 Junping Du , this looks to be important enough to put in 2.8.0. Is it okay to commit this to the release branch?
        Hide
        kihwal Kihwal Lee added a comment -

        I will review it today.

        Show
        kihwal Kihwal Lee added a comment - I will review it today.
        Hide
        daryn Daryn Sharp added a comment -

        Test failure not related.

        Show
        daryn Daryn Sharp added a comment - Test failure not related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 42s branch-2 passed
        +1 compile 1m 17s branch-2 passed with JDK v1.8.0_121
        +1 compile 1m 21s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 34s branch-2 passed
        +1 mvnsite 1m 23s branch-2 passed
        +1 mvneclipse 0m 28s branch-2 passed
        +1 findbugs 3m 36s branch-2 passed
        +1 javadoc 1m 13s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 1m 58s branch-2 passed with JDK v1.7.0_121
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 11s the patch passed
        +1 compile 1m 12s the patch passed with JDK v1.8.0_121
        +1 javac 1m 12s the patch passed
        +1 compile 1m 21s the patch passed with JDK v1.7.0_121
        +1 javac 1m 21s the patch passed
        -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 91 unchanged - 4 fixed = 93 total (was 95)
        +1 mvnsite 1m 20s the patch passed
        +1 mvneclipse 0m 23s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 55s the patch passed
        +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121
        +1 javadoc 1m 51s the patch passed with JDK v1.7.0_121
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121.
        -1 unit 48m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        136m 0s



        Reason Tests
        JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue HDFS-11379
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849748/HDFS-11379.branch-2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1ac3a94be4ef 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / d20dc86
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18286/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18286/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18286/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 42s branch-2 passed +1 compile 1m 17s branch-2 passed with JDK v1.8.0_121 +1 compile 1m 21s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 34s branch-2 passed +1 mvnsite 1m 23s branch-2 passed +1 mvneclipse 0m 28s branch-2 passed +1 findbugs 3m 36s branch-2 passed +1 javadoc 1m 13s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 58s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 1m 12s the patch passed with JDK v1.8.0_121 +1 javac 1m 12s the patch passed +1 compile 1m 21s the patch passed with JDK v1.7.0_121 +1 javac 1m 21s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project: The patch generated 2 new + 91 unchanged - 4 fixed = 93 total (was 95) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 55s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 51s the patch passed with JDK v1.7.0_121 +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121. -1 unit 48m 35s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 136m 0s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.datanode.metrics.TestDataNodeOutlierDetectionViaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11379 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12849748/HDFS-11379.branch-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1ac3a94be4ef 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d20dc86 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18286/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18286/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18286/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18286/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daryn Daryn Sharp added a comment -

        Main issue is getFinalizedBlockRange takes an offset and length. It will loop forever until it receives blocks that satisfy length, even if no locations were returned.

        Simplest change is to abort when no locations are returned in the expected range. Changed two duplicated code regions, sans whether to cache, to invoke a common method for fetching locations.

        Show
        daryn Daryn Sharp added a comment - Main issue is getFinalizedBlockRange takes an offset and length. It will loop forever until it receives blocks that satisfy length, even if no locations were returned. Simplest change is to abort when no locations are returned in the expected range. Changed two duplicated code regions, sans whether to cache, to invoke a common method for fetching locations.
        Hide
        daryn Daryn Sharp added a comment -

        Found due to hive jobs colliding. Tasks opened orc files, other tasks stomped on them, so when the original tasks attempted to read the footer (outside the initial fetch range) it went into an infinite loop requesting locations. Issue was difficult to isolate because by default the stream will fetch 10 blocks of locations so the issue only manifested for multi-GB files.

        Show
        daryn Daryn Sharp added a comment - Found due to hive jobs colliding. Tasks opened orc files, other tasks stomped on them, so when the original tasks attempted to read the footer (outside the initial fetch range) it went into an infinite loop requesting locations. Issue was difficult to isolate because by default the stream will fetch 10 blocks of locations so the issue only manifested for multi-GB files.

          People

          • Assignee:
            daryn Daryn Sharp
            Reporter:
            daryn Daryn Sharp
          • Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development