Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13405

Fix Connection Leak in OrcRawRecordMerger

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 1.3.0, 2.0.1, 2.1.0
    • Component/s: ORC
    • Labels:
      None
    • Target Version/s:

      Description

      In OrcRawRecordMerger.getLastFlushLength, if the opened stream throws an IOException on .available() or on .readLong(), the function will exit without closing the stream.

      This patch adds a try-with-resources to fix this.

      1. HIVE-13405.patch
        0.9 kB
        Thomas Poepping

        Activity

        Hide
        hiveqa Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12796903/HIVE-13405.patch

        ERROR: -1 due to no test(s) being added or modified.

        ERROR: -1 due to 41 failed/errored test(s), 9937 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
        org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
        org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
        org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
        org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
        org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
        org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
        org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
        org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
        org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
        org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
        org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLNoLock
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLShared
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadTable
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
        org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition
        org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
        org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
        org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
        org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
        org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
        org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
        org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
        org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
        org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
        

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7472/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7472/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7472/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 41 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12796903 - PreCommit-HIVE-TRUNK-Build

        Show
        hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12796903/HIVE-13405.patch ERROR: -1 due to no test(s) being added or modified. ERROR: -1 due to 41 failed/errored test(s), 9937 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLNoLock org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLShared org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadTable org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7472/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7472/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7472/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 41 tests failed This message is automatically generated. ATTACHMENT ID: 12796903 - PreCommit-HIVE-TRUNK-Build
        Hide
        poeppt Thomas Poepping added a comment -

        I'll look into these failures, but due to the size and content of this patch, it's unlikely that it caused the test failures.

        Show
        poeppt Thomas Poepping added a comment - I'll look into these failures, but due to the size and content of this patch, it's unlikely that it caused the test failures.
        Hide
        sershe Sergey Shelukhin added a comment -

        +0.9. Prasanth Jayachandran does this make sense?

        Show
        sershe Sergey Shelukhin added a comment - +0.9. Prasanth Jayachandran does this make sense?
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Yeah. This will work in jdk7 onwards. +1

        Show
        prasanth_j Prasanth Jayachandran added a comment - Yeah. This will work in jdk7 onwards. +1
        Hide
        ashutoshc Ashutosh Chauhan added a comment -

        Pushed to master. Thanks, Thomas Poepping

        Show
        ashutoshc Ashutosh Chauhan added a comment - Pushed to master. Thanks, Thomas Poepping
        Hide
        prasanth_j Prasanth Jayachandran added a comment -

        Backported to branch-2.0 and branch-1

        Show
        prasanth_j Prasanth Jayachandran added a comment - Backported to branch-2.0 and branch-1

          People

          • Assignee:
            poeppt Thomas Poepping
            Reporter:
            poeppt Thomas Poepping
          • Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development