Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • llap
    • None
    • None

    Attachments

      1. HIVE-10533.01.patch
        21 kB
        Sergey Shelukhin
      2. HIVE-10533.patch
        15 kB
        Sergey Shelukhin
      3. HIVE-10535.02.patch
        21 kB
        Sergey Shelukhin

      Activity

        Duplicate of HIVE-10456? I added cleanup code for map join case as well.

        prasanth_j Prasanth Jayachandran added a comment - Duplicate of HIVE-10456 ? I added cleanup code for map join case as well.

        This isolates LLAP caches by query, and on dag completion simply removes the cache for this query entirely. I am assuming by how other maps are treated that the event is pretty much guaranteed so this won't leak; otherwise periodic cleaner is needed (for this and other maps).

        gopalv sseth do you guys want to review?
        I haven't tested on cluster yet, might add some logging for that

        sershe Sergey Shelukhin added a comment - This isolates LLAP caches by query, and on dag completion simply removes the cache for this query entirely. I am assuming by how other maps are treated that the event is pretty much guaranteed so this won't leak; otherwise periodic cleaner is needed (for this and other maps). gopalv sseth do you guys want to review? I haven't tested on cluster yet, might add some logging for that

        Grrr, this doesn't work because the execution code just passes nulls everywhere for queryId. Great.

        sershe Sergey Shelukhin added a comment - Grrr, this doesn't work because the execution code just passes nulls everywhere for queryId. Great.

        This actually works, tested on cluster. I added a hook into TezProcessor to get dag name to queryId mapping

        sershe Sergey Shelukhin added a comment - This actually works, tested on cluster. I added a hook into TezProcessor to get dag name to queryId mapping
        hiveqa Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12744114/HIVE-10533.01.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4531/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4531/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4531/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
        + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4531/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ git = \s\v\n ]]
        + [[ git = \g\i\t ]]
        + [[ -z master ]]
        + [[ -d apache-github-source-source ]]
        + [[ ! -d apache-github-source-source/.git ]]
        + [[ ! -d apache-github-source-source ]]
        + cd apache-github-source-source
        + git fetch origin
        From https://github.com/apache/hive
           10dc20f..02e762f  master     -> origin/master
        + git reset --hard HEAD
        HEAD is now at 10dc20f HIVE-11016 : MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions) (Sergey Shelukhin, reviewed by Vikram Dixit K)
        + git clean -f -d
        Removing ql/src/test/queries/clientnegative/load_orc_negative_part.q
        Removing ql/src/test/queries/clientpositive/load_orc_part.q
        Removing ql/src/test/results/clientnegative/load_orc_negative_part.q.out
        Removing ql/src/test/results/clientpositive/load_orc_part.q.out
        + git checkout master
        Already on 'master'
        Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
        + git reset --hard origin/master
        HEAD is now at 02e762f HIVE-11179: HIVE should allow custom converting from HivePrivilegeObjectDesc to privilegeObject for different authorizers(Dapeng Sun, Reviewed by Ferdinand Xu)
        + git merge --ff-only origin/master
        Already up-to-date.
        + git gc
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12744114 - PreCommit-HIVE-TRUNK-Build

        hiveqa Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12744114/HIVE-10533.01.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4531/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4531/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4531/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4531/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin From https://github.com/apache/hive 10dc20f..02e762f master -> origin/master + git reset --hard HEAD HEAD is now at 10dc20f HIVE-11016 : MiniTez mergejoin test fails with Tez input error (issue in merge join under certain conditions) (Sergey Shelukhin, reviewed by Vikram Dixit K) + git clean -f -d Removing ql/src/test/queries/clientnegative/load_orc_negative_part.q Removing ql/src/test/queries/clientpositive/load_orc_part.q Removing ql/src/test/results/clientnegative/load_orc_negative_part.q.out Removing ql/src/test/results/clientpositive/load_orc_part.q.out + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 02e762f HIVE-11179: HIVE should allow custom converting from HivePrivilegeObjectDesc to privilegeObject for different authorizers(Dapeng Sun, Reviewed by Ferdinand Xu) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12744114 - PreCommit-HIVE-TRUNK-Build

        sseth now that you are back maybe you can review?

        sershe Sergey Shelukhin added a comment - sseth now that you are back maybe you can review?
        sseth Siddharth Seth added a comment -
        •  ObjectCacheFactory.removeLlapQueryCache(queryId == null ? savedQueryId : queryId); 

          This could end up being null if the query completes before a fragment starts executing. (possible when fragments are queued and the query is killed for example). Should either not be invoked or removeLlapQueryCache would need to check this (ConcurrentHashMap doesn't handle nulls).

        •  return getLlapObjectCache(queryId);

          Does this need to be wrapped in an ObjectCacheWrapper, to prefix the queryId correctly to the key ?

        • removeLlapQueryCache - Minor: if (isDebugEnabled) around the debug statement
        • makeKey in ObjectCacheWrapper. Does this end up changing the key from queryId__HASH_MAP to queryId_HASH_MAP. Not sure if this is significant

        Changing ExecutionContext is a little strange, but I see why it's required - since the queryId will only be available after parsing the payload. We'll likely be able to avoid this later once LLAP execution is a little more specialized, and parses the payloads up front.

        Rest looks good to me. Would prefer if vikram.dixit or hagleitn took a quick look at the changes to the cache and keys though.

        sseth Siddharth Seth added a comment - ObjectCacheFactory.removeLlapQueryCache(queryId == null ? savedQueryId : queryId); This could end up being null if the query completes before a fragment starts executing. (possible when fragments are queued and the query is killed for example). Should either not be invoked or removeLlapQueryCache would need to check this (ConcurrentHashMap doesn't handle nulls). return getLlapObjectCache(queryId); Does this need to be wrapped in an ObjectCacheWrapper, to prefix the queryId correctly to the key ? removeLlapQueryCache - Minor: if (isDebugEnabled) around the debug statement makeKey in ObjectCacheWrapper. Does this end up changing the key from queryId__HASH_MAP to queryId_HASH_MAP. Not sure if this is significant Changing ExecutionContext is a little strange, but I see why it's required - since the queryId will only be available after parsing the payload. We'll likely be able to avoid this later once LLAP execution is a little more specialized, and parses the payloads up front. Rest looks good to me. Would prefer if vikram.dixit or hagleitn took a quick look at the changes to the cache and keys though.

        ExecutionContext is the only hook I found into these things, if there was a better one I'd use that.
        Wrapper is not needed, LlapObjectCache handles its own queryId (the cache is by queryId).
        I don't think hash key change matters.
        Fixed the rest.

        sershe Sergey Shelukhin added a comment - ExecutionContext is the only hook I found into these things, if there was a better one I'd use that. Wrapper is not needed, LlapObjectCache handles its own queryId (the cache is by queryId). I don't think hash key change matters. Fixed the rest.

        The updated patch

        sershe Sergey Shelukhin added a comment - The updated patch
        sseth Siddharth Seth added a comment -

        Looks good to me.

        sseth Siddharth Seth added a comment - Looks good to me.
        hiveqa Hive QA added a comment -

        Overall: -1 no tests executed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12745510/HIVE-10535.02.patch

        Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4622/testReport
        Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4622/console
        Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4622/

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
        + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
        + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
        + cd /data/hive-ptest/working/
        + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4622/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ git = \s\v\n ]]
        + [[ git = \g\i\t ]]
        + [[ -z master ]]
        + [[ -d apache-github-source-source ]]
        + [[ ! -d apache-github-source-source/.git ]]
        + [[ ! -d apache-github-source-source ]]
        + cd apache-github-source-source
        + git fetch origin
        From https://github.com/apache/hive
           dc556bd..64d8582  branch-1   -> origin/branch-1
           4d984bd..240097b  master     -> origin/master
        + git reset --hard HEAD
        HEAD is now at 4d984bd HIVE-11223: CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan)
        + git clean -f -d
        + git checkout master
        Already on 'master'
        Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
        + git reset --hard origin/master
        HEAD is now at 240097b HIVE-11124. Move OrcRecordUpdater.getAcidEventFields to RecordReaderFactory.
        + git merge --ff-only origin/master
        Already up-to-date.
        + git gc
        + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hive-ptest/working/scratch/build.patch
        + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
        + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
        + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12745510 - PreCommit-HIVE-TRUNK-Build

        hiveqa Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12745510/HIVE-10535.02.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4622/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4622/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4622/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-4622/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin From https://github.com/apache/hive dc556bd..64d8582 branch-1 -> origin/branch-1 4d984bd..240097b master -> origin/master + git reset --hard HEAD HEAD is now at 4d984bd HIVE-11223: CBO (Calcite Return Path): MapJoin and SMBJoin conversion not triggered (Jesus Camacho Rodriguez, reviewed by Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 240097b HIVE-11124. Move OrcRecordUpdater.getAcidEventFields to RecordReaderFactory. + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12745510 - PreCommit-HIVE-TRUNK-Build

        committed to branch

        sershe Sergey Shelukhin added a comment - committed to branch

        People

          sershe Sergey Shelukhin
          sseth Siddharth Seth
          Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved: