Hive
  1. Hive
  2. HIVE-3596

Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0, 0.8.1, 0.9.0
    • Fix Version/s: 0.8.1, 0.9.0, 0.10.0
    • Component/s: Configuration
    • Labels:
      None

      Description

      With Hive 0.8.x, HiveConf was changed to utilize the private, static member "confVarURL" which points to /tmp/hive-<user>-<tmp_number>.xml for job configuration settings.

      During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job.

      1. HIVE-3596.patch
        1 kB
        Chris McConnell

        Issue Links

          Activity

          Hide
          Chris McConnell added a comment -

          Patch to remove static attached.

          Show
          Chris McConnell added a comment - Patch to remove static attached.
          Hide
          Chris McConnell added a comment -

          New patch, added comments for change as well.

          Show
          Chris McConnell added a comment - New patch, added comments for change as well.
          Hide
          Chris McConnell added a comment -
          Show
          Chris McConnell added a comment - Review: https://reviews.facebook.net/D6093
          Hide
          Carl Steinbach added a comment -

          Looks like the problem is still there after applying the patch:

          hive> set system:java.io.tmpdir;
          set system:java.io.tmpdir;
          system:java.io.tmpdir=/var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/
          hive> SHOW TABLES;
          ...
          hive> !ls ${system:java.io.tmpdir};
          ...
          hive-default-187929306310164734.xml
          hive-default-433053476043176173.xml
          hive-default-3093481938548070336.xml
          hive-default-159172276671008947.xml
          ...
          hive> !rm ${system:java.io.tmpdir}/hive-default*;
          hive> SHOW TABLES;
          Failed with exception java.io.IOException:java.lang.RuntimeException: java.io.FileNotFoundException: /var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/hive-default-187929306310164734.xml (No such file or directory)
          Time taken: 0.081 seconds
          

          It's also worth pointing out that previously we created a single copy of hive-default that was used by all HiveConf instances, but with this patch in place we create a new hive-default file for each HiveConf instance. Roughly speaking this means that we end up creating at least one new hive-default file for each statement that is executed.

          Show
          Carl Steinbach added a comment - Looks like the problem is still there after applying the patch: hive> set system:java.io.tmpdir; set system:java.io.tmpdir; system:java.io.tmpdir=/var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/ hive> SHOW TABLES; ... hive> !ls ${system:java.io.tmpdir}; ... hive-default-187929306310164734.xml hive-default-433053476043176173.xml hive-default-3093481938548070336.xml hive-default-159172276671008947.xml ... hive> !rm ${system:java.io.tmpdir}/hive-default*; hive> SHOW TABLES; Failed with exception java.io.IOException:java.lang.RuntimeException: java.io.FileNotFoundException: /var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/hive-default-187929306310164734.xml (No such file or directory) Time taken: 0.081 seconds It's also worth pointing out that previously we created a single copy of hive-default that was used by all HiveConf instances, but with this patch in place we create a new hive-default file for each HiveConf instance. Roughly speaking this means that we end up creating at least one new hive-default file for each statement that is executed.
          Hide
          Chris McConnell added a comment -

          Good catch Carl – I believe when I tested my rm of the data files required confirmation and perhaps I did not say "yes."

          Deeper digging the issue, will update when I have a clear option.

          Show
          Chris McConnell added a comment - Good catch Carl – I believe when I tested my rm of the data files required confirmation and perhaps I did not say "yes." Deeper digging the issue, will update when I have a clear option.
          Hide
          Carl Steinbach added a comment -

          @Chris: Turns out that Kevin has been working on this same problem in HIVE-3709. He was able to get a little bit farther, but his solution seems to have some concurrency issues. If you have time it may be worth looking at his solution and seeing if you can spot threading problem.

          Show
          Carl Steinbach added a comment - @Chris: Turns out that Kevin has been working on this same problem in HIVE-3709 . He was able to get a little bit farther, but his solution seems to have some concurrency issues. If you have time it may be worth looking at his solution and seeing if you can spot threading problem.
          Hide
          Carl Steinbach added a comment -

          Resolving this is a duplicate of HIVE-3709.

          Show
          Carl Steinbach added a comment - Resolving this is a duplicate of HIVE-3709 .
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              Chris McConnell
              Reporter:
              Chris McConnell
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development