Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Duplicate
-
Affects Version/s: 0.8.0, 0.8.1, 0.9.0
-
Component/s: Configuration
-
Labels:None
Description
With Hive 0.8.x, HiveConf was changed to utilize the private, static member "confVarURL" which points to /tmp/hive-<user>-<tmp_number>.xml for job configuration settings.
During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job.
Issue Links
- is duplicated by
-
HIVE-3709
Stop storing default ConfVars in temp file
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
New patch, added comments for change as well.
Looks like the problem is still there after applying the patch:
hive> set system:java.io.tmpdir;
set system:java.io.tmpdir;
system:java.io.tmpdir=/var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/
hive> SHOW TABLES;
...
hive> !ls ${system:java.io.tmpdir};
...
hive-default-187929306310164734.xml
hive-default-433053476043176173.xml
hive-default-3093481938548070336.xml
hive-default-159172276671008947.xml
...
hive> !rm ${system:java.io.tmpdir}/hive-default*;
hive> SHOW TABLES;
Failed with exception java.io.IOException:java.lang.RuntimeException: java.io.FileNotFoundException: /var/folders/6d/bmvmcxh938q6htnxtyg9jk5r0000gn/T/hive-default-187929306310164734.xml (No such file or directory)
Time taken: 0.081 seconds
It's also worth pointing out that previously we created a single copy of hive-default that was used by all HiveConf instances, but with this patch in place we create a new hive-default file for each HiveConf instance. Roughly speaking this means that we end up creating at least one new hive-default file for each statement that is executed.
Good catch Carl – I believe when I tested my rm of the data files required confirmation and perhaps I did not say "yes."
Deeper digging the issue, will update when I have a clear option.
@Chris: Turns out that Kevin has been working on this same problem in HIVE-3709. He was able to get a little bit farther, but his solution seems to have some concurrency issues. If you have time it may be worth looking at his solution and seeing if you can spot threading problem.
Resolving this is a duplicate of HIVE-3709.
This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
Patch to remove static attached.