I have the following setup on my cluster:
<property>
<name>hadoop.job.history.location</name>
<value>/mapred/history1</value>
<description> If job tracker is static the history files are stored in this single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
</description></property>
<property>
<name>mapred.job.tracker.history.completed.location</name>
<value>/mapred/history2/done</value>
<description> The completed job history files are stored at this single well known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
</description>
</property>
i.e. both hadoop.job.history.location and mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job completes and it retires, I get the following exception:
However, /mapred/history1 is created and the history files are very much present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are never moved to mapred.job.tracker.history.completed.location. The jobhistory is never viewable thereafter.
I have the following setup on my cluster:
<property>
<name>hadoop.job.history.location</name>
<value>/mapred/history1</value>
<description> If job tracker is static the history files are stored in this single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
</description></property>
<property>
<name>mapred.job.tracker.history.completed.location</name>
<value>/mapred/history2/done</value>
<description> The completed job history files are stored at this single well known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
</description>
</property>
i.e. both hadoop.job.history.location and mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job completes and it retires, I get the following exception:
However, /mapred/history1 is created and the history files are very much present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are never moved to mapred.job.tracker.history.completed.location. The jobhistory is never viewable thereafter.