Index: src/java/overview.html
===================================================================
--- src/java/overview.html (revision 932021)
+++ src/java/overview.html (working copy)
@@ -53,7 +53,7 @@
- - Java 1.6.x, preferably from Sun. Use the latest version available.
+ - Java 1.6.x, preferably from Sun. Use the latest version available except u18 (u19 is fine).
- This version of HBase will only run on Hadoop 0.20.x.
-
ssh must be installed and sshd must be running to use Hadoop's scripts to manage remote Hadoop daemons.
@@ -72,31 +72,13 @@
- Hosts must be able to resolve the fully-qualified domain name of the master.
-
- HBase currently is a file handle hog. The usual default of 1024 on *nix systems is insufficient
- if you are loading any significant amount of data into regionservers.
- See the FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?
- for how to up the limit. Also, as of 0.18.x Hadoop DataNodes have an upper-bound on the number of threads they will
- support (
dfs.datanode.max.xcievers). The default is 256 threads. Up this limit on your hadoop cluster.
-
- -
The clocks on cluster members should be in basic alignments. Some skew is tolerable but
wild skew could generate odd behaviors. Run NTP
on your cluster, or an equivalent.
-
- HBase servers put up 10 listeners for incoming connections by default.
- Up this number if you have a dataset of any substance by setting
hbase.regionserver.handler.count
- in your hbase-site.xml.
-
- -
This is the current list of patches we recommend you apply to your running Hadoop cluster:
+ -
+ HBase is a database, it uses a lot of files at the same time. The default ulimit -n of 1024 on *nix systems is insufficient.
+ Any significant amount of loading will lead you to
+ FAQ: Why do I see "java.io.IOException...(Too many open files)" in my logs?.
+ You will also notice errors like:
+
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException
+2010-04-06 03:04:37,542 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901
+
+ Do yourself a favor and change this to more than 10k using the FAQ.
+ Also, HDFS has an upper bound of files that it can serve at the same time, called xcievers (yes, this is misspelled). Again, before doing any loading,
+ make sure you configured Hadoop's conf/hdfs-site.xml with this:
+
+<property>
+ <name>dfs.datanode.max.xcievers</name>
+ <value>2047</value>
+</property>
+
+ See the background of this issue here: Problem: "xceiverCount 258 exceeds the limit of concurrent xcievers 256".
+ Failure to follow these instructions will result in data loss.
+
+