Index: src/docs/src/documentation/content/xdocs/loadstore.xml
===================================================================
--- src/docs/src/documentation/content/xdocs/loadstore.xml (revision 1378112)
+++ src/docs/src/documentation/content/xdocs/loadstore.xml (working copy)
@@ -27,34 +27,47 @@
The HCatLoader and HCatStorer interfaces are used with Pig scripts to read and write data in HCatalog managed tables. The HCatLoader and HCatStorer interfaces are used with Pig scripts to read
+and write data in HCatalog-managed tables. No HCatalog-specific setup is
+required for these interfaces. HCatLoader is used with Pig scripts to read data from HCatalog managed tables. HCatLoader is accessed via a Pig load statement. HCatLoader is used with Pig scripts to read data from HCatalog-managed tables. HCatLoader is accessed via a Pig load statement. Assumptions You must specify the table name in single quotes: LOAD 'tablename'. If you are using a non-default database you must specify your input as 'dbname.tablename'. If you are using Pig 0.9.2 or earlier, you must create your database and table prior to running the Pig script. Beginning with Pig 0.10 you can issue these create commands in Pig using the SQL command. The Hive metastore lets you create tables without specifying a database; if you
- created tables this way, then the database name is 'default' and is not required when
- specifying the table for HCatLoader. If the table is partitioned, you can indicate which partitions to scan by immediately following the load statement with a partition filter statement
- (see Examples). Assumptions You must specify the table name in single quotes: LOAD 'tablename'.
+If you are using a non-default database you must specify your input as
+'dbname.tablename'. If you are using Pig 0.9.2 or earlier, you must create
+your database and table prior to running the Pig script. Beginning with
+Pig 0.10 you can issue these create commands in Pig using the SQL command. The Hive metastore lets you create tables without specifying a database;
+if you created tables this way, then the database name is 'default' and is not
+required when specifying the table for HCatLoader. If the table is partitioned, you can indicate which partitions to scan by
+immediately following the load statement with a partition filter statement
+(see Load Examples below). Restrictions apply to the types of columns HCatLoader can read. HCatLoader can read only the data types listed in the table.
The table shows how Pig will interpret the HCatalog data type. Pig does not automatically pick up HCatalog jars. You will need tell Pig where your HCatalog jars are.
-These include the Hive jars used by the HCatalog client. To do this, you must define the environment
-variable PIG_CLASSPATH with the appropriate jars. HCat can tell you the jars it needs. In order to do this it
-needs to know where Hadoop is installed. Also, you need to tell Pig the URI for your metastore, in the PIG_OPTS
-variable. In the case where you have installed Hadoop and HCatalog via tar, you can do: Pig does not automatically pick up HCatalog jars. To bring in the necessary
+jars, you can either use a flag in the The -useHCatalog Flag To bring in the appropriate jars for working with HCatalog,
+simply include the following flag: Jars and Configuration Files For Pig commands that omit HCatalog can tell you the jars it needs. In order to do this
+it needs to know where Hadoop and Hive are installed.
+Also, you need to tell Pig the URI
+for your metastore, in the PIG_OPTS variable. In the case where you have installed Hadoop and HCatalog via tar,
+you can do this: Or you can pass the jars in your command line: The version number found in each filepath will be substituted for Authentication If you are using a secure cluster and a failure results in a message like "2010-11-03 16:17:28,225 WARN hive.metastore ... - Unable to connect metastore with URI thrift://..." in /tmp/<username>/hive.log, then make sure you have run "kinit <username>@FOO.COM" to get a Kerberos ticket and to be able to authenticate to the HCatalog server. If you are using a secure cluster and a failure results in a message
+like "2010-11-03 16:17:28,225 WARN hive.metastore ... - Unable to connect
+metastore with URI thrift://..." in /tmp/<username>/hive.log, then make
+sure you have run "kinit <username>@FOO.COM" to get a Kerberos ticket
+and to be able to authenticate to the HCatalog server. This load statement will load all partitions of the specified table. If only some partitions of the specified table are needed, include a partition filter statement immediately following the load statement in the data flow. (In the script, however, a filter statement might not immediately follow its load statement.) The filter statement can include conditions on partition as well as non-partition columns. If only some partitions of the specified table are needed, include a
+partition filter statement immediately following the
+load statement in the data flow. (In the script, however, a filter statement
+might not immediately follow its load statement.) The filter statement can
+include conditions on partition as well as non-partition columns. Notice that the schema is automatically provided to Pig; there's no need to
+declare name and age as fields, as if you were loading from a file. Notice that the schema is automatically provided to Pig, there's no need to declare name and age as fields, as if
-you were loading from a file. To scan a single partition of the table web_logs partitioned by the column
+datestamp, for example: To scan a single partition of the table web_logs, for example, partitioned by the column datestamp: Pig will split the above filter, pushing the datestamp portion to HCatalog and retaining the Pig will split the above filter, pushing the datestamp portion to HCatalog
+and retaining the ' Filter Operators HCatStorer is used with Pig scripts to write data to HCatalog managed tables. HCatStorer is accessed via a Pig store statement. HCatStorer is used with Pig scripts to write data to HCatalog-managed tables. HCatStorer is accessed via a Pig store statement. Assumptionspig command or set the
+environment variables PIG_CLASSPATH and PIG_OPTS as described below.-useHCatalog, you need to
+tell Pig where to find your HCatalog jars and the Hive jars used by the
+HCatalog client. To do this, you must define the environment variable
+PIG_CLASSPATH with the appropriate jars.*. For example, HCatalog release 0.4.0 uses these jars and conf files:
+
+
$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar$HIVE_HOME/lib/hive-metastore-0.9.0.jar$HIVE_HOME/lib/libthrift-0.7.0.jar$HIVE_HOME/lib/hive-exec-0.9.0.jar$HIVE_HOME/lib/libfb303-0.7.0.jar$HIVE_HOME/lib/jdo2-api-2.3-ec.jar$HIVE_HOME/conf$HADOOP_HOME/conf$HIVE_HOME/lib/slf4j-api-1.6.1.jar
-
@@ -138,9 +202,8 @@
-
+
-
+
+ user is not null part
-to apply itself. You can also give a more complex filter to retrieve a set of partitions.user is not null' part to apply itself.
+You can also give a more complex filter to retrieve a set of partitions.
For the USING clause, you can have a string argument that represents key/value pairs for partition. This is a mandatory argument when you are writing to a partitioned table -and the partition column is not in the output column. The values for partition keys -should NOT be quoted.
+and the partition column is not in the output column. The values for partition keys +should NOT be quoted.If partition columns are present in data they need not be specified as a STORE argument. Instead HCatalog will use these values to place records in the appropriate partition(s). It is valid to specify some partition keys in the STORE statement and have other partition keys in the data.
You can write to non-partitioned table simply by using HCatStorer. The contents of the table will be overwritten:
+You can write to a non-partitioned table simply by using HCatStorer. The contents of the table will be overwritten:
-To add one new partition to a partitioned table, specify the partition value in store function. Pay careful
+
To add one new partition to a partitioned table, specify the partition value in the store function. Pay careful attention to the quoting, as the whole string must be single quoted and separated with an equals sign:
To write into multiple partitions at one, make sure that the partition column is present in your data, then call +
To write into multiple partitions at once, make sure that the partition column is present in your data, then call HCatStorer with no argument:
Restrictions apply to the types of columns HCatStorer can write.
+Restrictions apply to the types of columns HCatStorer can write.
HCatStorer can write only the data types listed in the table. The table shows how Pig will interpret the HCatalog data type.