Index: contrib/templeton/LICENSE.txt =================================================================== --- contrib/templeton/LICENSE.txt (revision 0) +++ contrib/templeton/LICENSE.txt (revision 0) @@ -0,0 +1,398 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +APACHE HCATALOG SUBCOMPONENTS: + +The Apache HCatalog project contains subcomponents with separate copyright +notices and license terms. Your use of the source code for the these +subcomponents is subject to the terms and conditions of the following +licenses. + +For the ANTLR libraries: + +Copyright (c) 2003-2008, Terence Parr +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. Redistributions +in binary form must reproduce the above copyright notice, this list of +conditions and the following disclaimer in the documentation and/or +other materials provided with the distribution. Neither the name of +the author nor the names of its contributors may be used to endorse or +promote products derived from this software without specific prior +written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +For the StringTemplate library: + +Copyright (c) 2008, Terence Parr +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +Redistributions of source code must retain the above copyright notice, +this list of conditions and the following disclaimer. Redistributions +in binary form must reproduce the above copyright notice, this list of +conditions and the following disclaimer in the documentation and/or +other materials provided with the distribution. Neither the name of +the author nor the names of its contributors may be used to endorse or +promote products derived from this software without specific prior +written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +For the ASM library: + +Copyright (c) 2000-2005 INRIA, France Telecom +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: + +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +3. Neither the name of the copyright holders nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +THE POSSIBILITY OF SUCH DAMAGE. + + +For the org.json library: + +Copyright (c) 2002 JSON.org + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +The Software shall be used for Good, not Evil. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +For the JLine library: + +Copyright (c) 2002-2006, Marc Prud'hommeaux +All rights reserved. + +Redistribution and use in source and binary forms, with or +without modification, are permitted provided that the following +conditions are met: + +Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + +Redistributions in binary form must reproduce the above copyright +notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with +the distribution. + +Neither the name of JLine nor the names of its contributors +may be used to endorse or promote products derived from this +software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, +BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO +EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, +OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED +AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING +IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED +OF THE POSSIBILITY OF SUCH DAMAGE. + + +For the SLF4J library: + +Copyright (c) 2004-2008 QOS.ch +All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +For the DataNucleus library: + +DataNucleus is licensed under the Apache 2 license, as given above. + +For the Guava library: + +Guava is licensed under the Apache 2 license, as given above. Index: contrib/templeton/.gitignore =================================================================== --- contrib/templeton/.gitignore (revision 0) +++ contrib/templeton/.gitignore (revision 0) @@ -0,0 +1,5 @@ +/TAGS +/build +/foo.log +/ivy/ivy-2.2.0.jar +/templeton-site.xml Index: contrib/templeton/ivy.xml =================================================================== --- contrib/templeton/ivy.xml (revision 0) +++ contrib/templeton/ivy.xml (revision 0) @@ -0,0 +1,139 @@ + + + + + + + + Apache Templeton + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/ivy/ivysettings.xml =================================================================== --- contrib/templeton/ivy/ivysettings.xml (revision 0) +++ contrib/templeton/ivy/ivysettings.xml (revision 0) @@ -0,0 +1,61 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/ivy/libraries.properties =================================================================== --- contrib/templeton/ivy/libraries.properties (revision 0) +++ contrib/templeton/ivy/libraries.properties (revision 0) @@ -0,0 +1,17 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#This properties file lists the versions of the various artifacts used by hadoop and components. +#It drives ivy and the generation of a maven POM + +junit.version=4.0 +ivy.version=2.2.0 Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerCallbackTest.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerCallbackTest.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerCallbackTest.java (revision 0) @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test; + +import static org.junit.Assert.*; + +import org.apache.hcatalog.templeton.CallbackFailedException; +import org.apache.hcatalog.templeton.ServerCallback; +import org.junit.Test; + +public class ServerCallbackTest { + + @Test + public void test() { + ServerCallback.registerCallback("fred", "http://apache.org"); + ServerCallback.registerCallback("george", "https://www.google.com"); + ServerCallback.registerCallback("joe", "http://localhost:2345"); + ServerCallback.registerCallback("wilma", "http://localhost:2345/foo/bar"); + ServerCallback.registerCallback("betty", "http://localhost:2345/foo/bar?a=something&b=somethingelse"); + try { + ServerCallback.doCallback("fred"); + assertTrue(true); // We made it. + } catch (CallbackFailedException e) { + fail("Callback failed: " + e.getMessage()); + } + try { + ServerCallback.doCallback("george"); + assertTrue(true); // We made it. + } catch (CallbackFailedException e) { + fail("Callback failed: " + e.getMessage()); + } + try { + ServerCallback.doCallback("joe"); + fail("Didn't call server."); + } catch (CallbackFailedException e) { + assertTrue(e.getMessage().indexOf("http://localhost:2345?jobid=joe") > -1); + } + try { + ServerCallback.doCallback("wilma"); + fail("Didn't call server."); + } catch (CallbackFailedException e) { + assertTrue(e.getMessage().indexOf("http://localhost:2345/foo/bar?jobid=wilma") > -1); + } + try { + ServerCallback.doCallback("betty"); + fail("Didn't call server."); + } catch (CallbackFailedException e) { + assertTrue(e.getMessage().indexOf + ("http://localhost:2345/foo/bar?a=something&b=somethingelse&jobid=betty") > -1); + } + } + +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerTest.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerTest.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/ServerTest.java (revision 0) @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test; + +import java.io.IOException; + +import junit.framework.TestCase; + +import org.apache.hcatalog.templeton.ExecBean; +import org.apache.hcatalog.templeton.SimpleWebException; +import org.apache.hcatalog.templeton.test.mock.MockUriInfo; +import org.apache.hcatalog.templeton.test.mock.MockServer; + +/* + * Test that the server code exists. + */ +public class ServerTest extends TestCase { + + MockServer server; + + public void setUp() { + server = new MockServer(); + } + + public void testServer() { + assertNotNull(server); + } + + public void testStatus() { + assertTrue(server.status().indexOf("\"ok\"") > -1); + } + + public void testDDL() { + try { + validateDDL(server, "show tables", "admin", "777"); + validateDDL(server, "", "", ""); + validateDDL(server, "show tables", "", ""); + validateDDL(server, "show tables", "admin", ""); + } catch (SimpleWebException swe) { + swe.printStackTrace(); + fail("ddl execution caused a failure"); + } catch (Exception e) { + e.printStackTrace(); + fail("ddl execution caused a failure"); + } + } + + // More tests coming later. + + public void validateDDL(MockServer server, String command, String group, String permissions) + throws SimpleWebException, IOException { + ExecBean bean = server.ddl(command, group, permissions, new MockUriInfo()); + assertTrue(bean.stdout.endsWith("bin/hcat")); + String tmp = bean.stderr.substring(bean.stderr.indexOf("[") + 1, bean.stderr.indexOf("]")); + String[] parts = tmp.split(","); + assertTrue(parts[0].trim().equals("-e")); + assertTrue(parts[1].trim().equals(command)); + assertTrue(parts[2].trim().equals("-g")); + assertTrue(parts[3].trim().equals(group)); + assertTrue(parts[4].trim().equals("-p")); + assertTrue(parts[5].trim().equals(permissions)); + } +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockExecService.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockExecService.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockExecService.java (revision 0) @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test.mock; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import org.apache.commons.exec.ExecuteException; +import org.apache.hcatalog.templeton.ExecBean; +import org.apache.hcatalog.templeton.ExecService; +import org.apache.hcatalog.templeton.NotAuthorizedException; + +public class MockExecService implements ExecService { + + public ExecBean run(String user, String program, List args, + Map env) { + ExecBean bean = new ExecBean(); + bean.stdout = program; + bean.stderr = args.toString(); + return bean; + } + + @Override + public ExecBean runUnlimited(String user, String program, + List args, Map env) + throws NotAuthorizedException, ExecuteException, IOException { + ExecBean bean = new ExecBean(); + bean.stdout = program; + bean.stderr = args.toString(); + return null; + } +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockServer.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockServer.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockServer.java (revision 0) @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test.mock; + +import org.apache.hcatalog.templeton.Server; + +/* + * Test that the server code exists. + */ +public class MockServer extends Server { + + public MockServer() { + execService = new MockExecService(); + } + @Override + public String getUser() { + return System.getenv("USER"); + } +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockUriInfo.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockUriInfo.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/mock/MockUriInfo.java (revision 0) @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test.mock; + +import java.net.URI; +import java.net.URISyntaxException; +import java.util.List; + +import javax.ws.rs.core.MultivaluedMap; +import javax.ws.rs.core.PathSegment; +import javax.ws.rs.core.UriBuilder; +import javax.ws.rs.core.UriInfo; + +public class MockUriInfo implements UriInfo{ + + @Override + public URI getAbsolutePath() { + // TODO Auto-generated method stub + return null; + } + + @Override + public UriBuilder getAbsolutePathBuilder() { + // TODO Auto-generated method stub + return null; + } + + @Override + public URI getBaseUri() { + try { + return new URI("http://fakeuri/templeton"); + } catch (URISyntaxException e) { + e.printStackTrace(); + } + return null; + } + + @Override + public UriBuilder getBaseUriBuilder() { + // TODO Auto-generated method stub + return null; + } + + @Override + public List getMatchedResources() { + // TODO Auto-generated method stub + return null; + } + + @Override + public List getMatchedURIs() { + // TODO Auto-generated method stub + return null; + } + + @Override + public List getMatchedURIs(boolean arg0) { + // TODO Auto-generated method stub + return null; + } + + @Override + public String getPath() { + // TODO Auto-generated method stub + return null; + } + + @Override + public String getPath(boolean arg0) { + // TODO Auto-generated method stub + return null; + } + + @Override + public MultivaluedMap getPathParameters() { + // TODO Auto-generated method stub + return null; + } + + @Override + public MultivaluedMap getPathParameters(boolean arg0) { + // TODO Auto-generated method stub + return null; + } + + @Override + public List getPathSegments() { + // TODO Auto-generated method stub + return null; + } + + @Override + public List getPathSegments(boolean arg0) { + // TODO Auto-generated method stub + return null; + } + + @Override + public MultivaluedMap getQueryParameters() { + // TODO Auto-generated method stub + return null; + } + + @Override + public MultivaluedMap getQueryParameters(boolean arg0) { + // TODO Auto-generated method stub + return null; + } + + @Override + public URI getRequestUri() { + // TODO Auto-generated method stub + return null; + } + + @Override + public UriBuilder getRequestUriBuilder() { + // TODO Auto-generated method stub + return null; + } + +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TrivialExecServiceTest.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TrivialExecServiceTest.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TrivialExecServiceTest.java (revision 0) @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test.tool; + +import static org.junit.Assert.*; + +import java.io.BufferedReader; +import java.io.InputStreamReader; +import java.util.ArrayList; + +import org.apache.hcatalog.templeton.tool.TrivialExecService; +import org.junit.Test; + +public class TrivialExecServiceTest { + + @Test + public void test() { + ArrayList list = new ArrayList(); + list.add("echo"); + list.add("success"); + BufferedReader out = null; + BufferedReader err = null; + try { + Process process = TrivialExecService.getInstance().run(list); + out = new BufferedReader(new InputStreamReader( + process.getInputStream())); + err = new BufferedReader(new InputStreamReader( + process.getErrorStream())); + assertEquals("success", out.readLine()); + out.close(); + String line; + while ((line = err.readLine()) != null) { + fail(line); + } + process.waitFor(); + } catch (Exception e) { + e.printStackTrace(); + fail("Process caused exception."); + } finally { + try { + out.close(); + } catch (Exception ex) { + // Whatever. + } + try { + err.close(); + } catch (Exception ex) { + // Whatever + } + } + } + +} Index: contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TempletonUtilsTest.java =================================================================== --- contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TempletonUtilsTest.java (revision 0) +++ contrib/templeton/src/test/org/apache/hcatalog/templeton/test/tool/TempletonUtilsTest.java (revision 0) @@ -0,0 +1,257 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.test.tool; + +import static org.junit.Assert.*; + +import java.io.FileNotFoundException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.util.StringUtils; +import org.apache.hcatalog.templeton.tool.TempletonUtils; +import org.junit.Test; + +public class TempletonUtilsTest { + + @Test + public void testIssetString() { + assertFalse(TempletonUtils.isset((String)null)); + assertFalse(TempletonUtils.isset("")); + assertTrue(TempletonUtils.isset("hello")); + } + + @Test + public void testIssetTArray() { + assertFalse(TempletonUtils.isset((Long[]) null)); + assertFalse(TempletonUtils.isset(new String[0])); + String[] parts = new String("hello.world").split("\\."); + assertTrue(TempletonUtils.isset(parts)); + } + + @Test + public void testPrintTaggedJobID() { + //JobID job = new JobID(); + // TODO -- capture System.out? + } + + @Test + public void testExtractJobId() { + assertEquals(null, TempletonUtils.extractJobId("fred")); + assertEquals("fred", TempletonUtils.extractJobId + (TempletonUtils.JOB_ID_TAG + "fred")); + } + + @Test + public void testEncodeCliArg() { + assertEquals("*", TempletonUtils.encodeCliArg(null)); + assertEquals("*fred", TempletonUtils.encodeCliArg("fred")); + } + + @Test + public void testDecodeCliArg() { + assertEquals(null, TempletonUtils.decodeCliArg(null)); + assertEquals("", TempletonUtils.decodeCliArg("")); + assertEquals("fred", TempletonUtils.decodeCliArg( + TempletonUtils.encodeCliArg("fred"))); + } + + @Test + public void testEncodeArray() { + assertEquals(null, TempletonUtils.encodeArray(null)); + String[] tmp = new String[0]; + assertTrue(TempletonUtils.encodeArray(new String[0]).length() == 0); + tmp = new String[3]; + tmp[0] = "fred"; + tmp[1] = null; + tmp[2] = "peter,lisa,, barney"; + assertEquals("fred,,peter" + + StringUtils.ESCAPE_CHAR + ",lisa" + StringUtils.ESCAPE_CHAR + "," + + StringUtils.ESCAPE_CHAR + ", barney", TempletonUtils.encodeArray(tmp)); + } + + @Test + public void testDecodeArray() { + assertTrue(TempletonUtils.encodeArray(null) == null); + String[] tmp = new String[3]; + tmp[0] = "fred"; + tmp[1] = null; + tmp[2] = "peter,lisa,, barney"; + String[] tmp2 = TempletonUtils.decodeArray(TempletonUtils.encodeArray(tmp)); + try { + for (int i=0; i< tmp.length; i++) { + assertEquals((String) tmp[i], (String)tmp2[i]); + } + } catch (Exception e) { + fail("Arrays were not equal" + e.getMessage()); + } + } + + @Test + public void testEncodeCliArrayStringArray() { + assertEquals("*", TempletonUtils.encodeCliArray((String[]) null)); + // These have both been tested earlier in the test suite, so just + // test that this does what it says it does. + String[] tmp = new String[3]; + tmp[0] = "fred"; + tmp[1] = null; + tmp[2] = "peter,lisa,, barney"; + assertEquals(TempletonUtils.encodeCliArg(TempletonUtils.encodeArray(tmp)), + TempletonUtils.encodeCliArray(tmp)); + } + + @Test + public void testEncodeCliArrayListOfString() { + assertEquals(null, TempletonUtils.encodeCliArray((List) null)); + ArrayList tmplist = new ArrayList(); + tmplist.add("fred"); + tmplist.add(null); + tmplist.add("peter,lisa,, barney"); + String[] tmp = new String[3]; + tmp[0] = "fred"; + tmp[1] = null; + tmp[2] = "peter,lisa,, barney"; + assertEquals(TempletonUtils.encodeCliArray(tmp), + TempletonUtils.encodeCliArray(tmplist)); + + } + + @Test + public void testEncodeCliArrayString() { + assertEquals(null, TempletonUtils.encodeCliArray((String) null)); + assertEquals("*fred" + StringUtils.ESCAPE_CHAR + ",joe", + TempletonUtils.encodeCliArray("fred,joe")); + } + + @Test + public void testDecodeCliArray() { + String[] tmp = new String[3]; + tmp[0] = "fred"; + tmp[1] = null; + tmp[2] = "peter,lisa,, barney"; + String[] tmp2 = TempletonUtils.decodeCliArray(TempletonUtils.encodeCliArray(tmp)); + try { + for (int i=0; i() {{ put("exec", bean); }}); + } + +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/JarDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/JarDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/JarDelegator.java (revision 0) @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.List; +import org.apache.commons.exec.ExecuteException; +import org.apache.hcatalog.templeton.tool.TempletonUtils; +import org.apache.hcatalog.templeton.tool.TempletonControllerJob; + +/** + * Submit a job to the MapReduce queue. We do this by running the + * hadoop executable on the local server using the ExecService. + * This allows us to easily verify that the user identity is being + * securely used. + * + * This is the backend of the mapreduce/jar web service. + */ +public class JarDelegator extends TempletonDelegator { + public static final String JAR_CLASS = TempletonControllerJob.class.getName(); + + public JarDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public EnqueueBean run(String user, String jar, String mainClass, + String libjars, String files, + List jarArgs, List defines, + String statusdir) + throws NotAuthorizedException, BadParam, BusyException, QueueException, + ExecuteException, IOException + { + List args = makeArgs(jar, mainClass, + libjars, files, jarArgs, defines, + statusdir); + + ExecBean exec = execService.run(user, appConf.clusterHadoop(), args, null); + if (exec.exitcode != 0) + throw new QueueException("invalid exit code", exec); + String id = TempletonUtils.extractJobId(exec.stdout); + if (id == null) + throw new QueueException("Unable to get job id", exec); + + return new EnqueueBean(id, exec); + } + + private List makeArgs(String jar, String mainClass, + String libjars, String files, + List jarArgs, List defines, + String statusdir) + throws BadParam, IOException + { + ArrayList args = new ArrayList(); + try { + args.add("jar"); + args.add(appConf.templetonJar()); + args.add(JAR_CLASS); + String fullJar = TempletonUtils.hadoopFsFilename(jar, appConf); + args.add(TempletonUtils.encodeCliArray(fullJar)); + args.add(TempletonUtils.encodeCliArg(statusdir)); + args.add("--"); + args.add(appConf.clusterHadoop()); + args.add("jar"); + args.add(TempletonUtils.hadoopFsPath(jar, appConf).getName()); + if (TempletonUtils.isset(mainClass)) + args.add(mainClass); + if (TempletonUtils.isset(libjars)) { + args.add("-libjars"); + args.add(TempletonUtils.hadoopFsListAsString(libjars, appConf)); + } + if (TempletonUtils.isset(files)) { + args.add("-files"); + args.add(TempletonUtils.hadoopFsListAsString(files, appConf)); + } + + for (String d : defines) + args.add("-D" + d); + + args.addAll(jarArgs); + } catch (FileNotFoundException e) { + throw new BadParam(e.getMessage()); + } catch (URISyntaxException e) { + throw new BadParam(e.getMessage()); + } + + return args; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecService.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecService.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecService.java (revision 0) @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import org.apache.commons.exec.ExecuteException; + +public interface ExecService { + + public ExecBean run(String user, String program, List args, + Map env) + throws NotAuthorizedException, BusyException, ExecuteException, IOException; + + public ExecBean runUnlimited(String user, String program, List args, + Map env) + throws NotAuthorizedException, ExecuteException, IOException; +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/MaxByteArrayOutputStream.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/MaxByteArrayOutputStream.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/MaxByteArrayOutputStream.java (revision 0) @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.ByteArrayOutputStream; + +/** + * An output stream that will only accept the first N bytes of data. + */ +public class MaxByteArrayOutputStream extends ByteArrayOutputStream { + /** + * The max number of bytes stored. + */ + private int maxBytes; + + /** + * The number of bytes currently stored. + */ + private int nBytes; + + /** + * Create. + */ + public MaxByteArrayOutputStream(int maxBytes) { + this.maxBytes = maxBytes; + nBytes = 0; + } + + /** + * Writes the specified byte to this byte array output stream. + * Any bytes after the first maxBytes will be ignored. + * + * @param b the byte to be written. + */ + public synchronized void write(int b) { + if (nBytes < maxBytes) { + ++nBytes; + super.write(b); + } + } + + /** + * Writes len bytes from the specified byte array + * starting at offset off to this byte array output stream. + * Any bytes after the first maxBytes will be ignored. + * + * @param b the data. + * @param off the start offset in the data. + * @param len the number of bytes to write. + */ + public synchronized void write(byte b[], int off, int len) { + int storable = Math.min(maxBytes - nBytes, len); + if (storable > 0) { + nBytes += storable; + super.write(b, off, storable); + } + } + + +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleWebException.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleWebException.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleWebException.java (revision 0) @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import java.util.Map; +import java.util.HashMap; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.Response.ResponseBuilder; +import javax.ws.rs.core.Response; +import org.codehaus.jackson.map.ObjectMapper; + +/** + * Simple exception that will return a json error payload if thrown + * from a JAX web server. We skip using WebApplicationException and + * instead map our own so that Jersey doesn't log our exceptions as + * error in the output log. See SimpleExceptionMapper. + */ +public class SimpleWebException extends Throwable { + public int httpCode; + public Map params; + + public SimpleWebException(int httpCode, String msg) { + super(msg); + this.httpCode = httpCode; + } + + public SimpleWebException(int httpCode, String msg, Map params) { + super(msg); + this.httpCode = httpCode; + this.params = params; + } + + public Response getResponse() { + return buildMessage(httpCode, params, getMessage()); + } + + public static Response buildMessage(int httpCode, Map params, + String msg) + { + HashMap err = new HashMap(); + err.put("error", msg); + if (params != null) + err.putAll(params); + + String json = "\"error\""; + try { + json = new ObjectMapper().writeValueAsString(err); + } catch (IOException e) { + } + + return Response.status(httpCode) + .entity(json) + .type(MediaType.APPLICATION_JSON) + .build(); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/Server.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/Server.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/Server.java (revision 0) @@ -0,0 +1,264 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import java.util.List; +import javax.ws.rs.Consumes; +import javax.ws.rs.DELETE; +import javax.ws.rs.FormParam; +import javax.ws.rs.GET; +import javax.ws.rs.POST; +import javax.ws.rs.Path; +import javax.ws.rs.PathParam; +import javax.ws.rs.Produces; +import javax.ws.rs.QueryParam; +import javax.ws.rs.core.Context; +import javax.ws.rs.core.MediaType; +import javax.ws.rs.core.SecurityContext; +import javax.ws.rs.core.UriInfo; + +import org.apache.commons.exec.ExecuteException; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.hdfs.web.AuthFilter; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.authentication.client.PseudoAuthenticator; + +/** + * The Templeton Web API server. + */ +@Path("/v1") +public class Server { + public static final String STATUS_MSG + = "{\"status\": \"ok\", \"version\": \"v1\"}\n"; + + protected static ExecService execService = ExecServiceImpl.getInstance(); + private static AppConfig appConf = AppConfig.getInstance(); + + private @Context SecurityContext theSecurityContext; + + private static final Log LOG = LogFactory.getLog(Server.class); + + /** + * Check the status of this server. + */ + @GET + @Path("status.json") + @Produces({MediaType.APPLICATION_JSON}) + public String status() { + return STATUS_MSG; + } + + /** + * Execute an hcat ddl expression on the local box. It is run + * as the authenticated user and rate limited. + */ + @POST + @Path("ddl.json") + @Produces({MediaType.APPLICATION_JSON}) + public ExecBean ddl(@FormParam("exec") String exec, + @FormParam("group") String group, + @FormParam("permissions") String permissions, + @Context UriInfo info) + throws NotAuthorizedException, BusyException, BadParam, + ExecuteException, IOException + { + verifyUser(); + verifyParam(exec, "exec"); + // The calling URI to hit with a callback is info.getBaseUri() + LOG.info("Calling url: " + info.getBaseUri()); + + HcatDelegator d = new HcatDelegator(appConf, execService); + return d.run(getUser(), exec, group, permissions); + } + + /** + * Run a MapReduce Streaming job. + */ + @POST + @Path("mapreduce/streaming.json") + @Produces({MediaType.APPLICATION_JSON}) + public EnqueueBean mapReduceStreaming(@FormParam("input") List inputs, + @FormParam("output") String output, + @FormParam("mapper") String mapper, + @FormParam("reducer") String reducer, + @FormParam("file") List files, + @FormParam("define") List defines, + @FormParam("cmdenv") List cmdenvs, + @FormParam("arg") List args) + throws NotAuthorizedException, BusyException, BadParam, QueueException, + ExecuteException, IOException + { + verifyUser(); + verifyParam(inputs, "input"); + verifyParam(mapper, "mapper"); + verifyParam(reducer, "reducer"); + + StreamingDelegator d = new StreamingDelegator(appConf, execService); + return d.run(getUser(), inputs, output, mapper, reducer, + files, defines, cmdenvs, args); + } + + /** + * Run a MapReduce Jar job. + */ + @POST + @Path("mapreduce/jar.json") + @Produces({MediaType.APPLICATION_JSON}) + public EnqueueBean mapReduceJar(@FormParam("jar") String jar, + @FormParam("class") String mainClass, + @FormParam("libjars") String libjars, + @FormParam("files") String files, + @FormParam("arg") List args, + @FormParam("define") List defines, + @FormParam("statusdir") String statusdir) + throws NotAuthorizedException, BusyException, BadParam, QueueException, + ExecuteException, IOException + { + verifyUser(); + verifyParam(jar, "jar"); + verifyParam(mainClass, "class"); + + JarDelegator d = new JarDelegator(appConf, execService); + return d.run(getUser(), + jar, mainClass, + libjars, files, args, + defines, statusdir); + } + + /** + * Run a Pig job. + */ + @POST + @Path("pig.json") + @Produces({MediaType.APPLICATION_JSON}) + public EnqueueBean pig(@FormParam("execute") String execute, + @FormParam("file") String srcFile, + @FormParam("arg") List pigArgs, + @FormParam("files") String otherFiles, + @FormParam("statusdir") String statusdir) + throws NotAuthorizedException, BusyException, BadParam, QueueException, + ExecuteException, IOException + { + verifyUser(); + if (execute == null && srcFile == null) + throw new BadParam("Either execute or file parameter required"); + + PigDelegator d = new PigDelegator(appConf, execService); + return d.run(getUser(), + execute, srcFile, + pigArgs, otherFiles, + statusdir); + } + + /** + * Run a Hive job. + */ + @POST + @Path("hive.json") + @Produces({MediaType.APPLICATION_JSON}) + public EnqueueBean hive(@FormParam("execute") String execute, + @FormParam("file") String srcFile, + @FormParam("define") List defines, + @FormParam("statusdir") String statusdir) + throws NotAuthorizedException, BusyException, BadParam, QueueException, + ExecuteException, IOException + { + verifyUser(); + if (execute == null && srcFile == null) + throw new BadParam("Either execute or file parameter required"); + + HiveDelegator d = new HiveDelegator(appConf, execService); + return d.run(getUser(), execute, srcFile, defines, statusdir); + } + + /** + * Return the status of the jobid. + */ + @GET + @Path("queue/{jobid}.json") + @Produces({MediaType.APPLICATION_JSON}) + public QueueStatusBean showQueueId(@PathParam("jobid") String jobid) + throws NotAuthorizedException, BadParam, IOException + { + verifyUser(); + verifyParam(jobid, ":jobid"); + + StatusDelegator d = new StatusDelegator(appConf, execService); + return d.run(getUser(), jobid); + } + + /** + * Kill a job in the queue. + */ + @DELETE + @Path("queue/{jobid}.json") + @Produces({MediaType.APPLICATION_JSON}) + public QueueStatusBean deleteQueueId(@PathParam("jobid") String jobid) + throws NotAuthorizedException, BadParam, IOException + { + verifyUser(); + verifyParam(jobid, ":jobid"); + + DeleteDelegator d = new DeleteDelegator(appConf, execService); + return d.run(getUser(), jobid); + } + + /** + * Verify that we have a valid user. Throw an exception if invalid. + */ + public void verifyUser() + throws NotAuthorizedException + { + if (getUser() == null) { + String msg = "No user found."; + if (! UserGroupInformation.isSecurityEnabled()) + msg += " Missing " + PseudoAuthenticator.USER_NAME + " parameter."; + throw new NotAuthorizedException(msg); + } + } + + /** + * Verify that the parameter exists. Throw an exception if invalid. + */ + public void verifyParam(String param, String name) + throws BadParam + { + if (param == null) + throw new BadParam("Missing " + name + " parameter"); + } + + /** + * Verify that the parameter exists. Throw an exception if invalid. + */ + public void verifyParam(List param, String name) + throws BadParam + { + if (param == null || param.isEmpty()) + throw new BadParam("Missing " + name + " parameter"); + } + + public String getUser() { + if (theSecurityContext == null) + return null; + if (theSecurityContext.getUserPrincipal() == null) + return null; + return theSecurityContext.getUserPrincipal().getName(); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/AppConfig.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/AppConfig.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/AppConfig.java (revision 0) @@ -0,0 +1,203 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.File; +import java.net.URL; +import java.util.Map; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.util.VersionInfo; +import java.util.logging.Level; +import java.util.logging.Logger; +import com.sun.jersey.spi.container.servlet.WebComponent; + +/** + * The configuration for Templeton. This merges the normal Hadoop + * configuration with the Templeton specific variables. + * + * The Templeton configuration variables are described in + * templeton-default.xml + * + * The Templeton specific configuration is split into three layers + * + * 1. templeton-default.xml - All the configuration variables that + * Templeton needs. These are the defaults that ship with the app + * and should only be changed be the app developers. + * + * 2. templeton-dist.xml - The (possibly empty) configuration that can + * set variables for a particular distribution, such as an RPM + * file. + * + * 3. templeton-site.xml - The (possibly empty) configuration that the + * system administrator can set variables for their Hadoop cluster. + * + * The configuration files are loaded in this order with later files + * overriding earlier ones. + * + * To find the configuration files, we first attempt to load a file + * from the CLASSPATH and then look in the directory specified in the + * TEMPLETON_HOME environment variable. + * + * In addition the configuration files may access the special env + * variable env for all environment variables. For example, the + * hadoop executable could be specified using: + *
+ *      ${env.HADOOP_PREFIX}/bin/hadoop
+ *
+ */ +public class AppConfig extends Configuration { + public static final String[] HADOOP_CONF_FILENAMES = { + "core-default.xml", "core-site.xml", "mapred-default.xml", "mapred-site.xml" + }; + + public static final String[] HADOOP_PREFIX_VARS = { + "HADOOP_PREFIX", "HADOOP_HOME" + }; + + public static final String TEMPLETON_HOME_VAR = "TEMPLETON_HOME"; + + public static final String[] TEMPLETON_CONF_FILENAMES = { + "templeton-default.xml", + "templeton-dist.xml", + "templeton-site.xml" + }; + + public static final String EXEC_ENCODING_NAME = "templeton.exec.encoding"; + public static final String EXEC_ENVS_NAME = "templeton.exec.envs"; + public static final String EXEC_MAX_BYTES_NAME = "templeton.exec.max-output-bytes"; + public static final String EXEC_MAX_PROCS_NAME = "templeton.exec.max-procs"; + public static final String EXEC_TIMEOUT_NAME = "templeton.exec.timeout"; + public static final String HADOOP_NAME = "templeton.hadoop"; + public static final String HCAT_NAME = "templeton.hcat"; + public static final String HIVE_ARCHIVE_NAME = "templeton.hive.archive"; + public static final String HIVE_PATH_NAME = "templeton.hive.path"; + public static final String HIVE_PROPS_NAME = "templeton.hive.properties"; + public static final String PIG_ARCHIVE_NAME = "templeton.pig.archive"; + public static final String PIG_PATH_NAME = "templeton.pig.path"; + public static final String STREAMING_JAR_NAME = "templeton.streaming.jar"; + public static final String SUDO_NAME = "templeton.sudo"; + public static final String TEMPLETON_JAR_NAME = "templeton.jar"; + public static final String ZK_HOSTS = "templeton.zookeeper.hosts"; + public static final String ZK_SESSION_TIMEOUT + = "templeton.zookeeper.session-timeout"; + + private static final Log LOG = LogFactory.getLog(AppConfig.class); + + private static volatile AppConfig theSingleton; + + /** + * Retrieve the singleton. + */ + public static synchronized AppConfig getInstance() { + if (theSingleton == null) + theSingleton = new AppConfig(); + return theSingleton; + } + + public AppConfig() { + init(); + LOG.info("Using Hadoop version " + VersionInfo.getVersion()); + } + + private void init() { + for (Map.Entry e : System.getenv().entrySet()) + set("env." + e.getKey(), e.getValue()); + + String hadoopConfDir = getHadoopConfDir(); + for (String fname : HADOOP_CONF_FILENAMES) + loadOneFileConfig(hadoopConfDir, fname); + + String templetonDir = getTempletonDir(); + for (String fname : TEMPLETON_CONF_FILENAMES) + if (! loadOneClasspathConfig(fname)) + loadOneFileConfig(templetonDir, fname); + + // What a horrible place to do this. Needs to move into the + // logging config file. + Logger filterLogger = Logger.getLogger(WebComponent.class.getName()); + if (filterLogger != null) + filterLogger.setLevel(Level.SEVERE); + } + + public static String getHadoopPrefix() { + for (String var : HADOOP_PREFIX_VARS) { + String x = System.getenv(var); + if (x != null) + return x; + } + return null; + } + + + public static String getHadoopConfDir() { + String x = System.getenv("HADOOP_CONF_DIR"); + if (x != null) + return x; + + String prefix = getHadoopPrefix(); + if (prefix != null) + return new File(prefix, "conf").getAbsolutePath(); + + return null; + } + + public static String getTempletonDir() { + return System.getenv(TEMPLETON_HOME_VAR); + } + + private boolean loadOneFileConfig(String dir, String fname) { + if (dir != null) { + File f = new File(dir, fname); + if (f.exists()) { + addResource(new Path(f.getAbsolutePath())); + LOG.info("loaded config file " + f.getAbsolutePath()); + return true; + } + } + return false; + } + + private boolean loadOneClasspathConfig(String fname) { + URL x = getResource(fname); + if (x != null) { + addResource(x); + LOG.info("loaded config from classpath " + x); + return true; + } + + return false; + } + + public String templetonJar() { return get(TEMPLETON_JAR_NAME); } + public String clusterHadoop() { return get(HADOOP_NAME); } + public String clusterHcat() { return get(HCAT_NAME); } + public String pigPath() { return get(PIG_PATH_NAME); } + public String pigArchive() { return get(PIG_ARCHIVE_NAME); } + public String hivePath() { return get(HIVE_PATH_NAME); } + public String hiveArchive() { return get(HIVE_ARCHIVE_NAME); } + public String sudoPath() { return get(SUDO_NAME); } + public String zkHosts() { return get(ZK_HOSTS); } + public int zkSessionTimeout() { return getInt(ZK_SESSION_TIMEOUT, 30000); } + + public String streamingJar() { + return get(STREAMING_JAR_NAME); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/HcatDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/HcatDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/HcatDelegator.java (revision 0) @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import java.util.ArrayList; +import org.apache.commons.exec.ExecuteException; + +/** + * Run hcat on the local server using the ExecService. This is + * the backend of the ddl web service. + */ +public class HcatDelegator extends TempletonDelegator { + public HcatDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public ExecBean run(String user, String exec, String group, String permissions) + throws NotAuthorizedException, BusyException, ExecuteException, IOException + { + ArrayList args = new ArrayList(); + args.add("-e"); + args.add(exec); + if (group != null) { + args.add("-g"); + args.add(group); + } + if (permissions != null) { + args.add("-p"); + args.add(permissions); + } + + return execService.run(user, appConf.clusterHcat(), args, null); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/PigDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/PigDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/PigDelegator.java (revision 0) @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import org.apache.commons.exec.ExecuteException; +import org.apache.hcatalog.templeton.tool.TempletonControllerJob; +import org.apache.hcatalog.templeton.tool.TempletonUtils; + +/** + * Submit a Pig job. We do this by running the hadoop executable + * on the local server using the ExecService. This allows us to + * easily verify that the user identity is being securely used. + * + * This is the backend of the pig web service. + */ +public class PigDelegator extends TempletonDelegator { + public static final String JAR_CLASS = TempletonControllerJob.class.getName(); + + public PigDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public EnqueueBean run(String user, + String execute, String srcFile, + List pigArgs, String otherFiles, + String statusdir) + throws NotAuthorizedException, BadParam, BusyException, QueueException, + ExecuteException, IOException + { + List args = makeArgs(execute, + srcFile, pigArgs, + otherFiles, statusdir); + + ExecBean exec = execService.run(user, appConf.clusterHadoop(), args, null); + if (exec.exitcode != 0) + throw new QueueException("invalid exit code", exec); + String id = TempletonUtils.extractJobId(exec.stdout); + if (id == null) + throw new QueueException("Unable to get job id", exec); + + return new EnqueueBean(id, exec); + } + + private List makeArgs(String execute, String srcFile, + List pigArgs, String otherFiles, + String statusdir) + throws BadParam, IOException + { + ArrayList args = new ArrayList(); + try { + args.add("jar"); + args.add(appConf.templetonJar()); + args.add(JAR_CLASS); + args.add("-archives"); + args.add(appConf.pigArchive()); + + ArrayList allFiles = new ArrayList(); + if (TempletonUtils.isset(srcFile)) + allFiles.add(TempletonUtils.hadoopFsFilename(srcFile, appConf)); + if (TempletonUtils.isset(otherFiles)) { + String[] ofs = TempletonUtils.hadoopFsListAsArray(otherFiles, appConf); + allFiles.addAll(Arrays.asList(ofs)); + } + + args.add(TempletonUtils.encodeCliArray(allFiles)); + args.add(TempletonUtils.encodeCliArg(statusdir)); + args.add("--"); + args.add(appConf.pigPath()); + if (TempletonUtils.isset(execute)) { + args.add("-execute"); + args.add(execute); + } else if (TempletonUtils.isset(srcFile)) { + args.add("-file"); + args.add(TempletonUtils.hadoopFsPath(srcFile, appConf).getName()); + } + args.addAll(pigArgs); + } catch (FileNotFoundException e) { + throw new BadParam(e.getMessage()); + } catch (URISyntaxException e) { + throw new BadParam(e.getMessage()); + } + + return args; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/BadParam.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/BadParam.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/BadParam.java (revision 0) @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * Missing required or badly configured paramater. + */ +public class BadParam extends SimpleWebException { + public BadParam(String msg) { + super(400, msg); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/StatusDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/StatusDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/StatusDelegator.java (revision 0) @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import org.apache.hadoop.mapred.JobID; +import org.apache.hadoop.mapred.JobProfile; +import org.apache.hadoop.mapred.JobStatus; +import org.apache.hadoop.mapred.JobTracker; +import org.apache.hadoop.mapred.TempletonJobTracker; +import org.apache.hadoop.security.UserGroupInformation; + +/** + * Fetch the status of a given job id in the queue. + */ +public class StatusDelegator extends TempletonDelegator { + public StatusDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public QueueStatusBean run(String user, String id) + throws NotAuthorizedException, BadParam, IOException + { + UserGroupInformation ugi = UserGroupInformation.createRemoteUser(user); + TempletonJobTracker tracker = null; + try { + tracker = new TempletonJobTracker(ugi, + JobTracker.getAddress(appConf), + appConf); + JobID jobid = JobID.forName(id); + JobStatus status = tracker.getJobStatus(jobid); + JobProfile profile = tracker.getJobProfile(jobid); + return new QueueStatusBean(status, profile); + } catch (IllegalStateException e) { + throw new BadParam(e.getMessage()); + } finally { + if (tracker != null) + tracker.close(); + } + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecBean.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecBean.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecBean.java (revision 0) @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * ExecBean - The results of an exec call. + */ +public class ExecBean { + public String stdout; + public String stderr; + public int exitcode; + + public ExecBean() {} + + /** + * Create a new ExecBean. + * + * @param stdout standard output of the the program. + * @param stderr error output of the the program. + * @param exitcode exit code of the program. + */ + public ExecBean(String stdout, String stderr, int exitcode) { + this.stdout = stdout; + this.stderr = stderr; + this.exitcode = exitcode; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleExceptionMapper.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleExceptionMapper.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/SimpleExceptionMapper.java (revision 0) @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import javax.ws.rs.core.Response; +import javax.ws.rs.ext.ExceptionMapper; +import javax.ws.rs.ext.Provider; + +/** + * Map our exceptions to the Jersey response. This lets us have nice + * results in the error body. + */ +@Provider +public class SimpleExceptionMapper + implements ExceptionMapper +{ + public Response toResponse(SimpleWebException e) { + return e.getResponse(); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/NotAuthorizedException.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/NotAuthorizedException.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/NotAuthorizedException.java (revision 0) @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * Simple "user not found" type exception. + */ +public class NotAuthorizedException extends SimpleWebException { + public NotAuthorizedException(String msg) { + super(401, msg); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/QueueStatusBean.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/QueueStatusBean.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/QueueStatusBean.java (revision 0) @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import org.apache.hadoop.mapred.JobStatus; +import org.apache.hadoop.mapred.JobProfile; + +/** + * QueueStatusBean - The results of an exec call. + */ +public class QueueStatusBean { + public JobStatus status; + public JobProfile profile; + + public QueueStatusBean() {} + + /** + * Create a new QueueStatusBean + * + * @param status job status + * @param profile job profile + */ + public QueueStatusBean(JobStatus status, JobProfile profile) { + this.status = status; + this.profile = profile; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/CallbackFailedException.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/CallbackFailedException.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/CallbackFailedException.java (revision 0) @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * The callback failed when it tried to reach the callback URL. + */ +public class CallbackFailedException extends SimpleWebException { + /** + * + */ + private static final long serialVersionUID = 1L; + + public CallbackFailedException(String msg) { + super(400, msg); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/StreamingDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/StreamingDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/StreamingDelegator.java (revision 0) @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import org.apache.commons.exec.ExecuteException; +import org.apache.hcatalog.templeton.tool.TempletonUtils; +import org.apache.hcatalog.templeton.tool.TempletonStreamJob; + +/** + * Submit a streaming job to the MapReduce queue. We do this by + * running the hadoop executable on the local server using the + * ExecService. This allows us to easily verify that the user + * identity is being securely used. + * + * This is the backend of the mapreduce/streaming web service. + */ +public class StreamingDelegator extends TempletonDelegator { + public static final String STREAM_CLASS = TempletonStreamJob.class.getName(); + + public StreamingDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public EnqueueBean run(String user, + List inputs, String output, + String mapper, String reducer, + List files, List defines, + List cmdenvs, + List jarArgs) + throws NotAuthorizedException, BusyException, QueueException, + ExecuteException, IOException + { + ArrayList args = new ArrayList(); + args.add("jar"); + args.add(appConf.templetonJar()); + args.add(STREAM_CLASS); + args.add("-libjars"); + args.add(appConf.streamingJar()); + for (String input : inputs) { + args.add("-input"); + args.add(input); + } + args.add("-output"); + args.add(output); + args.add("-mapper"); + args.add(mapper); + args.add("-reducer"); + args.add(reducer); + + for (String f : files) + args.add("-file" + f); + for (String d : defines) + args.add("-D" + d); + for (String e : cmdenvs) + args.add("-cmdenv" + e); + args.addAll(jarArgs); + + HashMap env = new HashMap(); + env.put("HADOOP_CLASSPATH", appConf.streamingJar()); + + ExecBean exec = execService.run(user, appConf.clusterHadoop(), args, env); + if (exec.exitcode != 0) + throw new QueueException("invalid exit code", exec); + String id = TempletonUtils.extractJobId(exec.stdout); + if (id == null) + throw new QueueException("Unable to get job id", exec); + + return new EnqueueBean(id, exec); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/HiveDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/HiveDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/HiveDelegator.java (revision 0) @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import org.apache.commons.exec.ExecuteException; +import org.apache.hcatalog.templeton.tool.TempletonControllerJob; +import org.apache.hcatalog.templeton.tool.TempletonUtils; + +/** + * Submit a Hive job. We do this by running the hadoop executable + * on the local server using the ExecService. This allows us to + * easily verify that the user identity is being securely used. + * + * This is the backend of the pig web service. + */ +public class HiveDelegator extends TempletonDelegator { + public static final String JAR_CLASS = TempletonControllerJob.class.getName(); + + public HiveDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public EnqueueBean run(String user, + String execute, String srcFile, List defines, + String statusdir) + throws NotAuthorizedException, BadParam, BusyException, QueueException, + ExecuteException, IOException + { + List args = makeArgs(execute, srcFile, defines, statusdir); + + ExecBean exec = execService.run(user, appConf.clusterHadoop(), args, null); + if (exec.exitcode != 0) + throw new QueueException("invalid exit code", exec); + String id = TempletonUtils.extractJobId(exec.stdout); + if (id == null) + throw new QueueException("Unable to get job id", exec); + + return new EnqueueBean(id, exec); + } + + private List makeArgs(String execute, String srcFile, List defines, + String statusdir) + throws BadParam, IOException + { + ArrayList args = new ArrayList(); + try { + args.addAll(makeBasicArgs(execute, srcFile, statusdir)); + args.add("--"); + args.add(appConf.hivePath()); + args.add("--service"); + args.add("cli"); + for (String prop : appConf.getStrings(AppConfig.HIVE_PROPS_NAME)) { + args.add("--hiveconf"); + args.add(prop); + } + for (String prop : defines) { + args.add("--hiveconf"); + args.add(prop); + } + if (TempletonUtils.isset(execute)) { + args.add("-e"); + args.add(execute); + } else if (TempletonUtils.isset(srcFile)) { + args.add("-f"); + args.add(TempletonUtils.hadoopFsPath(srcFile, appConf).getName()); + } + } catch (FileNotFoundException e) { + throw new BadParam(e.getMessage()); + } catch (URISyntaxException e) { + throw new BadParam(e.getMessage()); + } + + return args; + } + + private List makeBasicArgs(String execute, String srcFile, String statusdir) + throws URISyntaxException, FileNotFoundException, IOException + { + ArrayList args = new ArrayList(); + + args.add("jar"); + args.add(appConf.templetonJar()); + args.add(JAR_CLASS); + args.add("-archives"); + args.add(appConf.hiveArchive()); + + ArrayList allFiles = new ArrayList(); + if (TempletonUtils.isset(srcFile)) + allFiles.add(TempletonUtils.hadoopFsFilename(srcFile, appConf)); + + args.add(TempletonUtils.encodeCliArray(allFiles)); + args.add(TempletonUtils.encodeCliArg(statusdir)); + return args; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/ServerCallback.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/ServerCallback.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/ServerCallback.java (revision 0) @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.BufferedReader; +import java.io.InputStreamReader; +import java.net.HttpURLConnection; +import java.net.URL; +import java.util.HashMap; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +/** + * A simple callback class that just registers a callback url when a job is started, and + * calls it back with the jobid when it's finished. + * + */ +public class ServerCallback { + // The map of jobids to urls + public static HashMap callbackMap = new HashMap(); + + // The logger + private static final Log LOG = LogFactory.getLog(ServerCallback.class); + + /** + * Register a callback url for a jobid. The jobid value is appended to the + * url: http://apache.org becomes http://apache.org?jobid=... + * + * @param jobid + * @param url + */ + public static void registerCallback(String jobid, String url) { + if (url.indexOf("?") > -1) { + url = url + "&jobid=" + jobid; + } else { + url = url + "?jobid=" + jobid; + } + callbackMap.put(jobid, url); + } + + /** + * Call the callback url with the jobid to let them know it's finished. + * + * @param jobid + * @throws CallbackFailedException + */ + public static void doCallback(String jobid) throws CallbackFailedException { + HttpURLConnection connection = null; + BufferedReader in = null; + try { + LOG.info("Calling " + callbackMap.get(jobid)); + URL url = new URL(callbackMap.get(jobid)); + connection = (HttpURLConnection) url.openConnection(); + in = new BufferedReader(new InputStreamReader(connection.getInputStream())); + String line; + while ((line = in.readLine()) != null) { + //LOG.info(line + "\n"); + } + HcatDelegator d = new HcatDelegator(AppConfig.getInstance(), ExecServiceImpl.getInstance()); + ExecBean bean = d.run(System.getenv("USER"), + "create table if not exists templeton_jobids (jobid STRING, url STRING)" + + " comment 'The templeton callback table' stored as textfile;", "admin", "777"); + + in.close(); + } catch (SimpleWebException ex) { + throw new CallbackFailedException("Unabled to connect to " + callbackMap.get(jobid)); + } catch (Exception e) { + throw new CallbackFailedException("Unable to connect to " + callbackMap.get(jobid)); + } finally { + try { + in.close(); + } catch (Exception e) { + // Can't do anything with it. + } + try { + connection.disconnect(); + } catch (Exception e) { + // Can't do anything with it. + } + } + } + +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecServiceImpl.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecServiceImpl.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/ExecServiceImpl.java (revision 0) @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Semaphore; +import org.apache.commons.exec.CommandLine; +import org.apache.commons.exec.DefaultExecutor; +import org.apache.commons.exec.ExecuteException; +import org.apache.commons.exec.ExecuteWatchdog; +import org.apache.commons.exec.PumpStreamHandler; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; + +/** + * Execute a local program. This is a singelton service that will + * execute programs as non-privileged users on the local box. See + * ExecService.run and ExecService.runUnlimited for details. + */ +public class ExecServiceImpl implements ExecService { + private static final Log LOG = LogFactory.getLog(ExecServiceImpl.class); + private static AppConfig appConf = AppConfig.getInstance(); + + private static volatile ExecServiceImpl theSingleton; + + /** + * Retrieve the singleton. + */ + public static synchronized ExecServiceImpl getInstance() { + if (theSingleton == null) { + theSingleton = new ExecServiceImpl(); + } + return theSingleton; + } + + private Semaphore avail; + + private ExecServiceImpl() { + avail = new Semaphore(appConf.getInt(AppConfig.EXEC_MAX_PROCS_NAME, 16)); + } + + /** + * Run the program synchronously as the given user. We rate limit + * the number of processes that can simultaneously created for + * this instance. + * + * @param user A valid user + * @param program The program to run + * @param env Any extra environment variables to set + * @returns The result of the run. + */ + public ExecBean run(String user, String program, List args, + Map env) + throws NotAuthorizedException, BusyException, ExecuteException, IOException + { + boolean aquired = false; + try { + aquired = avail.tryAcquire(); + if (aquired) { + return runUnlimited(user, program, args, env); + } else { + throw new BusyException(); + } + } finally { + if (aquired) { + avail.release(); + } + } + } + + /** + * Run the program synchronously as the given user. Warning: + * CommandLine will trim the argument strings. + * + * @param user A valid user + * @param program The program to run. + * @returns The result of the run. + */ + public ExecBean runUnlimited(String user, String program, List args, + Map env) + throws NotAuthorizedException, ExecuteException, IOException + { + DefaultExecutor executor = new DefaultExecutor(); + executor.setExitValues(null); + + // Setup stdout and stderr + int nbytes = appConf.getInt(AppConfig.EXEC_MAX_BYTES_NAME, -1); + ByteArrayOutputStream outStream = new MaxByteArrayOutputStream(nbytes); + ByteArrayOutputStream errStream = new MaxByteArrayOutputStream(nbytes); + executor.setStreamHandler(new PumpStreamHandler(outStream, errStream)); + + // Only run for N milliseconds + int timeout = appConf.getInt(AppConfig.EXEC_TIMEOUT_NAME, 0); + ExecuteWatchdog watchdog = new ExecuteWatchdog(timeout); + executor.setWatchdog(watchdog); + + CommandLine cmd = makeCommandLine(user, program, args, env); + + LOG.info("Running: " + cmd); + ExecBean res = new ExecBean(); + res.exitcode = executor.execute(cmd); + String enc = appConf.get(AppConfig.EXEC_ENCODING_NAME); + res.stdout = outStream.toString(enc); + res.stderr = errStream.toString(enc); + + return res; + } + + private CommandLine makeCommandLine(String user, String program, List args, + Map env) + throws NotAuthorizedException, IOException + { + String path = validateProgram(program); + + CommandLine cmd = new CommandLine(new File(appConf.sudoPath())); + cmd.addArgument("-u"); + cmd.addArgument(user); + + for (Map.Entry entry : sudoEnv(env).entrySet()) { + cmd.addArgument(entry.getKey() + "=" + entry.getValue()); + } + + cmd.addArgument(path); + if (args != null) + for (String arg : args) + cmd.addArgument(arg, false); + + return cmd; + } + + /** + * Build the environment used for all sudo calls. + * + * @return The environment variables. + */ + public Map sudoEnv(Map env) { + HashMap res = new HashMap(); + + for (String key : appConf.getStrings(AppConfig.EXEC_ENVS_NAME)) { + String val = System.getenv(key); + if (val != null) + res.put(key, val); + } + if (env != null) + res.putAll(env); + + return res; + } + + /** + * Given a program name, lookup the fully qualified path. Throws + * an exception if the program is missing or not authorized. + * + * @param path The path of the program. + * @return The path of the validated program. + */ + public String validateProgram(String path) + throws NotAuthorizedException, IOException + { + File f = new File(path); + if (f.canExecute()) { + return f.getCanonicalPath(); + } else { + throw new NotAuthorizedException("Unable to access program: " + path); + } + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/DeleteDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/DeleteDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/DeleteDelegator.java (revision 0) @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +import java.io.IOException; +import org.apache.hadoop.mapred.JobID; +import org.apache.hadoop.mapred.JobProfile; +import org.apache.hadoop.mapred.JobStatus; +import org.apache.hadoop.mapred.JobTracker; +import org.apache.hadoop.mapred.TempletonJobTracker; +import org.apache.hadoop.security.UserGroupInformation; + +/** + * Delete a job + */ +public class DeleteDelegator extends TempletonDelegator { + public DeleteDelegator(AppConfig appConf, ExecService execService) { + super(appConf, execService); + } + + public QueueStatusBean run(String user, String id) + throws NotAuthorizedException, BadParam, IOException + { + UserGroupInformation ugi = UserGroupInformation.createRemoteUser(user); + TempletonJobTracker tracker = null; + try { + tracker = new TempletonJobTracker(ugi, + JobTracker.getAddress(appConf), + appConf); + JobID jobid = JobID.forName(id); + tracker.killJob(jobid); + JobStatus status = tracker.getJobStatus(jobid); + JobProfile profile = tracker.getJobProfile(jobid); + return new QueueStatusBean(status, profile); + } catch (IllegalStateException e) { + throw new BadParam(e.getMessage()); + } finally { + if (tracker != null) + tracker.close(); + } + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/EnqueueBean.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/EnqueueBean.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/EnqueueBean.java (revision 0) @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * EnqueueBean - The results of a call that enqueues a Hadoop job. + */ +public class EnqueueBean { + public String id; + public ExecBean info; + + public EnqueueBean() {} + + /** + * Create a new EnqueueBean. + * + * @param id job id + * @param info result of the call that queued the job + */ + public EnqueueBean(String id, ExecBean info) { + this.id = id; + this.info = info; + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TrivialExecService.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TrivialExecService.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TrivialExecService.java (revision 0) @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.IOException; +import java.util.List; + +/** + * Execute a local program. This is a singelton service that will + * execute a programs on the local box. + */ +public class TrivialExecService { + private static volatile TrivialExecService theSingleton; + + /** + * Retrieve the singleton. + */ + public static synchronized TrivialExecService getInstance() { + if (theSingleton == null) + theSingleton = new TrivialExecService(); + return theSingleton; + } + + public Process run(List cmd) + throws IOException + { + System.err.println("templeton: starting " + cmd); + ProcessBuilder pb = new ProcessBuilder(cmd); + return pb.start(); + } + +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullSplit.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullSplit.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullSplit.java (revision 0) @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; +import org.apache.hadoop.io.Writable; +import org.apache.hadoop.mapreduce.InputSplit; + +/** + * An empty splitter. + */ +public class NullSplit extends InputSplit implements Writable { + public long getLength() { return 0; } + + public String[] getLocations() throws IOException { + return new String[]{}; + } + + @Override + public void write(DataOutput out) throws IOException {} + + @Override + public void readFields(DataInput in) throws IOException {} +} + Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonControllerJob.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonControllerJob.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonControllerJob.java (revision 0) @@ -0,0 +1,223 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.Arrays; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapreduce.Counter; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.Mapper; +import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; + +/** + * A Map Reduce job that will start another job. + */ +public class TempletonControllerJob extends Configured implements Tool { + static enum ControllerCounters { SIMPLE_COUNTER }; + + public static final String COPY_NAME = "templeton.copy"; + public static final String STATUSDIR_NAME = "templeton.statusdir"; + public static final String JAR_ARGS_NAME = "templeton.args"; + + public static final int WATCHER_TIMEOUT_SECS = 10; + public static final int KEEP_ALIVE_MSEC = 60 * 1000; + + private static TrivialExecService execService = TrivialExecService.getInstance(); + + public static class LaunchMapper + extends Mapper + { + protected Process startJob(Context context) + throws IOException, InterruptedException + { + Configuration conf = context.getConfiguration(); + copyLocal(COPY_NAME, conf); + String[] jarArgs + = TempletonUtils.decodeArray(conf.get(JAR_ARGS_NAME)); + + return execService.run(Arrays.asList(jarArgs)); + } + + private void copyLocal(String var, Configuration conf) + throws IOException + { + String[] filenames = TempletonUtils.decodeArray(conf.get(var)); + if (filenames != null) { + for (String filename : filenames) { + Path src = new Path(filename); + Path dst = new Path(src.getName()); + FileSystem fs = src.getFileSystem(conf); + System.err.println("templeton: copy " + src + " => " + dst); + fs.copyToLocalFile(src, dst); + } + } + } + + @Override + public void run(Context context) + throws IOException, InterruptedException + { + Process proc = startJob(context); + + Configuration conf = context.getConfiguration(); + String statusdir = conf.get(STATUSDIR_NAME); + Counter cnt = context.getCounter(ControllerCounters.SIMPLE_COUNTER); + + ExecutorService pool = Executors.newCachedThreadPool(); + executeWatcher(pool, conf, proc.getInputStream(), statusdir, "stdout"); + executeWatcher(pool, conf, proc.getErrorStream(), statusdir, "stderr"); + KeepAlive keepAlive = startCounterKeepAlive(pool, cnt); + + proc.waitFor(); + keepAlive.sendReport = false; + pool.shutdown(); + if (! pool.awaitTermination(WATCHER_TIMEOUT_SECS, TimeUnit.SECONDS)) + pool.shutdownNow(); + + if (proc.exitValue() != 0) { + System.err.println("templeton: job failed with exit code " + + proc.exitValue()); + System.exit(proc.exitValue()); + } + } + + private void executeWatcher(ExecutorService pool, Configuration conf, + InputStream in, String statusdir, String name) + throws IOException + { + Watcher w = new Watcher(conf, in, statusdir, name); + pool.execute(w); + } + + private KeepAlive startCounterKeepAlive(ExecutorService pool, Counter cnt) + throws IOException + { + KeepAlive k = new KeepAlive(cnt); + pool.execute(k); + return k; + } + } + + public static class Watcher implements Runnable { + private InputStream in; + private OutputStream out; + + public Watcher(Configuration conf, InputStream in, + String statusdir, String name) + throws IOException + { + this.in = in; + + if (name.equals("stderr")) + out = System.err; + else + out = System.out; + + if (TempletonUtils.isset(statusdir)) { + Path p = new Path(statusdir, name); + FileSystem fs = p.getFileSystem(conf); + out = fs.create(p); + System.err.println("templeton: Writing status to " + p); + } + } + + @Override + public void run() { + try { + byte[] buf = new byte[512]; + int len = 0; + while ((len = in.read(buf)) >= 0) + out.write(buf, 0, len); + } catch (IOException e) { + System.err.println("templeton: execute error: " + e); + } + } + } + + public static class KeepAlive implements Runnable { + private Counter cnt; + public boolean sendReport; + + public KeepAlive(Counter cnt) + { + this.cnt = cnt; + this.sendReport = true; + } + + @Override + public void run() { + try { + while (sendReport) { + cnt.increment(1); + Thread.sleep(KEEP_ALIVE_MSEC); + } + } catch (InterruptedException e) { + // Ok to be interrupted + } + } + } + + /** + * Enqueue the job and print out the job id for later collection. + */ + @Override + public int run(String[] args) + throws IOException, InterruptedException, ClassNotFoundException + { + Configuration conf = getConf(); + conf.set(COPY_NAME, TempletonUtils.decodeCliArg(args[0])); + conf.set(STATUSDIR_NAME, TempletonUtils.decodeCliArg(args[1])); + String[] childArgs = Arrays.copyOfRange(args, 2, args.length); + conf.set(JAR_ARGS_NAME, TempletonUtils.encodeArray(childArgs)); + Job job = new Job(conf); + job.setJarByClass(TempletonControllerJob.class); + job.setJobName("TempletonControllerJob"); + job.setMapperClass(LaunchMapper.class); + job.setMapOutputKeyClass(Text.class); + job.setMapOutputValueClass(Text.class); + job.setInputFormatClass(SingleInputFormat.class); + NullOutputFormat of + = new NullOutputFormat(); + job.setOutputFormatClass(of.getClass()); + job.setNumReduceTasks(0); + + job.submit(); + TempletonUtils.printTaggedJobID(job.getJobID()); + return 0; + } + + public static void main(String[] args) throws Exception { + int ret = ToolRunner.run(new TempletonControllerJob(), args); + if (ret != 0) + System.err.println("TempletonControllerJob failed!"); + System.exit(ret); + } +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonUtils.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonUtils.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonUtils.java (revision 0) @@ -0,0 +1,225 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.mapreduce.JobID; +import org.apache.hadoop.util.StringUtils; + +/** + * General utility methods. + */ +public class TempletonUtils { + public static final String JOB_ID_TAG = "templeton-job-id:"; + + /** + * Is the object non-empty? + */ + public static boolean isset(String s) { + return (s != null) && (s.length() > 0); + } + + /** + * Is the object non-empty? + */ + public static boolean isset(T[] a) { + return (a != null) && (a.length > 0); + } + + + /** + * Print a job id for later extraction. + */ + public static void printTaggedJobID(JobID jobid) { + System.out.println(); + System.out.println(JOB_ID_TAG + jobid); + } + + /** + * Extract the job id that we output earlier. + */ + public static String extractJobId(String stdout) { + String pat = "^" + Pattern.quote(JOB_ID_TAG) + "(\\S+)"; + Pattern re = Pattern.compile(pat, Pattern.MULTILINE); + + Matcher m = re.matcher(stdout); + if (! m.find()) + return null; + return m.group(1); + } + + + /** + * Encode a command line argument. We need to allow for empty + * arguments. + */ + public static String encodeCliArg(String s) { + if (TempletonUtils.isset(s)) + return "*" + s; + else + return "*"; + } + + /** + * Decode a command line argument. We need to allow for empty + * arguments. + */ + public static String decodeCliArg(String s) { + if (s != null && s.startsWith("*")) + return s.substring(1); + else + return s; + } + + /** + * Take an array of strings and encode it into one string. + */ + public static String encodeArray(String[] plain) { + if (plain == null) + return null; + + String[] escaped = new String[plain.length]; + + for (int i = 0; i < plain.length; ++i) { + if (plain[i] == null) { + plain[i] = ""; + } + escaped[i] = StringUtils.escapeString(plain[i]); + } + + return StringUtils.arrayToString(escaped); + } + + /** + * Take an encode strings and decode it into an array of strings. + */ + public static String[] decodeArray(String s) { + if (s == null) + return null; + + String[] escaped = StringUtils.split(s); + String[] plain = new String[escaped.length]; + + for (int i = 0; i < escaped.length; ++i) + plain[i] = StringUtils.unEscapeString(escaped[i]); + + return plain; + } + + /** + * Encode an array to be used on the command line. + */ + public static String encodeCliArray(String[] array) { + String x = encodeArray(array); + return encodeCliArg(x); + } + + /** + * Encode an array to be used on the command line. + */ + public static String encodeCliArray(List list) { + if (list == null) { + return null; + } + String[] array = new String[list.size()]; + String x = encodeArray(list.toArray(array)); + return encodeCliArg(x); + } + + /** + * Encode a string as a one element array to be used on the + * command line. + */ + public static String encodeCliArray(String s) { + if (s == null) + return null; + + String[] array = new String[1]; + array[0] = s; + return encodeCliArray(array); + } + + /** + * Decode a command line arg into an array of strings. + */ + public static String[] decodeCliArray(String s) { + String x = decodeCliArg(s); + return decodeArray(x); + } + + public static String[] hadoopFsListAsArray(String files, Configuration conf) + throws URISyntaxException, FileNotFoundException, IOException + { + if (files == null || conf == null) { + return null; + } + String[] dirty = files.split(","); + String[] clean = new String[dirty.length]; + + for (int i = 0; i < dirty.length; ++i) + clean[i] = hadoopFsFilename(dirty[i], conf); + + return clean; + } + + public static String hadoopFsListAsString(String files, Configuration conf) + throws URISyntaxException, FileNotFoundException, IOException + { + if (files == null || conf == null) { + return null; + } + return StringUtils.arrayToString(hadoopFsListAsArray(files, conf)); + } + + public static String hadoopFsFilename(String fname, Configuration conf) + throws URISyntaxException, FileNotFoundException, IOException + { + Path p = hadoopFsPath(fname, conf); + if (p == null) + return null; + else + return p.toString(); + } + + public static Path hadoopFsPath(String fname, Configuration conf) + throws URISyntaxException, FileNotFoundException, IOException + { + if (fname == null || conf == null) { + return null; + } + FileSystem defaultFs = FileSystem.get(conf); + URI u = new URI(fname); + Path p = new Path(u).makeQualified(defaultFs); + + FileSystem fs = p.getFileSystem(conf); + if (! fs.exists(p)) + throw new FileNotFoundException("File " + fname + " does not exist."); + + return p; + } + +} Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/SingleInputFormat.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/SingleInputFormat.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/SingleInputFormat.java (revision 0) @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapreduce.InputFormat; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.JobContext; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.TaskAttemptContext; + +/** + * An empty InputFormat. + */ +public class SingleInputFormat + extends InputFormat +{ + public List getSplits(JobContext job) + throws IOException + { + List res = new ArrayList(); + res.add(new NullSplit()); + return res; + } + + public RecordReader + createRecordReader(InputSplit split, + TaskAttemptContext context) + throws IOException + { + return new NullRecordReader(); + } +} + Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullRecordReader.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullRecordReader.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/NullRecordReader.java (revision 0) @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.IOException; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.TaskAttemptContext; + +/** + * An empty record reader. + */ +public class NullRecordReader + extends RecordReader +{ + @Override + public void initialize(InputSplit genericSplit, TaskAttemptContext context) + throws IOException + {} + + @Override + public void close() throws IOException {} + + @Override + public NullWritable getCurrentKey() { + return NullWritable.get(); + } + + @Override + public NullWritable getCurrentValue() { + return NullWritable.get(); + } + + @Override + public float getProgress() { return 1.0f; } + + @Override + public boolean nextKeyValue() throws IOException { + return false; + } +} + Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonStreamJob.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonStreamJob.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/tool/TempletonStreamJob.java (revision 0) @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton.tool; + +import java.io.IOException; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.conf.Configured; +import org.apache.hadoop.mapred.JobClient; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapred.RunningJob; +import org.apache.hadoop.streaming.StreamJob; +import org.apache.hadoop.util.Tool; +import org.apache.hadoop.util.ToolRunner; + +/** + * Submit a streaming job from the command line. The default Hadoop + * tool submits the job and waits for completion and we want to just + * put it in the queue and return a job id. + */ +public class TempletonStreamJob extends Configured implements Tool { + /** + * Enqueue the job and print out the job id for later collection. + */ + @Override + public int run(String[] args) + throws IOException, InterruptedException, ClassNotFoundException + { + JobConf conf = StreamJob.createJob(args); + Job job = new Job(conf); + job.submit(); + TempletonUtils.printTaggedJobID(job.getJobID()); + return 0; + } + + public static void main(String[] args) throws Exception { + int ret = ToolRunner.run(new TempletonStreamJob(), args); + if (ret != 0) + System.err.println("TempletonStreamJob failed!"); + System.exit(ret); + } +} + Index: contrib/templeton/src/java/org/apache/hcatalog/templeton/TempletonDelegator.java =================================================================== --- contrib/templeton/src/java/org/apache/hcatalog/templeton/TempletonDelegator.java (revision 0) +++ contrib/templeton/src/java/org/apache/hcatalog/templeton/TempletonDelegator.java (revision 0) @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hcatalog.templeton; + +/** + * The helper class for all the Templeton delegator classes. A + * delegator will call the underlying Templeton service such as hcat + * or hive. + */ +public class TempletonDelegator { + protected AppConfig appConf; + protected ExecService execService; + + public TempletonDelegator(AppConfig appConf, ExecService execService) { + this.appConf = appConf; + this.execService = execService; + } +} Index: contrib/templeton/src/java/org/apache/hadoop/mapred/TempletonJobTracker.java =================================================================== --- contrib/templeton/src/java/org/apache/hadoop/mapred/TempletonJobTracker.java (revision 0) +++ contrib/templeton/src/java/org/apache/hadoop/mapred/TempletonJobTracker.java (revision 0) @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.mapred; + +import java.io.IOException; +import java.net.InetSocketAddress; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.ipc.RPC; +import org.apache.hadoop.net.NetUtils; +import org.apache.hadoop.security.UserGroupInformation; + +/* + * Communicate with the JobTracker as a specific user. + */ +public class TempletonJobTracker { + private JobSubmissionProtocol cnx; + + /** + * Create a connection to the Job Tracker. + */ + public TempletonJobTracker(UserGroupInformation ugi, + InetSocketAddress addr, + Configuration conf) + throws IOException + { + cnx = (JobSubmissionProtocol) + RPC.getProxy(JobSubmissionProtocol.class, + JobSubmissionProtocol.versionID, + addr, + ugi, + conf, + NetUtils.getSocketFactory(conf, + JobSubmissionProtocol.class)); + } + + /** + * Grab a handle to a job that is already known to the JobTracker. + * + * @return Profile of the job, or null if not found. + */ + public JobProfile getJobProfile(JobID jobid) + throws IOException + { + return cnx.getJobProfile(jobid); + } + + /** + * Grab a handle to a job that is already known to the JobTracker. + * + * @return Status of the job, or null if not found. + */ + public JobStatus getJobStatus(JobID jobid) + throws IOException + { + return cnx.getJobStatus(jobid); + } + + + /** + * Kill a job. + */ + public void killJob(JobID jobid) + throws IOException + { + cnx.killJob(jobid); + } + + /** + * Close the connection to the Job Tracker. + */ + public void close() { + RPC.stopProxy(cnx); + } +} Index: contrib/templeton/src/docs/forrest.properties =================================================================== --- contrib/templeton/src/docs/forrest.properties (revision 0) +++ contrib/templeton/src/docs/forrest.properties (revision 0) @@ -0,0 +1,168 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +############## +# These are the defaults, un-comment them only if you need to change them. +# +# You can even have a completely empty file, to assist with maintenance. +# This file is required, even if empty. +# +# The file obtained from 'forrest seed-sample' shows the defaults. +############## + +# Prints out a summary of Forrest settings for this project +#forrest.echo=true + +# Project name (used to name .war file) +#project.name=my-project + +# Specifies name of Forrest skin to use +# See list at http://forrest.apache.org/docs/skins.html +#project.skin=pelt + +# codename: Dispatcher +# Dispatcher is using a fallback mechanism for theming. +# You can configure the theme name and its extension here +#project.theme-extension=.fv +#project.theme=pelt + + +# Descriptors for plugins and skins +# comma separated list, file:// is supported +#forrest.skins.descriptors=http://forrest.apache.org/skins/skins.xml,file:///c:/myskins/skins.xml +#forrest.plugins.descriptors=http://forrest.apache.org/plugins/plugins.xml,http://forrest.apache.org/plugins/whiteboard-plugins.xml + +############## +# behavioural properties +#project.menu-scheme=tab_attributes +#project.menu-scheme=directories + +############## +# layout properties + +# Properties that can be set to override the default locations +# +# Parent properties must be set. This usually means uncommenting +# project.content-dir if any other property using it is uncommented + +#project.status=status.xml +#project.content-dir=src/documentation +#project.raw-content-dir=${project.content-dir}/content +#project.conf-dir=${project.content-dir}/conf +#project.sitemap-dir=${project.content-dir} +#project.xdocs-dir=${project.content-dir}/content/xdocs +#project.resources-dir=${project.content-dir}/resources +#project.stylesheets-dir=${project.resources-dir}/stylesheets +#project.images-dir=${project.resources-dir}/images +#project.schema-dir=${project.resources-dir}/schema +#project.skins-dir=${project.content-dir}/skins +#project.skinconf=${project.content-dir}/skinconf.xml +#project.lib-dir=${project.content-dir}/lib +#project.classes-dir=${project.content-dir}/classes +#project.translations-dir=${project.content-dir}/translations + +#project.build-dir=${project.home}/build +#project.site=site +#project.site-dir=${project.build-dir}/${project.site} +#project.temp-dir=${project.build-dir}/tmp + +############## +# Cocoon catalog entity resolver properties +# A local OASIS catalog file to supplement the default Forrest catalog +#project.catalog=${project.schema-dir}/catalog.xcat + +############## +# validation properties + +# This set of properties determine if validation is performed +# Values are inherited unless overridden. +# e.g. if forrest.validate=false then all others are false unless set to true. +#forrest.validate=true +#forrest.validate.xdocs=${forrest.validate} +#forrest.validate.skinconf=${forrest.validate} + +# PIG-1508: Workaround for http://issues.apache.org/jira/browse/FOR-984 +# Remove when forrest-0.9 is available +forrest.validate.sitemap=false +forrest.validate.stylesheets=false +forrest.validate.skins.stylesheets=false + +#forrest.validate.stylesheets=${forrest.validate} +#forrest.validate.skins=${forrest.validate} +#forrest.validate.skins.stylesheets=${forrest.validate.skins} + +# *.failonerror=(true|false) - stop when an XML file is invalid +#forrest.validate.failonerror=true + +# *.excludes=(pattern) - comma-separated list of path patterns to not validate +# Note: If you do add an "excludes" list then you need to specify site.xml too. +# e.g. +#forrest.validate.xdocs.excludes=site.xml, samples/subdir/**, samples/faq.xml +#forrest.validate.xdocs.excludes=site.xml + + +############## +# General Forrest properties + +# The URL to start crawling from +#project.start-uri=linkmap.html + +# Set logging level for messages printed to the console +# (DEBUG, INFO, WARN, ERROR, FATAL_ERROR) +#project.debuglevel=ERROR + +# Max memory to allocate to Java +#forrest.maxmemory=64m + +# Any other arguments to pass to the JVM. For example, to run on an X-less +# server, set to -Djava.awt.headless=true +#forrest.jvmargs= + +# The bugtracking URL - the issue number will be appended +# Projects would use their own issue tracker, of course. +#project.bugtracking-url=http://issues.apache.org/bugzilla/show_bug.cgi?id= +#project.bugtracking-url=http://issues.apache.org/jira/browse/ + +# The issues list as rss +#project.issues-rss-url= + +#I18n Property. Based on the locale request for the browser. +#If you want to use it for static site then modify the JVM system.language +# and run once per language +#project.i18n=false +project.configfile=${project.home}/src/documentation/conf/cli.xconf + +# The names of plugins that are required to build the project +# comma separated list (no spaces) +# You can request a specific version by appending "-VERSION" to the end of +# the plugin name. If you exclude a version number, the latest released version +# will be used. However, be aware that this may be a development version. In +# a production environment it is recommended that you specify a known working +# version. +# Run "forrest available-plugins" for a list of plug-ins currently available. + +project.required.plugins=org.apache.forrest.plugin.output.pdf,org.apache.forrest.plugin.input.simplifiedDocbook + + +# codename: Dispatcher +# Add the following plugins to project.required.plugins: +#org.apache.forrest.plugin.internal.dispatcher,org.apache.forrest.themes.core,org.apache.forrest.plugin.output.inputModule + +# Proxy configuration +# - proxy.user and proxy.password are only needed if the proxy is an authenticated one... +# proxy.host=myproxy.myhost.com +# proxy.port= +# proxy.user= +# proxy.password= Index: contrib/templeton/src/docs/forrest.properties.dispatcher.properties =================================================================== --- contrib/templeton/src/docs/forrest.properties.dispatcher.properties (revision 0) +++ contrib/templeton/src/docs/forrest.properties.dispatcher.properties (revision 0) @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +############## +# Note: The reason for this "forrest.properties.dispatcher.properties" +# is to assist with automated testing (main/build.sh test). +# Its content redefines the project.required.plugins property which defines +# the list of required plugins for the dispatcher. +# To test the Dispatcher in development, simply replace the +# project.required.plugins property in the forrest.properties file by the one +# defined in this file. +# +project.required.plugins=org.apache.forrest.plugin.output.pdf,org.apache.forrest.plugin.internal.dispatcher,org.apache.forrest.themes.core,org.apache.forrest.plugin.output.inputModule Index: contrib/templeton/src/docs/src/documentation/conf/cli.xconf =================================================================== --- contrib/templeton/src/docs/src/documentation/conf/cli.xconf (revision 0) +++ contrib/templeton/src/docs/src/documentation/conf/cli.xconf (revision 0) @@ -0,0 +1,327 @@ + + + + + + + + . + WEB-INF/cocoon.xconf + ../tmp/cocoon-work + ../site + + + + + + + + + + + + + + + index.html + + + + + + + */* + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/src/docs/src/documentation/sitemap.xmap =================================================================== --- contrib/templeton/src/docs/src/documentation/sitemap.xmap (revision 0) +++ contrib/templeton/src/docs/src/documentation/sitemap.xmap (revision 0) @@ -0,0 +1,66 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/src/docs/src/documentation/skinconf.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/skinconf.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/skinconf.xml (revision 0) @@ -0,0 +1,431 @@ + + + + + + + + + + + + true + + false + + true + + true + + + true + + true + + true + + true + .at. + + true + + + Templeton + A RESTful web API for HCatalog and related Hadoop components. + + images/templetontitle.jpg + + + + + Templeton + + + images/templetonlogov2.png + + + + + + + + + + + + false + + + 2011 + The Apache Software Foundation + + + http://www.apache.org/licenses/ + + + + + + + + + + + + + + + + + + + + p.quote { + margin-left: 2em; + padding: .5em; + background-color: #f0f0f0; + font-family: monospace; + } + + #footer a { color: #0F3660; } + #footer a:visited { color: #009999; } + + + + #content h1 { + margin-bottom: .5em; + font-size: 200%; color: black; + font-family: arial; + } + h2, .h3 { font-size: 195%; color: black; font-family: arial; } + h3, .h4 { font-size: 140%; color: black; font-family: arial; margin-bottom: 0.5em; } + h4, .h5 { font-size: 125%; color: black; font-style: italic; font-weight: bold; font-family: arial; } + h5, h6 { font-size: 110%; color: #363636; font-weight: bold; } + + pre.code { + margin-left: 0em; + padding: 0.5em; + background-color: rgb(241,239,231); + font-family: monospace; + } + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Page 1 + + + 1in + 1in + 1.25in + 1in + + + false + + false + + + Index: contrib/templeton/src/docs/src/documentation/content/xdocs/status.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/status.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/status.xml (revision 0) @@ -0,0 +1,75 @@ + + + + + +
+ GET status +
+ + +
+ Description +

Returns the current status of the Templeton server. + Useful for heartbeat monitoring.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/status.json

+
+ +
+ Parameters +

Only the standard parameters + are accepted.

+
+ +
+ Results + + + + + + + + + + +
NameDescription
status"ok" if the Templeton server was contacted.
versionString containing the version number similar to "v1".
+
+ +
+ Example + +

Curl Command

+ +% curl 'http://www.myserver.com/templeton/v1/status.json?user.name=charlotte' + + +

JSON Output

+ +{ + "status": "ok", + "version": "v1" +} + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/queue.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/queue.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/queue.xml (revision 0) @@ -0,0 +1,115 @@ + + + + + +
+ GET queue/:jobid +
+ + +
+ Description +

Check the status of a job and get related job information given its job ID. + Substitute ":jobid" with the job ID received when the job was created.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/queue/:jobid.json

+
+ +
+ Parameters + + + + + + + + +
NameDescriptionRequired?Default
:jobidThe job ID to check. This is the ID received when then + job was created.RequiredNone
+
+ +
+ Results + + + + + + + + + +
NameDescription
statusA JSON object containing the job status information. + See the Hadoop documentation + (Class + JobStatus) for more information.
profileA JSON object containing the job profile information. + See the Hadoop documentation + (Class + JobProfile) for more information. +
+
+ +
+ Example + +

Curl Command

+ +% curl 'http://www.myserver.com/templeton/v1/queue/job_201111111311_0008.json?user.name=ctdean' + + +

JSON Output

+ +{ + "status": { + "startTime": 1321046803361, + "username": "ctdean", + "jobID": { + "jtIdentifier": "201111111311", + "id": 8 + }, + "jobACLs": { + }, + "schedulingInfo": "NA", + "failureInfo": "NA", + "jobId": "job_201111111311_0008", + "jobPriority": "NORMAL", + "runState": 2, + "jobComplete": true + }, + "profile": { + "url": "http://localhost:50030/jobdetails.jsp?jobid=job_201111111311_0008", + "user": "ctdean", + "jobID": { + "jtIdentifier": "201111111311", + "id": 8 + }, + "queueName": "default", + "jobFile": "hdfs://localhost:9000/tmp/hadoop-ctdean/mapred/staging/ctdean/.staging/job_201111111311_0008/job.xml", + "jobName": "streamjob2396312251369831644.jar", + "jobId": "job_201111111311_0008" + } +} + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/tabs.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/tabs.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/tabs.xml (revision 0) @@ -0,0 +1,35 @@ + + + + + + + + Index: contrib/templeton/src/docs/src/documentation/content/xdocs/example.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/example.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/example.xml (revision 0) @@ -0,0 +1,86 @@ + + + + + +
+ Example +
+ +

The following example, extracted from the HCatalog documentation, shows how people + might use HCatalog along with various other Hadoop tools to move data from the grid + into a database and ultimately analyze it.

+ +

Without Templeton there are three main steps to completing + the task.

+ +

First, Joe in data acquisition uses distcp to get data + onto the grid.

+ + +hadoop distcp file:///file.dat hdfs://data/rawevents/20100819/data + +hcat "alter table rawevents add partition 20100819 hdfs://data/rawevents/20100819/data" + + +

Second, Sally in data processing uses Pig to cleanse and prepare the + data. Oozie will be notified by HCatalog that data is available and can then + start the Pig job

+ + +A = load 'rawevents' using HCatLoader; +B = filter A by date = '20100819' and by bot_finder(zeta) = 0; +… +store Z into 'processedevents' using HCatStorer("date=20100819"); + + +

Third, Robert in client management uses Hive to analyze his + clients' results.

+ + +insert overwrite table 20100819events +select advertiser_id, count(clicks) +from processedevents +where date = ‘20100819’ +group by adverstiser_id; + + +

With Templeton all these steps can be easily performed programatcally + upon receipt of the initial data. Sally and Robert can still maintain their own scripts + and simply push them into HDFS to be accessed when required by Templeton.

+ + +??Still need to add web hdfs push! + +>POST /v1/templeton/ddl.json?exec="alter table rawevents add partition 20100819 hdfs://data/rawevents/20100819/data" +>{"result":"ok"} +> +>POST /v1/templeton/pig.json?src="hdfs://scripts/cleanse.pig" +>{"result": "ok", "jobid": "123"} +> +>... +>GET /v1/templeton/queue/123.json +>{"result": "ok", "status" "completed"} +> +>POST /v1/templeton/hive.json?src="hdfs://scripts/analyze.hive" +>{"result": "ok", "jobid": "456"} +> + + + +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/hive.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/hive.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/hive.xml (revision 0) @@ -0,0 +1,127 @@ + + + + + +
+ POST hive +
+ + +
+ Description +

Runs a Hive query or set of commands.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/hive.json

+
+ +
+ Parameters + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequired?Default
executeString containing an entire, short hive program to run.One of either "execute" or "file" is requiredNone
fileHDFS file name of a hive program to run.One of either "exec" or "file" is requiredNone
defineSet a Hive configuration variable using the syntax + define=NAME=VALUE.OptionalNone
statusdirA directory where Templeton will write the status of the + Hive job. If provided, it is the caller's responsibility + to remove this directory when done.OptionalNone
+
+ +
+ Results + + + + + + + + + + +
NameDescription
idA string containing the job id similar to "job_201110132141_0001".
infoA JSON object containing the information returned when the job was queued. + See the Hadoop documentation + (Class + TaskController) for more information.
+
+ +
+ Example + +

Curl Command

+ +% curl -d user.name=ctdean \ + -d execute="select+*+from+pokes;" \ + -d statusdir="pokes.output" \ + 'http://www.myserver.com/templeton/v1/hive.json' + + +

JSON Output

+ +{ + "id": "job_201111111311_0005", + "info": { + "stdout": "templeton-job-id:job_201111111311_0005 + ", + "stderr": "", + "exitcode": 0 + } +} + + +

Results

+ +% hadoop fs -ls pokes.output +Found 2 items +-rw-r--r-- 1 ctdean supergroup 610 2011-11-11 13:22 /user/ctdean/pokes.output/stderr +-rw-r--r-- 1 ctdean supergroup 15 2011-11-11 13:22 /user/ctdean/pokes.output/stdout + +% hadoop fs -cat pokes.output/stdout +1 a +2 bb +3 ccc + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/installation.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/installation.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/installation.xml (revision 0) @@ -0,0 +1,198 @@ + + + + + +
+ Installation +
+ + +
+ Introduction +

Templeton is deep in the middle of development and does not yet have a smooth + install procedure. As such, this version of Templeton should only be installed expert + developers.

+
+ +
+ Requirements +
    +
  • Ant, version 1.8 or higher
  • +
  • Tomcat, version 7.0 or higher
  • +
  • Hadoop, version 0.20.205.0
  • +
  • HCatalog, +
      +
    • To use the ddl resource, HCatalog must be installed in + /usr/local/hcat/bin/hcat
    • +
    • To use the hive resource, see instructions + below + for placing HCatalog into the Hadoop distributed cache.
    • +
    +
  • +
  • Pig, see instructions + below + for placing Pig into the Hadoop distributed cache.
  • +
+
+ +
+ Environment variables + + + + + + + + + + + + + + + + + + + +
VariableValue
HADOOP_PREFIXSet to the location of your Hadoop install
HCAT_PREFIXSet to the location of your HCatalog install
CATALINA_HOMESet to the location of your Tomcat install
TEMPLETON_HOMESet to the Templeton directory
TEMPLETON_JARSet to the location of the Templeton jar file. + After a build the location will be + $TEMPLETON_HOME/build/templeton/templeton-0.1.0.jar.
+
+ +
+ Procedure +
    +
  1. Ensure that the required related installations and environment + variables are in place.
  2. +
  3. Start Tomcat.
  4. +
  5. Build and install the Templeton war file using ant install-war + from $TEMPLETON_HOME. The Tomcat webapps directory must be writable.
  6. +
  7. Check that your local install works. Assuming that Tomcat is running + on port 8080, the following command would give output similar to that shown. + +% curl -i http://localhost:8080/templeton/v1/status.json +HTTP/1.1 200 OK +Server: Apache-Coyote/1.1 +Content-Type: application/json +Transfer-Encoding: chunked +Date: Tue, 18 Oct 2011 19:35:17 GMT + +{"status": "ok", "version": "v1"} +% +
  8. +
+
+ +
+ Authentication +

To run any of the authenticated API calls, sudo must be setup properly.

+

For Development

+
    +
  1. Run Tomcat as yourself.
  2. +
  3. Make sure that you can sudo as yourself. For example, + sudo -u $USER date
  4. +
+

For Production

+
    +
  1. Create user runner with permission to run the required commands. In the sudoers file: + +runner ALL=(%hadoop) NOPASSWD: command1[,command2, ...] + + allows runner to run the commands if they are owned by + a user in the group hadoop. +
  2. +
  3. Run Tomcat as runner.
  4. +
+
+ +
+ Hadoop Distributed Cache +

To avoid the installation of Pig and Hive everywhere on the cluster, + Templeton gathers a version of Pig or Hive from the + + Hadoop distributed cache whenever those resources are invoked. To install the + required components in the locations specificed by the default configuration, + follow these steps:

+ +
    +
  1. Create hive.tar. If you installed HCatalog as noted above, this + this is done as follows: + +cd /usr/local/hcat +tar cf /tmp/hive.tar + +
  2. +
  3. Place hive.tar into the cache. + +hadoop fs -put /tmp/hive.tar /user/templeton/hive.tar + +
  4. +
  5. Install Pig locally, for example in /usr/local/pig.
  6. +
  7. Create pig.tar: + +cd /usr/local/pig +tar cf /tmp/pig.tar + +
  8. +
  9. Place pig.tar into the cache. + +hadoop fs -put /tmp/pig.tar /user/templeton/pig.tar + +
  10. +
+ +

The location of these archives (tar files) in the cache, and the location + of the installations inside the archives can be specified using the following + Templeton configuration variables. (See the + Configuration documentation for more information + on changing Templeton configuration parameters.)

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDefaultDescription
templeton.pig.archivehdfs:///user/templeton/pig.tarThe path to the Pig archive.
templeton.pig.pathpig.tar/bin/pigThe path to the Pig executable.
templeton.hive.archivehdfs:///user/templeton/hive.tarThe path to the Hive archive.
templeton.hive.pathhive.tar/bin/hiveThe path to the Hive executable.
+
+ + +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducejar.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducejar.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducejar.xml (revision 0) @@ -0,0 +1,148 @@ + + + + + +
+ POST mapreduce/jar +
+ + + +
+ Description +

Creates and queues a standard + + Hadoop MapReduce job.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/mapreduce/jar.json

+
+ +
+ Parameters + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequired?Default
jarName of the jar file for Map Reduce to use.RequiredNone
className of the class for Map Reduce to use.RequiredNone
libjarsComma seperated jar files to include in the classpath.OptionalNone
filesComma separated files to be copied to the map reduce clusterOptionalNone
argSet a program argument.OptionalNone
defineSet an Hadoop configuration variable using the syntax + define=NAME=VALUEOptionalNone
statusdirA directory where Templeton will write the status of the + Map Reduce job. If provided, it is the caller's responsibility + to remove this directory when done.OptionalNone
+
+ +
+ Results + + + + + + + + + + +
NameDescription
idA string containing the job id similar to "job_201110132141_0001".
infoA JSON object containing the information returned when the job was queued. + See the Hadoop documentation + (Class + TaskController) for more information.
+
+ +
+ Example + +

Code and Data Setup

+ +% hadoop fs -put wordcount.jar . +% hadoop fs -put transform.jar . + +% hadoop fs -ls . +Found 2 items +-rw-r--r-- 1 ctdean supergroup 23 2011-11-11 13:29 /user/ctdean/wordcount.jar +-rw-r--r-- 1 ctdean supergroup 28 2011-11-11 13:29 /user/ctdean/transform.jar + + +

Curl Command

+ +% curl -d user.name=ctdean \ + -d jar=wordcount.jar \ + -d class=org.myorg.WordCount \ + -d libjars=transform.jar \ + -d arg=wordcount/input \ + -d arg=wordcount/output \ + 'http://www.myserver.com/templeton/v1/mapreduce/jar.json' + + +

JSON Output

+ +{ + "id": "job_201111121211_0001", + "info": { + "stdout": "templeton-job-id:job_201111121211_0001 + ", + "stderr": "", + "exitcode": 0 + } +} + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/ddl.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/ddl.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/ddl.xml (revision 0) @@ -0,0 +1,114 @@ + + + + + +
+ POST ddl +
+ + +
+ Description +

Performs an + HCatalog DDL command. The command is executed immediately upon request. + Responses are limited to 1MB. For requests which may return longer results + consider using the Hive resoure as an alternative.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/ddl.json

+
+ +
+ Parameters + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequired?Default
execThe HCatalog ddl string to executeRequiredNone
groupThe user group to use when creating a tableOptionalNone
permissionsThe permissions string to use when creating a table. The format is + "rwxrw-r-x".OptionalNone
+
+ +
+ Results + + + + + + + + + + + + + + +
NameDescription
stdoutA string containing the result HCatalog sent to standard out (possibly empty).
stderrA string containing the result HCatalog sent to standard error + (possibly empty).
exitcodeThe exitcode HCatalog returned.
+
+ +
+ Example + +

Curl Command

+ +% curl -d user.name=ctdean \ + -d 'exec=show tables;' \ + 'http://www.myserver.com/templeton/v1/ddl.json' + + +

JSON Output

+ +{ + "stdout": "important_table + my_other_table + my_table + my_table_2 + pokes + ", + "stderr": "WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated... + Hive history file=/tmp/ctdean/hive_job_log_ctdean_201111111258_2117356679.txt + OK + Time taken: 1.202 seconds + ", + "exitcode": 0 +} + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/site.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/site.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/site.xml (revision 0) @@ -0,0 +1,62 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/build-with-forrest-button.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/build-with-forrest-button.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/templetonlogov2.png =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/templetonlogov2.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-box.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-box.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/TempletonArch.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/TempletonArch.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/Templetontitle.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/Templetontitle.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/templetonlogo.png =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/templetonlogo.png ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-product.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-product.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-archt.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat-archt.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat.jpg =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Property changes on: contrib/templeton/src/docs/src/documentation/content/xdocs/images/hcat.jpg ___________________________________________________________________ Added: svn:mime-type + application/octet-stream Index: contrib/templeton/src/docs/src/documentation/content/xdocs/configuration.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/configuration.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/configuration.xml (revision 0) @@ -0,0 +1,161 @@ + + + + + +
+ Configuration +
+ +

The configuration for Templeton merges the normal Hadoop configuration with + the Templeton specific variables.

+ +

The Templeton specific configuration is split into three layers:

+ +
    +
  1. templeton-default.xml - All the configuration variables + that Templeton needs. This file sets the defaults that ship with Templeton and + should only be changed by the Templeton developers.
  2. + +
  3. templeton-dist.xml - The (possibly empty) configuration + file that can set variables for a particular distribution, such as an RPM file.
  4. + +
  5. templeton-site.xml - The (possibly empty) configuration + file in which the system administrator can set variables for their Hadoop cluster.
  6. +
+ +

The configuration files are loaded in this order with later files overriding + earlier ones.

+ +

To find the configuration files, Templeton first attempts to load a file from the + CLASSPATH and then looks in the directory specified in the + TEMPLETON_HOME environment variable.

+ +

In addition the configuration files may access the special environment variable + env for all environment variables. For example, the pig executable + could be specified using:

+ + +${env.PIG_HOME}/bin/pig + + +
+ Variables + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDefaultDescription
templeton.jar${env.TEMPLETON_HOME}/build/templeton/templeton-0.1.0-dev.jarThe path to the Templeton jar file.
templeton.streaming.jar${env.HADOOP_PREFIX}/share/hadoop/contrib/streaming/hadoop-streaming-0.20.205.0.jarThe path to the Hadoop streaming jar file.
templeton.hadoop${env.HADOOP_PREFIX}/bin/hadoopThe path to the Hadoop executable.
templeton.pig.archivehdfs:///user/templeton/pig.tarThe path to the Pig archive.
templeton.pig.pathpig.tar/bin/pigThe path to the Pig executable.
templeton.hcat${env.HCAT_PREFIX}/bin/hcatThe path to the Hcatalog executable.
templeton.hive.archivehdfs:///user/templeton/hive.tarThe path to the Hive archive.
templeton.hive.pathhive.tar/bin/hiveThe path to the Hive executable.
templeton.hive.properties +hive.metastore.local=false, +hive.metastore.uris=thrift://localhost:9933, +hive.metastore.sasl.enabled=falseProperties to set when running hive.
templeton.sudo/usr/bin/sudoThe path to the sudo program.
templeton.exec.encodingUTF-8The encoding of the stdout and stderr data.
templeton.exec.timeout10000How long in milliseconds a program is allowed to run on the + Templeton box. +
templeton.exec.max-procs16The maximum number of processes allowed to run at once.
templeton.exec.max-output-bytes1048576The maximum number of bytes from stdout or stderr stored in ram.
templeton.exec.envsHADOOP_PREFIX,HADOOP_HOME,JAVA_HOMEThe environment variables passed through to sudo.
+
+ + +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/index.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/index.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/index.xml (revision 0) @@ -0,0 +1,119 @@ + + + + + +
+ Templeton +
+ + +
+ Introduction +

Templeton provides a REST-like web API for + HCatalog + and related Hadoop components. + As shown in the figure below, developers make HTTP requests to access + Hadoop MapReduce, + Pig, + Hive, and + + HCatalog DDL from within applications. + Data and code used by Templeton is maintained in + HDFS. HCatalog DDL commands + are executed directly when requested. + MapReduce, Pig, and Hive jobs are placed in queue by + Templeton and can be monitored for progress or stopped as required. + Developers specify a location + in HDFS into which Templeton should place Pig, Hive, and MapReduce results.

+
+
+ +
+ URL format +

Templeton resources are accessed using the following URL format:

+

http://yourserver/templeton/v1/resource.format

+

where "yourserver" is replaced with your server name, and + "resource.format" is replaced by the Templeton + resource name and output response format. Note that in Templeton + version 0.1.0, JSON is the only supported response format.

+

For example, to check if the Templeton server is running you could + access the following URL:

+

http://www.myserver.com/templeton/v1/status.json

+
+ +
+ Security +

The current version of Templeton supports two types of security:

+
    +
  • Default security (without additional authentication)
  • +
  • Authentication via + Kerberos
  • +
+
+ Standard Parameters +

Every Templeton resource can accept the following parameters to + aid in authentication:

+
    +
  • user.name: The user name as a string. + Only valid when using default security.
  • +
  • SPNEGO credentials: When running with Kerberos authentication.
  • +
+
+
+ +
+ WebHDFS and Code Push +

Data and code that are used by Templeton resources must first be placed in + Hadoop. The current version of Templeton does not attempt to integrate or replace + existing web interfaces which can perform this task, like Web HDFS. (Integration + of these functions in some way, perhaps forwarding, is planned for a future + release.) This guide will sometimes refer to using Web HDFS to push code or data + into Hadoop, but in all cases + you can do this in whatever manner is most convienient for you.

+
+ +
+ Error Codes and Responses +

The Templeton server returns the following HTTP status codes.

+
    +
  • 200 OK: Success!
  • +
  • 400 Bad Request: The request was invalid.
  • +
  • 401 Unauthorized: Credentials were missing or incorrect.
  • +
  • 404 Not Found: The URI requested is invalid or the + resource requested does not exist.
  • +
  • 500 Internal Server Error: We received an unexpected result.
  • +
  • 503 Busy, please retry: The server is busy.
  • +
+

Other data returned directly by Templeton is currently returned in JSON format. + JSON responses are limited to 1MB in size. Responses over this limit must be + stored into HDFS using provided options instead of being directly returned. + If an HCatalog DDL command might return results greater than 1MB, it's + suggested that a corresponding Hive request be executed instead.

+
+ +
+ Project Name +

The Templeton project is named after the a character in the award-winning + children's novel Charlotte's Web, by E. B. White. + The novel's protagonist is a pig named + Wilber. Templeton is a rat who helps Wilber by running errands + and making deliveries.

+
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/queuedelete.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/queuedelete.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/queuedelete.xml (revision 0) @@ -0,0 +1,120 @@ + + + + + +
+ DELETE queue/:jobid +
+ + +
+ Description +

Kill a job given its job ID. + Substitute ":jobid" with the job ID received when the job was created.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/queue/:jobid.json

+
+ +
+ Parameters + + + + + + + + +
NameDescriptionRequired?Default
:jobidThe job ID to delete. This is the ID received when then job + job was created.RequiredNone
+
+ +
+ Results + + + + + + + + + +
NameDescription
statusA JSON object containing the job status information. + See the Hadoop documentation + (Class + JobStatus) for more information.
profileA JSON object containing the job profile information. + See the Hadoop documentation + (Class + JobProfile) for more information. +
+
+ + +
+ Example + +

Curl Command

+ +% curl -X DELETE 'http://www.myserver.com/templeton/v1/queue/job_201111111311_0009.json?user.name=ctdean' + + +

JSON Output

+ +{ + "status": { + "startTime": 1321047216471, + "username": "ctdean", + "jobID": { + "jtIdentifier": "201111111311", + "id": 9 + }, + "jobACLs": { + }, + "schedulingInfo": "NA", + "failureInfo": "NA", + "jobId": "job_201111111311_0009", + "jobPriority": "NORMAL", + "runState": 1, + "jobComplete": false + }, + "profile": { + "url": "http://localhost:50030/jobdetails.jsp?jobid=job_201111111311_0009", + "user": "ctdean", + "jobID": { + "jtIdentifier": "201111111311", + "id": 9 + }, + "queueName": "default", + "jobFile": "hdfs://localhost:9000/tmp/hadoop-ctdean/mapred/staging/ctdean/.staging/job_201111111311_0009/job.xml", + "jobName": "streamjob3322518350676530377.jar", + "jobId": "job_201111111311_0009" + } +} + +

Note: The job is not immediately deleted, therefore the + information returned may not reflect deletion, like in our example. + Use GET queue/:jobid + to monitor the job and confirm that it is eventually deleted.

+
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/resources.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/resources.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/resources.xml (revision 0) @@ -0,0 +1,54 @@ + + + + + +
+ Templeton Resources +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ResourceDescription
statusReturns the Templeton server status.
ddlPerforms an HCatalog DDL command.
pigCreates and queues Pig jobs.
mapreduce/streamingCreates and queues Hadoop streaming MapReduce jobs.
mapreduce/jarCreates and queues standard Hadoop MapReduce jobs.
hiveRuns Hive queries and commands.
GET queue/:jobidReturns the status of a job given its ID.
DELETE queue/:jobidKill a job given its ID.
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/pig.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/pig.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/pig.xml (revision 0) @@ -0,0 +1,136 @@ + + + + + +
+ POST pig +
+ + +
+ Description +

Create and queue a Pig job.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/pig.json

+
+ +
+ Parameters + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequired?Default
executeString containing an entire, short pig program to run.One of either "execcute" or "file" is requiredNone
fileHDFS file name of a pig program to run.One of either "exec" or "file" is requiredNone
argSet a program argument.OptionalNone
filesComma separated files to be copied to the map reduce clusterOptionalNone
statusdirA directory where Templeton will write the status of the + Pig job. If provided, it is the caller's responsibility + to remove this directory when done.OptionalNone
+
+ +
+ Results + + + + + + + + + + +
NameDescription
idA string containing the job id similar to "job_201110132141_0001".
infoA JSON object containing the information returned when the job was queued. + See the Hadoop documentation + (Class + TaskController) for more information.
+
+ +
+ Example + +

Code and Data Setup

+ +% cat id.pig +A = load 'passwd' using PigStorage(':'); +B = foreach A generate $0 as id; +dump B; + +% cat fake-passwd +ctdean:Chris Dean:secret +pauls:Paul Stolorz:good +carmas:Carlos Armas:evil +dra:Deirdre McClure:marvelous + +% hadoop fs -put id.pig . +% hadoop fs -put fake-passwd passwd + + +

Curl Command

+ +% curl -d user.name=ctdean \ + -d file=id.pig \ + -d arg=-v \ + 'http://www.myserver.com/templeton/v1/pig.json' + + +

JSON Output

+ +{ + "id": "job_201111101627_0018", + "info": { + "stdout": "templeton-job-id:job_201111101627_0018 + ", + "stderr": "", + "exitcode": 0 + } +} + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducestreaming.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducestreaming.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/xdocs/mapreducestreaming.xml (revision 0) @@ -0,0 +1,173 @@ + + + + + +
+ POST mapreduce/streaming +
+ + + +
+ Description +

Create and queue an + Hadoop + streaming MapReduce job.

+
+ +
+ URL +

http://www.myserver.com/templeton/v1/mapreduce/streaming.json

+
+ +
+ Parameters + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameDescriptionRequired?Default
inputLocation of the input data in Hadoop.RequiredNone
outputLocation in which to store the output data. If not specified, + Templeton will store the output in a location that can be discovered + using the queue resource.OptionalSee description
mapperLocation of the mapper program in Hadoop.RequiredNone
reducerLocation of the reducer program in Hadoop.RequiredNone
fileAdd an HDFS file to the distributed cache.OptionalNone
defineSet an Hadoop configuration variable using the syntax + define=NAME=VALUEOptionalNone
cmdenvSet an environment variable using the syntax + cmdenv=NAME=VALUEOptionalNone
argSet a program argument.OptionalNone
+
+ +
+ Results + + + + + + + + + + +
NameDescription
idA string containing the job id similar to "job_201110132141_0001".
infoA JSON object containing the information returned when the job was queued. + See the Hadoop documentation + (Class + TaskController) for more information.
+
+ +
+ Example + +

Code and Data Setup

+ +% cat mydata/file01 mydata/file02 +Hello World Bye World +Hello Hadoop Goodbye Hadoop + +% hadoop fs -put mydata/ . + +% hadoop fs -ls mydata +Found 2 items +-rw-r--r-- 1 ctdean supergroup 23 2011-11-11 13:29 /user/ctdean/mydata/file01 +-rw-r--r-- 1 ctdean supergroup 28 2011-11-11 13:29 /user/ctdean/mydata/file02 + + +

Curl Command

+ +% curl -d user.name=ctdean \ + -d input=mydata \ + -d output=mycounts \ + -d mapper=/bin/cat \ + -d reducer="/usr/bin/wc -w" \ + 'http://www.myserver.com/templeton/v1/mapreduce/streaming.json' + + +

JSON Output

+ +{ + "id": "job_201111111311_0008", + "info": { + "stdout": "packageJobJar: [] [/Users/ctdean/var/hadoop/hadoop-0.20.205.0/share/hadoop/contrib/streaming/hadoop-streaming-0.20.205.0.jar... + templeton-job-id:job_201111111311_0008 + ", + "stderr": "11/11/11 13:26:43 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments + 11/11/11 13:26:43 INFO mapred.FileInputFormat: Total input paths to process : 2 + ", + "exitcode": 0 + } +} + + +

Results

+ +% hadoop fs -ls mycounts +Found 3 items +-rw-r--r-- 1 ctdean supergroup 0 2011-11-11 13:27 /user/ctdean/mycounts/_SUCCESS +drwxr-xr-x - ctdean supergroup 0 2011-11-11 13:26 /user/ctdean/mycounts/_logs +-rw-r--r-- 1 ctdean supergroup 10 2011-11-11 13:27 /user/ctdean/mycounts/part-00000 + +% hadoop fs -cat mycounts/part-00000 + 8 + +
+ +
Index: contrib/templeton/src/docs/src/documentation/content/locationmap.xml =================================================================== --- contrib/templeton/src/docs/src/documentation/content/locationmap.xml (revision 0) +++ contrib/templeton/src/docs/src/documentation/content/locationmap.xml (revision 0) @@ -0,0 +1,72 @@ + + + + + + + + + + + + + + + + + + + + + + Index: contrib/templeton/src/docs/src/documentation/resources/images/ellipse-2.svg =================================================================== --- contrib/templeton/src/docs/src/documentation/resources/images/ellipse-2.svg (revision 0) +++ contrib/templeton/src/docs/src/documentation/resources/images/ellipse-2.svg (revision 0) @@ -0,0 +1,30 @@ + + + + + Ellipse + + + Index: contrib/templeton/src/docs/src/documentation/README.txt =================================================================== --- contrib/templeton/src/docs/src/documentation/README.txt (revision 0) +++ contrib/templeton/src/docs/src/documentation/README.txt (revision 0) @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +This is the base documentation directory. + +skinconf.xml # This file customizes Forrest for your project. In it, you + # tell forrest the project name, logo, copyright info, etc + +sitemap.xmap # Optional. This sitemap is consulted before all core sitemaps. + # See http://forrest.apache.org/docs/project-sitemap.html Index: contrib/templeton/src/docs/src/documentation/classes/CatalogManager.properties =================================================================== --- contrib/templeton/src/docs/src/documentation/classes/CatalogManager.properties (revision 0) +++ contrib/templeton/src/docs/src/documentation/classes/CatalogManager.properties (revision 0) @@ -0,0 +1,62 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#======================================================================= +# CatalogManager.properties for Catalog Entity Resolver. +# +# This is the default properties file for your project. +# This facilitates local configuration of application-specific catalogs. +# If you have defined any local catalogs, then they will be loaded +# before Forrest's core catalogs. +# +# See the Apache Forrest documentation: +# http://forrest.apache.org/docs/your-project.html +# http://forrest.apache.org/docs/validation.html + +# verbosity: +# The level of messages for status/debug (messages go to standard output). +# The setting here is for your own local catalogs. +# The verbosity of Forrest's core catalogs is controlled via +# main/webapp/WEB-INF/cocoon.xconf +# +# The following messages are provided ... +# 0 = none +# 1 = ? (... not sure yet) +# 2 = 1+, Loading catalog, Resolved public, Resolved system +# 3 = 2+, Catalog does not exist, resolvePublic, resolveSystem +# 10 = 3+, List all catalog entries when loading a catalog +# (Cocoon also logs the "Resolved public" messages.) +verbosity=1 + +# catalogs ... list of additional catalogs to load +# (Note that Apache Forrest will automatically load its own default catalog +# from main/webapp/resources/schema/catalog.xcat) +# Use either full pathnames or relative pathnames. +# pathname separator is always semi-colon (;) regardless of operating system +# directory separator is always slash (/) regardless of operating system +# The project catalog is expected to be at ../resources/schema/catalog.xcat +#catalogs=../resources/schema/catalog.xcat +# FIXME: Workaround FOR-548 "project DTD catalogs are not included +# when running as a servlet WAR". +# Same catalog, different path +catalogs=../resources/schema/catalog.xcat;../../project/src/documentation/resources/schema/catalog.xcat + +# relative-catalogs +# If false, relative catalog URIs are made absolute with respect to the +# base URI of the CatalogManager.properties file. This setting only +# applies to catalog URIs obtained from the catalogs property in the +# CatalogManager.properties file +# Example: relative-catalogs=[yes|no] +relative-catalogs=no Index: contrib/templeton/src/docs/forrest.properties.xml =================================================================== --- contrib/templeton/src/docs/forrest.properties.xml (revision 0) +++ contrib/templeton/src/docs/forrest.properties.xml (revision 0) @@ -0,0 +1,29 @@ + + + + + + + + + + Index: contrib/templeton/src/resources/templeton-default.xml =================================================================== --- contrib/templeton/src/resources/templeton-default.xml (revision 0) +++ contrib/templeton/src/resources/templeton-default.xml (revision 0) @@ -0,0 +1,133 @@ + + + + + + + + + + + templeton.jar + ${env.TEMPLETON_HOME}/build/templeton/templeton-0.1.0-dev.jar + The path to the Templeton jar file. + + + + templeton.streaming.jar + ${env.HADOOP_PREFIX}/share/hadoop/contrib/streaming/hadoop-streaming-0.20.205.0.jar + The path to the Hadoop streaming jar file. + + + + templeton.hadoop + ${env.HADOOP_PREFIX}/bin/hadoop + The path to the Hadoop executable. + + + + templeton.pig.archive + hdfs:///user/templeton/pig.tar + The path to the Pig archive. + + + + templeton.pig.path + pig.tar/bin/pig + The path to the Pig executable. + + + + templeton.hcat + ${env.HCAT_PREFIX}/bin/hcat + The path to the hcatalog executable. + + + + templeton.hive.archive + hdfs:///user/templeton/hive.tar + The path to the Hive archive. + + + + templeton.hive.path + hive.tar/bin/hive + The path to the Hive executable. + + + + templeton.hive.properties + hive.metastore.local=false,hive.metastore.uris=thrift://localhost:9933,hive.metastore.sasl.enabled=false + Properties to set when running hive. + + + + templeton.sudo + /usr/bin/sudo + The path to the sudo program. + + + + templeton.exec.encoding + UTF-8 + The encoding of the stdout and stderr data. + + + + templeton.exec.timeout + 10000 + + How long in milliseconds a program is allowed to run on the + Templeton box. + + + + + templeton.exec.max-procs + 16 + The maximum number of processes allowed to run at once. + + + + templeton.exec.max-output-bytes + 1048576 + + The maximum number of bytes from stdout or stderr stored in ram. + + + + + templeton.exec.envs + HADOOP_PREFIX,HADOOP_HOME,JAVA_HOME + The environment variables passed through to sudo. + + + + templeton.zookeeper.hosts + 127.0.0.1:2181 + ZooKeeper servers, as comma separated host:port pairs + + + + templeton.zookeeper.session-timeout + 30000 + ZooKeeper session timeout in milliseconds + + + Index: contrib/templeton/src/webapp/WEB-INF/web.xml =================================================================== --- contrib/templeton/src/webapp/WEB-INF/web.xml (revision 0) +++ contrib/templeton/src/webapp/WEB-INF/web.xml (revision 0) @@ -0,0 +1,55 @@ + + + + + Templeton Web API + + The Templeton Web API Server + + + + templeton-server + com.sun.jersey.spi.container.servlet.ServletContainer + + com.sun.jersey.config.property.packages + org.apache.hcatalog.templeton + + + com.sun.jersey.api.json.POJOMappingFeature + true + + 1 + + + + templeton-server + / + + + + authFilter + org.apache.hadoop.hdfs.web.AuthFilter + + + + authFilter + * + + + Index: contrib/templeton/bin/templeton-config.sh =================================================================== --- contrib/templeton/bin/templeton-config.sh (revision 0) +++ contrib/templeton/bin/templeton-config.sh (revision 0) @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Empty config file Index: contrib/templeton/bin/templeton_server.sh =================================================================== --- contrib/templeton/bin/templeton_server.sh (revision 0) +++ contrib/templeton/bin/templeton_server.sh (revision 0) @@ -0,0 +1,19 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +echo "Start the server!" + Index: contrib/templeton/etc/testing/Rakefile =================================================================== --- contrib/templeton/etc/testing/Rakefile (revision 0) +++ contrib/templeton/etc/testing/Rakefile (revision 0) @@ -0,0 +1,114 @@ +# +# Rakefile - Simple script to run through Templeton calls +# +# Chris Dean + +require 'cgi' + +RakeFileUtils.verbose(true) + +SERVER = "http://localhost:8080/templeton/v1" +USERNAME = `whoami`.chomp +USER = ["-d", "user.name=#{USERNAME}"] + +TESTS = [:status, :ddl, :streaming, :jar, :pig, :hive, + :queue, :queue_delete] + +task :default do + puts "Use rake test to run all the tests" +end + +desc "Run all the tests" +task :test do + TESTS.each do |t| + Rake::Task[t].invoke + puts "# ========================" + end +end + +desc "Run status.json" +task :status do + curl "status.json" +end + +desc "Run ddl.json" +task :ddl do + curl("ddl.json", + *USER, + "-d", "exec=show tables;") +end + +desc "Run mapreduce/streaming.json" +task :streaming do + stamp = `date '+%H%M%S'`.chomp + curl("mapreduce/streaming.json", + *USER, + "-d", "input=mydata", + "-d", "output=myoutput-#{stamp}", + "-d", "mapper=/bin/cat", + "-d", "reducer=/usr/bin/wc -w") +end + +desc "Run mapreduce/jar.json" +task :jar do + stamp = `date '+%H%M%S'`.chomp + curl("mapreduce/jar.json", + *USER, + "-d", "jar=wordcount.jar", + "-d", "class=org.myorg.WordCount", + "-d", "libjars=transform.jar,wordcount.jar", + "-d", "statusdir=wordcount/status-#{stamp}", + "-d", "arg=wordcount/input", + "-d", "arg=wordcount/output-#{stamp}") +end + +desc "Run pig.json" +task :pig do + script = < false} +end + +def quote(s) + if s.match(/[\s&?]/) + "'" + s + "'" + else + s + end +end Index: contrib/templeton/README.txt =================================================================== --- contrib/templeton/README.txt (revision 0) +++ contrib/templeton/README.txt (revision 0) @@ -0,0 +1,81 @@ +Templeton +========= + +Templeton is a webapi on top of HCatalog and other Hadoop services. + +Install +======= + +Templeton is deep in the middle of development and doesn't have a +smooth install procedure. As such, it should only be installed expert +developers. + +0. Requirements + - ant >= 1.8 + - tomcat >= 7.0 + - Hadoop == 0.20.205.0 + - Hcatalog + * To run the dll api call hcat must be installed in + "/usr/local/hcat/bin/hcat". (!) + * To run hive commands, hcat should additionally be installed + in the Hadoop distributed cache. (see below) + - Pig + * To run pig commands, pig should be installed in the + Hadoop distributed cache. (see below) + +1. Setup these environment variables: + + | Var | Value | + |----------------+---------------------------------------------------------| + | HADOOP_PREFIX | Set to your Hadoop install | + | CATALINA_HOME | Set to your Tomcat install | + | TEMPLETON_HOME | Set to the Templeton directory | + | TEMPLETON_JAR | The location of the Templeton jar file. | + | | After a build the location will be | + | | $TEMPLETON_HOME/build/templeton/templeton-0.1.0-dev.jar | + +2. Start tomcat, making sure that the above environment vars are set + for that process. + +3. Build and install the Templeton war file using "ant install-war" + from $TEMPLETON_HOME. The Tomcat webapps dir must be writable. + +4. Check that your local install works. Assuming that tomcat is + running on port 8080 on the local box, use: + + curl -i http://localhost:8080/templeton/v1/status.json + + A typical output is: + + HTTP/1.1 200 OK + Server: Apache-Coyote/1.1 + Content-Type: application/json + Transfer-Encoding: chunked + Date: Tue, 18 Oct 2011 19:35:17 GMT + + {"status": "ok", "version": "v1"} + +5. To run any of the authenticated api calls, sudo must be setup + properly. For development, you can follow these simple steps: + + - Run tomcat as yourself. + - Make sure that you can sudo as yourself. For example, + sudo -u $USER date + +6. To install pig and hive into the distributed cache: + + - Create a tar of the hcat installation (called hive.tar) and + place it into Hadoop + + cd /usr/local/hcat + tar cf /tmp/hive.tar + hadoop fs -put /tmp/hive.tar /user/templeton/hive.tar + + - Install pig locally, create a tar of the pig installation + and place it into Hadoop + + cd /usr/local/pig + tar cf /tmp/pig.tar + hadoop fs -put /tmp/pig.tar /user/templeton/pig.tar + + Index: contrib/templeton/build.xml =================================================================== --- contrib/templeton/build.xml (revision 0) +++ contrib/templeton/build.xml (revision 0) @@ -0,0 +1,351 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + You need Apache Ivy 2.0 or later from http://ant.apache.org/ + It could not be loaded from ${ivy_repo_url} + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Tests failed! + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +