[HBASE-13608] 413 Error with Stargate through Knox, using AD, SPNEGO, and Pre-Auth

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.12
• Fix Version/s: 0.98.13, 1.0.2, 1.2.0, 1.1.1, 2.0.0
• Component/s: REST
• Hadoop Flags: Reviewed
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 01/May/15 18:25
• Updated: 18/Dec/15 07:27
• Resolved: 01/May/15 21:32

Description

Jody Steadman reported that attempts to access Stargate via Apache Knox using curl ends with 413 error when the following conditions are met:

AD is being used directly for Kerberos
The knox principal is configured to require Kerberos pre-auth
Stargate is using SPNEGO
Pre-auth errors exist in Kerberos debug log output

sumit.gupta found that the gateway logs in the environment provided by Jody Steadman has the following error:

2015-04-20 14:24:21,660 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(273)) - >> GET /version?doAs=jsteadman HTTP/1.1
2015-04-20 14:24:21,661 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(276)) - >> Host: node1.jodylaptop.net:60080
2015-04-20 14:24:21,661 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(276)) - >> Connection: Keep-Alive
2015-04-20 14:24:21,661 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(276)) - >> User-Agent: Apache-HttpClient/4.2.5 (java 1.5)
2015-04-20 14:24:21,661 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(276)) - >> Authorization: Negotiate YIILlgYGKwYBBQUCoIILijCCC4agDTALBgkqhkiG9xIBAgKhBAMCAfaiggttBIILaWCCC2UGCSqGSIb3EgECAgEAboILVDCCC1CgAwIBBaEDAgEOogcDBQAgAAAAo4IEkGGCBIwwggSIoAMCAQWhEBsOSk9EWUxBUFRPUC5ORVSiJzAloAMCAQChHjAcGwRIVFRQGxRub2RlMS5qb2R5bGFwdG9wLm5ldKOCBEQwggRAoAMCARehAwIBA6KCBDIEggQu9ERbHVQt4WZMKK6OkjsnTtfCtSRkXUXFmrMaKugyQDQHQgHSNugwKlXQx5d1EQ9woHl5ivts6ETJBAPk3sxKqUdMssq6XVZIocnDcmSyK4xX+SnV1Hx5YXBrq0NUtMj8s7e5Avqd8LUuFIKvXkRfaG3vlHt103qdCEbGySrFchPdD7cInXhSI8mM7Ix86TCbs/Mu6PpNsoEQRZpbaBJuFWlYvuBYbu3H9vnfIX4eDMPk+mFHGOnXfg/qGORNuVn92z8JwV4ETbzyPpGgdQ61cpNQFs2bYj7ugJmAcv5JktEcmYypTzusCdUgRrNLyuIe40yKs41BYT+AytiWK6+YA4GAhepTGO8fBa9hJRXehKadHDdu0HBXm8dDhK0a9jtYkV+AKFLWq+hyLfaDoNYFAB4p17diShq7rMDgR/v65GVvIfsL+etM3Yewe85g9xtIFm86f8A/gNLRLvDLUsWoDjmmb4gzstOh6s3uPUgPE58SFL861b6JVYbEbmxCJLZe6Ni6UuNi1xXDgqBCb9duDYWYp+5n5hjy7/dxyz41CHSU/AYNg/AVXeQ3i5S5uc0ZbKiNI+ODTcNgXlHIjMXhuETHlgArbmqsVlZxbIoMdTYzHC7d54Up970SA3HbzQm0iJI/D1KlHp0Rby1wExcOw3TS/QhC/oyWoulgo9ZhtPEykOEvTYVt2USxEzfzxrmojwG27Fr30QOv1O597oQqic3TjfmyOImgp+sSgNkKSq2HTMvEelqCo7clUoll6V6zF6qwMVlXYZvyGOcgxiOvaoo2KfJPTs+VweCe2uX8K2VPVC7+VrzPj1w0K4NAOQX+p5HUfJ+fU2m/LQByNH23vzDErr8lWpxrZIh5awIKSTthIZ1IOn9fikdM1JkRKsKu4oOKyS/WquucCKG3n1l39VvkahgudFQaMUQal1Cu7X1RY8CUX8jrkahPBw4TkBp9/O3GDFiDgCw6BRGGxjqi0KuIGRdLmzUyRfqs1WFAi9WtXh7i6vQWA/vYGobWg+ONwsSmNf3/6ODbRvTXwzgzpi1r3Tsq/ptdAkoZ4El+rrV82NviDg2wNsYtsipDxCwau4RRxalioAxsIiitjl4F5pMhPYj2OJ91FXOi2ndBEY5VHx6ibxo0C+kXIHHR6cY/WuPkKKG/FFOCzf1bbhTIRJdQymq4q1Ekko8z/1x9Ehh5uKxHHRnXW2jXS5CwsJuK9qHSuZcNfa7iV1p55K6oGphWyaU3pBAI7uIlOPf8yj/P+do9AahoUzEdPSfg7MF9qAyTgodc5u0oRjVN5Hp/oNv+7OhdU6N7/eRunSKBPbEEBRN0WQFXI9VOK1lNlhc3wmORTJkiUaZFaMcNoaNPoJ6TEckJdXlVJV2oVr+rvqiry2s2M1f5lXb9MgcLMR23OVQUhlUtfGFjO930Bt+kggalMIIGoaADAgEXooIGmASCBpTibCP/GGPN1bwCd3fu8Wjb959JknZ6zrgPJfTWWRpljznACaMi+EXD3W1JrzPEgCaL6kduFTXNkHEag3igOASIOjNAp+W+fv8WJbCQXty68tbQCskvZBFYFwZHyqip/QfQQ8Y/lA88r9P8HZH2Y01ZQc0rEK0xjcHzBXqwA1UHMi7/dY6UUS78Rdt+P2QGABUcc+o9zdLYSeqlb9Gbpn2cVnYwl4G7A+j8oLmOTCH4yd7xVZ6lazeTezVs5p77+wtBq+h8COrGMa/0Ekq3wzN59X3DZrudKidPjolF4JNDj68w756neYxloGSIlF8JBYFuY54MNdEdQ94l5ekVjYXhiRulcpGUY/+VgCXcFbhKGRA/VsVGuM3S6KtBkj4keclSBnUaLHqhaSRRV456mj3lkeadd25huBrDRuZQsjObs/RPJexpNZQu6IvZLnqIQqD78lPc13vl6mrF+LHz90L0vt+caterKKqHzOunrNNqZ08jwviS6IE8xbHHqKOUn1wwG3j3GG35BDEo92WqOI9kOy7n2H/beJwFQY21O/OXQ6Vz+aeO/tHpIq2D+irNPHFZEfoFv5MXOFmSDb7TgYMNNQ9VuaJo1ncICGrlcD26DM05Q0gwgTZjoyfk1D74NsOqhKs99543mXF4OvA1KiBUikKmxItyqjIFdUoHMApEzN/Pljp30VR+bzVqfhJfYAJ3ooqUu/VR9CwSeZ0oycboHoRfMkaK6VkWFwgt9xaQM568wiOaZeZf0ejqv0gpzLN6U8tYGKNy0NEt+cA/ZrL5U8dfFns5wEh3wWajuR+RCJFxhiZB6ndXWn9g3r4QfMRV9o4saz5p5nhSQ0GXzspVbU4OsaDSjzduIaUJV1+7HwoKqTsBSrJYlUn/Qo06sIvJySx4Vwml2HLQbFKqdqtf8XUhNqq0yETuhN9aPFAPwnlIgXecAZEu3JP0QpSQQ1EmhnPWwo2WfFb0Yo1RR6f1V3wPidGagELv8DG9ezactULjVpozqoEIOo20o85DYcLWZn4EJduuJQZQ5znOyTfgq93qNHaUj1TuHrFT0Y3dUgckMSaRheR4P3oK+OKo6kIToijpbmks90KdANaBfFyqYt+48mPMQmmasFOybfUj2xziPNA7GJ7TYPrglAp0ECQMamuJtez9d9zx/5eaxI6YbAb/QBdUUeUjUX1C3BQVoyfU134stYsZX9c/dOUkZag37LHwYAuUzq7vdeFxFy5ttv0Bw2sQlDz6T8n8+GVR0I/lKHaW0jvZC/UYHE5wtzQcOnJ3rP1PKUQWL+DvCajrgQNU++/50dQou+3/CsDxdTa/vzSTCbiVCUKPPiLG0+eWya7pAqZA10RC+E60p1HlBczv8M+CH1SUZXGrm5MeLQofDIat19uopSd2HFbUwnvj0AJNn8AfE3JvH7RmDkspiUsVgiNdZnjWxWo7s/Avz7TZNcf2XQjDdK2U1ovXQqOEus37sxaEdcznrFBVchYBzy8EIG91OdTO1ZgJirh4FaG2/ZXdEpiBa6uOWFoV8C/mQhvY6MK0jbmNWRtLMASpcGZ/lS/3tWlAZoatUUZxpOxA/B1hiS6ukoDTtx0WeugieqoYEW3KpL7T8tgvkoXpsM650tGrnDHkmhPDz+sLtoPFr9kyoCX9pBNntSn+Hxqld/vHQvj7SPI5bWwes0eZs1OcfzeYOCRB4IqSeJ2Uqnnj3goI5t4gB92pDzWcT2XK6248TaLxgNhXYDoE+gBpoRGE+WhmK9rPGYSz7x6n1jHg+IYjmHDj2UGhHihR12PiuG/YDVEUKTKtZvoMblbg4zwIMMB0xW5ZpKC601qzKpvypEARjFJpvbgLRCz3Bw8z4rhUsfjlOUTlgHuMhDd4N/tAuTYpdJCF2h2hrLSnvJieX3Ltni37laJNRBHaFVA5Iikx3jWWT+ZDNHanG+VkfwtSrS8KvzFWLdGJeon5JsVb63QObIVL5DL7sTGSen7xJMUGWbWcwmc+0Z88J+8wwmNkRF2Q5Opfvqyh/9Ukewapdp7lgoJm2n8YhbsxbmU/rQhK6hadPKRxLSRrJa+J52xrG8iqqpPYkhu8JThhbkWQz19IZqQztAfTke8powJ7WqpJF89h2b7JPWLbHIpbzRfyPZDg62VjASeAUi2Sjpik2bYprNsSLkzcpH9v2WUntHMp24BDIK/rNlnX93lvUo4H3IKXrNWVTdhtzVs9WHjZ8T86oumoKkrW/JJ/eTdrGBsm
2015-04-20 14:24:21,664 DEBUG http.wire (Wire.java:wire(63)) - << "HTTP/1.1 413 FULL head[\r][\n]"
2015-04-20 14:24:21,665 DEBUG http.wire (Wire.java:wire(63)) - << "Connection: close[\r][\n]"
2015-04-20 14:24:21,666 DEBUG http.wire (Wire.java:wire(63)) - << "[\r][\n]"

Sumit suggested using large buffer size in hbase REST Server to get rid of the "FULL Head" error