[HDFS-14092] Remove two-step create/append in WebHdfsFileSystem

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Do
• Affects Version/s: 3.2.0
• Component/s: webhdfs
• Assignee: Siyao Meng
• Reporter: Daniel Templeton
• Created: 20/Nov/18 17:03
• Updated: 11/Dec/18 22:50
• Resolved: 11/Dec/18 22:50

Description

Per the javadoc on the WebHdfsFileSystem.connect() method:

    /**
     * Two-step requests redirected to a DN
     *
     * Create/Append:
     * Step 1) Submit a Http request with neither auto-redirect nor data.
     * Step 2) Submit another Http request with the URL from the Location header
     * with data.
     *
     * The reason of having two-step create/append is for preventing clients to
     * send out the data before the redirect. This issue is addressed by the
     * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
     * Unfortunately, there are software library bugs (e.g. Jetty 6 http server
     * and Java 6 http client), which do not correctly implement "Expect:
     * 100-continue". The two-step create/append is a temporary workaround for
     * the software library bugs.
     *
     * Open/Checksum
     * Also implements two-step connects for other operations redirected to
     * a DN such as open and checksum
     */

We should validate that it's safe to remove the two-step process and do so. FYI, smeng.

Issue Links

• is caused by: HDFS-2540 Change WebHdfsFileSystem to two-step create/append