Hadoop Common
  1. Hadoop Common
  2. HADOOP-7178

FileSystem should have an option to control the .crc file creations at Local.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When we copy the files from DFS to local, it is creating the .crc files in local filesystem for the verification of checksum. When user dont want to do any check sum verifications, this files will not be useful.

      Command line already has an option ignoreCrc, to control this.
      So, we should have the similar option on FileSystem also..... like fs.ignoreCrc().
      This should set the setVerifyChecksum to false and also should select the non CheckSumFileSystem as local fs.

      1. HADOOP-7178_COMMON.patch
        4 kB
        Uma Maheswara Rao G
      2. HADOOP-7178_HDFS.patch
        3 kB
        Uma Maheswara Rao G
      3. HADOOP-7178.patch
        4 kB
        Uma Maheswara Rao G
      4. HADOOP-7178.1.patch
        4 kB
        Uma Maheswara Rao G
      5. HADOOP-7178.2.patch
        3 kB
        Uma Maheswara Rao G
      6. HADOOP-7178.3.patch
        3 kB
        Uma Maheswara Rao G

        Issue Links

          Activity

          Uma Maheswara Rao G created issue -
          Uma Maheswara Rao G made changes -
          Field Original Value New Value
          Project Hadoop HDFS [ 12310942 ] Hadoop Common [ 12310240 ]
          Key HDFS-1733 HADOOP-7178
          Uma Maheswara Rao G made changes -
          Assignee Uma Maheswara Rao G [ umamaheswararao ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178_COMMON.patch [ 12473554 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178_HDFS.patch [ 12473555 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Uma Maheswara Rao G made changes -
          Summary copyToLocal API is creating .crc files in local, even after setting verifyChecksum to false at client side. FileSystem should have an option to control the .crc file creations at Local.
          Description When we copy the files from DFS to local, it is creating the .crc file in local filesystem for the verification of checksum even if we disable the checksum verification at client side.
                  When user does not want to do any checksum verification, then what will be the use in creating of these files in local file system.
          When we copy the files from DFS to local, it is creating the .crc files in local filesystem for the verification of checksum. When user dont want to do any check sum verifications, this files will not be useful.

          Command line already has an option ignoreCrc, to control this.
          So, we should have the similar option on FileSystem also..... like fs.ignoreCrc().
           This should set the setVerifyChecksum to false and also should select the non CheckSumFileSystem as local fs.
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178.patch [ 12488007 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178.1.patch [ 12488016 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Component/s fs [ 12310689 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178.2.patch [ 12488180 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Attachment HADOOP-7178.3.patch [ 12488398 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Uma Maheswara Rao G made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Affects Version/s 0.23.0 [ 12315569 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.23.0 [ 12315569 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Link This issue duplicates HADOOP-5179 [ HADOOP-5179 ]

            People

            • Assignee:
              Uma Maheswara Rao G
              Reporter:
              Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development