[HADOOP-16837] Spark-SQL test running on Windows: hadoop chgrp warnings

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.5
• Component/s: common, fs

Environment

Windows 10

Winutils 2.7.1: https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1

Oracle JavaSE 8

SparkSQL 2.4.4 / Hadoop 2.6.5

Using: -Dhive.exec.scratchdir=C:\Users\OSUser\hadoop\tmp\hive

Set: winutils chmod -R 777 \Users\OSUser\hadoop\tmp\hive

• Assignee: Unassigned
• Reporter: Guram Savinov
• Created: 04/Feb/20 07:28
• Updated: 04/Feb/20 15:40

Description

Running SparkSQL local embedded unit tests on Win10, using winutils.

Got warnings about 'hadoop chgrp'.

See environment info.

-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...

Related info on SO: https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210

The problem is: backslash character isn't included to allowedChars, see attached HadoopGroupTest.java

Original issue in Spark: https://issues.apache.org/jira/browse/SPARK-30701

Issue Links

• Blocked: SPARK-30701 SQL test running on Windows: hadoop chgrp warnings