[SPARK-30701] SQL test running on Windows: hadoop chgrp warnings

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.4
• Component/s: SQL
• Labels: WIndows hive unit-test

Environment

Windows 10

Winutils 2.7.1: https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1

Oracle JavaSE 8

SparkSQL 2.4.4 / Hadoop 2.6.5

Using: -Dhive.exec.scratchdir=C:\Users\OSUser\hadoop\tmp\hive

Set: winutils chmod -R 777 \Users\OSUser\hadoop\tmp\hive

• Assignee: Unassigned
• Reporter: Guram Savinov
• Created: 01/Feb/20 08:34
• Updated: 12/Dec/22 18:10
• Resolved: 04/Feb/20 04:28

Description

Running SparkSQL local embedded unit tests on Win10, using winutils.

Got warnings about 'hadoop chgrp'.

See environment info.

-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...

Related info on SO: https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive
Seems like the problem is here: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210

Issue Links

• Blocked: HADOOP-16837 Spark-SQL test running on Windows: hadoop chgrp warnings
• is a clone of: SPARK-20625 SQL test running on Windows hadoop command warning