Hive Partition sub folders HIVE_UNION_SUBDIR_1,HIVE_UNION_SUBDIR_2,HIVE_UNION_SUBDIR_8

 Hive Partition have sub folders like HIVE_UNION_SUBDIR_1,HIVE_UNION_SUBDIR_2,HIVE_UNION_SUBDIR_3

Problem :

When you use UNION ALL in query with Hive version 1.2.0 onward. UNION ALL will not supported

and we have TEZ engine setup in hive-site.xml or hive configuration file which is responsible to create  sub folders like HIVE_UNION_SUBDIR_1 etc  on HDFS.

When you use spark sql or other sql query to read Partition data then your resultant partition created with blank or 0 value.

Solution:

Please change the configuration file of hive and set below property. Its work for me in dataiku.

hive.execution.engine=mr

It will invoke Map Reduce process which will  bit slow to process the job but it will help to stop creating the extra sub folder on HDFS.