Generally GC Overhead limit exceeded error will come in below senories.
If the hdfs folder having more then 1 million files and if you tried to run hdfs commands like -ls,-du -h,-copyToLocal etc.
How to solve GC Overhead limit exceeded error
Hadoop commands we run on edgenode so increase hadoop client heap memory on fly .
Open hadoop edge node in putty
Login to your account
copy paste below command on your terminal to solve GC Overhead limit exceeded error in your hadoop environment.
export HADOOP_CLIENT_OPTS=”$HADOOP_OPTS -Xmx5g”
The above command will use full to fix GC Overhead limit exceeded error in hadoop and 5g indicates heap memory you can increase the value based on your requirement.
As per your requirement you want list of all external hive tables along with HDFS path location.
Database name,Table name,Table Type(External) and HDFS Location of hive External tables.
Use 3 tables TBLS,DBS and SDS tables , on top of this 3 tables we can apply joins on DB_ID and SD_ID .
select d.NAME,t_TBL_NAME,t.TBL_TYPE,s.LOCATION from TBLS t JOIN DBS d ON t.DB_ID=d.DB_ID JOIN SDS s ON t.SD_ID=s.SD_ID where t.TBL_TYPE='EXTERNAL_TABLE';
srilakshmi
Hello ,
srilakshmi
Hello Ashok,
As per your requirement you want list of all external hive tables along with HDFS path location.
Database name,Table name,Table Type(External) and HDFS Location of hive External tables.
Use 3 tables TBLS,DBS and SDS tables , on top of this 3 tables we can apply joins on DB_ID and SD_ID .
select d.NAME,t_TBL_NAME,t.TBL_TYPE,s.LOCATION from TBLS t JOIN DBS d ON t.DB_ID=d.DB_ID JOIN SDS s ON t.SD_ID=s.SD_ID where t.TBL_TYPE='EXTERNAL_TABLE';