Sign Up

Have an account?

Sign In

Don't have account,

Forgot Password

Lost your password? Please enter your email address. You will receive a link and will create a new password via email.

Have an account?

Sorry, you do not have a permission to add a question, You must login to ask question.

Need An Account,
Search
Ask A Question
Home / Questions / Q 161
In Process
admin

admin

  • HYDERABAD, India
View Profile
  • 6
Beginner
Asked: In:

What is GC Overhead Limit Exceeded Hadoop

  • 6

Hi All,

When i tried to list of files in my hdfs folder by using hdfs fs -ls /myfolder command  i am facing GC Overhead Limit exceeded exception error in hadoop.Please help me how to solve this error.

big datagc overhead limithadoop
Share

    2 Answers

    1. This answer was edited.

      Hello Srilakshmi,
       
      GC Overhead Limit exception error will come when doing data copy from local to hdfs or hdfs to local and setup a replication for hdfs  files at that time also this GC overhead limit exceeded error will come.
      We need to setup HADOOP_CLIENT_OPTS fix this error.
      export HADOOP_CLIENT_OPTS=”-XX:-UseGCOverheadLimit -Xmx4096m”

      • HADOOP_CLIENT_OPTS – All hadoop components run as a client.
      • This was needed because all the hadoop commands run as a client.
      • 3

    2. srilakshmi

      View Profile
      Beginner

      Hello ,

      Generally GC Overhead limit exceeded error will come in below senories.

      • If the hdfs folder having more then 1 million files and if you tried to run hdfs commands like -ls,-du -h,-copyToLocal etc.

      How to solve GC Overhead limit exceeded error

      Hadoop commands we run on edgenode so increase hadoop client heap memory on fly .

      • Open hadoop edge node in putty
      • Login to your account
      • copy paste below command on your terminal to solve GC Overhead limit exceeded error in your hadoop environment.

      export HADOOP_CLIENT_OPTS=”$HADOOP_OPTS -Xmx5g”

      The above command will use full to fix GC Overhead limit exceeded error in hadoop and 5g indicates heap memory you can increase the value based on your requirement.

      • 3
    Leave an answer

    Leave an answer

    Browse
    Browse

    Choose from here the video type.

    Put here the video id : https://www.youtube.com/watch?v=sdUUx5FdySs Ex: 'sdUUx5FdySs'.

    What is the capital of Egypt? ( Cairo )