top button
Flag Notify
    Connect to us
      Facebook Login
      Site Registration Why to Join

Facebook Login
Site Registration

Hadoop: mapreduce.jobtracker.expire.trackers.interval no effect

+2 votes
225 views

we want to set the heartbeat timout for a tasktracker.

If the tasktracker does not send heartbeats for 60 seconds he should be marked as lost. I found the parameter mapreduce.jobtracker.expire.trackers.interval which sounds right to me.

I defined

mapreduce.jobtracker.expire.trackers.interval 60000

in the mapred-site.xml on all servers and restarted the jobtracker and all tasktrackers.

I started a benchmark "hadoop jar hadoop-examples.jar randomwriter rand" and every tasktracker gets 2 jobs. It is a small test environment.

On one tasktracker i stopped the network. On the jobtracker i could see the "Seconds since heartbeat"
increasing. But after 60 seconds the tasktracker was still in the overview. Even in the log of the jobtracker I found nothing.

After over 600 seconds i found the message

org.apache.hadoop.mapred.JobTracker: Lost tracker .....

And the tasktracker wasn't shown any more on the jobtracker. Isn't this the right setting?

posted Dec 3, 2013 by Jagan Mishra

Share this question
Facebook Share Button Twitter Share Button Google+ Share Button LinkedIn Share Button Multiple Social Share Button
We use Cloudera 2.0.0-mr1-cdh4.2.0

1 Answer

+2 votes

I did a small test, and I a setting mapred.tasktracker.expiry.interval=60000 worked for me (TT became considered as lost after around 66 seconds).

May be the formula: mapred.tasktracker.expiry.interval + 2 * some-heartbeat-interval-that-is-3-sec-by-default? Otherwise, is the 6 sec some kind of time needed to make a decision to consider TT as lost?

answer Dec 3, 2013 by Ahmed Patel
I think I have found the reason. I looked at the job.xml and found the parameter
mapred.tasktracker.expiry.interval 6000000
and
mapreduce.jobtracker.expire.trackers.interval 30000

So I tried the deprecated parameter mapred.tasktracker.expiry.interval in my configuration and it works!

Why they write that the parameter is deprecated when the new one is not working and will be overwritten by the old one with the default value?
I tried the deprecated parameter mapred.tasktracker.expiry.interval in my configuration and it worked...
Similar Questions
+1 vote

I use Hadoop 2.2 and I want to run MapReduce web UI. So I visit following url: http://x.x.x.x:50030/jobtracker.jsp

Unable to connect Firefox can't establish a connection to the server at x.x.x.x:50030. Where am I wrong?

+1 vote

I would like to know if you have any examples or tutorials where I can learn hadoop mapReduce on mongodb in java?

+2 votes
public class MaxMinReducer extends Reducer {
int max_sum=0; 
int mean=0;
int count=0;
Text max_occured_key=new Text();
Text mean_key=new Text("Mean : ");
Text count_key=new Text("Count : ");
int min_sum=Integer.MAX_VALUE; 
Text min_occured_key=new Text();

 public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
       int sum = 0;           

       for (IntWritable value : values) {
             sum += value.get();
             count++;
       }

       if(sum < min_sum)
          {
              min_sum= sum;
              min_occured_key.set(key);        
          }     


       if(sum > max_sum) {
           max_sum = sum;
           max_occured_key.set(key);
       }          

       mean=max_sum+min_sum/count;
  }

 @Override
 protected void cleanup(Context context) throws IOException, InterruptedException {
       context.write(max_occured_key, new IntWritable(max_sum));   
       context.write(min_occured_key, new IntWritable(min_sum));   
       context.write(mean_key , new IntWritable(mean));   
       context.write(count_key , new IntWritable(count));   
 }
}

Here I am writing minimum,maximum and mean of wordcount.

My input file :

high low medium high low high low large small medium

Actual output is :

high - 3------maximum

low - 3--------maximum

large - 1------minimum

small - 1------minimum

but i am not getting above output ...can anyone please help me?

Contact Us
+91 9880187415
sales@queryhome.net
support@queryhome.net
#280, 3rd floor, 5th Main
6th Sector, HSR Layout
Bangalore-560102
Karnataka INDIA.
QUERY HOME
...