MapReduce Scheduler Configuration

MapReduce v1 is configured by default to use the FairScheduler, (in both CDH 4 and CDH 5). Through Cloudera Manager you can change scheduler type (FIFO and Capacity Scheduler are supported). You can also modify the configuration for Fair Scheduler and Capacity Scheduler.

You can modify the task scheduler configuration from the JobTracker Role Group:

  1. From the MapReduce service page, under the Configuration tab, select View and Edit.
  2. Under the JobTracker Default Group, go to the Classes category.
  3. By default, the Task Scheduler (mapred.jobtracker.taskScheduler) is set to the FairScheduler (org.apache.hadoop.mapred.FairScheduler).

    To change this, click anywhere in the Value field — this displays the list of scheduler types. Select the one you want to use and then Save Changes.

  4. The actual configuration of the Fair Scheduler or the Capacity Scheduler (allocation, pool configuration, pre-emption and so on) can be managed under the Jobs category of the JobTracker Role Group.

    Make your changes as appropriate and Save Changes.

    You must refresh the JobTracker to have the new configuration take effect.

  5. Go to the MapReduce service page.
  6. Under the Instances tab, click the JobTracker role link to go to the JobTracker role Status page.
  7. From the JobTracker's Actions menu, click Refresh JobTracker.
  8. Confirm that you want to do this by clicking Refresh JobTracker in the confirmation pop-up.

For more information about these schedulers, see http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/FairScheduler.html or http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html.