Scheduling

Scheduling

A responsibility of YARN is scheduling user jobs. Scheduling is important because the cluster has finite resources to allocate to users’ jobs based on some policy. Without scheduling, a rogue job can hog the entire cluster and starve other jobs of resources. There’s no silver bullet when it comes to scheduling, as no one policy satisfies all possible usecases. YARN comes with a choice of schedulers and configuration knobs. The three schedulers in YARN are:

• FIFO Scheduler stands for first-in-first-out scheduler. Using this scheduler the jobs are placed in a queue and executed in the order of their submission. Though simple, the FIFO scheduler isn’t suitable for a shared cluster. A large job could deny smaller jobs from running.