Tune the performance of Apache Yarn

 

There are several ways that you can tune the performance of Apache Yarn, the resource management platform in Hadoop:

1. Increase the number of resources available: By increasing the amount of memory and CPU available to Yarn, you can improve the performance of your applications. You can do this by increasing the values of the yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores configuration parameters.
2. Increase the number of NodeManagers: Adding more NodeManagers to your cluster can improve the performance of Yarn by allowing it to schedule tasks across more resources. You can do this by installing additional NodeManagers on new machines and adding them to the cluster.
3. Enable preemption: By enabling preemption, Yarn will be able to reclaim resources from applications that are using them inefficiently and allocate them to other applications that need them. You can enable preemption by setting the yarn.resourcemanager.scheduler.monitor.enable configuration parameter to true.
4. Use Yarn's resource types feature: Yarn's resource types feature allows you to specify different types of resources, such as GPUs or FPGAs, and allocate them to applications that need them. This can improve the performance of resource-intensive applications by ensuring that they have access to the resources they need.
5. Enable short-circuit reads: Short-circuit reads allow MapReduce tasks to read data directly from local disks, rather than transferring it over the network. This can improve the performance of MapReduce jobs that read large amounts of data by

Increase the number of resources available with example

To increase the number of resources available to Yarn, you can adjust the yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores configuration parameters.

For example, to increase the amount of memory available to Yarn to 8 GB and the number of CPU cores to 4, you can set the following configuration parameters in the yarn-site.xml file:

<property>

  <name>yarn.nodemanager.resource.memory-mb</name>

  <value>8192</value>

</property>

<property>

  <name>yarn.nodemanager.resource.cpu-vcores</name>

  <value>4</value>

</property>

These values will apply to all NodeManagers in the cluster. You will need to restart the NodeManagers for the changes to take effect.

It's important to note that increasing the resources available to Yarn can improve the performance of your applications, but it can also increase the resource requirements of your cluster. You will need to carefully evaluate the specific requirements and constraints of your environment to determine the optimal values for these parameters.

Increase the number of NodeManagers with example

To increase the number of NodeManagers in your Yarn cluster, you will need to install additional NodeManagers on new machines and add them to the cluster.

Here is an example of how you might do this:

1. Install Hadoop on the new machines and configure the hadoop-env.sh and yarn-env.sh files to specify the necessary environment variables, such as the Java home directory and the location of the Hadoop configuration files.
2. Start the NodeManager daemon on the new machines by running the yarn-daemon.sh script:

$ yarn-daemon.sh start nodemanager

3. Add the new NodeManagers to the yarn-site.xml file on the ResourceManager machine, specifying the hostnames and ports of the new NodeManagers:

<property>

  <name>yarn.resourcemanager.nodes.include-path</name>

  <value>/etc/hadoop/conf/nodes</value>

</property>

# In the nodes file:

node1

node2

node3

4. Restart the ResourceManager and the NodeManagers for the changes to take effect:

$ yarn-daemon.sh restart resourcemanager

$ yarn-daemon.sh restart nodemanager

This is just one example of how you can increase the number of NodeManagers in a Yarn cluster. There are many other factors to consider, such as network configuration, security, and resource allocation. You will need to carefully evaluate the specific requirements and constraints of your environment to determine the best approach.

Enable preemption with example

To enable preemption in Yarn, you will need to set the yarn.resourcemanager.scheduler.monitor.enable configuration parameter to true.

Here is an example of how you might do this in the yarn-site.xml file:

<property>

  <name>yarn.resourcemanager.scheduler.monitor.enable</name>

  <value>true</value>

</property>

Once you have set this configuration parameter, you will need to restart the ResourceManager and NodeManagers for the changes to take effect:

$ yarn-daemon.sh restart resourcemanager

$ yarn-daemon.sh restart nodemanager

It's important to note that enabling preemption can improve the performance of Yarn by allowing it to reclaim resources from applications that are using them inefficiently and allocate them to other applications that need them. However, it can also lead to increased resource contention, since applications may be preempted to make way for other applications. You will need to carefully evaluate the specific requirements and constraints of your environment to determine whether enabling preemption is a suitable solution.

Use Yarn's resource types feature

Yarn's resource types feature allows you to specify different types of resources, such as GPUs or FPGAs, and allocate them to applications that need them. This can improve the performance of resource-intensive applications by ensuring that they have access to the resources they need.

To use Yarn's resource types feature, you will need to do the following:

1. Define the resource types in the yarn-site.xml file on the ResourceManager machine. For example:

<property>

  <name>yarn.resource-types</name>

  <value>gpu,fpga</value>

</property>

2. Configure the NodeManagers to expose the resource types that they have available. For example:

<property>

  <name>yarn.nodemanager.resource.gpu.enabled</name>

  <value>true</value>

</property>

<property>

  <name>yarn.nodemanager.resource.gpu.path</name>

  <value>/usr/local/nvidia</value>

</property>

<property>

  <name>yarn.nodemanager.resource.fpga.enabled</name>

  <value>true</value>

</property>

<property>

  <name>yarn.nodemanager.resource.fpga.path</name>

  <value>/usr/local/altera</value>

</property>

3. Update the application submission script to request the specific resource types that the application requires. For example:

# In the application submission script:

 

--resources gpu=2,fpga=1

This will request 2 GPUs and 1 FPGA for the application.

It's important to note that Yarn's resource types feature is a powerful tool for improving the performance of resource-intensive applications, but it can also add an additional layer of complexity to your cluster. You will need to carefully evaluate the specific requirements and constraints of your environment to determine whether it is a suitable solution.

Enable short-circuit reads

Short-circuit reads allow MapReduce tasks to read data directly from local disks, rather than transferring it over the network. This can improve the performance of MapReduce jobs that read large amounts of data by reducing the amount of data transfer required.

To enable short-circuit reads in Yarn, you will need to do the following:

1. Set the dfs.client.read.shortcircuit configuration parameter to true in the hdfs-site.xml file on the client machines. This will enable short-circuit reads for HDFS:

<property>

  <name>dfs.client.read.shortcircuit</name>

  <value>true</value>

</property>

2. Set the yarn.nodemanager.disk-health-checker.enable configuration parameter to false in the yarn-site.xml file on the NodeManager machines. This will disable the disk health checker, which is necessary for short-circuit reads to work:

<property>

  <name>yarn.nodemanager.disk-health-checker.enable</name>

  <value>false</value>

</property>

3. Restart the NodeManagers for the changes to take effect:

$ yarn-daemon.sh restart nodemanager

It's important to note that enabling short-circuit reads can improve the performance of MapReduce jobs that read large amounts of data, but it can also increase the load on the local disks and may not be suitable for all environments. You will need to carefully evaluate the specific requirements and constraints of your environment to determine whether it is a suitable solution.