Overview

Yet Another Resource Negotiator is a universal resource management system for the upper y'n

YARM architecture

image.png

Client: Submit tasks to RM, kill tasks, etc. ResourceManager: In the cluster, there is only one that provides external services at the same time, and is responsible for resource-related ApplicationMaster: each application corresponds to an AM, and AM applies for resources to RM for use on NM Start the corresponding Task. Data segmentation, and apply for resource Container from RM for each task. NodeManager: Start and execute tasks, send heartbeat information to RM, task execution, process requests from clients: submit start/monitor AM, monitor NM

Configuration file

cd app/hadoop-2.6.0-cdh5.7.0/etc/hadoop
vi mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
 </property>
</configuration>

vi yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
 </property>
</configuration>
~

Start yarn

$cd app/hadoop-2.6.0-cdh5.7.0/sbin
$./start-yarn.sh
$ jps
27500 NodeManager
27389 ResourceManage

Interface browse http://{hostname}:8088/cluster

image.png

Start job

hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount/input/wc/hello.txt/output/wc/hello/
19/04/07 07:19:44 INFO client.RMProxy: Connecting to ResourceManager at/0.0.0.0:8032
19/04/07 07:19:46 INFO input.FileInputFormat: Total input paths to process: 1
19/04/07 07:19:46 INFO mapreduce.JobSubmitter: number of splits:1

View Results

 $ hadoop fs -ls/output/wc/hello/
19/07/12 08:25:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 hadoop supergroup 0 2019-07-12 08:22/output/wc/hello/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 26 2019-07-12 08:22/output/wc/hello/part-r-00000
 $hadoop fs -text/output/wc/hello/part-r-00000