Driven Agent Guide: Driven Agent for MapReduce

version 2.2.6

Driven Agent Guide: Driven Agent for MapReduce

1. Prerequisites: 1.1. Check your team’s demands and expectations

1.2. Check your system privileges and network access
2. Installing the Driven Agent: 2.1. Downloading the Driven Agent

2.2. Installing the Driven Agent

2.3. Confirming Installation
3. Configuring the Driven Agent: 3.1. Testing the Agent

3.2. Agent Common Options

3.3. Agent Advanced Options
4. Driven Agent for MapReduce: 4.1. Running on Hadoop or YARN

4.2. MapReduce Versions
5. Driven Agent for Hive: 5.1. Hive Version Requirements

5.2. Metadata Support

5.3. Using the Hive Agent Artifact

5.4. What Cluster Work Is Monitored
6. Driven Agent for Apache Spark: 6.1. Spark Version Requirements

6.2. Supported APIs

6.3. Spark Runtimes

6.4. Supported Runtimes
7. Using Driven Agent with Apache Oozie: 7.1. Driven Agent Configuration
8. Advanced Installation: 8.1. Scripted Installation

8.2. Amazon Elastic MapReduce
9. Troubleshooting the Driven Agent: 9.1. Applications cannot send data to the Driven Server

Driven Agent for MapReduce

The agent for MapReduce enables Driven to perform real-time monitoring of any application written as one or more MapReduce jobs.

For instructions on downloading and installing the Driven Agent see sections on downloading and installing the agent.

If you plan to use the agent with Apache Oozie, see the additional installation documentation in Using Driven Agent with Apache Oozie.

Running on Hadoop or YARN

When running on Apache Hadoop, the -javaagent argument must be passed down to the java executable. There are two ways to accomplish this depending on how Hadoop applications are launched.

When running a standard Hadoop job jar, one of the following commands are invoked:

$ hadoop jar <jar-path> <app arguments>
$ yarn jar <jar-path> <app arguments>

For example, to run the MapReduce PI example application using the YARN launch script:

$ yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 16 1000

So, to set the -javaagent argument on the Java executable, one of the following environment variables must be set, depending on how the application is launched:

When using the hadoop launch script:

export YARN_CLIENT_OPTS="\"-javaagent:/path/to/driven-agent-[framework]-<version>.jar=drivenURL=<url>\""

Or when using the yarn launch script:

export HADOOP_OPTS="\"-javaagent:/path/to/driven-agent-[framework]-<version>.jar=drivenURL=<url>\""

Note	The above export statements escape additional quotes within the environment variable — using `\"`. If setting more than one option on the Driven Agent, the semicolon (;) delimiter must be within quotes, otherwise the `yarn` or `hadoop` launch scripts may fail to run.

MapReduce Versions

The Driven Agent works for both the mapred.* and mapreduce.* APIs (sometimes known as the old and new APIs) on either Hadoop 1.x or 2.x releases.