Driven Agent Guide: Driven Agent for MapReduce

version 2.2.6

Driven Agent for MapReduce

The agent for MapReduce enables Driven to perform real-time monitoring of any application written as one or more MapReduce jobs.

For instructions on downloading and installing the Driven Agent see sections on downloading and installing the agent.

If you plan to use the agent with Apache Oozie, see the additional installation documentation in Using Driven Agent with Apache Oozie.

Running on Hadoop or YARN

When running on Apache Hadoop, the -javaagent argument must be passed down to the java executable. There are two ways to accomplish this depending on how Hadoop applications are launched.

When running a standard Hadoop job jar, one of the following commands are invoked:

  • $ hadoop jar <jar-path> <app arguments>

  • $ yarn jar <jar-path> <app arguments>

For example, to run the MapReduce PI example application using the YARN launch script:

$ yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 16 1000

So, to set the -javaagent argument on the Java executable, one of the following environment variables must be set, depending on how the application is launched:

When using the hadoop launch script:

export YARN_CLIENT_OPTS="\"-javaagent:/path/to/driven-agent-[framework]-<version>.jar=drivenURL=<url>\""

Or when using the yarn launch script:

export HADOOP_OPTS="\"-javaagent:/path/to/driven-agent-[framework]-<version>.jar=drivenURL=<url>\""
The above export statements escape additional quotes within the environment variable — using \". If setting more than one option on the Driven Agent, the semicolon (;) delimiter must be within quotes, otherwise the yarn or hadoop launch scripts may fail to run.

MapReduce Versions

The Driven Agent works for both the mapred.* and mapreduce.* APIs (sometimes known as the old and new APIs) on either Hadoop 1.x or 2.x releases.