Driven Agent Guide: Installing the Driven Agent

version 2.2.6

Driven Agent Guide: Installing the Driven Agent

1. Prerequisites: 1.1. Check your team’s demands and expectations

1.2. Check your system privileges and network access
2. Installing the Driven Agent: 2.1. Downloading the Driven Agent

2.2. Installing the Driven Agent

2.3. Confirming Installation
3. Configuring the Driven Agent: 3.1. Testing the Agent

3.2. Agent Common Options

3.3. Agent Advanced Options
4. Driven Agent for MapReduce: 4.1. Running on Hadoop or YARN

4.2. MapReduce Versions
5. Driven Agent for Hive: 5.1. Hive Version Requirements

5.2. Metadata Support

5.3. Using the Hive Agent Artifact

5.4. What Cluster Work Is Monitored
6. Driven Agent for Apache Spark: 6.1. Spark Version Requirements

6.2. Supported APIs

6.3. Spark Runtimes

6.4. Supported Runtimes
7. Using Driven Agent with Apache Oozie: 7.1. Driven Agent Configuration
8. Advanced Installation: 8.1. Scripted Installation

8.2. Amazon Elastic MapReduce
9. Troubleshooting the Driven Agent: 9.1. Applications cannot send data to the Driven Server

Installing the Driven Agent

Review this chapter before any of the framework specific chapters as most configuration and installation procedures are identical.

Note	To monitor only Cascading applications with Driven, the Driven Agent is not necessary. However, installation of the Driven Plugin JAR is required. See Driven documentation for details.

Downloading the Driven Agent

Select the agent library that is applicable to your framework.

Download directly from:

Note	The Driven Agent maintenance version may not match the current Driven Server maintenance version.

Or from a terminal/shell, execute one of the following commands.

# latest Hive agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-hive.txt

# latest MapReduce agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-mr.txt

# latest Spark agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-spark.txt

Note

The -i switch downloads the latest-driven-agent-[framework].txt file, and then downloads the link in the file, which is always the latest maintenance release for the current minor version of Driven. This is useful for automating installation of the Driven Agent on new host systems or keeping existing hosts up-to-date.

Installing the Driven Agent

Installing the Driven Agent for Java involves adding it as a javaagent (Java Programming Language Agent) on your JVM and configuring the Driven API key and Driven host parameters for it to report data to the Driven Server.

The Driven Agent for Java uses dynamic bytecode injection to instrument a JVM and it runs as a part of the JVM process.

Install the Driven Agent as the same user or administrator of the JVM. Otherwise the agent may not have the correct write permissions for the system. The agent directories must have write permission so that the Agent can update the logs and other agent files.

The following assume that Hadoop applications are being launched from:

the command line, via bin/yarn jar … or bin/hadoop jar …
an interactive shell like Beeline when using Hive with a "thick" client
jobs that start from a long-running server like Hive Server or from an application server like Tomcat, JBoss, Spring, etc.

The Driven API key and Driven Host Configuration

The Driven team API key and the Driven server host are required to start using the Driven Agent.

You need to log into the Driven UI to obtain the API key -

On the Welcome Page, you can find your team API key
API key is also listed on the Teams view which can be accessed by entering the 'My Accounts' on the top right corner of any page.

For more information the usage for the Driven team API key, please see section on Driven Team API Key for details.

The Driven host is the Driven server url.

The Driven host for the cloud service is https://cloud.driven.io/
For the onsite installation, the default host is http://localhost:8080

Using the Driven Agent

In the current console or within a bash script the command that corresponds to your framework and platform combination.

Framework Platform Installation Start Command Example / Notes

Framework	Platform	Installation	Start Command Example / Notes
Native MapReduce	Apache Hadoop 1.x	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`$ hadoop jar path/to/your/app.jar`
Native MapReduce	Apache Hadoop 2.x / YARN	`export YARN_CLIENT_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`$ yarn jar path/to/your/app.jar`
Hive on MapReduce	Apache Hadoop 1.x or Apache Hadoop 2.x / YARN	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`YARN_CLIENT_OPTS` is not honored by Hive
Hive on Apache Tez	Apache Hadoop 2.x / YARN	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	requires Timeline/History Server running. `YARN_CLIENT_OPTS` is not honored by Hive
Apache Spark	Apache Hadoop 2.x / YARN - client/cluster mode, Spark standalone - client/cluster mode	Add to `spark-submit` `--driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"` and `--jars /path/to/driven-agent-spark-<version>.jar`	`$ spark-submit \ --master yarn-client \ --num-executors 3 \ --driver-memory 4g \ --executor-memory 2g \ --executor-cores 1 \ --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" \ --jars /path/to/driven-agent-spark-<version>.jar \ --class org.apache.spark.examples.SparkPi \ ${SPARK_HOME}/lib/spark-examples*.jar 100`

Native MapReduce

Apache Hadoop 1.x

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ hadoop jar path/to/your/app.jar

Native MapReduce

Apache Hadoop 2.x / YARN

export YARN_CLIENT_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ yarn jar path/to/your/app.jar

Hive on MapReduce

Apache Hadoop 1.x or Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

YARN_CLIENT_OPTS is not honored by Hive

Hive on Apache Tez

Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

requires Timeline/History Server running. YARN_CLIENT_OPTS is not honored by Hive

Apache Spark

Apache Hadoop 2.x / YARN - client/cluster mode, Spark standalone - client/cluster mode

Add to spark-submit

--driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" and
--jars /path/to/driven-agent-spark-<version>.jar

$ spark-submit \
  --master yarn-client \
  --num-executors 3 \
  --driver-memory 4g \
  --executor-memory 2g \
  --executor-cores 1 \
  --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" \
  --jars /path/to/driven-agent-spark-<version>.jar \
  --class org.apache.spark.examples.SparkPi \
  ${SPARK_HOME}/lib/spark-examples*.jar 100

Note	Adding the Driven Agent to the CLASSPATH has no effect.

Note	`[framework]` stands for Hive (`hive`) or MapReduce (`mr`), Spark (`spark`), and `<version>` stands for the current Driven Agent version

Confirming Installation

You can confirm your application is sending data by reviewing your application logs for any ERRORs or WARNINGs.

If the logs show now apparent issues, either logging in to the Driven Server and navigating to the correct team, or by clicking on the URL, provided by the Driven Agent, in the application logs.

The current application can be viewed in real-time on the Driven Server before and after the application exits.