Driven Agent Guide: Installing the Driven Agent

version 2.2.6

Installing the Driven Agent

Review this chapter before any of the framework specific chapters as most configuration and installation procedures are identical.

To monitor only Cascading applications with Driven, the Driven Agent is not necessary. However, installation of the Driven Plugin JAR is required. See Driven documentation for details.

Downloading the Driven Agent

Select the agent library that is applicable to your framework.

The Driven Agent maintenance version may not match the current Driven Server maintenance version.
  • Or from a terminal/shell, execute one of the following commands.

# latest Hive agent
$ wget -i

# latest MapReduce agent
$ wget -i

# latest Spark agent
$ wget -i
The -i switch downloads the latest-driven-agent-[framework].txt file, and then downloads the link in the file, which is always the latest maintenance release for the current minor version of Driven. This is useful for automating installation of the Driven Agent on new host systems or keeping existing hosts up-to-date.

Installing the Driven Agent

Installing the Driven Agent for Java involves adding it as a javaagent (Java Programming Language Agent) on your JVM and configuring the Driven API key and Driven host parameters for it to report data to the Driven Server.

The Driven Agent for Java uses dynamic bytecode injection to instrument a JVM and it runs as a part of the JVM process.

Install the Driven Agent as the same user or administrator of the JVM. Otherwise the agent may not have the correct write permissions for the system. The agent directories must have write permission so that the Agent can update the logs and other agent files.

The following assume that Hadoop applications are being launched from:

  • the command line, via bin/yarn jar …​ or bin/hadoop jar …​

  • an interactive shell like Beeline when using Hive with a "thick" client

  • jobs that start from a long-running server like Hive Server or from an application server like Tomcat, JBoss, Spring, etc.

The Driven API key and Driven Host Configuration

The Driven team API key and the Driven server host are required to start using the Driven Agent.

You need to log into the Driven UI to obtain the API key -

  • On the Welcome Page, you can find your team API key

  • API key is also listed on the Teams view which can be accessed by entering the 'My Accounts' on the top right corner of any page.

For more information the usage for the Driven team API key, please see section on Driven Team API Key for details.

The Driven host is the Driven server url.

Using the Driven Agent

In the current console or within a bash script the command that corresponds to your framework and platform combination.

Framework Platform Installation Start Command Example / Notes

Native MapReduce

Apache Hadoop 1.x

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ hadoop jar path/to/your/app.jar

Native MapReduce

Apache Hadoop 2.x / YARN

export YARN_CLIENT_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ yarn jar path/to/your/app.jar

Hive on MapReduce

Apache Hadoop 1.x or Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

YARN_CLIENT_OPTS is not honored by Hive

Hive on Apache Tez

Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

requires Timeline/History Server running. YARN_CLIENT_OPTS is not honored by Hive

Apache Spark

Apache Hadoop 2.x / YARN - client/cluster mode, Spark standalone - client/cluster mode

Add to spark-submit

  • --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" and

  • --jars /path/to/driven-agent-spark-<version>.jar

$ spark-submit \
  --master yarn-client \
  --num-executors 3 \
  --driver-memory 4g \
  --executor-memory 2g \
  --executor-cores 1 \
  --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" \
  --jars /path/to/driven-agent-spark-<version>.jar \
  --class org.apache.spark.examples.SparkPi \
  ${SPARK_HOME}/lib/spark-examples*.jar 100
Adding the Driven Agent to the CLASSPATH has no effect.
[framework] stands for Hive (hive) or MapReduce (mr), Spark (spark), and <version> stands for the current Driven Agent version

Confirming Installation

You can confirm your application is sending data by reviewing your application logs for any ERRORs or WARNINGs.

If the logs show now apparent issues, either logging in to the Driven Server and navigating to the correct team, or by clicking on the URL, provided by the Driven Agent, in the application logs.

The current application can be viewed in real-time on the Driven Server before and after the application exits.