Getting Started Guide: Driven Plugin and Agent Installation

version 2.2.6

Getting Started Guide: Driven Plugin and Agent Installation

1. Introduction: 1.1. For Driven Online usage

1.2. For Driven Onsite installation
2. Agent and Plugin Installation and Configuration: 2.1. Download the Driven Plugin or Driven Agent

2.2. Find your Driven API key

2.3. Select the Driven Host

2.4. Configure the Driven Plugin or Driven Agent
3. Server Installation and Configuration: 3.1. Download the Driven Server

3.2. Unzip the archive

3.3. Run the start up script

3.4. Verify if the server is running

3.5. Set the license key

3.6. Log in

Agent and Plugin Installation and Configuration

Download the Driven Plugin or Driven Agent

# latest Cascading plugin
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-plugin/latest-jar.txt

# latest Hive agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-hive.txt

# latest MapReduce agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-mr.txt

# latest Spark agent
$ wget -i http://files.concurrentinc.com/driven/2.2/driven-agent/latest-driven-agent-spark.txt

Find your Driven API key

By using a team API key when you run an application, Driven can collect and display a wider range of application performance information. The API also enables collaborative features, such as team-sharing of views.

You need to log into the Driven UI to obtain the API key -

On the Welcome Page, you can find your team API key
API key is also listed on the Teams view which can be accessed by entering the 'My Accounts' on the top right corner of any page.

Select the Driven Host

The Driven host for the cloud service is https://cloud.driven.io/
For the onsite installation, the default host is http://localhost:8080

Configure the Driven Plugin or Driven Agent

In the current console or within a bash script the command that corresponds to your framework and platform combination.

Framework Platform Installation Start Command Example / Notes

Framework	Platform	Installation	Start Command Example / Notes
Cascading	Apache Hadoop 1.x/YARN/Tez	Set the Driven plugin jar in the cascading-services.properties file: `echo cascading.management.service.jar=/path/to/driven-plugin-2.1.4.jar >> $HADOOP_CONF/cascading-service.properties` Set the API key: `echo cascading.management.document.service.apikey=<driven api key> >> ${HADOOP_CONF}/cascading-service.properties` Set the Driven server host: `echo cascading.management.document.service.hosts=<driven host> >> ${HADOOP_CONF}/cascading-service.properties`	Ensure that the HADOOP_CONF variable is correctly set.
Native MapReduce	Apache Hadoop 1.x	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`$ hadoop jar path/to/your/app.jar`
Native MapReduce	Apache Hadoop 2.x / YARN	`export YARN_CLIENT_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`$ yarn jar path/to/your/app.jar`
Hive on MapReduce	Apache Hadoop 1.x or Apache Hadoop 2.x / YARN	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	`YARN_CLIENT_OPTS` is not honored by Hive
Hive on Apache Tez	Apache Hadoop 2.x / YARN	`export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"`	requires Timeline/History Server running. `YARN_CLIENT_OPTS` is not honored by Hive
Apache Spark	Apache Hadoop 2.x / YARN - client/cluster mode, Spark standalone - client/cluster mode	Add to `spark-submit` `--driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"` and `--jars /path/to/driven-agent-spark-<version>.jar`	`$ spark-submit \ --master yarn-client \ --num-executors 3 \ --driver-memory 4g \ --executor-memory 2g \ --executor-cores 1 \ --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" \ --jars /path/to/driven-agent-spark-<version>.jar \ --class org.apache.spark.examples.SparkPi \ ${SPARK_HOME}/lib/spark-examples*.jar 100`

Cascading

Apache Hadoop 1.x/YARN/Tez

Set the Driven plugin jar in the cascading-services.properties file:

echo cascading.management.service.jar=/path/to/driven-plugin-2.1.4.jar >> $HADOOP_CONF/cascading-service.properties

Set the API key:

echo cascading.management.document.service.apikey=<driven api key> >> ${HADOOP_CONF}/cascading-service.properties

Set the Driven server host:

echo cascading.management.document.service.hosts=<driven host> >> ${HADOOP_CONF}/cascading-service.properties

Ensure that the HADOOP_CONF variable is correctly set.

Native MapReduce

Apache Hadoop 1.x

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ hadoop jar path/to/your/app.jar

Native MapReduce

Apache Hadoop 2.x / YARN

export YARN_CLIENT_OPTS="-javaagent:/path/to/driven-agent-mr-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

$ yarn jar path/to/your/app.jar

Hive on MapReduce

Apache Hadoop 1.x or Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

YARN_CLIENT_OPTS is not honored by Hive

Hive on Apache Tez

Apache Hadoop 2.x / YARN

export HADOOP_OPTS="-javaagent:/path/to/driven-agent-hive-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>"

requires Timeline/History Server running. YARN_CLIENT_OPTS is not honored by Hive

Apache Spark

Apache Hadoop 2.x / YARN - client/cluster mode, Spark standalone - client/cluster mode

Add to spark-submit

--driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" and
--jars /path/to/driven-agent-spark-<version>.jar

$ spark-submit \
  --master yarn-client \
  --num-executors 3 \
  --driver-memory 4g \
  --executor-memory 2g \
  --executor-cores 1 \
  --driver-java-options "-javaagent:/path/to/driven-agent-spark-<version>.jar=drivenHosts=<driven host>;drivenAPIkey=<driven api key>" \
  --jars /path/to/driven-agent-spark-<version>.jar \
  --class org.apache.spark.examples.SparkPi \
  ${SPARK_HOME}/lib/spark-examples*.jar 100