Getting Started

version 1.3.8

Getting Started

The Driven Agent is a collection JVM level agent libraries that allows monitoring of Apache Hadoop applications like Apache Hive or native MapReduce jobs within Driven.

The Agent is currently available for the following frameworks:

And is compatible with the following scheduler:

Apache Oozie

The Driven Agent is a Java agent wrapper for the Driven Plugin, and is available in two different forms, one that bundles the Driven plugin, and one that does not. See the Driven Documentation on details on the Driven Plugin.

Using the bundled agent is the quickest and simplest way to get started if the Driven Plugin has not already been installed for existing Cascading applications. In that case, the un-bundled agent can be used, which allows for the plugin to be upgraded and configured independently of the agent installation.

Downloading the Java agent

Select the agent library that is applicable to your framework.

From a terminal/shell, execute one of the following commands.

# latest Hive agent bundle
> wget -i http://files.concurrentinc.com/driven-agent/1.3/latest-driven-agent-hive-bundle.txt

# latest MapReduce agent bundle
> wget -i http://files.concurrentinc.com/driven-agent/1.3/latest-driven-agent-mr-bundle.txt

Note	The `-i` switch downloads the `latest-…-bundle.txt` file, and then downloads the link in the file, which is always the latest release for the current version of Driven.

Install the Java agent

To install the Agent:

Download the correct Driven Agent for your application type.
In your home directory, create a new directory named driven-agent.
Copy the downloaded file into the driven-agent directory.
Execute your app with the -javaagent:/path/to/driven-agent-[framework]-<version>.jar argument passed to the JVM. See framework specific instructions below.
Log in to the Driven Server to see your application’s performance information.

Note, the Agent has to be enabled via a JVM level command line switch of the form below:

java -javaagent:/path/to/driven-agent-[framework]-<version>.jar[=key1=value1;key2=value2,value3] <other java arguments>

Where [framework] denotes Hive (hive) or MapReduce (mr), and <version> is the current agent version.

Putting the agent on the runtime CLASSPATH will have no effect.

Note	To work, place the `-javaagent:/path/to/driven-agent-[framework]-<version>.jar` switch on the JVM command line before the application jar.

Configuring the Java agent

The Driven Agent accepts a number for configuration options after the path to the Driven Agent jar file.

java -javaagent:/path/to/driven-agent-[framework]-<version>.jar[=key1=value1;key2=value2,value3] <other java arguments>

Available Agent options can be printed to the console by running the Java Agent Jar with the following command:

java -jar /path/to/driven-agent-[framework]-<version>.jar

The Agent also accepts a properties file via the optionsFile option. To generate a template file with defaults, run the following command (with a dash as the only argument):

java -jar /path/to/driven-agent-[framework]-<version>.jar - > driven-agent.properties

Note	This re-directs the output to the file `driven-agent.properties` in the current directory.

Tip	The file specified by `optionsFile` will be treated relative to the Java JVM current working directory, and if not found, will be relative to the Java agent directory, unless the path is absolute.

The configuration options break down into three groups.

Note	Some options may not be available for different frameworks.

Agent Specific Options

optionsFile: file to read options values from, all values take precedence. relative to current dir, else agent jar directory
agentDisableLogging: disable all logging in agent and plugin
agentDebugLogging: enable debug logging in agent and plugin
agentExitOnlyOnJobCompletion: force jvm to remain active until the monitored jobs complete, fail, or are killed. appCompletionTimeout is not honored
agentKillJobsOnExit: on jvm exit, kill all running jobs. work will be marked as stopped if System.exit is called when detaching the client
agentLogSystemExitCalls: enable logging the stacktrace making System and Runtime exit calls. this installs a custom SecurityManager if none installed

Plugin Specific Options

drivenHosts: the server host names and ports to send data to: host1:80,host2:8080
drivenAPIKey: the API key to associate application executions with
drivenArchiveDir: the local directory to store copies of transmitted data
drivenDisabled: disable sending data to the server
drivenSuppressSliceData: disable sending slice level data and detailed low level performance visualizations, overrides server settings. this may reduce network traffic, load on any history servers, and indexing latency
drivenContinuousSliceData: enable frequent updates of slice level data before slice completion, update on completion is the default, overrides server settings. this may increase network traffic, load on any history server, and indexing latency
drivenSuppressJavaCommandData: disable sending the command line argument data, overrides server settings. this prevents sending sensitive information that may appear on the command line

Application Specific Options

appName: then name of this application. default is the jar name without version information
appVersion: then version of this application. default is parsed from the jar name
appTags: tags that should be associated with this application: cluster:prod,dept:engineering
appCompletionTimeout: timeout in millis to wait to send all completed application details
appFailedOnAnyUoWFail: if any unit-of-work fails, mark the application as failed. default is if the last unit-of-work fails the app is marked as failed
appFailedOnAnyUoWPending: if any unit-of-work has not been started, mark the application as failed. default is if the last unit-of-work fails the app is marked as failed