Driven Agent Guide: Configuring the Driven Agent

version 2.2.6

Driven Agent Guide: Configuring the Driven Agent

1. Prerequisites: 1.1. Check your team’s demands and expectations

1.2. Check your system privileges and network access
2. Installing the Driven Agent: 2.1. Downloading the Driven Agent

2.2. Installing the Driven Agent

2.3. Confirming Installation
3. Configuring the Driven Agent: 3.1. Testing the Agent

3.2. Agent Common Options

3.3. Agent Advanced Options
4. Driven Agent for MapReduce: 4.1. Running on Hadoop or YARN

4.2. MapReduce Versions
5. Driven Agent for Hive: 5.1. Hive Version Requirements

5.2. Metadata Support

5.3. Using the Hive Agent Artifact

5.4. What Cluster Work Is Monitored
6. Driven Agent for Apache Spark: 6.1. Spark Version Requirements

6.2. Supported APIs

6.3. Spark Runtimes

6.4. Supported Runtimes
7. Using Driven Agent with Apache Oozie: 7.1. Driven Agent Configuration
8. Advanced Installation: 8.1. Scripted Installation

8.2. Amazon Elastic MapReduce
9. Troubleshooting the Driven Agent: 9.1. Applications cannot send data to the Driven Server

Configuring the Driven Agent

The Driven Agent accepts various configuration options after the path to the Driven Agent JAR file.

-javaagent:/path/to/driven-agent-[framework]-<version>.jar[=key1=value1;key2=value2,value3]

Note	Key pairs are separated by a semi-colon (;). In some cases the `-javaagent:…` argument may need to be surrounded by quotes.

Available agent options can be printed to the console by running the Driven Agent JAR with the following command:

java -jar /path/to/driven-agent-[framework]-<version>.jar

The agent also accepts a properties file via the optionsFile option. To generate a template file with defaults, run the following command (with a dash as the only argument):

java -jar /path/to/driven-agent-[framework]-<version>.jar - > driven-agent.properties

This creates a driven-agent.properties template in the current directory.

To use this file, set:

-javaagent:/path/to/driven-agent-[framework]-<version>.jar=optionsFile=driven-agent.properties

Note	Some of the following configuration options might not be available for all frameworks.

Testing the Agent

If having issues with a specific Driven Agent, it might be useful to test the MapReduce Agent with an existing Hadoop example application.

See the section on Running On Hadoop or YARN.

Agent Common Options

drivenHosts: Specifies the server host names and ports where data is to be sent. Values should be entered in this format: host1:80,host2:8080. The http:// or https:// prefix may be placed before the host name.

Note

It is a best practice for production systems to setup multiple Driven Servers to allow for failover. Each host name should be added to drivenHosts. By default the plugin will randomize access to the servers, and iterate over all servers once before failing on a socket timeout exception. The plugin will re-attempt all hosts after a delay. See drivenMaxServerUnreachableTimeout.

drivenAPIKey: Specifies the API key that is associated with application executions.

Note	If you are using the EAP or the Hosted Trial, `drivenAPIKey` must be set in order to see your applications in Driven after logging in. This requires an account, which you can get on the Driven Trial Options website.

agentDisableLogging: Disables all Driven Agent and Driven Plugin logging.
agentDebugLogging: Enables debug logging in the Driven Agent and the Driven Plugin.
appName: Names an application. The default name is the JAR file name without version information.
appVersion: Specifies the version of an application. The default version is parsed from the JAR file name.
appTags: Assigns tags that should be associated with the application, for example: cluster:prod,dept:engineering

Agent Advanced Options

optionsFile: Specifies the file that provides option values for the Driven Agent. All values take precedence over the agent argument values. The file is relative to the current directory. If no file in current directory is found, the file is relative to the directory the Driven Agent JAR file is located.
appCompletionTimeout: Specifies timeout (in milliseconds) to wait to send all completed application details before shutdown.
appFailedOnAnyUoWFail: Indicates that if any Unit of Work fails, then the application is marked as FAILED. The default is to mark an app as FAILED only if the last Unit of Work fails.
appFailedOnAnyUoWPending: Indicates that if any Unit of Work is not started, then the application is marked as FAILED. The default is to mark an app as FAILED only if the last Unit of Work does not start.
uoWSanitizeStatements: Enable query statement sanitization. This option is enabled by default.
agentExitOnlyOnJobCompletion: Forces the JVM to remain active until the monitored jobs complete, fail, or are killed. The appCompletionTimeout option is not supported. Default is TRUE.
agentKillJobsOnExit: Kills all running jobs when JVM is exited. Work is marked as STOPPED if System.exit is called when detaching the client.
agentLogSystemExitCalls: Enables logging of the stack trace making System and Runtime exit calls. The option also installs a custom SecurityManager if no other SecurityManager has been installed.
agentLogSubmitStackTrace: Enables logging of the stack trace making submit() calls to the cluster, which helps in diagnosing the root main class and function.
drivenHostRandomizationDisabled: Disable randomizing the host names on connection attempts if set to true.
drivenBufferFlushDelay: Delay in milliseconds to wait between buffer flush calls. Default is 400ms. Note the buffer flushing is continuous until it is empty.
drivenMaxConnectionRetry: Max retry delay, in milliseconds, to wait between attempts to re-connect to the Driven Server hosts. Retry attempts will back off incrementally until this value is reached.

Note	All configured `drivenHosts` will be attempted before there is a retry delay.

drivenMaxServerUnreachableTimeout: Max duration across all server connection attempts to wait before shutting down the plugin. Resets after a successful connection. By default retry attempts will run indefinitely (-1), but will not prevent the application from shutting down.
drivenSocketTimeout: Socket timeout in milliseconds. Defaults to 7 seconds (7000 ms).
drivenConnectionTimeout: Connection timeout in milliseconds. Defaults to 5 seconds (5000 ms).
drivenBufferPath: Directory to place the disk buffer file. Must have sufficient available space and write privileges. Will be the first path to attempt unless drivenUseBufferPathOnly is true.
drivenUseBufferPathOnly: Only use the drivenBufferPath location, if unavailable, disable the agent. false by default.
drivenUseTempPath: Attempt to use the currently configured 'temp' directory as provided by the Java System properties. true by default.
drivenUseCWDPath: Attempt to use the current working directory as provided by the Java System properties. true by default.
drivenUseHomePath: Attempt to use the current user home directory as provided by the Java System properties. true by default.
drivenMaxBufferFileSize: The maximum file size that can be created by the disk buffer, in MB. Default is 100MB.
drivenLowDiskSizeWarn: The percentage of available disk space that will issue an infrequent warning in the logs. Default 5%, 0 disables messages. Do not include the % sign.
drivenLowDiskSizeError: The percentage of available disk space that will issue a frequent warning in the logs. Default 2.5%, 0 disables warnings. Do not include the % sign
drivenMinAvailableDiskSize: The minimum available disk space, in MB, that will prevent a disk/partition from being used during the path search, or disables the plugin once active on a found path. Default 200MB.

Note	Once the plugin disables itself, any disk space used will be reclaimed losing any buffered data. Applications will need to be 'marked as stopped' in the Driven UI.

drivenArchiveDir: Indicates the local directory where copies of transmitted data are to be stored.
drivenDisabled: Disables the sending of data to the Driven Server.
drivenSuppressSliceData: Disables sending slice-level data and detailed low-level performance visualizations; overrides server settings. This option can reduce network traffic, load on any history servers, and indexing latency.
drivenContinuousSliceData: Enables frequent updates of slice-level data before slice completion (update on completion is the default); overrides server settings. This option can increase network traffic, load on any history server, and indexing latency.

Note	Some platforms do not support retrieving intermediate results at this level.

drivenSuppressJavaCommandData: Disables sending command-line argument data; overrides server settings. This option prevents sending sensitive information that might appear on the command line.