Driven Plugin Guide: Advanced Installing and Configuring

version 2.2.6

Driven Plugin Guide: Advanced Installing and Configuring

1. Prerequisites: 1.1. Check your team’s demands and expectations

1.2. Check your system privileges and network access
2. Installing the Driven Plugin: 2.1. Downloading

2.2. Using the Driven Plugin
3. Configuring: 3.1. Driven Server Hosts

3.2. API Key

3.3. Retry Rules

3.4. Disk Buffer

3.5. Default Application Tags

3.6. Application Heartbeat Frequency

3.7. Slice Data Suppression

3.8. Slice Data on Completion

3.9. Java Command Suppression

3.10. Failed Flows

3.11. Pending Flows

3.12. Control Plugin Logging

3.13. Disable Plugin

3.14. Plugin Property Reference
4. Advanced Installing and Configuring: 4.1. Scripted Installation

4.2. Integrating the Driven Plugin for Amazon Elastic MapReduce
5. Troubleshooting the Driven Agent: 5.1. Applications cannot send data to the Driven Server

5.2. Driven Agent is making the application seem slower

Advanced Installing and Configuring

Advanced users may wish to script the Driven Agent installation, or use the Driven Agent with Amazon Elastic MapReduce.

Scripted Installation

For advanced users, the Driven Plugin can be installed with the following script:

# to download the script
$ wget http://files.concurrentinc.com/driven/2.2/driven-plugin/install-driven-plugin.sh

# install the plugin
$ bash install-driven-plugin.sh

Alternately, as a one-liner:

$ curl http://files.concurrentinc.com/driven/2.2/driven-plugin/install-driven-plugin.sh | sh

This script will create a .driven-plugin directory in the current user’s home directory, download the latest Driven Plugin JAR, and create a symbolic link referencing the latest versions of the driven-plugin.jar.

Re-running the script can be used to safely upgrade the agent.

Note	`driven-plugin.jar` is a unix symbolic link to the latest downloaded version of the agent jar file. This link is created or updated by the install script.

Integrating the Driven Plugin for Amazon Elastic MapReduce

If you run your applications in an Amazon Web Services Elastic MapReduce (AWS EMR) cluster, use a bootstrap action to install the plugin. The bootstrapping works with both persistent EMR clusters as well as auto-terminating clusters.

You can bootstrap by using either the AWS command-line interface (CLI) or the AWS Management Console. Bootstrapping installs the plugin on the EMR master node so that Cascading applications launched from the AWS CLI or the Management Console automatically operate with the plugin.

EMR 4.x

Amazon introduced a set of changes in EMR version 4.0, that have a direct influence on how to install the Driven plugin. One important change is that bootstrap actions can no longer modify the installation of Hadoop, since Hadoop is only deployed after all bootstrap actions have been executed.

The new way of changing the Hadoop installation with user defined settings is using the application configuration feature.

Driven provides a configuration JSON file whose usage is detailed in the sections below for command-line or AWS management console.

Amazon Web Services Command-Line Interface

The following code example shows you how to bootstrap Driven to an EMR cluster if you want to use the AWS CLI. Note the following about the code example:

The argument "--api-key,${DRIVEN_API_KEY}" appears in the following code. Using a Driven API key is optional. Omit this argument if you do not want to enable team features in Driven.
${DRIVEN_SERVER_HOST} and ${DRIVEN_API_KEY} are variables, which must be replaced with your real values for the Driven Server host URL and API key.

--bootstrap-actions Path=s3://files.concurrentinc.com/driven/2.2/driven-plugin/install-driven-plugin.sh,Args="--host,${DRIVEN_SERVER_HOST},--api-key,${DRIVEN_API_KEY}" \
--configurations http://files.concurrentinc.com/driven/2.2/driven-plugin/configurations-cascading.json

Note	The --configurations switch for the emr commandline client only supports the `file://` and `http(s)` protocols. Using `s3://` protocol like in the AWS Management Console will cause an error.

Amazon Web Services Management Console

To bootstrap the Driven Plugin using the AWS Management Console:

The AWS Management Console requires the s3 protocol for configuration files. This currently causes cross account and cross region access issues even with publicly hosted files. You have two options for using the provided configuration file at http://files.concurrentinc.com/driven/2.2/hosted/driven-plugin/configurations-cascading.json with the AWS Management Console.

Download the configuration file and upload it into your own s3 bucket. Then use the s3 url for the file in your bucket for the configuration from s3 option.
Download the configuration file, copy the provided JSON from the file, and then paste it into the JSON configuration text window.
1. Navigate to the Create Cluster window advanced options.
2. In the Software Configuration window either point to your own copy of the configuration file on s3 or paste the JSON into the JSON configuration text window.
3. Select the option to add a custom bootstrap action.
4. Enter the configuration for the new bootstrap action:
  1. Name: Driven Bootstrap Action
  2. S3 Location: s3://files.concurrentinc.com/driven/2.2/hosted/driven-plugin/install-driven-plugin.sh
  3. Arguments: --host ${DRIVEN_SERVER_HOST} --api-key ${DRIVEN_API_KEY} or just --host ${DRIVEN_SERVER_HOST} if you do not want to pass an API key. (Substitute ${DRIVEN_SERVER_HOST} and ${DRIVEN_API_KEY} with your values.)