Driven Plugin Guide: Installing the Driven Plugin

version 2.2.6

Driven Plugin Guide: Installing the Driven Plugin

1. Prerequisites: 1.1. Check your team’s demands and expectations

1.2. Check your system privileges and network access
2. Installing the Driven Plugin: 2.1. Downloading

2.2. Using the Driven Plugin
3. Configuring: 3.1. Driven Server Hosts

3.2. API Key

3.3. Retry Rules

3.4. Disk Buffer

3.5. Default Application Tags

3.6. Application Heartbeat Frequency

3.7. Slice Data Suppression

3.8. Slice Data on Completion

3.9. Java Command Suppression

3.10. Failed Flows

3.11. Pending Flows

3.12. Control Plugin Logging

3.13. Disable Plugin

3.14. Plugin Property Reference
4. Advanced Installing and Configuring: 4.1. Scripted Installation

4.2. Integrating the Driven Plugin for Amazon Elastic MapReduce
5. Troubleshooting the Driven Agent: 5.1. Applications cannot send data to the Driven Server

5.2. Driven Agent is making the application seem slower

Installing the Driven Plugin

The Driven Plugin only works with Cascading 2.x and 3.x based applications.

To use the Driven Plugin, the Plugin JAR file must be downloaded and placed in the Cascading application CLASSPATH.

And the Driven API key and Driven Host must be set.

Downloading

Downloading Directly

To download the Plugin to the local disk, download directly from:

http://files.concurrentinc.com/driven/2.2/driven-plugin/driven-plugin-2.2.6.jar

Or from a terminal, run the following command:

$ wget -i http://files.concurrentinc.com/driven/2.2/driven-plugin/latest-jar.txt

The Plugin JAR file includes all of its required dependencies, all of which are shaded to prevent any CLASSPATH errors with existing application Java library dependencies.

Note	The Driven Plugin does not include a copy of Cascading.

Downloading Using Maven

If your project uses Maven to manage dependencies, you can optionally include the Plugin directly in your application build for testing and development purposes.

We recommend adding the Plugin to the runtime scope or another optional scope like testing.

In the <repositories> block add the following:

<repository>
  <id>conjars</id>
  <url>http://conjars.org/repo</url>
</repository>

Add the following dependency:

<dependency>
   <groupId>driven</groupId>
   <artifactId>driven-plugin</artifactId>
   <version>2.2.6</version>
   <scope>runtime</scope>
</dependency>

Downloading Using Gradle

If your project uses Gradle to manage dependencies, you can optionally include the Plugin directly in your application build for testing and development purposes.

We recommend adding the Plugin to the runtime scope or another optional scope like testing.

Add the repository:

repositories {
  maven { url 'http://conjars.org/repo' }
}

Add the dependency:

runtime group: 'driven', name: 'driven-plugin', version: '2.2.6'

Using the Driven Plugin

The Driven Plugin only needs to be added to the application CLASSPATH.

For testing and development, using Maven and Gradle is the simplest method.

For production deployments, a shared installation, not a per application, installation is recommended.

Note	The Plugin is a fully shaded fat JAR and will not cause any CLASSPATH issues with existing libraries.

For Use During Development

The Driven Plugin is published to the Conjars Maven repository making it trivial to add the plugin directly to a project or build, aiding in testing and developing Cascading applications.

See the sections on Maven and Gradle on how to add the Plugin directly to your project for development and testing. This method, when using the testing scope, is very helpful when running Cascading applications from within test, or an IDE like IntelliJ IDEA and Eclipse.

Hadoop Jar Packaging Note

Only add the Driven Plugin to the Hadoop application job JAR for testing purposes.

For production use, the Driven Plugin should be placed on the Hadoop CLASSPATH as described below.

For testing and development, it may be included directly in the application job JAR.

When including the plugin (and other dependencies) in your own submittable job JAR for Hadoop, the best practice is to place them in a lib subdirectory of the submittable JAR.

For example, with Gradle, add this to the jar or fatJar task:

doFirst {
    into( 'lib' ) {
    from '/full/path/to/plugin.jar'
  }
}

Note	Using the embedded `lib` folder is a recommendation from the Apache Hadoop documentation.

Install the Plugin to Run with Cascading Local-Mode Applications

If you have an existing application that runs in Cascading local mode, you can do this by one of the following methods:

Set the CLASSPATH environment variable.
Alternatively, specify the plugin on the java command line using the -cp flag. To do this, enter a command in the following format:

$ java -cp your-app.jar:driven-plugin-2.2.6.jar your.app.Main

For Use in Production

So that the Plugin can be upgraded and configured from a single place on the host machine, it is recommended the Driven Plugin be downloaded and placed on the Hadoop client CLASSPATH, and not embedded within each application.

When the application starts, it checks for the Driven Plugin JAR file, or for the cascading-service.properties file in the current Java CLASSPATH. If the cascading-service.properties is found, the cascading.management.service.jar property will instruct Cascading to load the Plugin JAR from the given location.

Note	The Plugin should not be placed on the slaves of the Hadoop cluster, this will have no effect.

If your Cascading applications are run via the bin/hadoop jar … or bin/yarn jar … commands on a remote production cluster, follow these instructions.

There are three ways make the Driven Plugin visible to a Cascading application so that Cascading can dynamically load it.

Add the `cascading-service.properties` file to the `HADOOP_CONF` directory.

This method allows the Plugin to be updated without updating the environment for every dependent application. To upgrade the Plugin or change any global properties or settings, only this file needs to be edited for all applications.

Simply place the cascading-service.properties in your Hadoop configuration directory located at echo $HADOOP_CONF. The HADOOP_CONF directory is automatically added to the HADOOP_CLASSPATH by the yarn and hadoop start scripts. The Plugin JAR file can be placed anywhere on the host file system.

$ echo cascading.management.service.jar=/plugin/download/driven-plugin-2.2.6.jar >> ${HADOOP_CONF}/cascading-service.properties

Add the `cascading-service.properties` file to the `HADOOP_CLASSPATH` environment variable

Create or update the cascading-service.properties file by calling:

$ echo cascading.management.service.jar=/plugin/download/driven-plugin-2.2.6.jar >> /path/to/cascading-service.properties

The add the directory containing cascading-service.properties file to the HADOOP_CLASSPATH. For example:

$ export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/

Note	When adding a resource to the `CLASSPATH`, the directory containing the resource must be added, not the file itself (as you would a JAR file).

Add the Plugin directly to the `HADOOP_CLASSPATH` environment variable.

This method is only recommended if one of the other previous methods is not suitable for the given environment.

$ export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/driven-plugin-2.2.6.jar

The Driven API key and Driven Host Configuration

The Driven team API key and the Driven server host are required to start using the Driven Plugin.

You need to log into the Driven UI to obtain the API key -

On the Welcome Page, you can find your team API key
API key is also listed on the Teams view which can be accessed by entering the 'My Accounts' on the top right corner of any page.

For more information the usage for the Driven team API key, please see section on Driven Team API Key for details.

The Driven host is the Driven server url.

The Driven host for the cloud service is https://cloud.driven.io/
For the onsite installation, the default host is http://localhost:8080

There are four ways make the Driven API key and Driven host name visible to the Driven Plugin.

Store the property in the cascading-service.properties file.

$ echo driven.management.document.service.hosts=<driven host> >> ${HADOOP_CONF}/cascading-service.properties
$ echo driven.management.document.service.apikey=<driven api key> >> ${HADOOP_CONF}/cascading-service.properties

See using the the Plugin in production section on where to place the cascading-service.properties file.

Set the API key and the host name as environment variables. Example:

$ export DRIVEN_SERVER_HOSTS=<driven server host>
$ export DRIVEN_API_KEY=<driven api key>

As a Java System property, for example:

System.setProperty( "driven.management.document.service.hosts", "<driven host>" )
System.setProperty( "driven.management.document.service.apikey", "<driven api key>" )

Or, if applicable:

$ java -Ddriven.management.document.service.hosts=<driven host> <your app args>
$ java -Ddriven.management.document.service.apikey=<driven api key> <your app args>

As a Cascading FlowConnector property, for example:

Properties properties = new Properties();

properties.setProperty( "driven.management.document.service.hosts", "<driven host>" );
properties.setProperty( "driven.management.document.service.apikey", "<driven api key>" );


FlowConnector flowConnector = new HadoopFlowConnector( properties );