Driven Plugin Guide: Installing the Driven Plugin

version 2.2.6

Installing the Driven Plugin

The Driven Plugin only works with Cascading 2.x and 3.x based applications.

To use the Driven Plugin, the Plugin JAR file must be downloaded and placed in the Cascading application CLASSPATH.

And the Driven API key and Driven Host must be set.


Downloading Directly

To download the Plugin to the local disk, download directly from:

Or from a terminal, run the following command:

$ wget -i

The Plugin JAR file includes all of its required dependencies, all of which are shaded to prevent any CLASSPATH errors with existing application Java library dependencies.

The Driven Plugin does not include a copy of Cascading.

Downloading Using Maven

If your project uses Maven to manage dependencies, you can optionally include the Plugin directly in your application build for testing and development purposes.

We recommend adding the Plugin to the runtime scope or another optional scope like testing.

In the <repositories> block add the following:


Add the following dependency:


Downloading Using Gradle

If your project uses Gradle to manage dependencies, you can optionally include the Plugin directly in your application build for testing and development purposes.

We recommend adding the Plugin to the runtime scope or another optional scope like testing.

Add the repository:

repositories {
  maven { url '' }

Add the dependency:

runtime group: 'driven', name: 'driven-plugin', version: '2.2.6'

Using the Driven Plugin

The Driven Plugin only needs to be added to the application CLASSPATH.

For testing and development, using Maven and Gradle is the simplest method.

For production deployments, a shared installation, not a per application, installation is recommended.

The Plugin is a fully shaded fat JAR and will not cause any CLASSPATH issues with existing libraries.

For Use During Development

The Driven Plugin is published to the Conjars Maven repository making it trivial to add the plugin directly to a project or build, aiding in testing and developing Cascading applications.

See the sections on Maven and Gradle on how to add the Plugin directly to your project for development and testing. This method, when using the testing scope, is very helpful when running Cascading applications from within test, or an IDE like IntelliJ IDEA and Eclipse.

Hadoop Jar Packaging Note

Only add the Driven Plugin to the Hadoop application job JAR for testing purposes.

For production use, the Driven Plugin should be placed on the Hadoop CLASSPATH as described below.

For testing and development, it may be included directly in the application job JAR.

When including the plugin (and other dependencies) in your own submittable job JAR for Hadoop, the best practice is to place them in a lib subdirectory of the submittable JAR.

For example, with Gradle, add this to the jar or fatJar task:

doFirst {
    into( 'lib' ) {
    from '/full/path/to/plugin.jar'
Using the embedded lib folder is a recommendation from the Apache Hadoop documentation.
Install the Plugin to Run with Cascading Local-Mode Applications

If you have an existing application that runs in Cascading local mode, you can do this by one of the following methods:

  • Set the CLASSPATH environment variable.

  • Alternatively, specify the plugin on the java command line using the -cp flag. To do this, enter a command in the following format:

$ java -cp your-app.jar:driven-plugin-2.2.6.jar

For Use in Production

So that the Plugin can be upgraded and configured from a single place on the host machine, it is recommended the Driven Plugin be downloaded and placed on the Hadoop client CLASSPATH, and not embedded within each application.

When the application starts, it checks for the Driven Plugin JAR file, or for the file in the current Java CLASSPATH. If the is found, the property will instruct Cascading to load the Plugin JAR from the given location.

The Plugin should not be placed on the slaves of the Hadoop cluster, this will have no effect.

If your Cascading applications are run via the bin/hadoop jar …​ or bin/yarn jar …​ commands on a remote production cluster, follow these instructions.

There are three ways make the Driven Plugin visible to a Cascading application so that Cascading can dynamically load it.

Add the file to the HADOOP_CONF directory.

This method allows the Plugin to be updated without updating the environment for every dependent application. To upgrade the Plugin or change any global properties or settings, only this file needs to be edited for all applications.

Simply place the in your Hadoop configuration directory located at echo $HADOOP_CONF. The HADOOP_CONF directory is automatically added to the HADOOP_CLASSPATH by the yarn and hadoop start scripts. The Plugin JAR file can be placed anywhere on the host file system.

$ echo >> ${HADOOP_CONF}/
Add the file to the HADOOP_CLASSPATH environment variable

Create or update the file by calling:

$ echo >> /path/to/

The add the directory containing file to the HADOOP_CLASSPATH. For example:

When adding a resource to the CLASSPATH, the directory containing the resource must be added, not the file itself (as you would a JAR file).
Add the Plugin directly to the HADOOP_CLASSPATH environment variable.

This method is only recommended if one of the other previous methods is not suitable for the given environment.

$ export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/driven-plugin-2.2.6.jar

The Driven API key and Driven Host Configuration

The Driven team API key and the Driven server host are required to start using the Driven Plugin.

You need to log into the Driven UI to obtain the API key -

  • On the Welcome Page, you can find your team API key

  • API key is also listed on the Teams view which can be accessed by entering the 'My Accounts' on the top right corner of any page.

For more information the usage for the Driven team API key, please see section on Driven Team API Key for details.

The Driven host is the Driven server url.

There are four ways make the Driven API key and Driven host name visible to the Driven Plugin.

  • Store the property in the file.

$ echo<driven host> >> ${HADOOP_CONF}/
$ echo<driven api key> >> ${HADOOP_CONF}/

See using the the Plugin in production section on where to place the file.

  • Set the API key and the host name as environment variables. Example:

$ export DRIVEN_SERVER_HOSTS=<driven server host>
$ export DRIVEN_API_KEY=<driven api key>
  • As a Java System property, for example:

System.setProperty( "", "<driven host>" )
System.setProperty( "", "<driven api key>" )

Or, if applicable:

$ java<driven host> <your app args>
$ java<driven api key> <your app args>
  • As a Cascading FlowConnector property, for example:

Properties properties = new Properties();

properties.setProperty( "", "<driven host>" );
properties.setProperty( "", "<driven api key>" );

FlowConnector flowConnector = new HadoopFlowConnector( properties );