Driven Administrator Guide

version 2.1.4

Integrating the Driven Plugin for Amazon Elastic MapReduce

If you run your applications in an Amazon Web Services Elastic MapReduce (AWS EMR) cluster, use a bootstrap action to install the plugin. The bootstrapping works with both persistent EMR clusters as well as auto-terminating clusters.

You can bootstrap by using either the AWS command-line interface (CLI) or the AWS Management Console. Bootstrapping installs the plugin on the EMR master node so that Cascading applications launched from the AWS CLI or the Management Console automatically operate with the plugin.

EMR 4.x

Amazon introduced a set of changes in EMR version 4.0, that have a direct influence on how to install the Driven plugin. One important change is that bootstrap actions can no longer modify the installation of Hadoop, since Hadoop is only deployed after all bootstrap actions have been executed.

The new way of changing the Hadoop installation with user defined settings is using the application configuration feature.

Driven provides a configuration JSON file whose usage is detailed in the sections below for command-line or AWS management console.

Amazon Web Services Command-Line Interface

The following code example shows you how to bootstrap Driven to an EMR cluster if you want to use the AWS CLI. Note the following about the code example:

The argument "--api-key,${DRIVEN_API_KEY}" appears in the following code. Using a Driven API key is optional. Omit this argument if you do not want to enable team features in Driven.
${DRIVEN_SERVER_HOST} and ${DRIVEN_API_KEY} are variables, which must be replaced with your real values for the Driven Server host URL and API key.

--bootstrap-actions Path=s3://files.concurrentinc.com/driven/2.1/driven-plugin/install-driven-plugin.sh,Args="--host,${DRIVEN_SERVER_HOST},--api-key,${DRIVEN_API_KEY}" --configurations http://files.concurrentinc.com/driven/2.1/driven-plugin/configurations-cascading.json

NOTE: The --configurations switch for the emr commandline client only supports
the `file://` and `http(s)` protocols. Using `s3://` protocol like in the AWS
Management Console will cause an error.

Amazon Web Services Management Console

To bootstrap the Driven Plugin using the AWS Management Console:

The AWS Management Console requires the s3 protocol for configuration files. This currently causes cross account and cross region access issues even with publicly hosted files. You have two options for using the provided configuration file at http://files.concurrentinc.com/driven/2.1/hosted/driven-plugin/configurations-cascading.json with the AWS Management Console.

Download the configuration file and upload it into your own s3 bucket. Then use the s3 url for the file in your bucket for the configuration from s3 option.
Download the configuration file, copy the provided JSON from the file, and then paste it into the JSON configuration text window.
1. Navigate to the Create Cluster window advanced options.
2. In the Software Configuration window either point to your own copy of the configuration file on s3 or paste the JSON into the JSON configuration text window.
3. Select the option to add a custom bootstrap action.
4. Enter the configuration for the new bootstrap action:
  1. Name: Driven Bootstrap Action
  2. S3 Location: s3://files.concurrentinc.com/driven/2.1/hosted/driven-plugin/install-driven-plugin.sh
  3. Arguments: --host ${DRIVEN_SERVER_HOST} --api-key ${DRIVEN_API_KEY} or just --host ${DRIVEN_SERVER_HOST} if you do not want to pass an API key. (Substitute ${DRIVEN_SERVER_HOST} and ${DRIVEN_API_KEY} with your values.)

For More Information

See Driven Plugin Configuration Parameters if you need to further customize the plugin for the Driven deployment in your environment. Otherwise, you can return to the Driven Administrator Guide sitemap to review other topics.