Driven Administrator Guide
version 2.1.4Driven Plugin Configuration Parameters
The Driven Plugin is the component that collects data from a running Cascading application and sends the information to the Driven Server for processing and analysis.
In general, you do not need to customize the Driven Plugin beyond that needed to make the plugin receive data from running applications. The following information is a reference for the Driven Plugin configuration parameters, which you might need depending on your environment and preferences.
Make the plugin available to the job by placing a file named cascading-service.properties in Hadoop’s classpath with the following contents: ---- cascading.management.service.jar=${PATH}/driven-plugin.jar ----
Tip
|
If installation and configuration of the plugin appear to fail, check your Cascading application logs to ensure that the ServiceLoader, CascadingService, and DrivenDocumentService properties point to the same path for the Driven Plugin JAR file. |
Driven Server URL
If not using the default hosted Driven Server, the Driven Plugin must be configured to connect to your own installation of the Driven Server.
Either configure the cascading-service.properties file:
cascading.management.document.service.hosts=http<s>://${HOST_NAME}<:${PORT}>
Or set the environment variable:
$ export DRIVEN_SERVER_HOSTS=http<s>://${HOST_NAME}<:${PORT}>
Retry Rules
The following properties control the Driven Plugin’s behavior as it attempts to communicate with the Driven Server.
Set the initial retry wait time using the following property:
cascading.management.document.service.initial.retry.millis=400
This property has a default value of 400 milliseconds, which means that the Driven Plugin will wait 400 milliseconds for a response from the server before retrying. With subsequent retries, it will gradually increase the wait time using a progressive back-off strategy until it reaches the maximum default wait time of 4000 milliseconds. To change the default maximum wait time, use the following property:
cascading.management.document.service.max.retry.millis=4000
Once the interval between retries reaches the value of cascading.management.document.service.max.retry.millis, the Driven Plugin will retry at an interval equal to that value. To set the Driven Plugin to stop retrying after a certain number of milliseconds, use the following property:
cascading.management.document.service.max.server.unreachable.millis=-1
By default, this property is turned off (set to "-1").
In case of failed connections, the Driven Plugin keeps a backlog of the batched items until they can be sent, with a default of 100000 items. To change the default, use the following property:
cascading.management.document.service.max.backlog.size=100000
By default, the Driven Plugin will attempt to connect to the Driven Server an indefinite number of times while continuing to record data. To set a maximum number at which the Driven Plugin stops retries (and stops recording data), give the following property a positive integer value (default value is "-1"):
cascading.management.document.service.max.failed.attempts=-1
When the value you set is reached, the Driven Plugin stops retrying the Driven Server (and stops recording data).
API Key
API keys map to teams in Driven. An application that is associated with an API key is searchable by members of the team that own the key. See Configuring Teams for Collaboration in the User Guide for more information.
Either configure the cascading-service.properties file:
cascading.management.document.service.apikey=${API_KEY}
Or set the environment variable:
$ export DRIVEN_API_KEY=${API_KEY}
Default Application Tags
As part of the Driven Plugin configuration, you can specify a tag or group of tags to associate with all applications that run with the plugin. See Managing Applications with Tags in the User Guide for best practices with tags and for how to apply tags on particular applications.
In the following lines, replace CATEGORY:INSTANCE1 with your real tag. It is recommended to use the "category label:specific value" capability of tags, which requires using the colon as a separator. To specify multiple tags, enter them as comma-separated values.
Either configure the cascading-service.properties file:
driven.protocol.tags=CATEGORY:INSTANCE1
Or set the environment variable:
$ export DRIVEN_DEFAULT_TAGS=CATEGORY:INSTANCE1
Application Heartbeat Frequency
Once an application JVM is running and initial data is sent by the Driven Plugin, then an application heartbeat signal is sent by the Driven Plugin to the Driven Server. To set the frequency (or interval, in milliseconds) of this signal, use the following property:
driven.protocol.app.heartbeat.frequency=<value>
Slice Data Suppression
Slice data transmission can be suppressed or resumed by setting a configuration parameter. The default setting of this parameter is false, which means slice data is transmitted unless you change the setting. Setting this property to true overrides a server-side configuration for slice-data transmission.
Either configure the cascading-service.properties file:
driven.protocol.slice.suppress=[true|false]
Or configure the environment variable:
$ export DRIVEN_SUPPRESS_SLICE_DATA=[true|false]
Slice Data on Completion
A configuration parameter can be set to send or to not send data when the slice state changes to completed. The default setting for this parameter is true, which means data is sent only when the slice state reaches completed state.
Either configure the cascading-service.properties file:
driven.protocol.slice.state_change_only=[true|false]
Or set the environment variable:
$ export DRIVEN_SLICE_STATE_CHANGE_ONLY=[true|false]
Java Command Suppression
You can suppress the command-line arguments used to launch the Cascading job by setting the parameter for Java command suppression to true. This property overrides any server-side configuration for Java command suppression. The default setting for this parameter is false.
Either configure the cascading-service.properties file:
driven.protocol.command.suppress=[true|false]
Or set the environment variable:
$ export DRIVEN_SUPPRESS_JAVA_COMMAND=[true|false]
Archive Mode
The Driven Plugin can be run in archive mode. If archive mode is enabled, all records that are sent to the Driven Server are written to disk even if the server is unreachable. This can be useful if the Driven Server is not available or unreachable as the archive can later be replayed when the server is reachable.
Either configure the cascading-service.properties file:
cascading.management.document.service.archive.dir=/${ARCHIVE_DIRECTORY_PATH}
Or set the environment variable:
$ export DRIVEN_ARCHIVE_DIR=${ARCHIVE_DIRECTORY_PATH}
To prevent the plugin from uploading the on-disk data to the Driven server, also set the following property:
cascading.management.document.service.archive.only=true
Or set the environment variable:
$ export DRIVEN_ARCHIVE_ONLY=true
When cascading.management.document.service.archive.only is true, the plugin will save records locally, allowing you to view plugin data during initial testing or troubleshooting situations when sending to the Driven server isn’t possible or required.
Failed Flows
By default, a flow that has FAILED status does not automatically set the end state of the application to FAILED. You can configure Driven so that an application is put in FAILED state when a flow fails by setting the following parameter to true.
Configure the cascading-service.properties file:
cascading.management.state.service.appstats.failonflow=[true|false]
Pending Flows
By default, if an application finishes running but still has a flow that has PENDING status, the end state of the application is SUCCESSFUL. You can configure Driven so that an application is put in FAILED state when a flow remains in PENDING state by setting the following parameter to true.
Configure the cascading-service.properties file:
cascading.management.state.service.appstats.failonpending=[true|false]
Control Plugin Logging
By default, the Driven Plugin logs events. To disable all Driven Plugin logging, set the following property:
cascading.management.document.service.log.disabled=true
Or set the environment variable:
$ export DRIVEN_LOG_DISABLED=true
To enable the Driven Plugin debug logging, set the following property:
cascading.management.document.service.log.debug=true
Or set the environment variable:
$ export DRIVEN_LOG_DEBUG=true
To force the Driven Plugin to log INFO messages, set the following property:
cascading.management.document.service.log.force=true
Disable Plugin
To disable the Driven Plugin while leaving it installed, set the following property:
cascading.management.state.service.disabled=true
Or set the environment variable:
$ export DRIVEN_DISABLED=TRUE
Next
Learn about how a Driven admin can further customize and monitor the deployment in Using the Driven Admin Console.