Driven Administrator Guide: Integrating with Third-Party Monitoring Applications

version 2.2.6

Driven Administrator Guide: Integrating with Third-Party Monitoring Applications

1. Prerequisite: 1.1. Upgrading

1.2. Check your team’s demands and expectations

1.3. Check your hardware and software
2. Planning a Driven Deployment: 2.1. Understanding the Deployment Architecture

2.2. External Dependencies

2.3. Selecting a Deployment Topology
3. Installing and Configuring the Driven Server: 3.1. Downloading the Driven Server

3.2. Installing the Driven Server

3.3. Configuring the Driven Server
4. Starting and Stopping the Driven Server: 4.1. Starting the Driven Server

4.2. Stopping the Driven Server
5. Uninstalling the Driven Server: 5.1. Stopping the Server

5.2. Removing All Files and Data
6. Using the Driven Admin Features: 6.1. Manage Users

6.2. Managing A Single User

6.3. Manage Teams

6.4. Manage System
7. The Driven Command Line Client: 7.1. Installing the Driven CLI Client

7.2. Backing Up and Restoring Driven Repositories

7.3. Backing Up Your Driven Data

7.4. Extracting Data with the scope Command

7.5. Examples of Command Usage

7.6. Common Command-Line Options

7.7. How-To Tips
8. Integrating with Third-Party Monitoring Applications: 8.1. Using Java Management Extensions (JMX) with Driven

8.2. Using Nagios Plugins with Driven
9. Upgrading Driven Server: 9.1. Before WAR Replacement

9.2. Upgrade the Driven Server
10. Troubleshooting Driven: 10.1. The Driven Server installation generates a Java incompatibility error

10.2. Can not install a valid license

10.3. Transmission of application metrics data to Driven stops but app status remains in RUNNING state

Integrating with Third-Party Monitoring Applications

While Driven provides a superior interface to visualize your application in development and operational stages, often it is necessary to relay higher-level insights to a common monitoring platform.

Driven supports integration with other frameworks to extend representation of application performance in different monitoring platforms. Two approaches for utilizing this extensibility are to integrate with the following technologies:

Using Java Management Extensions (JMX) with Driven - The JMX API is a standard API for managing and monitoring of resources such as applications, devices, services, and the Java Virtual Machine (JVM). The API can enable any tool supporting the JMX interface to monitor precanned statistics from Driven.
Using Nagios Plugins with Driven - Nagios is an application for monitoring IT infrastructure, including Hadoop clusters. By developing Nagios plugins for Driven, you can pull monitoring information from Driven to gain an alternative representation of application status.

Using Java Management Extensions (JMX) with Driven

The Java Management Extensions (JMX) API is a standard API for managing and monitoring of resources, such as applications, devices, services, and the Java Virtual Machine (JVM). Typical uses of the JMX technology include:

Gathering statistics about application behavior
Notifying of state changes and erroneous conditions

Note	The JMX API includes remote access so that a remote management program can interact with a running application for these purposes.

Using a JMX Interface to a Driven Server

The scope CLI supports setting up a JMX interface for proxy scope queries to a Driven Server. This enables any tool supporting the JMX interface to monitor pre-canned statistics from Driven.

Prerequisite: Ensure that Driven is monitoring applications before completing the following steps.

Step 1: Start the scope CLI in daemon mode

In the driven-client directory, enter:

$ driven scope --jmx

Make sure that Scope JMX has started.

started Scope JMX

Step 2: Open a monitoring tool

For example, the JConsole monitoring tool helps you manage Driven operations.

With a web browser window open, start the command terminal and enter:

$ JConsole

The JConsole:New Connection window appears.

Figure 1. JConsole displays the "driven.management.scope.Scope --jmx" local process

Connect to driven.management.scope.Scope --jmx

Step 3: Navigate to Driven operations in the console

Click MBeans tab > driven.management.scope.Scope --jmx option > Operations option.

Figure 2. The Operations list of driven.management.scope.jmx

Step 4: View details about an operation

Highlight an option in the Operations list to view the MBeans operation information. Use the Operation invocation pane to launch specific details of the Driven operation.

API Details

The scope CLI supports several categories of methods with the JMX API.

Utility Methods

getProcessTypes();: getProcessTypes(); returns all process types in Driven. The types values are app, flow, step, and slice.
getTimeUnits();: getTimeUnits(); returns all time units. Values can be in DAYS, HOURS, or MINUTES.
getStatuses();: getStatuses(); returns all application statuses. Status can be SKIPPED, STARTED, SUBMITTED, RUNNING, SUCCESSFUL, STOPPED, or FAILED.

public List<String> getProcessTypes();
public List<String> getTimeUnits();
public List<String> getStatuses();

Operation Method

statusCounts( final String type );: statusCounts( final Stringtype ); returns the number of your applications that are in FAILED, SUCCESSFUL, and STOPPED states.

public Map<String, Long> statusCounts( final String type );

Application-Specific Operations

findApps( String status, Integer amount, String timeUnit );: findApps ( String status, Integer amount, String timeUnit ); returns all applications with the values that you provided in your query. These attributes consist of the application’s status (SKIPPED, STARTED, SUBMITTED, RUNNING, SUCCESSFUL, STOPPED, or FAILED), time unit (DAYS, HOURS, or MINUTES), and the amount in the specified time unit.
findAppsWithDurationGreaterThan( String status, Integer duration, String timeUnit );: findAppsWithDurationGreaterThan( String status, Integer duration, String timeUnit ); returns the IDs for all applications in a particular status for more time than the duration, timeUnit parameters specify. For example, findAppsWithDurationGreaterThan ( RUNNING, 3, HOURS ) returns all IDs for applications that were in RUNNING state for more than 3 hours.
getAppDetail( String id );: getAppDetail( String id ) returns the application name and ID, start and finish times, its duration, and current status.

public List<String> findApps( String status, Integer amount, String timeUnit );
public List<String> findAppsWithDurationGreaterThan( String status, Integer duration, String timeUnit );
public Map getAppDetail( String id );

Attributes

The following attributes for JMX API integration can be automatically graphed in VisualVM.

public Long getTotalAppsRunning();
public Long getTotalFlowsRunning();
public Long getTotalStepsRunning();
public Long getTotalSlicesRunning();

Code Example of Method Call to Scope CLI Client

/**
 * An example of calling a method on Driven client scope JMX MBean
 */
public class JMXExample
  {
  public static void main( String[] args )
    {
    String host = "localhost";

    // The port can be set in $DRIVEN_CLIENT_HOME/driven-env.sh
    int port = 9010;

    String url = "service:jmx:rmi:///jndi/rmi://" + host + ":" + port + "/jmxrmi";

    try
      {
      JMXServiceURL serviceUrl = new JMXServiceURL( url );
      JMXConnector jmxConnector = JMXConnectorFactory.connect( serviceUrl, null );
      ObjectName objectName = new ObjectName( "driven.management.scope.jmx:type=DrivenScope" );
      try
        {

        MBeanServerConnection conn = jmxConnector.getMBeanServerConnection();

        /**
         * Example of getting a system attribute
         */
        long count = (long) conn.getAttribute( objectName, "totalAppsRunning" );
        System.out.format( "%d apps are currently in RUNNING state\n", count );

        /**
         * Example of invoking a method on the MBean
         */
        Map statuses = (Map) conn.invoke( objectName, "statusCounts", new Object[]{ "app" }, new String[]{ "java.lang.String" } );
        for ( Object item : statuses.keySet() )
          System.out.format("%s->%s\n", item, statuses.get(item) );
        }
      finally
        {
        if( jmxConnector != null )
          jmxConnector.close();
        }
      }
    catch( Exception exception )
      {
      exception.printStackTrace();
      }
    }

  }

Using Nagios Plugins with Driven

Nagios can be used for monitoring IT infrastructure, including a Hadoop cluster. With Nagios, you can either use the Bash shell or the JMX interface to develop custom plugins for monitoring applications through the Driven interface.

For more information about developing Nagios plugins, refer to Nagios online documentation. There are multiple tutorials available in the Nagios community describing how to build Bash-based plugins by calling Driven scope commands.

Nagios Bash Plugin Examples

Here are some simple examples of Nagios Bash plugins developed using the scope CLI:

This is an example of tracking all the applications that did not complete successfully.

#!/bin/bash
driven scope --fields type,id,name,owner,status,failedCause | awk '$5!="SUCCESSFUL" { print $1 "\t"  $2 "\t" $3 "\t" $4 "\t" $5 "\t" $6}'
exit 1

To track the longest running applications:

#!/bin/bash
driven scope --fields type,id,name,owner,status,startTime:submitTime,startTime:finishedTime,duration | awk 'BEGIN { OFS = "\t" } NR == 1 { $9 = "diff." } NR >= 3 { $9 = $7 - $6 } 1' | sort -rgk 9 | head -n 10
exit 1

To track the applications that are consuming the most storage on the cluster:

#!/bin/bash
driven scope --fields type,id,name,owner,status,tuplesWritten,bytesWritten | awk '{ print $1 "\t"  $2 "\t" $3 "\t" $4 "\t" $5 "\t" $6 "\t" $7}' | sort -rgk 7 | head -n 10
exit 1