Driven User Guide

version 2.1.4

Configuring Teams for Collaboration

Applications are not deployed and monitored in isolation. Usually, application development is a collaboration between the IT developers, engineering, business analysts, and other data scientists. When applications become operational, it becomes vital to create collaboration to investigate job behavior and observe characteristics. Organizing application execution data by groups of stakeholders can help you and your colleagues focus on metrics that you are targeting.

On the other hand, most organizations want to categorize and restrict viewing of application metrics for governance, regulatory compliance, developing models for chargebacks, or other reasons. For example, in a bank you may have the Hadoop production cluster as a single enterprise capability, shared between different lines of businesses such as marketing, fraud, operational risk, and across organizations in different geographies. The bank runs many applications that contain confidential data. In this scenario, it is possible that information about the xyz application should be revealed only to the fraud and operational risk business units because the xyz application processes confidential data that needs to be monitored by these units.

Using teams, you can both share and restrict views of specific application metrics. In Driven a team is associated with an API key. For any application using the API key, the data from the application is visible only to users who are team members and to Driven administrators. As a result, when a team member saves a status or application view with programmed API keys, the view is accessible to other team members and the Driven administrator.

Tip	For the sake of comparing multiple applications that are restricted by teams, you might need a user with administrator privileges to perform this analysis efficiently.

Access a team view by clicking My Teams and selecting one of the links that appear in that section. Alternatively, select a team from the All Teams drop-down menu.

Figure 1. Alternative ways to access a team view

By default, a user who creates a team is the team leader. However, the default team leader can transfer the role to another user on the team. Team leaders can also invite users to join the team, remove users, and delete the team altogether.

Creating and Managing Teams

The following procedure permits users who are not logged in to Driven as an administrator to create a team and assign existing user accounts to the team. If you have administrator privileges, you can create new Driven users while you create teams. See Using the Driven Admin Console in the Driven Administrator Guide for more information about administrator configuration of teams and users.

Step 1: Create a team

Perform the following tasks to create a team:

If you do not see a list of your teams, expand My Teams in the side panel.
Click Create a new team.
Enter a team name.
Click Create Team.
Record or copy the API key that appears next to the newly created team. You need this to configure the Driven Plugin to display application run data to Driven users who belong to the team with the API key.

Step 2: Add another user to the team

Click the team details icon in the side panel.
Enter the user email address.
Click Invite User.

Note	A message appears if the SMTP service is not enabled on your Driven Server. You must resend the invitation after the SMTP service is enabled.

The team details page is the place where you can manage other aspects of the team, such as deleting a team from Driven.

Figure 2. Team details page

Associating an Application with a Team

Before you can run the application to display information to a specific team, you need to set the API key that is associated with the team when the application is executed.

Set the API key for an application by one of the following methods, substituting ${API_KEY} in the code examples with your real key:

Option 1: Create or update the cascading-service.properties file and ensure it is in your HADOOP_CLASSPATH. The easiest way to do this is to place the file in your Hadoop configuration directory ($HADOOP_CONF). For Hadoop 1.x, this may be $HADOOP_HOME/conf. For Hadoop 2.x, the directory could be $HADOOP_INSTALL/etc/hadoop.

$ echo cascading.management.document.service.apikey=${API_KEY}>> $HADOOP_CONF/cascading-service.properties

Option 2: - Set the DRIVEN_API_KEY environment variable:

$ export DRIVEN_API_KEY=${API_KEY}

Option 3: Add a JVM flag to the application invocation:

-Dcascading.management.document.service.apikey=${API_KEY}

Click the options listed under the My Teams headings of the side panel to see the application metrics to which you have access. You can see that your applications are categorized by team and can be viewed accordingly. If you were added to other teams, you also have access to these teams in this area of the sidebar.

External Issue-Tracking

Driven 1.2 has the capability to link directly to Atlassian JIRA if your team reports and track issues with an up-and-running implementation of JIRA. If external issue-tracking is enabled by the team leader or Driven administrator, the link to the JIRA instance appears on views containing application and flow details.

When you click on the link, JIRA opens and prompts you to file an issue. The URL of the page that you are viewing is automatically placed in the description field for the issue. JIRA project ID and issue type are also prepopulated, based on the settings that the team leader or Driven administrator defined.

The JIRA link icon jira-link appears on the top right corner of application details and flow details views if external issue-tracking is enabled.

Integrating JIRA Issue Links

Prerequisites: Know the JIRA base URL, project ID number, and issue type ID number that you want to associate with issues that are filed with the JIRA link in Driven.

To enable external issue-tracking links:

Open the team details page if you are not there already.
Click Add external link. The Add External Link page appears. The value in the Type field cannot be changed because Driven 1.2 supports only JIRA here.
Enter a name for the link. The name will appear only on the team details page in Driven.
Enter the base URL, project ID number, and issue type number.
Click Add link. The JIRA link icon appears on pages with application- and flow-level details. Information about the link appears on the team details page, where you can also remove the link if you no longer want it to appear.

Automating Notifications Delivered on Messaging Platforms

Driven can be integrated with third-party messaging platforms to automatically notify people when an application reaches one of the following states:

SUCCESSFUL status
FAILED status
Completed state (SUCCESSFUL, FAILED, or STOPPED status)

Driven 1.2 supports webhooks that communicate with Slack, HipChat, and PagerDuty. The notifications that users receive typically contain a URL hyperlink to the application details page for the event.

The following procedure documents setup of both a new webhook connection and a new messaging ruleset. If you have an existing messaging ruleset that you want to apply to another webhook connection that was created in Driven previously, you can skip the Webhook Connections section and click Add rule to assign the ruleset to the connection.

To configure webhook notifications:

Open the team details page if you are not there already. The team details page is opened by clicking the icon in the My Teams list of the side panel.
Click Add connection in the Webhook Connections section. The Add Webhook page appears.
Select the messaging platform in the Type field, and assign a name for the webhook connection to appear in Driven. Other parameters in the Add Webhook page depends on the messaging platform that you select:
- Slack: Refer to Slack API Incoming Webhooks documentation.
- HipChat: Refer to https://mydomain.hipchat.com/account/api, replacing "mydomain" with your HipChat domain name. On this web page you can create an access token and find information about API access.
- PagerDuty: See Creating a Generic API Service for how to generate a service API key, which you use to fill in the Service Key field in the Add Webhook window. The API key enables automatic notification of application events to PagerDuty team members. If you have an existing PagerDuty incident where you want to record application events, you can fill in the Incident Key field to write status messages automatically to the incident log in addition to notifications that are sent to PagerDuty team members.
Click the Add Webhook button. The team details page appears.
Click Add rule in the Messaging Rules section. The Add Message Rule window appears.
In the On Event field, select the application event to trigger a message.
In the Connection field, select the webhook connection to which you want to apply the message rule.
Optional: Enter text in the Description field that will appear in the Messaging Rules part of the team details page.
Accept the default text in the Message Template field, or customize the message. The string "driven_url" in this field is treated as a token for the application details page URL. If you include "driven_URL", it will be replaced by a URL hyperlink to the Driven page with specific details about an application event.
Click the Add Rule button.

Enabling Automatic Email Notifications

You can configure a team so that Driven emails team members when an application reaches one of the following states:

SUCCESSFUL status
FAILED status
Completed state (SUCCESSFUL, FAILED, or STOPPED status)

To enable and configure email notifications:

Open the team details page if you are not there already. The team details page is opened by clicking the icon in the My Teams list of the side panel.
Click Add configuration in the Email Notifications section.
In the Add Email Configuration page:
1. Enter a name for the email configuration. The name does not appear in the emails, but rather is a reference when viewing the team details page.
2. Select Current Team to send emails to all team members, or select Team Members to designate individual team members rather than the whole team as email recipients.
3. Optional: Enter a description. The description does not appear in the emails, but rather is a reference when viewing the team details page.
4. Click Add.
Click Add rule in the Messaging Rules section of the team details page.
In the Add Message Rule page:
1. Select the application event to trigger a message in the On Event field.
2. Select the email notification configuration to apply in the Connection field.
3. Optional: Enter text in the Description field that will appear in the Messaging Rules part of the team details page.
4. Enter a subject line for the automatic emails based on the current message rule in the Subject field.
5. Accept the default text in the Message Template field, or customize the message. The string "driven_url" in this field is treated as a token for the application details page URL. If you include "driven_URL", it is replaced by a URL hyperlink to the Driven page with specific details about an application event.
6. Click Add Rule.

Using Annotations with Cascading Applications