Driven User Guide

A search box appears near the top of the page if you want to filter what units of work are displayed. If you are inspecting a complex or long-running application and know the value of a unit of work parameter listed in Figure 1, then the search can help pinpoint runtime factors that pertain to your goals. For a less fine-tuned search, you can also filter the search results that are displayed by using the Status drop-down menu.

Each application instance is represented as a directed acyclic graph (DAG). The graph renders an interactive diagram of the units of work and steps that can reveal underlying slice performance issues. Units of work, step, and slice information are particularly useful to monitor instances of application execution over a period of time as the application grows in complexity and size. In addition, units of work and step details on the DAG and in the table below the graph can be used to:

When you execute your application, the underlying framework builds a state model to optimally execute the unit of work on the Hadoop cluster. The Driven Plugin transmits the state model to the Driven application, which represents the execution plan as a DAG.

The DAG representation of each application execution can be useful to stakeholders responsible for documenting how an application has been developed and has performed, such as a documentation analyst. Over a period of time, the analyst might not be able to track and record relevant application details. Because Driven has a persistence layer to store application execution data, past application performance can be recreated by generating a DAG on demand. Without such an interface, it can be difficult to map business needs to technical implementation, especially in work environments that involve large teams that are spread across different regions.

In the graph, each node corresponds to a step or a processing function in your application code. You can refer to the specific code for a step by clicking on the node link. The pop-up window with detailed information is an annotation. See Using Annotations for more information.

The DAG on the app details page can be viewed in three different ways:

- The Contracted View is useful for complex and large applications.

- The Logical View (default) shows all the steps and resources (excluding implicit resources) and built-in functions.

- The Physical View shows all the steps including the implicit resources and built-in functions. The Physical View may show more details than the Logical View, if any exist.

Driven represents Cascading’s pipes metaphor as lines connecting steps in the DAG. A step is dependent on another step only if it relies on the execution of the previous step. The Driven Plugin dynamically determines the dependencies between the units of work. If the output (sink) of one unit of work is consumed by another unit of work (as a source), Driven notates that dependency by connecting the two units of work.

Visualizing your end-to-end application as a DAG along with operational data, such as read/write data processed at each step, can provide important insights into improving the performance of the application. For example, reviewing the DAG can expose opportunities to introduce Filter functions in your code upstream to reduce the volumes of the data being processed by the pipes or to make the Join functions more efficient.

Driven can refresh the displayed information as updates stream in from the plugin. This includes display of real-time progress of your application, which includes highlighting the current steps being executed, number of completed steps, and read/write data.

Getting the most current information can be very useful. You might discover in Driven that a long-running job is not executing properly, which could be a signal to terminate the application. Also, for example, if you see sudden slow-down in the progress of your application, you may want to immediately start investigating the reason (network storm or a rogue job submitted to the cluster).

One of the most interesting insights is the ability to track the percentage of applications that have completed in real time. For long-running applications, it is often useful to spot-check the behavior to ensure that there are no anomalies.

Ensure that the Auto Update slider in the top right corner is enabled to allow the displayed Driven data to auto-refresh in real time.

The table under the DAG provides a detailed breakdown of each unit of work in the application run. Some key monitoring assets of the tabular interface include the following capabilities: