Driven User Guide: Understanding the Unit of Work Details Page

The Status panel displays a color-coded bar representing the time the Unit of Work spent in each of the states it was in. The timeline is a graphical way to view the amount of time the Unit of Work spent in each of the states it entered, including the current one. The states illustrated by the bar are labeled with begin and end times (time entered and time exited), or dates for long-running units of work. For detailed information on these states, see the State Model

Below the timeline is the Progress section, which displays the total number of steps the unit of work has, along with tables that categorize the unit of work’s steps by state. The Active table’s columns are named for the active states that a step could be in, while the Completed table’s columns are named for the end states that a step could be in. The values in these columns are simple counters that show how many steps are in the specified state.

If a unit of work is still active, a Slice Rate graph appears below the Progress section, along with the current number of active slices and the time of last update. Mousing over the graph updates the time and number of active slices to those of the chosen point. For more information, see Common Counters.

The Metrics Panel displays unit-of-work-level metrics. These same metrics (from each unit of work) are aggregated to provide the values for the equivalent application-level metrics. For more information on Driven metrics, see Counters.

The Properties Panel reports on the last time data was received about the unit of work.

If the unit of work is part of a Hive-based application, it may display a Statement property containing the SQL executed by the unit of work.

The Environment Panel shows the platform on which the unit of work is being executed.

A Hive query DAG will be drawn showing a query icon between the source table(s) and output table(s). The user can click on the source and sink resource nodes to see more detailed information about the tables touched by the query. Hive units of work only have one processing vertex representing the HQL query. The details of the query are listed in the Properties tab of the status panel.

Native map reduce DAGs will show map, shuffle, and reduce vertices between the source and sink resource vertices.

Spark DAGs will show resilient distributed dataset (RDD) vertices between the source and sink resource vertices.

A key component of a Cascading application is the query planner. When the Cascading application executes, the query planner compiles all the data-processing steps, analyzes dependencies of the steps, and develops the DAG for the application.

The Cascading query planner iterates through the DAG, breaking it into smaller and smaller graphs - called expression graphs - until the graph matches a pattern associated with a unit of work, such as a mapper or a reducer.

In the slice performance dashboard, you can see the slice (a unit of work such as a map or a reduce task) information at the individual or at an aggregate level.

Observe if any of your slices are skewed. In a MapReduce application, the data is divided and processed in equal-sized chunks. If certain slices are taking more time to finish processing a similar type of task with (assumed) similarly sized data, then it is an anomaly and could indicate application execution problems.

Often, these skews indicate that applications are processing a large number of small files, which usually means that you need to optimize the environment. In other cases, depending on the skew dimension, they could indicate a network issue, which can delay the shuffle-sort operations in MapReduce.

For further details on bottlenecks, click View Slice Waterfall to open a waterfall, or gantt-like timeline, of the slices in the selected node.

From this view, the number of active slices running on the cluster concurrently is shown upon hovering the mouse over the diagram. A low number of active slices can indicate congestion on your cluster. Also look out for: