Spark Job Metrics

Metrics published by every Onehouse-managed Spark job, for tracking job health, resource usage, and throughput in your own dashboards.

Query them from the Prometheus and Grafana running in your cloud environment — see Advanced Monitoring for access. For the ingestion and table-service metrics that describe what your Flows and Tables are doing, see the Datadog Metrics Reference. Both are in the same Prometheus, so you can combine them on one dashboard.

Executor metrics

The primary family for tracking a Spark job. Labeled with application_id, application_name, and executor_id. The driver reports itself as executor_id="driver" — filter with executor_id!="driver" for executors only.

Tasks

Metric	Description
metrics_executor_activeTasks	Tasks currently running
metrics_executor_maxTasks	Maximum concurrent tasks the executor can run
metrics_executor_totalCores	Cores available to the executor
metrics_executor_completedTasks_total	Tasks completed successfully (cumulative)
metrics_executor_failedTasks_total	Tasks that failed (cumulative)
metrics_executor_totalTasks_total	All tasks launched (cumulative)
metrics_executor_totalDuration_seconds_total	Total task execution time (cumulative)

Memory

Metric	Description
metrics_executor_JVMHeapMemory_bytes	JVM heap in use
metrics_executor_JVMOffHeapMemory_bytes	JVM off-heap in use
metrics_executor_memoryUsed_bytes	Memory used for cached blocks
metrics_executor_maxMemory_bytes	Memory available for cached blocks
metrics_executor_OnHeapExecutionMemory_bytes	On-heap memory used for execution (shuffles, joins, aggregations)
metrics_executor_OnHeapStorageMemory_bytes	On-heap memory used for storage
metrics_executor_OffHeapExecutionMemory_bytes	Off-heap memory used for execution
metrics_executor_OffHeapStorageMemory_bytes	Off-heap memory used for storage
metrics_executor_DirectPoolMemory_bytes	NIO direct buffer pool
metrics_executor_MappedPoolMemory_bytes	NIO mapped buffer pool

Process memory

Memory as seen by the OS. Useful for diagnosing container OOM kills, which JVM heap metrics alone will not explain.

Metric	Description
metrics_executor_ProcessTreeJVMRSSMemory_bytes	JVM resident memory
metrics_executor_ProcessTreeJVMVMemory_bytes	JVM virtual memory
metrics_executor_ProcessTreePythonRSSMemory_bytes	Python resident memory (PySpark)
metrics_executor_ProcessTreePythonVMemory_bytes	Python virtual memory (PySpark)

Garbage collection

Metric	Description
metrics_executor_totalGCTime_seconds_total	Total GC time across all collectors (cumulative)
metrics_executor_MinorGCCount_total	Minor (young generation) GC count
metrics_executor_MinorGCTime_seconds_total	Minor GC time
metrics_executor_MajorGCCount_total	Major (full) GC count
metrics_executor_MajorGCTime_seconds_total	Major GC time

Shuffle, I/O, and storage

Metric	Description
metrics_executor_totalInputBytes_bytes_total	Bytes read from input sources (cumulative)
metrics_executor_totalShuffleRead_bytes_total	Shuffle bytes read (cumulative)
metrics_executor_totalShuffleWrite_bytes_total	Shuffle bytes written (cumulative)
metrics_executor_diskUsed_bytes	Disk used for cached blocks
metrics_executor_rddBlocks	Cached RDD blocks held by the executor

Driver metrics

Job-level state from the Spark driver.

note

Driver metrics are not labeled by application. Filter on the pod label, which is named managed-spark-job-<job-uuid>-driver, to attribute them to a specific job.

Metric	Description
metrics_onehouse_driver_jvm_heap_usage_Value	Driver heap in use, as a fraction between 0 and 1
metrics_onehouse_driver_JVMCPU_jvmCpuTime_Value	Driver JVM CPU time
metrics_onehouse_driver_DAGScheduler_job_activeJobs_Value	Spark jobs currently running
metrics_onehouse_driver_appStatus_jobDuration_Value	Duration of the most recent Spark job
metrics_onehouse_driver_appStatus_jobs_succeededJobs_Count	Spark jobs that succeeded (cumulative)
metrics_onehouse_driver_appStatus_jobs_failedJobs_Count	Spark jobs that failed (cumulative)
metrics_onehouse_driver_appStatus_stages_completedStages_Count	Stages completed (cumulative)
metrics_onehouse_driver_appStatus_stages_failedStages_Count	Stages that failed (cumulative)
metrics_onehouse_driver_ExecutorAllocationManager_executors_numberAllExecutors_Value	Executors currently allocated
metrics_onehouse_driver_BlockManager_memory_memUsed_MB_Value	Driver memory used for cached blocks, in MB

The driver publishes several hundred additional metrics covering Spark internals. They are not listed here because they vary between Spark versions. To explore them, run this in the Prometheus query console:

group by (__name__) ({__name__=~"metrics_onehouse_driver_.*"})

Job and cluster status

Metric	Description
spark_info	Spark version and build info — always 1, read the labels
spark_driver_health	Driver health indicator
spark_job_failure_status	Job failure status
spark_job_inconsistent_state	Job is in an inconsistent state
auto_scale_target_executors	Target executor count chosen by the autoscaler
auto_scale_trigger	Autoscaling trigger events
auto_scale_events	Autoscaling events

Labels

Label	Present on	Meaning
application_id	executor metrics	Spark application ID — unique per job run
application_name	executor metrics	Spark application name — stable across runs
executor_id	executor metrics	Executor number, or driver
pod	all	Driver pod name — the only way to identify the job on driver metrics
namespace	all	Kubernetes namespace the job runs in
sparkClusterId	auto_scale_ metrics	Cluster the autoscaler is acting on

note

On auto_scale_ metrics the Spark job name appears as exported_job, not job — Prometheus renames the metric's own job label because it collides with the scrape target label.

Example queries

Active tasks per executor

sum by (application_name, executor_id) (metrics_executor_activeTasks)

Task failure rate per job

sum by (application_name) (rate(metrics_executor_failedTasks_total[5m]))

Percentage of executor time spent in GC

100 * sum by (application_name) (rate(metrics_executor_totalGCTime_seconds_total[5m]))
    / sum by (application_name) (rate(metrics_executor_totalDuration_seconds_total[5m]))

Executor JVM heap in use

sum by (application_name, executor_id) (metrics_executor_JVMHeapMemory_bytes)

Shuffle read and write throughput

sum by (application_name) (rate(metrics_executor_totalShuffleRead_bytes_total[5m]))

Input bytes read per job

sum by (application_name) (rate(metrics_executor_totalInputBytes_bytes_total[5m]))

Failed stages over the last hour

sum by (pod) (increase(metrics_onehouse_driver_appStatus_stages_failedStages_Count[1h]))

Executor count over time

metrics_onehouse_driver_ExecutorAllocationManager_executors_numberAllExecutors_Value

Things to know

• Metrics exist only while the job runs. Series disappear when a job ends, and executor series churn as executors scale up and down. Use rate() or increase() over a window rather than instant values for stable panels.
• Cumulative counters reset on restart. Anything ending in _total or _Count resets to zero when a driver or executor restarts. rate() and increase() handle this; raw subtraction does not.
• Aggregate before charting. Executor metrics are per-executor — use sum by (application_name) or a busy job renders one line per executor.
• maxMemory_bytes is not total heap. It is the memory reserved for cached blocks, so pair it with memoryUsed_bytes, not with JVMHeapMemory_bytes.
• Watch cardinality. application_id is unique per job run and unbounded over time. Group by application_name or pod for dashboards that stay stable across restarts.

If you need a metric that is not listed here, file a support ticket with Onehouse.