Run a Job

Job run states

When you trigger a Job run, it will go through the following possible states:

• Queued: Each Job run starts in this state when it is triggered, and stays until the Spark driver starts.
• Running: The Job run is actively running on the specified Cluster.
• Completed: The Job run completed without errors.
• Failed: The Job run failed with error(s).
• Canceled: The Job was manually canceled while it was queued or running.

Trigger Job runs

Trigger a one-time run

After a Job is created, you can trigger a run.

1. In the Onehouse console, navigate to the Jobs page.
2. Open the Job you'd like to run, then click Actions > Run. You can also use the RUN JOB API command.

info

• When you run a Job, Onehouse will pick up the latest version of the JAR or Python script from the cloud storage bucket path in the Job definition. If the path does not exist, the Job run will fail.
• By default a Job runs one run at a time: while a run is Queued or Running, you cannot trigger a new run. To allow more, see Concurrent Job runs.

Trigger recurring runs

You can set up your own orchestration to trigger a Job run on a recurring basis using the RUN JOB API command.

Onehouse does not yet offer native orchestration for Jobs, but integrates with most orchestration tools. For example, follow this guide to set up orchestration with Apache Airflow.

Create temporary Clusters for Job runs

The Onehouse APIs enable you to spin up a temporary Cluster, run one or more Jobs, then spin down the Cluster. This pattern is sometimes called "Job flows", and may be a cost-efficient approach when you don't need a persistent cluster.

Perform the following steps, using Onehouse API commands, you can do the following:

1. Create a Cluster with CREATE CLUSTER.
2. Create Job definitions with CREATE JOB.
3. Run the Jobs with RUN JOB
   1. The API will return a Job run ID.
   2. If using the Airflow HTTP operator, this response will be captured.
4. Check for Job run completion with DESCRIBE JOB_RUN.
   1. Poll repeatedly for completion.
   2. If using the Airflow HTTP operator, you can use the response_check parameter.
5. Delete Cluster with DELETE CLUSTER.

Concurrent Job runs

A Job has a Max concurrent runs setting that caps how many of its runs can be active (Queued or Running) at the same time. It defaults to 1, which is the one-run-at-a-time behavior described above. Raise it when the same Job definition needs to process several inputs in parallel — for example one run per source table, per region, or per backfill date, triggered by your orchestrator.

Enablement

Concurrent runs must be enabled for your project before you can set the value above 1. Contact Onehouse support to enable it, and to raise the maximum allowed for your project if you need more.

Set the limit

In the Onehouse console, set Max concurrent runs in the Job configuration when you create or edit a Job. The field only appears once concurrent runs are enabled for your project.

With the API, use the MAX_CONCURRENT_RUNS clause:

CREATE JOB `count_transactions`
    TYPE = 'JAR'
    PARAMETERS = ( '--class', 'com.example.MySparkApp', 's3://onehouse-customer-bucket-12345/jobs/my-spark-app.jar' )
    CLUSTER = 'onehouse_cluster_spark'
    MAX_CONCURRENT_RUNS = 3

ALTER JOB `count_transactions`
    SET MAX_CONCURRENT_RUNS = 3

See CREATE JOB, CREATE AND RUN JOB, and ALTER JOB. DESCRIBE JOB returns the current value as maxConcurrentRuns.

Lowering the limit does not cancel runs that are already active — it only applies to runs triggered after the change. Setting it back to 1 returns the Job to one run at a time.

Trigger concurrent runs

Each run is triggered the same way as a single run, with RUN JOB or Actions > Run in the console, and gets its own run ID, driver, logs, and status. Runs of the same Job do not coordinate with each other.

When you trigger runs over the API, an Idempotency-Key header is required for any Job whose limit is above 1:

--header 'Idempotency-Key: count_transactions-2026-08-19'

The key identifies one logical run. If the request is retried with the same key, Onehouse returns the run ID of the run it already started rather than starting a second one. Raising the limit above 1 removes the "one run at a time" guard that used to stop duplicates, so the key is what takes over that job — which is why it is required there, and a good idea even at 1. Orchestrator identifiers work well: the Airflow task-attempt run_id, or a date-based key for a daily run. See RUN JOB for the full header rules.

Two caveats on keys:

• Reuse a key only for a retry of the same submission. Sending the same key with different per-run argument overrides is rejected as Idempotency key <key> was already used with different arguments.
• A key that belongs to a run that has already finished is not reused. Resubmitting it starts a new run.

Override arguments for one run

A run can override the Job's arguments for that run only — in the console's Run job dialog under Argument overrides, or over the API on RUN JOB. An override changes the value of an argument or adds a new one; arguments you do not name are inherited from the Job.

Arguments only

Overrides are limited to the Job's arguments. They cannot change Spark properties (--conf) or Hudi configurations (--hudi-conf) — those belong to the Job configuration and are applied from the Job definition on every run. Change them on the Job itself instead, in the console or with ALTER JOB.

What happens at the limit

A run triggered while the Job is already at its limit is rejected, and the Status API reports API_OPERATION_STATUS_FAILED with Spark Job <job_id> already has <N> active run(s) (max <N>). Nothing is queued behind the limit — your orchestrator decides whether to retry later or fail the task.

Sizing and monitoring

• Every concurrent run is a separate Spark driver plus its executors on the Job's Cluster. Size the Cluster (and its autoscaling limits) for the number of runs you allow, otherwise runs sit in Queued waiting for capacity. See Clusters.
• Runs of the same Job are independent writers. Give each run its own output — one table, or one set of partitions — unless you have set up concurrency control (a lock provider) for the table they share.
• The Job's page in the console lists every active run separately. See Monitor Jobs.

Read the Job run ID from your code

Each Job run has a unique run ID, returned by RUN JOB and shown in the console. Onehouse injects that ID into the Spark runtime, so your Job code can read it — for example to tag output paths, correlate your own logs with a Job run, or write audit records. If you are migrating from AWS Glue, this is the equivalent of Glue's --JOB_RUN_ID argument.

The ID is available two ways, on both the driver and the executors:

Type	Name
Environment variable	ONEHOUSE_JOB_RUN_ID
Spark configuration	spark.onehouse.jobRunId

Python:

import os

run_id = os.environ["ONEHOUSE_JOB_RUN_ID"]
# or, from the Spark session
run_id = spark.conf.get("spark.onehouse.jobRunId")

Scala:

val runId = sys.env("ONEHOUSE_JOB_RUN_ID")
// or
val runId = spark.conf.get("spark.onehouse.jobRunId")

info

• In code that runs on the executors (for example inside mapPartitions), read the environment variable. The Spark session and its configuration are only available on the driver.
• The run ID is injected for Python and JAR Jobs. It is not injected into ingestion (Stream Capture) runs.