Skip to main content

CREATE AND RUN JOB

Description​

Create a new Job, then immediately trigger a Job run.

Note that the SQL statement does not end with ;

Syntax​

CREATE AND RUN JOB `<job_name>`
    TYPE = { 'JAR' | 'PYTHON' }
    PARAMETERS = <array>
    CLUSTER = <string>
    [ MAX_CONCURRENT_RUNS = <integer> ]

Example​

Example command for creating and running a Job.

CREATE AND RUN JOB `count_transactions`
    TYPE = 'JAR'
    PARAMETERS = ( '--class', 'com.example.MySparkApp', '/path/to/my-spark-app.jar', 'arg1', 'arg2' )
    CLUSTER = 'onehouse_cluster_spark'

Example command for creating a Job that allows up to 3 runs at the same time, and triggering the first run.

CREATE AND RUN JOB `count_transactions`
    TYPE = 'JAR'
    PARAMETERS = ( '--class', 'com.example.MySparkApp', '/path/to/my-spark-app.jar', 'arg1', 'arg2' )
    CLUSTER = 'onehouse_cluster_spark'
    MAX_CONCURRENT_RUNS = 3

Required parameters​

  • <job_name>: Unique name to identify the Job.
  • TYPE: Specify the type of Job - this can be a JAR (for Java or Scala code) or Python script.
  • PARAMETERS: Specify an array of Strings to pass as parameters to the Job, which will be used in a spark-submit (see Apache Spark docs). This should include the following:
    • [Required] For JAR Jobs, you must include the --class parameter.
    • [Required] Include the cloud storage bucket path containing the code for your Job. The Onehouse agent must have access to read this path.
    • [Optional] Include any other Spark properties you'd like the Job to use.
    • [Optional] Include Apache Hudi configurations as --hudi-conf <key>=<value> pairs — see Set Apache Hudi configurations on a Job.
    • [Optional] Include any arguments you'd like to pass to the Job.
  • CLUSTER: Specify the name of an existing Onehouse Cluster with type Spark to run the Job.

Optional parameters​

  • MAX_CONCURRENT_RUNS: Maximum number of runs of this Job that can be active (Queued or Running) at the same time. Defaults to 1, which is the behavior when the clause is omitted. Values greater than 1 require concurrent runs to be enabled for your project — see CREATE JOB for the validation rules and Concurrent Job runs for how the runs behave.
    • Send an Idempotency-Key header with this command — required when MAX_CONCURRENT_RUNS is greater than 1 — so that a retried request returns the run that was already started instead of creating a second one. See RUN JOB for the rules on that header.

Status API​

Status API response​

  • API_OPERATION_STATUS_SUCCESS from Status API confirms that the Job has been created and submitted to the cluster, but does not reflect the status of the submitted Job. To monitor the Job's status, send a request to the DESCRIBE JOB_RUN API, and refer to sparkJobRun.status field from its Status API response.
  • API_OPERATION_STATUS_FAILED from Status API indicates that the Job has not been created or that the Job has been created but its submission to the cluster failed. To check whether the Job has been created, send a request to DESCRIBE JOB API, and refer to sparkJobRun.latestJobStatus and sparkJobRun.latestJobRunSubmittedAt fields from its Status API response. You can also get the Job run id from the sparkJobRun.latestJobRunId. To monitor the Job's status, send a request to the DESCRIBE JOB_RUN API, and refer to sparkJobRun.status field from its Status API response.

Example Status API response​

The Status API response of a successful creation and submission of a Job.

{
    "apiStatus": "API_OPERATION_STATUS_SUCCESS",
    "apiResponse": {
        "runJobApiResponse": {
            "jobRunId": "<job_run_id>"
        }
    }
}