Set Apache Hudi configurations on a Job

A Job can carry its own hoodie.* configurations. They apply to every run of that Job, without changing your application code and without affecting other Jobs on the same Cluster.

Use this when the Job needs a Hudi setting that differs from the platform default — for example pointing AWS Glue sync at a catalog in another account (hoodie.datasource.meta.sync.glue.catalogId), choosing the sync target database, or setting write semantics for the table the Job writes.

Enablement

Hudi configurations on Jobs are enabled per project, and only an allow-listed set of configurations can be set. Contact Onehouse support to enable the feature for your project — see Which configurations you can set.

Set the configurations

Hudi configurations are part of the Job's Parameters, passed as --hudi-conf <key>=<value> pairs.

Each --hudi-conf carries exactly one configuration, so repeat the flag for each one:

--hudi-conf hoodie.x1=y1 --hudi-conf hoodie.x2=y2

A single --hudi-conf holding several configurations is not supported. The pair is split at the first =: everything before it is the key, everything after it is the value. So --hudi-conf hoodie.x1=y1,hoodie.x2=y2 sets one configuration — hoodie.x1 — to the literal value y1,hoodie.x2=y2.

In the console

Add the pairs to the Parameters field when you create or edit the Job:

["--class", "com.example.MySparkApp",
 "--hudi-conf", "hoodie.datasource.meta.sync.glue.catalogId=123456789012",
 "--hudi-conf", "hoodie.datasource.hive_sync.database=analytics",
 "s3://onehouse-customer-bucket-12345/jobs/my-spark-app.jar"]

With the API

CREATE JOB `count_transactions`
    TYPE = 'JAR'
    PARAMETERS = ( '--class', 'com.example.MySparkApp',
                   '--hudi-conf', 'hoodie.datasource.meta.sync.glue.catalogId=123456789012',
                   's3://onehouse-customer-bucket-12345/jobs/my-spark-app.jar' )
    CLUSTER = 'onehouse_cluster_spark'

To add, change, or remove a configuration on an existing Job, re-submit its full parameter list with ALTER JOB:

ALTER JOB `count_transactions`
    SET PARAMETERS = ( '--class', 'com.example.MySparkApp',
                       '--hudi-conf', 'hoodie.datasource.meta.sync.glue.catalogId=123456789012',
                       '--hudi-conf', 'hoodie.datasource.hive_sync.database=analytics',
                       's3://onehouse-customer-bucket-12345/jobs/my-spark-app.jar' )

Position matters

--hudi-conf is only a Job configuration when it appears before the path to your application file. After that path, everything is passed to your application as an argument instead:

--class Foo --hudi-conf hoodie.a=b   s3://bucket/app.jar   --hudi-conf hoodie.c=d   arg1
└──────── Job configuration ──────┘  └── application ───┘  └── your app's own arguments ──┘

A misplaced pair is not an error — a Hudi Streamer-style application legitimately takes Hudi configs on its own command line — so nothing warns you. Confirm with DESCRIBE JOB (below).

Confirm what the Job stores

DESCRIBE JOB returns the parsed configurations in hudiConfigs:

"sparkJob": {
    "name": "count_transactions",
    "hudiConfigs": {
        "hoodie.datasource.meta.sync.glue.catalogId": "123456789012"
    }
}

An expected key missing from hudiConfigs means it was parsed as an application argument — check its position.

--hudi-conf vs --conf

Both live in the Job's parameters, but they configure different systems and are not interchangeable.

	--conf spark.x=y	--hudi-conf hoodie.x=y
Configures	Spark, via spark-submit	Apache Hudi, via the properties Hudi reads at runtime
Shown by DESCRIBE JOB as	sparkConfigs	hudiConfigs
Accepts	spark.* keys	hoodie.* keys only
A hoodie.* key here	Ignored — move it to --hudi-conf	Correct place

Which configurations you can set

Only configurations on your project's allow-list are applied. The list starts empty, so a configuration you have not requested is not applied even when the feature is enabled — it is stored on the Job, and rejected outright when you create a Job with it.

To use a configuration that is not yet allowed, contact Onehouse support and say what you need. A request can cover:

Request	Effect	Example
A configuration family	Every configuration under a prefix. Name it with a trailing .	hoodie.datasource.meta.sync. — all catalog-sync configurations, including hoodie.datasource.meta.sync.glue.catalogId
One configuration	That exact key only	hoodie.datasource.write.operation

Families are the usual choice: one request covers the related configurations you are likely to need next, instead of a round trip per key. Include what the Job is trying to do — Onehouse reviews the request against the settings the platform manages for your tables, and the reserved configurations below are never granted.

Scope and precedence

• Configurations belong to the Job, so they apply to all of its runs, including concurrent runs. They cannot be overridden for a single run: a run may override the Job's arguments, but not its Spark or Hudi configurations.

• A Job configuration takes precedence over the Cluster-level default for the same key.

• Configurations Onehouse manages for correctness always win, and some keys cannot be set at all:

  Key	Why
  hoodie.write.concurrency.*, hoodie.write.lock.*, hoodie.cleaner.policy.failed.writes	Onehouse manages concurrency control; changing these risks data loss when another writer touches the table
  hoodie.metrics.*	Reserved for platform metrics collection
  hoodie.write.config.freeze	Controls which configurations are protected in the first place

  These are the same configurations you should not set in your application code — see Working with Apache Hudi.

Errors

Message	Cause
--hudi-conf is not enabled for this organization. Remove the --hudi-conf parameters, or contact Onehouse support to enable Hudi config overrides on Spark jobs.	The feature is not enabled for the project.
--hudi-conf keys are not permitted for this organization: <keys>. Remove them, or contact Onehouse support to allow them.	The key is outside the project's allow-list.
--hudi-conf only accepts hoodie.* keys, got '<key>'. Use --conf for Spark configs.	A non-Hudi key was passed to --hudi-conf.
--hudi-conf must be followed by hudi config <key>=<value> / Hudi config must be of format <key>=<value>	The pair is missing its value or its =.
--hudi-conf value for '<key>' is empty. Remove the config to fall back to the platform default.	An empty value would override the platform default with an empty string rather than leave the key unset.
--hudi-conf entry for '<key>' contains a line break or leading/trailing whitespace, which would not survive the run's properties file intact.	Remove the line break or the padding around the key or value.

Editing a Job re-submits its whole parameter list, so an edit that carries existing --hudi-conf pairs is accepted even if the feature or the allow-list has since narrowed — the Job stays editable, but those configurations stop being applied to its runs.

Common questions

I set a configuration and the run behaved as before. Check its position in the parameters first, then confirm the key appears in DESCRIBE JOB → hudiConfigs. If it is there and still not applied, the key may be outside your project's allow-list — contact support.

Can I set a Hudi configuration for one run only? No. Hudi configurations are Job-level. A run can override the Job's arguments, but not its configurations.

My application also sets the configuration in code. An option your code passes on the write takes precedence for that write. Use the Job configuration for settings your code does not set itself.