Amazon Kinesis (Limited availability)

Continuously stream data from Amazon Kinesis Data Streams into Onehouse tables.

Kinesis ingestion is enabled per project — contact Onehouse to enable it for yours.

Click Sources > Add New Source > Amazon Kinesis. Then, follow the setup guide within the Onehouse console to configure your source.

Cloud Provider Support

• AWS: ✅ Supported
• GCP: Not supported

Reading Kinesis Records

Onehouse supports the following data formats for Kinesis stream records:

Data Format	Schema Registry	Description
JSON	Optional	Deserializes stream records in JSON format.
Avro	Required	Deserializes stream records in Avro format. Requires a connected schema registry.
Parquet	Optional	Deserializes stream records in Parquet format.

Schema Registry Support

Kinesis sources support the following schema registries:

• AWS Glue Schema Registry — native integration for AWS-based deployments.
• Confluent Schema Registry — for environments using Confluent for schema management.
• File-Based Schema Registry — for custom or local schema definitions.

When a schema registry is configured, select the schema name for each stream to ensure correct deserialization.

Streams in Another AWS Account

A Kinesis stream can live in a different AWS account than your Onehouse data plane. Kinesis has no resource-based policy, so cross-account reads go through IAM role assumption: you create a role in the stream's account, and Onehouse assumes it (via AWS STS) to discover and read the stream.

Set the optional IAM Role ARN field when adding the source (kinesis.cross.account.role.arn in CREATE SOURCE). Leave it empty for streams in the data plane's own account — that is the default and needs no extra setup.

	Same account (default)	Different account
IAM Role ARN	Leave empty	Role ARN in the stream's account
Credentials used	The data plane's own node role	The assumed cross-account role
Customer stack setup	kinesisDataLoad.enabled = true	Above, plus the role ARN in kinesisDataLoad.crossAccountRoleArns (stack version 1.0.46+)

Before adding the source, complete the IAM setup in Onehouse Cross Account Data Access — the role's trust policy, its Kinesis read permissions, and the crossAccountRoleArns allow-list entry. Onehouse assumes the role while the source is being created in order to list the streams, so a role that is not set up correctly fails at that point with the AWS authorization error naming the role, and no source is created.

One Source per AWS Account and Region

Each Kinesis source covers one AWS account — identified by its IAM Role ARN — in one AWS region. Both are fixed when the source is created. A project can have one source without a role ARN (the data plane's own account) plus one source per cross-account role; adding a second source that reuses a role ARN already in use is rejected. Create the source once per role and reuse it for every stream it can reach.

To ingest from streams in several regions of the same account, create a separate IAM role per region and one Onehouse source per role. This is also what the least-privilege policy in the cross-account guide produces, since it scopes the Kinesis read actions to stream ARNs in a single region.

Schema Registry in the Stream's Account

If the stream's records are described by a Glue Schema Registry that lives in the same account as the stream, Onehouse reads that registry with the same assumed role — there is no separate ARN to configure. The role's permissions policy must grant the Glue Schema Registry read actions in addition to the Kinesis read actions; both are covered by the policy in the cross-account guide.

Creating a Flow from a Kinesis Source

Select Streams

After selecting your Kinesis data source, Onehouse automatically discovers available streams. From the stream list, select the streams you want to ingest.

Auto-Capture

Enable auto-capture to continuously monitor and ingest new streams that match a filter pattern. When auto-capture is active, Onehouse periodically scans for new streams and automatically creates flows for any that match the configured regex filter. A common schema, key, and transformation configuration is applied to all auto-captured streams.

Starting Sequence Number

Configure where Onehouse begins reading from each shard:

Option	Description
Latest (default)	Start reading from the most recent sequence number in each shard. New records published after flow creation are ingested.
Trim Horizon	Start reading from the oldest available sequence number in each shard, ingesting all retained records from the beginning of the shard's retention period.

Per-Stream Configuration

For each selected stream, configure:

• Destination Table Name — the name of the Hudi table created in your lakehouse.
• Record Keys — columns that uniquely identify each record.
• Precombine Key — the field used to resolve duplicate records in merge-on-read tables.
• Partition Keys — columns used to partition the destination table.
• Transformations — optional data transformations applied during ingestion (see Transformations).
• Quarantine — optionally route invalid records to a quarantine table for inspection.

Permissions

The following role-based access control (RBAC) actions govern Kinesis flow operations:

Action	Description
Create Stream	Create a new flow from a Kinesis source.
Clone Stream	Duplicate an existing Kinesis flow configuration.
Edit Stream	Modify the configuration of an existing Kinesis flow.

Usage Notes

• Kinesis sources are available only for AWS-linked projects.
• Stream discovery requires that the linked cloud account has read access to the Kinesis Data Streams service.
• The starting sequence number configuration applies to all shards within a stream.
• A single source covers exactly one AWS account and one region — see Streams in Another AWS Account and Onehouse Cross Account Data Access for the required cross-account IAM role setup.
• The IAM Role ARN is shown but not editable when editing an existing source in the console. To read a stream through a different role, add a new source for that account.