Snowflake Integration

Integration of Snowflake with the Data Mesh Manager.

When setting up an integration between Data Mesh Manager and Snowflake, we provide two options:

1. Ingestion based Integration

The integration is managed inside of Data Mesh Manager, you configure the connection and a sync schedule, and you are ready to start syncing with Snowflake.

No additional deployments are needed.

2. Connector based Integration (see Snowflake Connector)

Our connectors are based on the Datamesh Manager SDK and provide support for more use cases. You get the asset syncing capabilities of the ingestor based integration. And in addition you can integrate directly with the permission component of Snowflake to dynamically apply changes through the Access Approval Workflow on the data platform.

If you have a complex network topology or direct integration with Data Mesh Manager is not possible due to governance restrictions, the Snowflake connector can be deployed in a DMZ of your network to provide additional security.

Snowflake Ingestion based Integration

You can directly integrate Snowflake with the Data Mesh Manager.

Prerequisites

You need to have a Data Mesh Manager Enterprise License or use the Cloud Edition. To enable the integration, you need to set APPLICATION_INGESTIONS_ENABLED to true in your environment. See Configuration for more information.

To start, navigate to Organization Settings > Integrations > Add Integration This will open a wizard that supports you to configure the integration.

Select the Integration Type

Configure the Credentials

Data Mesh Manager will ask you to provide the credentials to access Snowflake. At the moment, we support Programmatic Access Tokens and Key Pair Authentication. To find out more about the different authentication methods, please refer to the Snowflake documentation - Key Pair Authentication resp. Programmatic Access Tokens.

Note: The credentials are stored encrypted in the Data Mesh Manager database. If you want encryption enabled in your environment, please set a 64 hex character long APPLICATION_ENCRYPTION_KEYS in your environment (also see Configuration).

Configure Filters

You can configure filters to limit the assets that are synchronized. Include- and exclude filters are supported. For Snowflake, we support filters on Databases, Schemas, Tables and Views.

The filters can contain '*' as wildcard characters. This will match any number of characters.

Configure Schedule

Direct integrations support a schedule to synchronize the assets. We provide some predefined schedules, but you can also define your own schedule in the cron expression format.

Note: All schedules are configured in UTC, so make sure to take this into account when configuring your schedule. Please do not synchronize the assets more than once or twice per day. We reserve the right to disable the integration if this happens. You will be able to trigger a synchronization manually if you need an immediate update.

Complete the Integration configuration

Make sure to choose a unique name for the integration, check your configuration and click Save.

Next Steps

The integration is now configured and will be executed according to the schedule. To check the status of the integration, navigate to Organization Settings > Integrations. You find the status of the integration, together with the last 10 runs of the integrations there.

You will also be able to adjust the configuration of the integration at any time. Please note that we save the configuration in YAML format. The editor supports you with syntax validation.

Furthermore, you will be able to change and test the credentials in the edit view as well.

Note: The credentials will not be displayed in the edit view for security reasons. If you want to change them, you can add the new credentials and save the integration.

If you deselect the Enabled checkbox, the integration will be disabled and will not be executed anymore.

Snowflake Connector

see Snowflake Connector