Azure Data Factory
Azure Data Factory is the platform that solves such data scenarios. It is the cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale.
Guzzle only supports ADF as an External job.
Steps to create Datastore for Azure Data Factory#
Click on the action button (
) from the Datastores section in Left Navigation and select Azure Data Factory connector. Alternatively users can launch from Create New Datastore link in Activity authoring UI.Enter the Datastore name for the new datastore and click Ok
Update the connection name or leave the default. You can refer to Connection and Environments for more details
Enter the additional properties for the ADF as described below:
| Property | Description | Required |
|---|---|---|
| Credential Type | Authentication type to use when connecting to ADF. Following mechanisms are supported: Service principal : To use Service principal which to access selected container or folder in the storage account. Follow the steps at Register your application with an Azure AD tenant to create Application Registration and capture following information like Application (client) ID, Client secret, and Directory (Tenant) ID. Also ensure following permission: As source: Grant Execute permission for all parent folders, along with Read permission for the files to copy. Alternatively, in Access control (IAM), grant at least the Storage Blob Data Reader role at container or storage account level. As sink(target): Grant Execute permission for all parents folders, along with Write permission for the sink folders. Alternatively, in Access control (IAM), grant at least the Storage Blob Data Contributor role at container or storage account level. | Yes |
| Client ID | Service principal’s client id | Yes |
| Client Secret | Specify the service principal secret.. For providing client secret the following options are available: 1. Manual: Provide client secret directly. 2. Azure Key Vault: To use Azure key vault feature user have to integrate Key Vault with Guzzle for that visit here. Give value of the key vault name and secret name where client secret is stored in Azure Key Vault instance. | Yes |
| Tenant ID | Directory ID of the service principal | Yes |
| Subscription ID | Users Azure subscription ID | Yes |
| Resource Group Name | Specify the resource group name under which created Azure Data Factory Instance | Yes |
| Factory Name | Specify the name of Data Factory | Yes |
- Save the Datastore config. Optionally you can also Test the connection.
Interface for ADF datastore#
