Skip to main content

Azure Data Factory

Azure Data Factory is the platform that solves such data scenarios. It is the cloud-based ETL and data integration service that allows you to create data-driven workflows for orchestrating data movement and transforming data at scale.

Guzzle only supports ADF as an External job.

Steps to create Datastore for Azure Data Factory#

  1. Click on the action button (image alt text) from the Datastores section in Left Navigation and select Azure Data Factory connector. Alternatively users can launch from Create New Datastore link in Activity authoring UI.

  2. Enter the Datastore name for the new datastore and click Ok

  3. Update the connection name or leave the default. You can refer to Connection and Environments for more details

  4. Enter the additional properties for the ADF as described below:

Credential TypeAuthentication type to use when connecting to ADF.
Following mechanisms are supported:

Service principal : To use Service principal which to access selected container or folder in the storage account. Follow the steps at Register your application with an Azure AD tenant to create Application Registration and capture following information like Application (client) ID, Client secret, and Directory (Tenant) ID.

Also ensure following permission:
As source: Grant Execute permission for all parent folders, along with Read permission for the files to copy. Alternatively, in Access control (IAM), grant at least the Storage Blob Data Reader role at container or storage account level.
As sink(target): Grant Execute permission for all parents folders, along with Write permission for the sink folders. Alternatively, in Access control (IAM), grant at least the Storage Blob Data Contributor role at container or storage account level.
Client IDService principal’s client idYes
Client SecretSpecify the service principal secret..
For providing client secret the following options are available:
1. Manual: Provide client secret directly.
2. Azure Key Vault: To use Azure key vault feature user have to integrate Key Vault with Guzzle for that visit here. Give value of the key vault name and secret name where client secret is stored in Azure Key Vault instance.
Tenant IDDirectory ID of the service principalYes
Subscription IDUsers Azure subscription IDYes
Resource Group NameSpecify the resource group name under which created Azure Data Factory InstanceYes
Factory NameSpecify the name of Data FactoryYes
  1. Save the Datastore config. Optionally you can also Test the connection.

Interface for ADF datastore#