Azure Key Vault is a cloud-based service that helps to store cryptographic keys and secrets used by apps and services. It is centralizing storage of application secrets that allows user to control their distribution.
When using Key Vault, Guzzle no longer need to store secrets and passwords value in application. User can configure Key Vault name and Secret key instead of native secrets and password in Guzzle. Azure key vault requires proper authentication and authorization to get access of secrets. Guzzle uses the managed identity to access Azure Key Vault resource.
- Guzzle VM
- User-Assigned Managed Identity
- Azure Key Vault
- Databricks Workspace
- Create user-assigned managed identity in resource group by click on
- Assign a user-assigned managed identity to Guzzle VM by navigating to the Guzzle VM and click Identity, User assigned and then
+Add. for more detailed steps click here
- Create azure key vault in resource group by click on
- Add Access Policy for user-assigned managed identity and databricks workspace by navigating to Azure Key Vault and click on Access Policy,
+ Add Access Policy.
- Select Configure from template as Secret Management and Select user-assigned managed identity as Principle click on
Addto create access policy.
- Do same thing for AzureDatabricks principle.
- Create Databricks Workspace in resource group by click on
- After that we create an Azure Key Vault-backed secret scope using the UI.
- Go to
https://<databricks-instance>#secrets/createScope. replace databricks-instance with your databricks instance url.
- Enter azure key vault name in scope name field. Scope name must be same as key vault name.
- Enter the DNS Name and Resource ID of Key Vault. These properties you will find by navigating to Azure Key Vault and click on Properties.
Createbutton to create secret scope. For more information visit the databricks create secret scope documentation.
When the secret value is required inside API, Batch, and Pipeline Execution it will use Microsoft Azure Key Vault API to fetch the secret value.
When jobs in running in databricks cluster and requires secrets value, it will get it using databricks key vault instance.