Key Vault Firewall access by Azure App Services

More than a few support cases are created when Key Vault users wisely decide to enable the Firewall settings on their vault.  Then the problem begins:  Azure App Service websites are no longer able to access the Key Vault, and end up creating an error message like the following:

Operation returned an invalid status code 'Forbidden'

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

Forbidden is means that the HTTP response code is 403.  In the past, 403 has always meant:

  1. The identity was confirmed
  2. The resource is correct
  3. Either the lack of an access policy OR an access policy itself prevents the resource from being obtained by the confirmed identity

However, since the Azure Key Vault Firewall And Virtual Networks feature was released, it can also mean that the client is not allowed by virtue of the calling location. Here's a look at that part of Key Vault:

If the Key Vault Firewall/VNet is activated, there are exactly three ways to get into the Key Vault (given that an access policy is also in place):

  1. Be on the same Virtual Network as the Key Vault
  2. Be on the Firewall IP address whitelist
  3. Be a "Trusted Microsoft Service"

I think that the first two are self-explanatory.  However, #3 is causing a headache for many customers.

This is the explicit list of "Trusted Microsoft Services" (as of the time of this authoring):

Trusted services

Here's a list of trusted services that are allowed to access a key vault if the Allow trusted services option is enabled.

Trusted service Usage scenarios
Azure Virtual Machines deployment service Deploy certificates to VMs from customer-managed Key Vault.
Azure Resource Manager template deployment service Pass secure values during deployment.
Azure Disk Encryption volume encryption service Allow access to BitLocker Key (Windows VM) or DM Passphrase (Linux VM), and Key Encryption Key, during virtual machine deployment. This enables Azure Disk Encryption.
Azure Backup Allow backup and restore of relevant keys and secrets during Azure Virtual Machines backup, by using Azure Backup.
Exchange Online & SharePoint Online Allow access to customer key for Azure Storage Service Encryption with Customer Key.
Azure Information Protection Allow access to tenant key for Azure Information Protection.
Azure App Service Deploy Azure Web App Certificate through Key Vault.
Azure SQL Database Transparent Data Encryption with Bring Your Own Key support for Azure SQL Database and Data Warehouse.
Azure Storage Storage Service Encryption using customer-managed keys in Azure Key Vault.
Azure Data Lake Store Encryption of data in Azure Data Lake Store with a customer-managed key.

 

You will look at the list and find "Azure App Services".  However, it's the next column which is just as important:  the only scenario where an App Service is trusted is for deployment of App Service Certificates.

That's all. That's it.

  

To show this, I enabled the firewall with the trusted Microsoft Services, as well as Audit logging in my Azure Key Vault and attempted to access it from my Azure App Service.  I get this message in my website:

Looking in my audit logs, I find the following:

    "time": "2019-01-03T19:14:18.2665709Z",
    "category": "AuditEvent",
    "operationName": "SecretGet",
    "resultType": "Success",
    "resultDescription": "Client address (00.00.00.00) is not authorized and caller is not a trusted service",
    "correlationId": "c8e4f5ef-6def-411d-9a24-ab43f00d0566",
    "callerIpAddress": "00.00.00.00",

As soon as I added the IP address of my web site...



to the Firewall whitelist...



the site worked as expected. 





This is by design as an App Service is not a "Trusted Microsoft Service".

I hope this relieves your frustration with the setting.

Comments (1) -

Wow! your post saved my project! Spent hours investigating this. Thanks.

Add comment