This blog will have a slight change in scope

This blog was originally intended to be a place for my previous support team (the Azure Identity Incubation Team) to post content related to the various technologies that we worked on.  It turned out to be a place where I have been posting about Azure Key Vault for the past year.  To reflect this, I am changing the focus of this blog to be exclusively Key Vault.  I will be leaving the existing non-Key Vault posts in place until a better location is found for them.  Thank you.

 

Getting It Right: Key Vault Access Policies

Introduction

Azure customers of all sizes are using ARM templates, Powershell, and CLI in order to create Registered Applications/Service Principals and then assign them to an Access Policy in the Key Vault in order to perform operations.  In performing these assignments, it is frequent that customers call into Key Vault support trying to understand why their Service Principal is receiving an HTTP 403 (Forbidden) when attempting to access the Key Vault data plane.  I want to clear up what's going on with this so that you have a better understanding and can avoid this problem.

Key Vault Access Policy In the Portal

Let's review a Registered Application that I've newly created using the portal.  We can clearly see all of the GUIDs associated with this Registered Application which are important to creating the Access Policy.

So now I create an Access Policy for this Registered Application in my Key Vault:

 

Next, let's get an access token using the Application ID (Client Id) and Client Secret (Key) that I created in the Application Registration blade in AAD:

Request

POST https://login.microsoftonline.com/52e2bc67-bdf5-4b30-88e1-6081e5331b3e/oauth2/token HTTP/1.1
Accept: application/json
Content-Type: application/x-www-form-urlencoded
Host: login.microsoftonline.com
Content-Length: 168
Expect: 100-continue

resource=https%3a%2f%2fvault.azure.net&client_id=70a3cf30-ef51-43ab-a08f-7cccac109b3c&client_secret=<removed>&grant_type=client_credentials

 

Response (some info removed)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Fri, 17 May 2019 14:29:02 GMT
Content-Length: 1324

{"token_type":"Bearer","expires_in":"3599","ext_expires_in":"3599","expires_on":"1558106943","not_before":"1558103043","resource":"https://vault.azure.net","access_token":"<removed>"}


Now that I've got the token, I can make a successful request to Key Vault for a secret:

Request

GET https://akvaccesspolicy.vault.azure.net/secrets/MySecret/?api-version=7.0 HTTP/1.1
x-ms-client-request-id: 7b431436-fb67-4f44-b7c3-b91822f5d0aa
accept-language: en-US
Authorization: Bearer <removed>
User-Agent: FxVersion/4.7.3324.0 OSName/Windows10Enterprise OSVersion/6.3.17763 Microsoft.Azure.KeyVault.KeyVaultClient/3.0.0.1
Host: akvaccesspolicy.vault.azure.net


Response (some info removed)

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Expires: -1
Date: Fri, 17 May 2019 14:55:24 GMT
Content-Length: 223

{"value":"SecretValue123","id":"https://akvaccesspolicy.vault.azure.net/secrets/MySecret/79511656b229401bbae219d4d21fc4b6","attributes":{"enabled":true,"created":1558104763,"updated":1558104763,"recoveryLevel":"Purgeable"}}


You may be looking at this and wondering why I am walking you through a standard setup and request/response.  It's because I want to show you that this works as expected, even though the next step is the confusing part - the values that are kept in the Key Vault for an access policy. 

Key Vault Properties

Now look at the Key Vault using Powershell:

And more specifically, check out the Access Policy for the AKVAccessPolicyServicePrincipal:

There are three important things to note here:

  • There is no Application Id listed as a property of this Access Policy
  • The ObjectID listed here (92ba...) does not match the ObjectId (7da4...) shown in the registered application (see above)
  • The Application Id (70a3...) of the registered application from above is shown in the Display Name section

At this point you should be wondering what is going on with this, so let's figure out where the ObjectID 92ba... came from.

Enterprise Applications

Go back into the Azure portal, AAD Service, then into the "Enterprise Registrations" (have you ever even looked here before?):

           

                           

Now we see that this Enterprise Application - with the same name and Application Id as my registered application - but a different ObjectID - is what we've registered in the Key Vault Access Policy!  But what in the world is an Enterprise Application?

Here are a couple of important definitions from official documentation:

  • Application objects - Although there are exceptions, application objects can be considered the definition of an application.
  • Service principals - Can be considered an instance of an application. Service principals generally reference an application object, and one application object can be referenced by multiple service principals across directories.

One additional really important piece of information from the above link:
You can manage service principals in the Azure portal through the Enterprise Applications experience. 

Now we understand - a Service Principal is NOT the same as a Registered Application and for Key Vault, we do not give an access policy to a Registered Application but to a Service Principal related to the Registered Application.  

 

Key Vault Access Policy via Powershell

Let's get back to Powershell and properly creating Access Policies.  I am going to delete the above Access Policy and attempt to recreate it using Powershell. There are three ways of creating an Access Policy:

  1. Set-AzureRmKeyVaultAccessPolicy [-VaultName] <String> [-ApplicationId <Guid>] -ObjectId <String> -PermissionsToKeys all
  2. Set-AzureRmKeyVaultAccessPolicy [-VaultName] <String> -PermissionsToKeys all -ServicePrincipalName <String>
  3. Set-AzureRmKeyVaultAccessPolicy [-VaultName] <String> -PermissionsToKeys all -UserPrincipalName <String>

For most people with an Application Id and Object Id, #1 might seem to do the trick, so let's see what happens when we use this.  Let's also have a reminder of what it looks like when properly created in the portal:



Upon running this cmdlet, we are immediately met with a request for an ObjectId.  I can put in the ObjectId of the Registered Application and it seems fine until I try to use it (I won't show that).  Let's see what that looks like in the Key Vault:

So that really might look ok to most people, but as you noticed above, that's not what we really want.  The Object Id is not correct (we know that now), and the correct working Access Policy does not have an Application Id but this one does.  I reminded you what a properly registered Access Policies looks like, so let's look at the one we just created:



You can see that the name is correct, but the icon and coloring is different.  This should be a clue that something is wrong.  Let's look at doing this correctly in one of two ways:

  1. Use the Object ID of the Enterprise App.  This is identical to the Access Policy we created earlier in the portal, and the icon looks correct:


  2. Use the Application Id of the Registered Application as the Service Principal name. This automatically extracts the Enterprise Application Object ID and places it into Object ID of the Key Vault properties, and also populates the Display Name - exactly like above.

 

Application Id

At this point you are probably asking what an Application Id is for in an Access Policy.  Go back to the portal and into the new Access Policy blade.  Notice the "Authorized Application" part at the bottom of this image:



The Application Id property of a Key Vault refers directly to that "Authorized Application" part of an Access Policy. This is for On-Behalf-Of Authorization scenarios which means that authorization is granted to a specific user only via a specific application. Without other Access Policies, the user cannot access the Key Vault without the app, and the app cannot access the Key Vault without the user.  Most organizations will not use this feature, but I know that some have. 

Conclusion

It's important to know exactly the correct Object ID to use in an Access Policy.  The difference between a registered application and an Enterprise App/Service Principal seems to be the #1 most confusing part of this.  However, I hope this blog post clears up the issues around this and makes your life easier when troubleshooting your Access Policy issues.

Finally: I want to thank Scott Cottrille, Principal Group Software Engineering Manager of Azure Key Vault for the inspiration for this post.

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

 

Azure Key Vault Documentation "Try It" Pages

A number of customers call into Microsoft Support when they go to the documentation for Azure Key Vault REST API and use the "Try It" pages.  This is one of those pages:

https://docs.microsoft.com/en-us/rest/api/keyvault/createkey/createkey#code-try-0

 

They go through the process of putting in the correct information only to find that they get a 401 or a 404 or some error saying that it's not working.  Their real concern is that  they do not understand what's going on, they've done something wrong and when they go to implement code to call Key Vault in production, it's not going to work.  Unfortunately, it's the "Try It" page that is not working correctly, and this is why:

The "Try It" test harness is built using Swagger. These "Try It" pages are automatically generated, and given that Azure REST APIs mainly work with ARM, the resource that Swagger uses to generate is always https://management.core.windows.net/.  However, as I've previously described, the correct resource to use is https://vault.azure.net (no slash!).  Let's look at the resource (also known as the audience) of the token generated for the above request:

Therefore this request can never work from the Try It Pages.  The product team is aware of this issue and is working to fix it. There is no need to open a support request.

Hope it helps! Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

 

 

Maximum Number of Objects in a Key Vault

This one is short and sweet:  people want to know the maximum number of objects in an Azure Key Vault.  Per the Azure Key Vault product team:

  • There is no limit on number of keys, secrets, or certificates in a Key Vault.
  • There is no maximum number of versions of secrets, keys or certificates.
  • The maximum number of access policies in a Key Vault as of March 2019 is 1024.

I hope this helps.

Key Vault Firewall access by Azure App Services

More than a few support cases are created when Key Vault users wisely decide to enable the Firewall settings on their vault.  Then the problem begins:  Azure App Service websites are no longer able to access the Key Vault, and end up creating an error message like the following:

Operation returned an invalid status code 'Forbidden'

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'

Forbidden is means that the HTTP response code is 403.  In the past, 403 has always meant:

  1. The identity was confirmed
  2. The resource is correct
  3. Either the lack of an access policy OR an access policy itself prevents the resource from being obtained by the confirmed identity

However, since the Azure Key Vault Firewall And Virtual Networks feature was released, it can also mean that the client is not allowed by virtue of the calling location. Here's a look at that part of Key Vault:

If the Key Vault Firewall/VNet is activated, there are exactly three ways to get into the Key Vault (given that an access policy is also in place):

  1. Be on the same Virtual Network as the Key Vault
  2. Be on the Firewall IP address whitelist
  3. Be a "Trusted Microsoft Service"

I think that the first two are self-explanatory.  However, #3 is causing a headache for many customers.

This is the explicit list of "Trusted Microsoft Services" (as of the time of this authoring):

Trusted services

Here's a list of trusted services that are allowed to access a key vault if the Allow trusted services option is enabled.

Trusted service Usage scenarios
Azure Virtual Machines deployment service Deploy certificates to VMs from customer-managed Key Vault.
Azure Resource Manager template deployment service Pass secure values during deployment.
Azure Disk Encryption volume encryption service Allow access to BitLocker Key (Windows VM) or DM Passphrase (Linux VM), and Key Encryption Key, during virtual machine deployment. This enables Azure Disk Encryption.
Azure Backup Allow backup and restore of relevant keys and secrets during Azure Virtual Machines backup, by using Azure Backup.
Exchange Online & SharePoint Online Allow access to customer key for Azure Storage Service Encryption with Customer Key.
Azure Information Protection Allow access to tenant key for Azure Information Protection.
Azure App Service Deploy Azure Web App Certificate through Key Vault.
Azure SQL Database Transparent Data Encryption with Bring Your Own Key support for Azure SQL Database and Data Warehouse.
Azure Storage Storage Service Encryption using customer-managed keys in Azure Key Vault.
Azure Data Lake Store Encryption of data in Azure Data Lake Store with a customer-managed key.

 

You will look at the list and find "Azure App Services".  However, it's the next column which is just as important:  the only scenario where an App Service is trusted is for deployment of App Service Certificates.

That's all. That's it.

  

To show this, I enabled the firewall with the trusted Microsoft Services, as well as Audit logging in my Azure Key Vault and attempted to access it from my Azure App Service.  I get this message in my website: