AZIdentity

Something that I've seen a bunch of times in Key Vault support cases is that the customer tries to use a token previously obtained to perform operations on Azure Services such as VMs, Websites, and even Key Vault to also access keys, secrets or certificates inside the Key Vault.  They get understandably confused because the requests made about the Key Vault work fine. It's just that the requests made for Key Vault values fail with 401.

Azure's OAUTH client credentials grant protocol requires that the resource of the Web API being used is passed to the authentication server.  The documentation states this:

This is the POST sample in the same documentation:

POST /contoso.com/oauth2/token HTTP/1.1
Host: login.microsoftonline.com
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&client_id=625bc9f6-3bf6-4b6d-94ba-e97cf07a22de&client_secret=qkDwDJlDfig2IpeuUZYKH1Wb8q1V0ju6sILxQQqhJ+s=&resource=https%3A%2F%2Fservice.contoso.com%2F

The highlighted area above is what I am referring to for this issue.  There are multiple resources in Azure; here are a few that I am aware of:
 

Here is an actual OAUTH request I made using the Key Vault resource URI (with some values removed):

POST https://login.microsoftonline.com/<tenant removed>/oauth2/token HTTP/1.1
Accept: application/json
Content-Type: application/x-www-form-urlencoded
Host: login.microsoftonline.com
Content-Length: 182
Expect: 100-continue

resource=https%3a%2f%2fvault.azure.net&client_id=<removed>&client_secret=<removed>&grant_type=client_credentials

HTTP/1.1 200 OK
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
x-ms-request-id: 1c4a97a5-00f5-4d32-bbc2-4be61201c201
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: fpc=AXuxcQCE2KRImJ4RCkjegB_EqXEQAQDjhy1f3lbWCA; expires=Sun, 30-Dec-2018 16:10:39 GMT; path=/; secure; HttpOnly
Set-Cookie: x-ms-gateway-slice=013; path=/; secure; HttpOnly
Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly
Date: Fri, 30 Nov 2018 16:10:39 GMT
Content-Length: 1324

{"token_type":"Bearer","expires_in":"3600","ext_expires_in":"3600","expires_on":"1543597839","not_before":"1543593939","resource":"https://vault.azure.net","access_token":"<removed>"}

You can also see this value in the access token when it's decoded:

It's important to note that the resource value does not have a trailing slash!  There are other resource URIs that work fine either with or without the trailing slash, but the Key Vault resource URI does not work with it.

Why are there two Resource URIs for Key Vault?

It's because Key Vault has two different sets of resources which it uses:  the Management Plane and the Data Plane.

The Management plane is the set of operations which are performed on the Key Vault itself, such as:

• Creating/changing access policies
• Creating a new Key Vault
• Adding IP restrictions/VNet
• Setting Tags
• Access Control
• etc

Access to ARM APIs for Key Vault are set by RBAC.

The Data plane is the set of operations which allows others to retrieve/set the values of objects contained inside the Key Vault:

• Keys
• Secrets
• Certificates
• Storage Objects

These objects are governed by the Access Policies, and therefore require a different set of APIs than the Management APIs.

I hope this clears up any confusion about making calls to the Key Vault using REST. 

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

Today's blog post comes from Fabian Gonzales, a CSS support engineer working in our new Costa Rica support center.

I was recently working with a customer that was configuring a PowerShell Run Book for an Automation Account, the purpose of that script was to Encrypt Virtual Machines Disks using the Automation account to authenticate with Azure AD and then perform the encryption operation.

The connection looked similar to this:

The customer was getting 'Forbidden'  status code: 403 when running the script. It is known that the 403 code is mainly related to access policies issues, but the customer did not know to which identity he needed to provide a policy, he had already added himself.

I reproduced the customer issue and found out that when you create an Automation Account and you want it to have an Identity(Service Principal) in your Azure AD. You must select 'Yes' to 'Create Azure Run As account'

The 'Run as account' will generate a new Service Principal for the Automation Account(And adds it to the subscription level as a contributor) that you will be able to use for authenticating to Azure AD. if you go to Azure Active Directory > App Registrations > My apps you will see the Service Principal, the name format of the Service Principal will be: automationaccountname + _ + SP key

Then you just need to add the Service Principal as part of you Azure Key Vault Access policies and then you will be able to use the Runbook + Key Vault for the purpose you have. In this case, we added the SP to the Access policies using the Key management template and then the customer was able to Encrypt his Virtual Machines successfully using the script.

Note: This Run as account can be used for different purposes, you can also limit the account by removing it from the subscription contributor role. For additional information about the 'Run As accounts' refer to https://docs.microsoft.com/en-us/azure/automation/manage-runas-account

Thank you Fabian!

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

Hello, my name is Nate Harris and I am part of the Azure Identity Incubation team.  My role on the team is to become a SME with Role Based Access Control (RBAC), and ensure that Support has all the tools and knowledge necessary to deliver top notch support to our customers.  To that end I will be creating these types of Blogs posts as I come across issues that I am working in and find that there is a clear knowledge gap with our customers and/or the support organization.  What follows is one such example:

Scenario: You are the Global Admin for your Azure subscription and have created an Azure AD Group called TestGroup02.  This group will contain your networking team members who will manage your load balancers.  You assign the Reader role to the group for the Resource Group that the load balancers are found within.  At the Load Balancer resource you add the group to the Network Contributor role so that they can manage the Load Balancer resources.  However when a user from that group logs in and attempts to manage the existing Inbound NAT rules, they are met with this error message:

With RBAC the permissions are compartmentalized to a Resource or Resources and flow from Parent to child.  As we cannot, today, assign Deny permissions, RBAC permissions are therefore cumulative in nature.  In this scenario it appears that the User should have permissions to modify a Child resource, in this case an Inbound NAT rule on the Load Balancer, but that is not the case.  Let's walk through the configuration around this scenario.

Here is the Azure AD group and it's membership shown.

This group has been added as a Reader to the Resource Group to which the Load Balancer belongs

At the Load Balancer we have added this Group to the Network Contributor role.  So you can see the Implicit role (Reader) and Explicit role (Network Contributor).

So what's the cause, you ask?  Well a clue can be found within other Inbound NAT rules that the admins are able to view.  When they look into one of these other NAT rules that the Admin created we see a key difference in the destination choices:

What the LoadBalancerAdmin group member is able to view:

And what the Subscription Admin is able to view:

We see that the LoadBalancerAdmin member sees fewer VMs than the Subscription Admin, which ultimately is key to solving our issue.  While the LoadBalancerAdmin AD group is a Reader for the Resource Group with which the Load Balancer resides, it does not have at least Reader role rights outside of that Resource Group.  So for Inbound NAT rules that have been created by the Subscription Admin that point to resources outside of that Resource Group, the LoadBalancerAdmin group member would only ever see Access Denied when attempting to open the properties of those rules.

Solution: Grant the User/Group at least Reader role rights to all other necessary resources.

Today's blog post comes from Jason Fritts, a support engineer on the Azure Identity Support Team in Microsoft CSS.

 I was recently working with a customer who was trying to automate some Key Vault management tasks such as updating a Key Vault's access policy but was running into access errors like this one:

The customer was authenticating to Azure via service principal clientID\clientSecret like the following:
             

$clientid = "<service principal client id">

$password = ConvertTo-SecureString "<service principal client secret>" -AsPlainText -Force

$psCred = New-Object System.Management.Automation.PSCredential($clientId, $password)

Login-AzureRMAccount -Credential $pscred -ServicePrincipal -TenantID <Tenant ID>

 We verified that the customer had already granted management plane (RBAC) access for this service principal to the Key Vault. We also verified that the customer had granted data plane (Key Vault access policy) permissions to this service principal.

 
After troubleshooting, it was determined that the permissions required were related to Graph calls made to Azure Active Directory which require at least read access.  By default, a user principal has the necessary permissions to make Graph calls to its own Azure Active Directory tenant, but service principals do not.  In order to grant a service principal the permissions to perform directory read operations, you can add the service principal to the Azure Active Directory "Directory Readers" built-in role with the following:

$sp = Get-AzureADServicePrincipal -ObjectId <object ID of service principal>

Add-AzureADDirectoryRoleMember -ObjectId (Get-AzureADDirectoryRole | where-object {$_.DisplayName -eq "Directory Readers"}).Objectid -RefObjectId $sp.ObjectId

NOTE: This process can only be performed in PowerShell currently with the Azure AD PowerShell module, you can not add service principals to AAD roles via the portal today.

Reference: https://docs.microsoft.com/en-us/powershell/azure/active-directory/signing-in-service-principal?view=azureadps-2.0#give-the-service-principal-reader-access-to-the-current-tenant-get-azureaddirectoryrole

After granting the service principal the required permissions, you can now run operations such as Set-AzureRmKeyVaultAccessPolicy with service principal credentials.

Thank you Jason!

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

Today's post comes via information from Steve Lewis, a Sr. Azure Cloud Engineer (ACE) in CXP.  He's delivering support to our top Azure customers.

An interesting aspect of being a Support Engineer (or an Azure Cloud Engineer) is that we learn how to read error messages better than anyone.  Customers have grown to ignore error messages as they so often are not useful.  However, this post shows a really good error message from Azure that you should know about.

In case you're not familiar with it, CloudShell is an awesome part of the Azure portal that allows you to execute PowerShell cmdlets (as well as Bash) in the browser.  Pressing on the ">_" button in the top of the screen gets you to the CloudShell environment:

We had a customer that recently opened up a critical situation support case regarding Azure Key Vault and CloudShell. He'd used CloudShell to create the vault and then immediately went to create the secret inside of it, but was denied access to do so.

An interesting note about Key Vault in the Azure Portal:  the creator of a Key Vault is automatically granted an Access Policy.  In fact, you can see it in the blade before you create it:

Even more:  when a Key Vault is created using Powershell on the desktop, an Access Policy is created as well:

 However, we have a different experience when running the same in CloudShell:

You can clearly see that there are no access policies set for this Key Vault upon creation.  Not only that, it tells us that there are no access policies set and how to set them.  That's why when I try to set a secret value inside the key vault, it does not work:



This is one of those cases where if the customer had carefully read the output, they would have understood that they need to set an access policy.  It's been known that there are sometimes really arcane and useless error messages that come from Microsoft resources, but this is not one of those times.  However, before doing that, let's look at the principal we are working under:

It's highly unlikely that you would want to set an access policy for this user, but would rather do it for your own security principal. Let's log in properly:

Now we can can work properly with this.  Let's change to the right subscription, set the access policy (using your account name), and check the vault again:

Now it has an access policy:

And we can finally set the secret:

Again, thanks to Steve Lewis for figuring this out and sharing with me for this blog.

 I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

A couple of support cases have asked the question:  how to change the sku of an existing Key Vault?  

To be clear about the skus - there are two pricing tiers available when creating a new Key Vault: standard and premium.  The premium level adds Thales HSM (Hardware Security Modules) to your Key Vault.  See more about that feature here.

The process is simple but not obvious.  Powershell is always our friend when it comes to making changes in Key Vault, as this ability is not available in the portal.

I can see in the KVExplorer that my present SKU is "Standard":

To change that to Premium, I can run this Powershell script:

Login-AzAccount
$vaultResourceId = (Get-AzureRMKeyVault -VaultName "AzIdentity").ResourceId
$vault = Get-AzureRmResource -ResourceId $vaultResourceId -ExpandProperties
$vault.Properties.sku.name = "Premium" # or "Standard"
Set-AzureRMResource -ResourceId $vaultResourceId -Properties $vault.Properties

A refresh of the KVExplorer shows that I now have the Premium level:

It's important to note that this is a safe operation.  If you were to move from Premium to Standard, HSM keys will not longer be available for your Key Vault service, but they are not deleted.  Moving back to Premium allows the keys to be used once again.

While attempting to create (or recreate) a Key Vault, have you ever seen the error message "The name '<key vault name>' is already in use?

It can be frustrating as you may be trying to use a name which you know isn't being used anywhere else, or which you used recently on a key vault, deleted, and now want to recreate. 

This commonly occurs for these reasons:

1. The Key Vault that you just deleted has "Soft Delete" enabled on it. This means that the Key Vault still exists, but is not accessible.  It will remain this way for 90 days, at which point you can recover the key vault name.  This setting is in case the key vault is actually needed again.  You can check for this condition using the Powershell Get-AzureRmKeyVault cmdlet to find soft-deleted vaults:
   

   Get-AzureRMKeyVault -InRemovedState

   After this, you may choose to either revive the key vault using cmdlet "Undo-AzureRmKeyVaultRemoval":
   

   Undo-AzureRmKeyVaultRemoval -VaultName <vaultname> -ResourceGroupName <groupname> -Location <location>

   Or you can "purge" the key vault in order to ensure it's really gone:
   

   Remove-AzureRmKeyVault -VaultName <vaultname> -Location <location> -InRemovedState

   
   
   

2. NOTE:  The Key Vault Product team has changed the validation scheme for domain names.  If your domain name is not available, but you know that it is because you recently deleted it, it may have gotten into the "orphaned DNS" scenario below.  However, the validation has changed to allow the DNS resolution to check if the Key Vault exists.  If it does not exist, the name should become available after 10 minutes.  If it still does not work, then your problem is likely that it's either being used by another Key Vault or it's soft-deleted.

   The Another reason for this condition is an "orphaned" DNS value caused by deleting/recreating the key vault too fast.   You can confirm this problem by trying to access the vault in the browser: https://<vaultname>.vault.azure.net.  You should get an HTTP 403, indicating that the key vault still exists in the DNS, but isn't available in actuality:
   
   
   
   This is a known condition which the Azure Key Vault is working on.  The fix for this right now is to email the product team directly: AzureKeyVault _@_ Microsoft.com, or open a support case and we will help you with this problem.
   
   
   
3. It's possible that the name you've chosen for your Key Vault is already taken.  If the name you've chosen is more like "TestKeyVault" or "AzCloudKv" than "WE-NameOfMyCompany-RG-Eagle-18h6", and you have never had this name on a Key Vault previously, then it's probably used by someone else.  The test above of navigating to the Key Vault to get an HTTP 403 will still hold true, but it will mean you don't have access to that Key Vault rather than it is an orphaned DNS record.
   
   
   

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

Today I helped a customer confused about how to properly download a certificate from Key Vault that contains both the public and private keys.  The current method for downloading a certificate will retrieve only the public key.  Having both parts of the certificate is essential for SSL binding and is necessary for situations such as sending a client certificate for authentication.

As I have shown in previous posts, but perhaps not described fully, is that the necessary method for retrieving the certificate with both keys is to download it as a secret. It does not mean that you re-upload the certificate into the Secrets blade.  It means that you leave the certificate exactly where it is, yet when downloading the certificate as a secret, use the secrets endpoint.  Here's my example:

First, let's establish that I have a certificate, listed as a certificate, in my Key Vault:

The certificate "KVExplorerCom" is an App Service Certificate uploaded to a Key Vault named "kvexplorer".  

Using C# to download the certificate as a certificate reveals the following information:

First, I get a really nice CertificateBundle object displaying the downloaded certificate in a pleasant form.  The values are readily observable.  Another nice and important feature is a byte array called Cer which describes the certificate, and I can use this to convert to an X509 certificate, which I do in the next line of code.  Let's look at that now...

Now that I've created an X509 certificate in memory, examining it seems to go well, until you see that there's no Private Key.  That's the part many applications require for proper SSL binding.  This makes more sense when you realize that the public key for a certificate has a file extension which is ".cer", while the private extension is ".key".  

You can see more about the widely-used certificate file extensions here. https://blogs.msdn.microsoft.com/kaushal/2010/11/04/various-ssltls-certificate-file-typesextensions/

Now let's see what happens when I download the certificate as a secret.

This time I get a SecretBundle object, with the Content-Type set to a certificate, and the Value which is my secret in Base-64 encoding.  Once I Base-64 decode the Value of this secret, I can then convert it to a complete X509 certificate, with public key:

Final note about this downloaded certificate:  if you save the X509 certificate as a PFX, it will not have a password.  This is because of the way that Key Vault stores the certificate internally.  You can see my previous post for instructions on how to download the certificate using Powershell and simultaneously generate a password for it as it's being written to disk.

I am providing feedback to our product team that this is a confusing mechanism in Key Vault and could use improvement.  

Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

Some customers use CLI instead of Powershell as they are working in non-Windows environments.  A support case came in where a customer asked how to download an App Service Certificate from his Key Vault using CLI alone.  The Support Engineer who owned the support case downloaded the certificate using the certificate download CLI command:

# az keyvault certificate download -f certificatename.pfx -n edgeui --vault-name keyvaultname -e DER

What he noticed was that this command only downloads the public part of the certificate.  This is documented:
https://docs.microsoft.com/en-us/cli/azure/keyvault/certificate?view=azure-cli-latest#az-keyvault-certificate-download

Of course, the customer wants to be able to download the entire certificate, both public and private. I've never used CLI previously, but set out to find the right way to download this certificate.  It was far easier than I expected, and it is accomplished by downloading the certificate as a secret, like I described in my previous post.  The exact command is:

# az keyvault secret download --file <path>\<nameofcert>.pfx --name <NameOfSecretOrCertificate> --vault-name <VaultName> 

It is important to note that the downloaded PFX has a blank password.  It will install easily and contain the private key. 

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

A support case that we have seen come through a few times is the question about the BYOK toolset needed to transfer HSM-protected keys into Azure Key Vault. 

The question is whether that toolset is available in 32-bit version, as some customers are using 32-bit workstations to generate the HSM key.  Unfortunately, the product team has confirmed that there is no 32-bit version of this toolset.  It comes in 64-bit mode only.

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity