AZIdentity

Today's blog post comes from Jason Fritts, a support engineer on the Azure Identity Support Team in Microsoft CSS.

 I was recently working with a customer who was trying to automate some Key Vault management tasks such as updating a Key Vault's access policy but was running into access errors like this one:

The customer was authenticating to Azure via service principal clientID\clientSecret like the following:
             

$clientid = "<service principal client id">

$password = ConvertTo-SecureString "<service principal client secret>" -AsPlainText -Force

$psCred = New-Object System.Management.Automation.PSCredential($clientId, $password)

Login-AzureRMAccount -Credential $pscred -ServicePrincipal -TenantID <Tenant ID>

 We verified that the customer had already granted management plane (RBAC) access for this service principal to the Key Vault. We also verified that the customer had granted data plane (Key Vault access policy) permissions to this service principal.

 
After troubleshooting, it was determined that the permissions required were related to Graph calls made to Azure Active Directory which require at least read access.  By default, a user principal has the necessary permissions to make Graph calls to its own Azure Active Directory tenant, but service principals do not.  In order to grant a service principal the permissions to perform directory read operations, you can add the service principal to the Azure Active Directory "Directory Readers" built-in role with the following:

$sp = Get-AzureADServicePrincipal -ObjectId <object ID of service principal>

Add-AzureADDirectoryRoleMember -ObjectId (Get-AzureADDirectoryRole | where-object {$_.DisplayName -eq "Directory Readers"}).Objectid -RefObjectId $sp.ObjectId

NOTE: This process can only be performed in PowerShell currently with the Azure AD PowerShell module, you can not add service principals to AAD roles via the portal today.

Reference: https://docs.microsoft.com/en-us/powershell/azure/active-directory/signing-in-service-principal?view=azureadps-2.0#give-the-service-principal-reader-access-to-the-current-tenant-get-azureaddirectoryrole

After granting the service principal the required permissions, you can now run operations such as Set-AzureRmKeyVaultAccessPolicy with service principal credentials.

Thank you Jason!

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

Today's post comes via information from Steve Lewis, a Sr. Azure Cloud Engineer (ACE) in CXP.  He's delivering support to our top Azure customers.

An interesting aspect of being a Support Engineer (or an Azure Cloud Engineer) is that we learn how to read error messages better than anyone.  Customers have grown to ignore error messages as they so often are not useful.  However, this post shows a really good error message from Azure that you should know about.

In case you're not familiar with it, CloudShell is an awesome part of the Azure portal that allows you to execute PowerShell cmdlets (as well as Bash) in the browser.  Pressing on the ">_" button in the top of the screen gets you to the CloudShell environment:

We had a customer that recently opened up a critical situation support case regarding Azure Key Vault and CloudShell. He'd used CloudShell to create the vault and then immediately went to create the secret inside of it, but was denied access to do so.

An interesting note about Key Vault in the Azure Portal:  the creator of a Key Vault is automatically granted an Access Policy.  In fact, you can see it in the blade before you create it:

Even more:  when a Key Vault is created using Powershell on the desktop, an Access Policy is created as well:

 However, we have a different experience when running the same in CloudShell:

You can clearly see that there are no access policies set for this Key Vault upon creation.  Not only that, it tells us that there are no access policies set and how to set them.  That's why when I try to set a secret value inside the key vault, it does not work:



This is one of those cases where if the customer had carefully read the output, they would have understood that they need to set an access policy.  It's been known that there are sometimes really arcane and useless error messages that come from Microsoft resources, but this is not one of those times.  However, before doing that, let's look at the principal we are working under:

It's highly unlikely that you would want to set an access policy for this user, but would rather do it for your own security principal. Let's log in properly:

Now we can can work properly with this.  Let's change to the right subscription, set the access policy (using your account name), and check the vault again:

Now it has an access policy:

And we can finally set the secret:

Again, thanks to Steve Lewis for figuring this out and sharing with me for this blog.

 I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and tweet about it!
@WinDevMatt @AzIdentity

A couple of support cases have asked the question:  how to change the sku of an existing Key Vault?  

To be clear about the skus - there are two pricing tiers available when creating a new Key Vault: standard and premium.  The premium level adds Thales HSM (Hardware Security Modules) to your Key Vault.  See more about that feature here.

The process is simple but not obvious.  Powershell is always our friend when it comes to making changes in Key Vault, as this ability is not available in the portal.

I can see in the KVExplorer that my present SKU is "Standard":

To change that to Premium, I can run this Powershell script:

Login-AzAccount
$vaultResourceId = (Get-AzureRMKeyVault -VaultName "AzIdentity").ResourceId
$vault = Get-AzureRmResource -ResourceId $vaultResourceId -ExpandProperties
$vault.Properties.sku.name = "Premium" # or "Standard"
Set-AzureRMResource -ResourceId $vaultResourceId -Properties $vault.Properties

A refresh of the KVExplorer shows that I now have the Premium level:

It's important to note that this is a safe operation.  If you were to move from Premium to Standard, HSM keys will not longer be available for your Key Vault service, but they are not deleted.  Moving back to Premium allows the keys to be used once again.

While attempting to create (or recreate) a Key Vault, have you ever seen the error message "The name '<key vault name>' is already in use?

It can be frustrating as you may be trying to use a name which you know isn't being used anywhere else, or which you used recently on a key vault, deleted, and now want to recreate. 

This commonly occurs for these reasons:

1. The Key Vault that you just deleted has "Soft Delete" enabled on it. This means that the Key Vault still exists, but is not accessible.  It will remain this way for 90 days, at which point you can recover the key vault name.  This setting is in case the key vault is actually needed again.  You can check for this condition using the Powershell Get-AzureRmKeyVault cmdlet to find soft-deleted vaults:
   

   Get-AzureRMKeyVault -InRemovedState

   After this, you may choose to either revive the key vault using cmdlet "Undo-AzureRmKeyVaultRemoval":
   

   Undo-AzureRmKeyVaultRemoval -VaultName <vaultname> -ResourceGroupName <groupname> -Location <location>

   Or you can "purge" the key vault in order to ensure it's really gone:
   

   Remove-AzureRmKeyVault -VaultName <vaultname> -Location <location> -InRemovedState

   
   
   

2. NOTE:  The Key Vault Product team has changed the validation scheme for domain names.  If your domain name is not available, but you know that it is because you recently deleted it, it may have gotten into the "orphaned DNS" scenario below.  However, the validation has changed to allow the DNS resolution to check if the Key Vault exists.  If it does not exist, the name should become available after 10 minutes.  If it still does not work, then your problem is likely that it's either being used by another Key Vault or it's soft-deleted.

   The Another reason for this condition is an "orphaned" DNS value caused by deleting/recreating the key vault too fast.   You can confirm this problem by trying to access the vault in the browser: https://<vaultname>.vault.azure.net.  You should get an HTTP 403, indicating that the key vault still exists in the DNS, but isn't available in actuality:
   
   
   
   This is a known condition which the Azure Key Vault is working on.  The fix for this right now is to email the product team directly: AzureKeyVault _@_ Microsoft.com, or open a support case and we will help you with this problem.
   
   
   
3. It's possible that the name you've chosen for your Key Vault is already taken.  If the name you've chosen is more like "TestKeyVault" or "AzCloudKv" than "WE-NameOfMyCompany-RG-Eagle-18h6", and you have never had this name on a Key Vault previously, then it's probably used by someone else.  The test above of navigating to the Key Vault to get an HTTP 403 will still hold true, but it will mean you don't have access to that Key Vault rather than it is an orphaned DNS record.
   
   
   

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

Today I helped a customer confused about how to properly download a certificate from Key Vault that contains both the public and private keys.  The current method for downloading a certificate will retrieve only the public key.  Having both parts of the certificate is essential for SSL binding and is necessary for situations such as sending a client certificate for authentication.

As I have shown in previous posts, but perhaps not described fully, is that the necessary method for retrieving the certificate with both keys is to download it as a secret. It does not mean that you re-upload the certificate into the Secrets blade.  It means that you leave the certificate exactly where it is, yet when downloading the certificate as a secret, use the secrets endpoint.  Here's my example:

First, let's establish that I have a certificate, listed as a certificate, in my Key Vault:

The certificate "KVExplorerCom" is an App Service Certificate uploaded to a Key Vault named "kvexplorer".  

Using C# to download the certificate as a certificate reveals the following information:

First, I get a really nice CertificateBundle object displaying the downloaded certificate in a pleasant form.  The values are readily observable.  Another nice and important feature is a byte array called Cer which describes the certificate, and I can use this to convert to an X509 certificate, which I do in the next line of code.  Let's look at that now...

Now that I've created an X509 certificate in memory, examining it seems to go well, until you see that there's no Private Key.  That's the part many applications require for proper SSL binding.  This makes more sense when you realize that the public key for a certificate has a file extension which is ".cer", while the private extension is ".key".  

You can see more about the widely-used certificate file extensions here. https://blogs.msdn.microsoft.com/kaushal/2010/11/04/various-ssltls-certificate-file-typesextensions/

Now let's see what happens when I download the certificate as a secret.

This time I get a SecretBundle object, with the Content-Type set to a certificate, and the Value which is my secret in Base-64 encoding.  Once I Base-64 decode the Value of this secret, I can then convert it to a complete X509 certificate, with public key:

Final note about this downloaded certificate:  if you save the X509 certificate as a PFX, it will not have a password.  This is because of the way that Key Vault stores the certificate internally.  You can see my previous post for instructions on how to download the certificate using Powershell and simultaneously generate a password for it as it's being written to disk.

I am providing feedback to our product team that this is a confusing mechanism in Key Vault and could use improvement.  

Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

Some customers use CLI instead of Powershell as they are working in non-Windows environments.  A support case came in where a customer asked how to download an App Service Certificate from his Key Vault using CLI alone.  The Support Engineer who owned the support case downloaded the certificate using the certificate download CLI command:

# az keyvault certificate download -f certificatename.pfx -n edgeui --vault-name keyvaultname -e DER

What he noticed was that this command only downloads the public part of the certificate.  This is documented:
https://docs.microsoft.com/en-us/cli/azure/keyvault/certificate?view=azure-cli-latest#az-keyvault-certificate-download

Of course, the customer wants to be able to download the entire certificate, both public and private. I've never used CLI previously, but set out to find the right way to download this certificate.  It was far easier than I expected, and it is accomplished by downloading the certificate as a secret, like I described in my previous post.  The exact command is:

# az keyvault secret download --file <path>\<nameofcert>.pfx --name <NameOfSecretOrCertificate> --vault-name <VaultName> 

It is important to note that the downloaded PFX has a blank password.  It will install easily and contain the private key. 

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

A support case that we have seen come through a few times is the question about the BYOK toolset needed to transfer HSM-protected keys into Azure Key Vault. 

The question is whether that toolset is available in 32-bit version, as some customers are using 32-bit workstations to generate the HSM key.  Unfortunately, the product team has confirmed that there is no 32-bit version of this toolset.  It comes in 64-bit mode only.

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

I've seen several support cases recently where someone wants to update a Key Vault via template but not disturb the existing access policies. The problem is that if you deploy a resource of type Microsoft.KeyVault/vaults, that will replace any existing access policy.

This is by-design of ARM Templates.  

In speaking with the product team, I learned of a special child resource for ARM template which allows the addition of new access policies without affecting existing access policies.

When you deploy a resource of type Microsoft.KeyVault/vaults/accessPolicies with the name “add”, it will merge in your changes. This special child resource type was created to allow Managed Service Identity scenarios where you don’t know the identity of a VM until the VM is deployed and you want to give that identity access to the vault during deployment. 

An incremental deployment can be used along with this json to achieve the objective:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vaultName": {
            "type": "string"
        }
    },
    "resources": [
        {
            "type": "Microsoft.KeyVault/vaults/accessPolicies",
            "name": "[concat(parameters('vaultName'), '/add')]",
            "apiVersion": "2016-10-01",
            "properties": {
                "accessPolicies": [
                    {
                        "tenantId": "dfe47ca8-acfc-4539-9519-7d195a9e79e4",
                        "objectId": "5abe9358-10ae-4195-ba23-d34111430329",
                        "permissions": {
                            "keys": ["all"],
                            "secrets": ["all"],
                            "certificates": ["all"],
                            "storage": ["all"]
                        }
                    }
                ]
            }
        }
    ],
    "outputs": {
    }
}

I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity

Several support cases have come in where an Azure customer purchases an App Service Certificate via the portal and then wants to download the certificate from the Key Vault and use it in other systems.  Three things commonly occur in these cases:

1. The customer is unable to find the App Service Certificate in the Key Vault
2. The customer is unable to use a certificate downloaded from the Key Vault because the password does not work
3. The customer needs the certificate in .cer/.key format rather than .pfx

Luckily, I have the solution for all three of these scenarios.

1. You are unable to find the App Service Certificate in the Key Vault
   Many customers who are using Azure App Services do not have much experience with an Azure Key Vault, but most every website has a need for an SSL certificate.  Going through the process of purchasing an Azure App Service Certificate requires you to store the certificate in a Key Vault, which is how most people become first familiar with the Key Vault.
   
   
   
   After purchasing the certificate, customers sometimes want to download it for use elsewhere, so they go into the Key Vault, look under the certificates blade, and find the following (the Key Vault pictured here does in fact hold an App Service Certificate):
   
   
   This is definitely confusing to the person unfamiliar with Key Vault. The reason behind this is simple:  when the programming for App Service Certificates was being done, certificates as objects that could be stored within the Key Vault was not available.  Since then, Certificates are now available, but the process behind the App Service Certificate purchase has not changed yet.  This is coming in the future.
   
   Customers should be able to find their App Service Certificates in the Secrets blade, and I do see the secret of type pkcs12 (certificate) stored there:
   
   
   
2. You are unable to use a certificate downloaded from the Key Vault because the password does not work
   Now that you've found the certificate, you want to download it and use it on another machine.  This should be simply done by using this button:
   
   
   
   It's easy to download this certificate as a .pfx file.  You do so, but when you want to install it, you have no idea what the password is:
   
   
   
   In fact, even if you uploaded the .pfx as a secret yourself, you may find that the password that you do know does not work.  The solution is to download the certificate (as a secret) from the Key Vault using Powershell, then convert that secret into an actual certificate with its own password. 
   
   

   Login-AzureRmAccount
   
   $vaultName  = "<NameOfKeyVault>"
   $keyVaultSecretName = "<NameOfTheSecretWhereCertificateIsStored>"
   
   $secret = Get-AzureKeyVaultSecret -VaultName $VaultName -Name $keyVaultSecretName
   
   $pfxCertObject = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -ArgumentList @([Convert]::FromBase64String($secret.SecretValueText),"",[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)
   
   $pfxPassword = -join ((65..90) + (97..122) + (48..57) | Get-Random -Count 50 | % {[char]$_})
   
   $currentDirectory = (Get-Location -PSProvider FileSystem).ProviderPath
   [Environment]::CurrentDirectory = (Get-Location -PSProvider FileSystem).ProviderPath
   [io.file]::WriteAllBytes(".\KeyVaultCertificate.pfx", $pfxCertObject.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12, $pfxPassword))
   
   Write-Host "Created an App Service Certificate copy at: $currentDirectory\KeyVaultCertificate.pfx"
   Write-Warning "For security reasons, do not store the PFX password. Use it directly from the console as required."
   Write-Host "PFX password: $pfxPassword" 
   

   
   Now you have the certificate and its password for use anywhere you want.
   
   
3. You need the certificate in .cer/.key format rather than .pfx
   There are times when you want the certificate in a .cer and .key format, such as for use on the Apache web server.  This can be accomplished by the use of OpenSSL.  A Windows version of the open-source binaries can be found here: http://slproweb.com/products/Win32OpenSSL.html.
   
   Once installed, it is only necessary to run the following two commands at a command prompt in order to convert your files:
   

   openssl pkcs12 -in KeyVaultCertificate.pfx -nocerts -out KVCert.key
   
   openssl pkcs12 -in KeyVaultCertificate.pfx -clcerts -nokeys -out KVCert.cer

   Note that your .pfx file may have a different name.  You should also plan on having the certificate in the same folder as the openssl command, or use the full path to the certificate.
   
   You will be required to use the password for the .pfx when converting.  Here's a successful run of these commands:
   
   
   Notice that there's no "private key" associated with this certificate - that's because the .cer file does not contain the key - the whole purpose of this exercise.
   
   I hope this helps you out when using the Azure Key Vault!  Please follow us on Twitter and retweet!
   @WinDevMatt @AzIdentity

There are occasions when the Incubation team gets support cases for problems that occur in the Azure portal. It's greatly preferable to document these for public consumption rather than have you open a support case to figure out what to do. This problem sees quite a few support cases:

"The directory currently selected differs from this key vault's directory"

Although it's technically correct, it does not explain what happened and why you're seeing the problem.  It occurs when the internal "Tenant Id" value of the Key Vault does not match the Tenant Id of the subscription that currently owns the Key Vault.  It's probable that one of these things happened:

1. The Key Vault was deployed to a subscription using an ARM Template that contains a different Tenant Id
2. The subscription that owns the Key Vault was moved to a different Tenant (much more likely).

When the subscription is moved to a new tenant, the Key Vault's internal "Tenant Id" is not changed, and that's so the existing access policies (which point to Service Principals in the in old tenant) do not automatically fail.  When the Tenant Id of the Key Vault is changed to the new value, access policies for Service Principals in the old Tenant Id will no longer work.

It is possible to update the Tenant Id and get rid of this message by following these instructions:
https://docs.microsoft.com/en-us/azure/key-vault/key-vault-subscription-move-fix

Update on 8/15/2019
The Key Vault product team has made a change to the DNS validation process.  If a DNS name is in an orphaned state, on the first attempt at claiming it, it will still fail.  However, after 20 minutes, a second attempt will allow you to reuse the name if it's in the same region as the original vault.

Please follow us on Twitter and retweet!
@WinDevMatt @AzIdentity