More than a few support cases are created when Key Vault users wisely decide to enable the Firewall settings on their vault. Then the problem begins: Azure App Service websites are no longer able to access the Key Vault, and end up creating an error message like the following:
Operation returned an invalid status code 'Forbidden'
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: Microsoft.Azure.KeyVault.Models.KeyVaultErrorException: Operation returned an invalid status code 'Forbidden'
Forbidden is means that the HTTP response code is 403. In the past, 403 has always meant:
- The identity was confirmed
- The resource is correct
- Either the lack of an access policy OR an access policy itself prevents the resource from being obtained by the confirmed identity
However, since the Azure Key Vault Firewall And Virtual Networks feature was released, it can also mean that the client is not allowed by virtue of the calling location. Here's a look at that part of Key Vault:

If the Key Vault Firewall/VNet is activated, there are exactly three ways to get into the Key Vault (given that an access policy is also in place):
- Be on the same Virtual Network as the Key Vault
- Be on the Firewall IP address whitelist
- Be a "Trusted Microsoft Service"
I think that the first two are self-explanatory. However, #3 is causing a headache for many customers.
This is the explicit list of "Trusted Microsoft Services" (as of the time of this authoring):
Trusted services
Here's a list of trusted services that are allowed to access a key vault if the Allow trusted services option is enabled.
You will look at the list and find "Azure App Services". However, it's the next column which is just as important: the only scenario where an App Service is trusted is for deployment of App Service Certificates.
That's all. That's it.
To show this, I enabled the firewall with the trusted Microsoft Services, as well as Audit logging in my Azure Key Vault and attempted to access it from my Azure App Service. I get this message in my website:

Looking in my audit logs, I find the following:
"time": "2019-01-03T19:14:18.2665709Z",
"category": "AuditEvent",
"operationName": "SecretGet",
"resultType": "Success",
"resultDescription": "Client address (00.00.00.00) is not authorized and caller is not a trusted service",
"correlationId": "c8e4f5ef-6def-411d-9a24-ab43f00d0566",
"callerIpAddress": "00.00.00.00",
As soon as I added the IP address of my web site...

to the Firewall whitelist...

the site worked as expected.

This is by design as an App Service is not a "Trusted Microsoft Service".
I hope this relieves your frustration with the setting.