Azure KeyVault won't allow you to download private key of certificate. It forces you to write a program, so I'll give you one.

I collected everything from stackoverflow.

Tools

You need python, and you have to run pip install azure-keyvault-secrets azure-identity. You also need openssl and azure-cli.

Authenticate

Set environment variables for auth in python, like this

export AZURE_CLIENT_ID=xxx
export AZURE_CLIENT_SECRET=xxx
export AZURE_TENANT_ID=xxx

And authenticate azure-cli like this

az login --service-principal --username "$AZURE_CLIENT_ID" --tenant "$AZURE_TENANT_ID" --password "$AZURE_CLIENT_SECRET"

Do it

Assuming you're downloading MyCertName.pfx from MyKeyVault, run this python script

from azure.identity import DefaultAzureCredential                                                              
from azure.keyvault.secrets import SecretClient                                                                
credential = DefaultAzureCredential()                                                                          
secret_client = SecretClient(vault_url="https://MyKeyVault.vault.azure.net/", credential=credential)

from cryptography.hazmat.primitives.serialization import pkcs12
import base64
certificate_secret = secret_client.get_secret(name="MyCertName")
cert_bytes = base64.b64decode(certificate_secret.value)
private_key, public_certificate, additional_certificates = pkcs12.load_key_and_certificates(data=cert_bytes,password=None)

################################

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.serialization import load_pem_private_key

def save_key(pk, filename):
    pem = pk.private_bytes(
        encoding=serialization.Encoding.PEM, 
        format=serialization.PrivateFormat.TraditionalOpenSSL,
        encryption_algorithm=serialization.NoEncryption()
    )
    with open(filename, 'wb') as pem_out:
        pem_out.write(pem)

save_key(private_key, "/tmp/shitpriv.pem")

Then run

az keyvault certificate download --vault-name MyKeyVault -n MyCertName --file /tmp/shitpub.pem --encoding PEM
openssl pkcs12 -inkey /tmp/shitpriv.pem -in /tmp/shitpub.pem  -export -out fuckyou.pfx

You're all set. It sucks, fuck you.