Unfortunately the Windows crypto API, and consequently the .NET Core crypto API, don't provide very helpful error messages.
From the stack trace, I see that the SAML assertion is being decrypted. This is done using the private key associated with the X.509 certificate.
The most likely causes are either the private key is missing or the calling application doesn't have read access to the private key.
Make sure that the permissions are set on the associated private key container.
The following link refers to the private key container and setting permissions on it.https://www.componentspace.com/Forums/29/Troubleshooting-Loading-X.509-Certificates
Is the X509Certificate2 constructor code you supplied how you're loading the certificate?
If the certificates are in a database, are you copying them to the file system?