Feel like sharing?

It’s been forever since I have posted a developer blog, to the point that I almost considered this site dead. However, once in a while you hit a problem that you spend just enough time on you feel compelled to help your community.

I am an absolute fanboy of everything VoloSoft does, their frameworks for building multi-tenant SaaS applications are hands down the most comprehensive tools available. The latest generation framework, abp.io is the killer app for any .NET developer and I highly recommend it to anyone starting a new project.

That said, because of the framework’s size, sometimes certain aspects of it can be hard to wire-up. For years I have found most of my problem-solving with ABP to be dealing with scaffolding and deployment issues, not developer challenges. The last few days have been no Exception();

We are currently starting the deployment process for a small abp.io app and, as usual, were planning on using Azure to host the service. As this was my first time deploying abp.io to Azure, I was expecting a few

headbangers ball problems deploying abp to azure

sessions to get things configured properly.

However, what I wasn’t expecting was a big chunk of missing code from abp.io that prevents applications from running at all. Admittedly, this may be because I have become so spoiled with the framework traditionally wrapping and hiding these things. But after struggling with this for a few days, I think abp.io should be a bit more out-of-the-box compatible with Azure.

The Problem: Deploy to Azure

If you get to the point where you have successfully set up and deployed your native abp.io application, you will inevitably run into what I consider the most dreaded type of Azure error screen. When you try accessing your app from your browser, you will almost certainly see this HTTP Error 500.30 ASP.NET Core app failed to start screen:

The problem I always encounter with this error message is hunting down more information. Because I don’t see this error often, I typically forget where to go for more context. I spend too much time looking through output screens and logs, forgetting that this screen is a result of start-up logic failing. In which case, the best (and perhaps only) place to trace down what’s happening is the AppService’s event log. To get to your EventLog, you should select Diagnose Problems, then Application Event Logs

If you dig into the Event Log, you will eventually find this error:

Application '/LM/W3SVC/1877369640/ROOT' with physical root 'C:\home\site\wwwroot\' has exited from Program.Main with exit code = '1'. First 30KB characters of captured stdout and stderr logs:
[13:33:27 INF] Starting web host.
[13:34:10 FTL] Host terminated unexpectedly!
Volo.Abp.AbpInitializationException: An error occurred during ConfigureServicesAsync phase of the module Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule, Volo.Abp.OpenIddict.AspNetCore, Version=6.0.2.0, Culture=neutral, PublicKeyToken=null. See the inner exception for details.
 ---> Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: The specified network password :[PASSWORD] not correct.
   at Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(ReadOnlySpan`1 rawData, SafePasswordHandle password, PfxCertStoreFlags pfxCertStoreFlags)
   at Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(ReadOnlySpan`1 rawData, String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at Internal.Cryptography.Pal.CertificatePal.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password)
   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
   at MyApp.Web.MyAppWebModule.GetEncryptionCertificate(IWebHostEnvironment hostingEnv, IConfiguration configuration) in C:\src\abpio\src\MyApp.Web\MyAppWebModule.cs:line 170
   at MyApp.Web.MyAppWebModule.<>c__DisplayClass0_0.<PreConfigureServices>b__1(OpenIddictServerBuilder builder) in C:\src\MyApp-4\aspnet-core\src\MyApp.Web\MyAppWebModule.cs:line 96
   at Volo.Abp.Options.PreConfigureActionList`1.Configure(TOptions options)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionPreConfigureExtensions.ExecutePreConfiguredActions[TOptions](IServiceCollection services, TOptions options)
   at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.<>c__DisplayClass1_0.<AddOpenIddictServer>b__0(OpenIddictServerBuilder builder)
   at Microsoft.Extensions.DependencyInjection.OpenIddictServerExtensions.AddServer(OpenIddictBuilder builder, Action`1 configuration)
   at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.AddOpenIddictServer(IServiceCollection services)
   at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.ConfigureServices(ServiceConfigurationContext context)
   at Volo.Abp.Modularity.AbpModule.ConfigureServicesAsync(ServiceConfigurationContext context)
   at Volo.Abp.AbpApplicationBase.ConfigureServicesAsync()
   --- End of inner exception stack trace ---
   at Volo.Abp.AbpApplicationBase.ConfigureServicesAsync()
   at Volo.Abp.AbpApplicationFactory.CreateAsync[TStartupModule](IServiceCollection services, Action`1 optionsAction)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionApplicationExtensions.AddApplicationAsync[TStartupModule](IServiceCollection services, Action`1 optionsAction)
   at Microsoft.Extensions.DependencyInjection.WebApplicationBuilderExtensions.AddApplicationAsync[TStartupModule](WebApplicationBuilder builder, Action`1 optionsAction)
   at MyApp.Web.Program.Main(String[] args) in C:\src\MyApp-4\aspnet-core\src\MyApp.Web\Program.cs:line 36

Process Id: 4688.
File Version: 16.0.22296.11. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: 617d594f2bf75a8904d3d0e7d68a0bacf8e6763a

This error is a result of a set of missing certificates that are required by the OpenIddict libraries used by abp.io. While the issue is mentioned in the abp.io documentation (click here) it’s not necessarily where I was looking when trying to debug a deployment problem. Additionally, I found the documentation on how to resolve this a bit scattered and incomplete. After finally pulling it all together, I wanted to document a more comprehensive step-by-step solution for the rest of the abp.io community.

The Solution

To solve this problem, we need to generate and give our application access to a pair of certificates that are suitable for a production environment. For our solution, we are going to simply check for the existence of two .pfx files in the application’s host directory. If the files don’t exist, or they are over 180 days old, we will (re-)create them.

Step 1: Load your abp.io solution and navigate to the MyApp.Web project.

Step 2: Load your MyWebWebModule.cs (located in the project root).

Step 3: Add the following methods to your code:

private X509Certificate2 GetSigningCertificate(IWebHostEnvironment hostingEnv,
                            IConfiguration configuration)
{
    var fileName = $"cert-signing.pfx";
    var passPhrase = configuration["MyAppCertificate:X590:PassPhrase"]; 
    var file = Path.Combine(hostingEnv.ContentRootPath, fileName);        
    if (File.Exists(file))
    {
        var created = File.GetCreationTime(file);
        var days = (DateTime.Now - created).TotalDays;
        if (days > 180)          
            File.Delete(file);
        else
            return new X509Certificate2(file, passPhrase,
                         X509KeyStorageFlags.MachineKeySet);
    }

    // file doesn't exist or was deleted because it expired
    using var algorithm = RSA.Create(keySizeInBits: 2048);
    var subject = new X500DistinguishedName("CN=Fabrikam Signing Certificate");
    var request = new CertificateRequest(subject, algorithm, 
                        HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
    request.CertificateExtensions.Add(new X509KeyUsageExtension(
                        X509KeyUsageFlags.DigitalSignature, critical: true));
    var certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow, 
                        DateTimeOffset.UtcNow.AddYears(2));
    File.WriteAllBytes(file, certificate.Export(X509ContentType.Pfx, string.Empty));
    return new X509Certificate2(file, passPhrase, 
                        X509KeyStorageFlags.MachineKeySet);
}

private X509Certificate2 GetEncryptionCertificate(IWebHostEnvironment hostingEnv,
                             IConfiguration configuration)
{
    var fileName = $"cert-encryption.pfx";
    var passPhrase = configuration["MyAppCertificate:X590:PassPhrase"]; 
    var file = Path.Combine(hostingEnv.ContentRootPath, fileName);
    if (File.Exists(file))
    {
        var created = File.GetCreationTime(file);
        var days = (DateTime.Now - created).TotalDays;
        if (days > 180)
            File.Delete(file);
        else
            return new X509Certificate2(file, passPhrase, 
                            X509KeyStorageFlags.MachineKeySet);
    }

    // file doesn't exist or was deleted because it expired
    using var algorithm = RSA.Create(keySizeInBits: 2048);
    var subject = new X500DistinguishedName("CN=Fabrikam Encryption Certificate");
    var request = new CertificateRequest(subject, algorithm, 
                        HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
    request.CertificateExtensions.Add(new X509KeyUsageExtension(
                        X509KeyUsageFlags.KeyEncipherment, critical: true));
    var certificate = request.CreateSelfSigned(DateTimeOffset.UtcNow,
                        DateTimeOffset.UtcNow.AddYears(2));
    File.WriteAllBytes(file, certificate.Export(X509ContentType.Pfx, string.Empty));
    return new X509Certificate2(file, passPhrase, X509KeyStorageFlags.MachineKeySet);
}

Step 4: From within that same file, modify the PreConfigureServices method by ensuring the two methods above are called when your application is not running in a production environment:

 if (!hostingEnvironment.IsDevelopment())
        {
            PreConfigure<AbpOpenIddictAspNetCoreOptions>(options =>
            {
                options.AddDevelopmentEncryptionAndSigningCertificate = false;
            });

            PreConfigure<OpenIddictServerBuilder>(builder =>
            {
                // In production, it is recommended to use two RSA certificates, 
                // one for encryption, one for signing.
                builder.AddEncryptionCertificate(
                        GetEncryptionCertificate(hostingEnvironment, context.Services.GetConfiguration()));
                builder.AddSigningCertificate(
                        GetSigningCertificate(hostingEnvironment, context.Services.GetConfiguration()));
            });
        }

Step 5: Add a custom passphrase to your appsettings.json or Azure configuration:

{
  
  "MyAppCertificate": {
    "X590": "[custom string]"
  }
}

Developer Notes

The above code comes mostly from existing documentation within the abp.io community sites, some issues reported on their GitHub issues page, and the OpenIddict documentation. I found that there were still a few issues with the code samples, primarily with regard to where the .pfx files are stored and how they are loaded.

  • A number of suggestions were made to store these in the Azure Key Vault, or other locations. However, I prefer my applications to be a bit more self-contained, so I am creating and using local files.
  • Without the additional parameter to the X509Certificate2 method (X509KeyStorageFlags.MachineKeySet) I was still getting start-up errors in production. With a little help from StackOverFlow, I found that method still looks in the private key store, even when a specific file path is provided.

Addendum

It turns out the latest release, 6.0.2 of abp.io also has an incompatibility with ApplicationInsights. Luckily, someone solved it a while ago Issue running Abp application on an Azure App Service #3030 | Support Center | ABP Commercial

If you turn on ApplicationInsights, you cannot enable “Show local variables for your application when an exception is thrown.” within the Snapshot Debugger settings. If that is enabled, you will receive a 502 response when accessing pages such as /account/login

</end> Done!

That’s all you need to do. Now when your app is deployed to an Azure AppService, the above code will create the needed certificates, and load them on demand.

Please follow me on Twitter @jasenf as I hopelessly try to grow my followers

Feel like sharing?

Last modified: December 27, 2022

Author