Chef: Unable to upload cookbooks after SSL certificate change

We recently started using Chef server and about half way through our POC, we decided that things were going so good that we should give it a trusted SSL certificate from our internal CA.  We were also accessing the server via a CNAME alias, so the new cert was created using this name instead of the hostname of the server itself.  We did this and things went great at first it seemed.  However, the next time I tried to upload cookbooks via knife, I was greeted with a 500 Internal Server Error.  I did some digging into the Chef server log files and found the following…

==> /var/log/opscode/opscode-erchef/crash.log <==
 2016-01-06 13:11:57 =ERROR REPORT====
 SSL: certify: ssl_handshake.erl:438:Fatal error: certificate unknown
 2016-01-06 13:11:57 =ERROR REPORT====
 Checking presence of file (checksum: <<"34660b767dc2726427140f73ceb8f8c8">>) for org <<"f1cccf515db16b3743623de1620ed231">> from bucket "bookshelf" (key: "organization-f1cccf515db16b3743623de1620ed231/checksum-34660b767dc2726427140f73ceb8f8c8") raised exception error:{aws_error,{socket_error,{conn_failed,{error,{tls_alert,"certificate unknown"}}}}}

…And when running chef-server-ctl test, the following errors were reported:

Failures:

  1) Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: make_cookbook_artifact_with_recipes(cookbook_name, cookbook_identifier, [recipe_spec])
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:222:in `make_cookbook_artifact_with_recipes'
     # ./spec/api/cookbook_artifacts/delete_spec.rb:90:in `block (5 levels) in <top (required)>'

  2) Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/cookbooks/delete_spec.rb:91:in `block (5 levels) in <top (required)>'

  3) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  4) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  5) Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK
     Failure/Error: response.should look_like expected_response
       Response should have HTTP status code 200 ('OK'), but it was actually 500 ('Internal Server Error')
         Reponse Body: {"error":["internal service error"]}
     # ./spec/api/sandboxes/complete_endpoint_spec.rb:239:in `block (4 levels) in <top (required)>'

Finished in 1 minute 25.7 seconds
148 examples, 5 failures, 2 pending

Failed examples:

rspec ./spec/api/cookbook_artifacts/delete_spec.rb:95 # Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/cookbooks/delete_spec.rb:94 # Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/sandboxes/complete_endpoint_spec.rb:234 # Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK

After researching for several hours and running across multiple threads with no real solution, I found that the original self-signed certificate that was generated by the Chef server was signed using the sha256RSA signature algorithm while our new certificate was signed using RSASSA-PSS which as it turns out is not supported by Erlang and therefore not supported for use with Chef (which uses Erlang).

cert

This should be a common problem among Chef admins who use internal Microsoft Certificate Authorities as RSASSA-PSS is now the default signing algorithm as of Windows 2008 R2.  The solution to our issue was to generate a new certificate using the sha256RSA signature algorithm from our internal CA.  In order to do that, we had to do the following…

  1. Logon to the CA and change HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\CertSvc\Configuration\IssuingCA\CSP\AlternateSignatureAlgorithm from a 1 to a 0.
  2. Net stop certsvc
  3. Net start certsvc
  4. Re-issue the certificate
  5. Change the regkey back and restart certsvc again if you so desire.
  6. Install the new certificate onto the Chef server.

After doing this, knife can once again upload cookbooks and chef-server-ctl test returns no errors.