Chef: Unable to upload cookbooks after SSL certificate change

We recently started using Chef server and about half way through our POC, we decided that things were going so good that we should give it a trusted SSL certificate from our internal CA.  We were also accessing the server via a CNAME alias, so the new cert was created using this name instead of the hostname of the server itself.  We did this and things went great at first it seemed.  However, the next time I tried to upload cookbooks via knife, I was greeted with a 500 Internal Server Error.  I did some digging into the Chef server log files and found the following…

==> /var/log/opscode/opscode-erchef/crash.log <==
 2016-01-06 13:11:57 =ERROR REPORT====
 SSL: certify: ssl_handshake.erl:438:Fatal error: certificate unknown
 2016-01-06 13:11:57 =ERROR REPORT====
 Checking presence of file (checksum: <<"34660b767dc2726427140f73ceb8f8c8">>) for org <<"f1cccf515db16b3743623de1620ed231">> from bucket "bookshelf" (key: "organization-f1cccf515db16b3743623de1620ed231/checksum-34660b767dc2726427140f73ceb8f8c8") raised exception error:{aws_error,{socket_error,{conn_failed,{error,{tls_alert,"certificate unknown"}}}}}

…And when running chef-server-ctl test, the following errors were reported:

Failures:

  1) Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: make_cookbook_artifact_with_recipes(cookbook_name, cookbook_identifier, [recipe_spec])
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:222:in `make_cookbook_artifact_with_recipes'
     # ./spec/api/cookbook_artifacts/delete_spec.rb:90:in `block (5 levels) in <top (required)>'

  2) Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/cookbooks/delete_spec.rb:91:in `block (5 levels) in <top (required)>'

  3) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  4) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  5) Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK
     Failure/Error: response.should look_like expected_response
       Response should have HTTP status code 200 ('OK'), but it was actually 500 ('Internal Server Error')
         Reponse Body: {"error":["internal service error"]}
     # ./spec/api/sandboxes/complete_endpoint_spec.rb:239:in `block (4 levels) in <top (required)>'

Finished in 1 minute 25.7 seconds
148 examples, 5 failures, 2 pending

Failed examples:

rspec ./spec/api/cookbook_artifacts/delete_spec.rb:95 # Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/cookbooks/delete_spec.rb:94 # Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/sandboxes/complete_endpoint_spec.rb:234 # Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK

After researching for several hours and running across multiple threads with no real solution, I found that the original self-signed certificate that was generated by the Chef server was signed using the sha256RSA signature algorithm while our new certificate was signed using RSASSA-PSS which as it turns out is not supported by Erlang and therefore not supported for use with Chef (which uses Erlang).

cert

This should be a common problem among Chef admins who use internal Microsoft Certificate Authorities as RSASSA-PSS is now the default signing algorithm as of Windows 2008 R2.  The solution to our issue was to generate a new certificate using the sha256RSA signature algorithm from our internal CA.  In order to do that, we had to do the following…

  1. Logon to the CA and change HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\CertSvc\Configuration\IssuingCA\CSP\AlternateSignatureAlgorithm from a 1 to a 0.
  2. Net stop certsvc
  3. Net start certsvc
  4. Re-issue the certificate
  5. Change the regkey back and restart certsvc again if you so desire.
  6. Install the new certificate onto the Chef server.

After doing this, knife can once again upload cookbooks and chef-server-ctl test returns no errors.

 

Shrinking XenServer VDI’s

This one is more of a note to myself…

1. With a third party utility shrink the windows partiton to the desired size.
2. Shut down the virtual machine
3. Next step is the use of the "lvreduce" command to get back the extra space. Make sure you leave a little room for overhead just in case. I usually go 1GB bigger than the partition size then resize again to fill the extra gig:

· in the shell of the host console type "lvm"

· then "lvdisplay" to display all the informations of your logical volumes. Find your logical volume, select and copy the LV Name associated

· then "lvreduce <LVName> –size 16G " (replace LV Name by the one you copied, just right click and paste. replace 16G with desired size)

4. Power on your VM and make sure windows starts correctly, verify disk partitions and disk sizes in disk management.
5. Run the command “xe-toolstack-restart”. I had to wait a few hours for the space to get freed up after running this command. If you reboot the host server it will be immediate and negates the need to run this command.

Public Folder Recovery when Deleted Item Retention does not work in Outlook

I ran into a problem today where Outlook gave me an error when attempting to recovery a public folder from deleted item retention that was deleted by a user.  The error was…

Outlook was unable to recover some or all of the items in this folder. Make sure you have the required permissions to recover items in this folder and try again. If the problem persists contact your administrator.

It turns out that you can get around this error by recovering the folder via webmail.  Just use the following URL to show items in deleted item retention and restore from there…

http://<servername>/public/<path to parent folder>/?cmd=showdeleted&btnClose=1

MSI Manager: Reinstall Applications Assigned by Group Policy

Software installation via Group Policy is a great feature that can save any administrator HOURS of time over installing apps one by one on all machines within the network. But what happens when those applications go bad? Microsoft has not provided a way to force re-installation of GPO-Managed software on a SINGLE machine, opting instead to only give you the option to redeploy the application on ALL machines. On top of this, if you remove the application from Add/Remove Programs the application does NOT get reinstalled! 

Enter MSI Manager… Continue reading

How To: Delete files that are in Use

Ever needed to delete a file but couldn’t do so even after a fresh boot?  Hate having to go into Safe Mode to delete pesky files?  Can’t delete the damn thing from Safe Mode anyway?  You need the PendingFileRenameOperations registry value.

This is great for deleting spyware and viruses from your system if you know the name of the offending EXE.

HERE is a good summary of the PendingFileRenameOperations registry value,

and HERE is a freeware utility that adds an option to delete files on the next reboot to the right-click context menu.

Now go delete that bastard!

Standardized Outlook Signatures…For FREE

Well, apparently 10 bucks a user isn’t cheap enough for some people, so I had to figure out a cheaper (read: FREE) way to push out standard signatures to our users. It doesn’t make sense to create signatures for each employee by hand, and I can’t really trust users to follow directions in an e-mail telling them what signature they have to use, so what we have here is a completely automated way to push out a standardized company signature to your Outlook 2003 users leveraging the information you have in Active Directory.

Continue reading