Chef: Unable to upload cookbooks after SSL certificate change

We recently started using Chef server and about half way through our POC, we decided that things were going so good that we should give it a trusted SSL certificate from our internal CA.  We were also accessing the server via a CNAME alias, so the new cert was created using this name instead of the hostname of the server itself.  We did this and things went great at first it seemed.  However, the next time I tried to upload cookbooks via knife, I was greeted with a 500 Internal Server Error.  I did some digging into the Chef server log files and found the following…

==> /var/log/opscode/opscode-erchef/crash.log <==
 2016-01-06 13:11:57 =ERROR REPORT====
 SSL: certify: ssl_handshake.erl:438:Fatal error: certificate unknown
 2016-01-06 13:11:57 =ERROR REPORT====
 Checking presence of file (checksum: <<"34660b767dc2726427140f73ceb8f8c8">>) for org <<"f1cccf515db16b3743623de1620ed231">> from bucket "bookshelf" (key: "organization-f1cccf515db16b3743623de1620ed231/checksum-34660b767dc2726427140f73ceb8f8c8") raised exception error:{aws_error,{socket_error,{conn_failed,{error,{tls_alert,"certificate unknown"}}}}}

…And when running chef-server-ctl test, the following errors were reported:

Failures:

  1) Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: make_cookbook_artifact_with_recipes(cookbook_name, cookbook_identifier, [recipe_spec])
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:222:in `make_cookbook_artifact_with_recipes'
     # ./spec/api/cookbook_artifacts/delete_spec.rb:90:in `block (5 levels) in <top (required)>'

  2) Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/cookbooks/delete_spec.rb:91:in `block (5 levels) in <top (required)>'

  3) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  4) /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
     Failure/Error: before(:each) { setup_cookbooks(cookbooks) }
     RuntimeError:
       bad response code 500 in response: {"error":["internal service error"]}
     # ./lib/pedant/rspec/common.rb:419:in `ensure_2xx'
     # ./lib/pedant/rspec/cookbook_util.rb:65:in `commit_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:74:in `upload_files_to_sandbox'
     # ./lib/pedant/rspec/cookbook_util.rb:429:in `save_dummy_cookbook_with_recipes'
     # ./lib/pedant/rspec/cookbook_util.rb:448:in `block (2 levels) in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:447:in `block in setup_cookbooks'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `each'
     # ./lib/pedant/rspec/cookbook_util.rb:446:in `setup_cookbooks'
     # ./spec/api/environments/recipes_spec.rb:98:in `block (3 levels) in <top (required)>'

  5) Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK
     Failure/Error: response.should look_like expected_response
       Response should have HTTP status code 200 ('OK'), but it was actually 500 ('Internal Server Error')
         Reponse Body: {"error":["internal service error"]}
     # ./spec/api/sandboxes/complete_endpoint_spec.rb:239:in `block (4 levels) in <top (required)>'

Finished in 1 minute 25.7 seconds
148 examples, 5 failures, 2 pending

Failed examples:

rspec ./spec/api/cookbook_artifacts/delete_spec.rb:95 # Cookbook Artifacts API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/cookbooks/delete_spec.rb:94 # Cookbooks API endpoint DELETE /cookbooks/<name>/<version> for existing cookbooks when deleting existent version of an existing cookbook should cleanup unused checksum data in s3/bookshelf
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from a non-default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/environments/recipes_spec.rb:46 # /environments/ENVIRONMENT/recipes API endpoint with multiple versions of multiple cookbooks with no environment constraints when fetching recipes from _default environment should respond with 200 OK and recipes from the latest version of all cookbooks within the environment
rspec ./spec/api/sandboxes/complete_endpoint_spec.rb:234 # Sandboxes API Endpoint Sandboxes Endpoint, PUT when committing a sandbox after uploading files should respond with 200 OK

After researching for several hours and running across multiple threads with no real solution, I found that the original self-signed certificate that was generated by the Chef server was signed using the sha256RSA signature algorithm while our new certificate was signed using RSASSA-PSS which as it turns out is not supported by Erlang and therefore not supported for use with Chef (which uses Erlang).

cert

This should be a common problem among Chef admins who use internal Microsoft Certificate Authorities as RSASSA-PSS is now the default signing algorithm as of Windows 2008 R2.  The solution to our issue was to generate a new certificate using the sha256RSA signature algorithm from our internal CA.  In order to do that, we had to do the following…

  1. Logon to the CA and change HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\CertSvc\Configuration\IssuingCA\CSP\AlternateSignatureAlgorithm from a 1 to a 0.
  2. Net stop certsvc
  3. Net start certsvc
  4. Re-issue the certificate
  5. Change the regkey back and restart certsvc again if you so desire.
  6. Install the new certificate onto the Chef server.

After doing this, knife can once again upload cookbooks and chef-server-ctl test returns no errors.

 

Shrinking XenServer VDI’s

This one is more of a note to myself…

1. With a third party utility shrink the windows partiton to the desired size.
2. Shut down the virtual machine
3. Next step is the use of the "lvreduce" command to get back the extra space. Make sure you leave a little room for overhead just in case. I usually go 1GB bigger than the partition size then resize again to fill the extra gig:

· in the shell of the host console type "lvm"

· then "lvdisplay" to display all the informations of your logical volumes. Find your logical volume, select and copy the LV Name associated

· then "lvreduce <LVName> –size 16G " (replace LV Name by the one you copied, just right click and paste. replace 16G with desired size)

4. Power on your VM and make sure windows starts correctly, verify disk partitions and disk sizes in disk management.
5. Run the command “xe-toolstack-restart”. I had to wait a few hours for the space to get freed up after running this command. If you reboot the host server it will be immediate and negates the need to run this command.

Public Folder Recovery when Deleted Item Retention does not work in Outlook

I ran into a problem today where Outlook gave me an error when attempting to recovery a public folder from deleted item retention that was deleted by a user.  The error was…

Outlook was unable to recover some or all of the items in this folder. Make sure you have the required permissions to recover items in this folder and try again. If the problem persists contact your administrator.

It turns out that you can get around this error by recovering the folder via webmail.  Just use the following URL to show items in deleted item retention and restore from there…

http://<servername>/public/<path to parent folder>/?cmd=showdeleted&btnClose=1

MSI Manager: Reinstall Applications Assigned by Group Policy

Software installation via Group Policy is a great feature that can save any administrator HOURS of time over installing apps one by one on all machines within the network. But what happens when those applications go bad? Microsoft has not provided a way to force re-installation of GPO-Managed software on a SINGLE machine, opting instead to only give you the option to redeploy the application on ALL machines. On top of this, if you remove the application from Add/Remove Programs the application does NOT get reinstalled! 

Enter MSI Manager… Continue reading

Entering Expert Mode automatically on log in

This post is for dealing with Check Point SPLAT firewalls and other SPLAT based appliances. 

Why would I need this?  Well, for one, if you plan to use SCP to access your SPLAT box you must have this enabled.  Other than that, typically you only enter the SPLAT box command line interface if you need to do something, not just for fun.  Any changes or commands you need to run outside of the ‘sysconfig’ or ‘cpconfig’ menu systems, requires you to be in Expert Mode.  You used to have to edit a file using ‘vi’ which isn’t the most user friendly editing program, but there is now a new method.  I recently found this and wanted to share it since it SO much easier than before.  Enjoy!

 To enter Expert mode automatically on each login, perform the following steps:

  1. Enter Expert mode.
  2. Run the ‘chsh -s /bin/tcsh admin’ command (to work in tsch).
    Run the ‘chsh -s /bin/bash admin’ command (to work in bash).

To revert back to the default login shell:

    Run the ‘chsh -s /bin/cpshell admin’ command
    Note: ‘admin’ is the user name in the above commands and can be substituted for any user name you have on the box.

Cannot copy file_name: The path is too deep

Ever get this error “Cannot copy file_name: The path is too deep” when trying to drag-and-drop a file?  Were you going through a firewall?

I recently came across this error and it had us stumped for quite a while.  I found several articles online that didn’t quite identify it correctly or didn’t apply to my situation.  After digging around on the knowledge base of the firewall manufacture, Check Point in this case, I came across a real solution that worked.  It wasn’t easy to find, even on their site because the situation wasn’t the same, but I figured “what the heck, I’ll try it”, and it worked. 

The problem had to do with the default windowing size allowed through the firewall.  If you aren’t familiar, “windowing” is how TCP negotiates the transfer of data.  It is variable and starts out slow until it can negotiate an acceptable packet-to-acknowledgement rate for both parties.  For example, first we exchange packets by me giving  you one packet and you responding (acknowledging) that you received it.  Then we try say 10 packets to one, if that worked without corruption, we increase it.  So on and so forth until we get to a maximum agreeable rate that both of us are comfortable with and we get data transfered at a much higher speed.  All that to say this…

Check Point firewalls have a max windowing size of 10K by default.  This sometimes gives you the “Path is too deep” error, espescially when on a LAN going to a DMZ or some other interface on the firewall.  To fix it you will want to do the following:

To increase the window size, run the fw ctl set int fwtcpstr_max_window 65536 command.

Note: This command does not survive a reboot.

To make the command survive a reboot:

  1. On Linux or SecurePlatform, edit the $FWDIR/boot/modules/fwkern.conf file using vi.
  2. Set a parameter name to a value, e.g.,
    fwtcpstr_max_window=65536
  3. Run the fw ctl get int fwtcpstr_max_window command to verify whether the new value is applied on the OS properly.

After the procedure completes, users should be able to successfully copy the files.