[chef] Unable to use SSL cert from in-house Chef Server w/ knife

[Ivan Suftin]

Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

First, I run the check:
$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org:443
ERROR: The SSL certificate of chef.owicloud.org could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=***@usgs.gov

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1k 8 Jan 2015
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"

TO FIX THIS ERROR:

[ 
 the usual text we know and love 
]

Ok, so obviously that doesn’t work. I then try to run a knife ssl fetch:
$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `[]' for nil:NilClass

I’ve always had this issue so I’ve never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

Server:
$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°╯ ┻━┻
Ivan Suftin - Applications Developer - ***@usgs.gov
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

[Daniel DeLeo]

Post by Ivan Suftin
Ohai Chefs!
We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

Post by Ivan Suftin
$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org (http://chef.owicloud.org):443
ERROR: The SSL certificate of chef.owicloud.org (http://chef.owicloud.org) could not be verified
* Version: OpenSSL 1.0.1k 8 Jan 2015
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"
[ … the usual text we know and love …]

Did you get a message about "There are invalid certificates in your trusted_certs_dir.” from `knife ssl check`? (Your certs would have to exist in your trusted_certs_dir before you run that command for `knife ssl check` to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we’ve seen this when users follow an outdated blog post for setting "Subject Alternative Names,” but you could have stumbled upon this issue as well.

Post by Ivan Suftin

Post by Ivan Suftin
$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org (http://chef.owicloud.org) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `[]' for nil:NilClass

This looks like an actual bug with `knife ssl fetch`. If you run `knife ssl fetch URL -VV`, it will show the backtrace. You should then copy all of that into a bug report at https://github.com/chef/chef/issues so we can fix the command.

Post by Ivan Suftin

Post by Ivan Suftin
$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?
__________________________ (╯°□°）╯︵ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo

[Ivan Suftin]

Hi Daniel,

I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir”. Do you have more information regarding the "Subject Alternative Names” issue? I could get more information for you if needed in order to better analyze the issue.

In regards to the knife issue, I’ve opened this ticket: https://github.com/chef/chef/issues/2919 <https://github.com/chef/chef/issues/2919>

__________________________ (╯°□°╯ ┻━┻
Ivan Suftin - Applications Developer - ***@usgs.gov
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

Post by Ivan Suftin
Ohai Chefs!
We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

Post by Ivan Suftin
$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org (http://chef.owicloud.org):443
ERROR: The SSL certificate of chef.owicloud.org (http://chef.owicloud.org) could not be verified
* Version: OpenSSL 1.0.1k 8 Jan 2015
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"
[ 
 the usual text we know and love 
]

Did you get a message about "There are invalid certificates in your trusted_certs_dir.” from `knife ssl check`? (Your certs would have to exist in your trusted_certs_dir before you run that command for `knife ssl check` to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we’ve seen this when users follow an outdated blog post for setting "Subject Alternative Names,” but you could have stumbled upon this issue as well.

Post by Ivan Suftin

Post by Ivan Suftin
$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org (http://chef.owicloud.org) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `[]' for nil:NilClass

This looks like an actual bug with `knife ssl fetch`. If you run `knife ssl fetch URL -VV`, it will show the backtrace. You should then copy all of that into a bug report at https://github.com/chef/chef/issues so we can fix the command.

Post by Ivan Suftin

Post by Ivan Suftin
$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?
__________________________ (╯°□°╯ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo

[Tensibai]

I'll try to resume what I know with SSL validation.

I had issues with self-signed certificates for the chef-server and our
proxy certificate wich do ssl interception.

What solved nearly all for me is to add the certificates in the
cacert.pem file.

For the self signed certificate of the chef server:

openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null
2>/dev/null|openssl x509 -outform PEM >>
/opt/chef/embedded/ssl/certs/cacert.pem

For my proxy I had to add our internal root Certificate authority cert
(took from a browser and exporting certificate to disk)

for some other tools I had to do `export
SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem` to enforce
openssl lib to use my updated certificate list.

After that, one cookbook to distribute it and update chef conf after the
file is in place (for nodes).

Hope it may help.

Post by Ivan Suftin
Hi Daniel,
I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir". Do you have more information regarding the "Subject Alternative Names" issue? I could get more information for you if needed in order to better analyze the issue.
In regards to the knife issue, I've opened this ticket: https://github.com/chef/chef/issues/2919 [1]
__________________________ (╯°□°╯ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/ [2]
United States Geological Survey
8505 Research Way, Middleton, WI 53562
Ohai Chefs!
We have a Chef 12 server set up with a self-signed cert. I'm trying to get knife to communicate using peer verification with the server.
$ knife ssl check https://chef.owicloud.org/organizations/cida [3]
Connecting to host chef.owicloud.org [4] (http://chef.owicloud.org [4]):443
ERROR: The SSL certificate of chef.owicloud.org [4] (http://chef.owicloud.org [4]) could not be verified
* Version: OpenSSL 1.0.1k 8 Jan 2015
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"
[ ... the usual text we know and love ...]

Did you get a message about "There are invalid certificates in your
trusted_certs_dir." from `knife ssl check`? (Your certs would have to
exist in your trusted_certs_dir before you run that command for `knife
ssl check` to detect this issue). There are certain properties on
certificates, which if they are not set, will make OpenSSL refuse to
verify the certificate against itself. In the past we've seen this when
users follow an outdated blog post for setting "Subject Alternative
Names," but you could have stumbled upon this issue as well.

Post by Ivan Suftin

$ knife ssl fetch https://chef.owicloud.org/ [5]
WARNING: Certificates from chef.owicloud.org [4] (http://chef.owicloud.org [4]) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `[]' for nil:NilClass

This looks like an actual bug with `knife ssl fetch`. If you run `knife
ssl fetch URL -VV`, it will show the backtrace. You should then copy all
of that into a bug report at https://github.com/chef/chef/issues [6] so
we can fix the command.

Post by Ivan Suftin

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?
__________________________ (╯°□°╯ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/ [2]
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo



Links:
------
[1] https://github.com/chef/chef/issues/2919
[2] http://cida.usgs.gov/
[3] https://chef.owicloud.org/organizations/cida
[4] http://chef.owicloud.org
[5] https://chef.owicloud.org/
[6] https://github.com/chef/chef/issues

[Ivan Suftin]

So this actually turned out to be an issue with our VPC perimeter firewall we use at AWS. When VPN’d into the VPC, past the firewall and using the direct IP of the machine, the knife command acts as expected.

We will be switching to a dedicated VPN soon to connect our datacenter into the VPC in a bidirectional way. At that point this will not be an issue, so for now I will continue turning off peer verification.

Thank you for your help!
__________________________ (╯°□°╯ ┻━┻
Ivan Suftin - Applications Developer - ***@usgs.gov
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

Post by Tensibai
I'll try to resume what I know with SSL validation.
I had issues with self-signed certificates for the chef-server and our proxy certificate wich do ssl interception.
What solved nearly all for me is to add the certificates in the cacert.pem file.
openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >> /opt/chef/embedded/ssl/certs/cacert.pem
For my proxy I had to add our internal root Certificate authority cert (took from a browser and exporting certificate to disk)
for some other tools I had to do `export SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem` to enforce openssl lib to use my updated certificate list.
After that, one cookbook to distribute it and update chef conf after the file is in place (for nodes).
Hope it may help.

Post by Ivan Suftin
Hi Daniel,
I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir". Do you have more information regarding the "Subject Alternative Names" issue? I could get more information for you if needed in order to better analyze the issue.
In regards to the knife issue, I've opened this ticket: https://github.com/chef/chef/issues/2919 <https://github.com/chef/chef/issues/2919>
__________________________ (╯°□°╯ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/ <http://cida.usgs.gov/>
United States Geological Survey
8505 Research Way, Middleton, WI 53562

Post by Ivan Suftin
Ohai Chefs!
We have a Chef 12 server set up with a self-signed cert. I'm trying to get knife to communicate using peer verification with the server.

$ knife ssl check https://chef.owicloud.org/organizations/cida <https://chef.owicloud.org/organizations/cida>
Connecting to host chef.owicloud.org <http://chef.owicloud.org/> (http://chef.owicloud.org <http://chef.owicloud.org/>):443
ERROR: The SSL certificate of chef.owicloud.org <http://chef.owicloud.org/> (http://chef.owicloud.org <http://chef.owicloud.org/>) could not be verified
* Version: OpenSSL 1.0.1k 8 Jan 2015
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"
[ ... the usual text we know and love ...]

Did you get a message about "There are invalid certificates in your trusted_certs_dir." from `knife ssl check`? (Your certs would have to exist in your trusted_certs_dir before you run that command for `knife ssl check` to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we've seen this when users follow an outdated blog post for setting "Subject Alternative Names," but you could have stumbled upon this issue as well.

Post by Ivan Suftin

$ knife ssl fetch https://chef.owicloud.org/ <https://chef.owicloud.org/>
WARNING: Certificates from chef.owicloud.org <http://chef.owicloud.org/> (http://chef.owicloud.org <http://chef.owicloud.org/>) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the `-VV` option before filing a bug report.
Exception: NoMethodError: undefined method `[]' for nil:NilClass

This looks like an actual bug with `knife ssl fetch`. If you run `knife ssl fetch URL -VV`, it will show the backtrace. You should then copy all of that into a bug report at https://github.com/chef/chef/issues <https://github.com/chef/chef/issues> so we can fix the command.

Post by Ivan Suftin

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?
__________________________ (╯°□°╯ ┻━┻
Office: [phone] - Cell : [phone]
Center for Integrated Data Analytics - http://cida.usgs.gov/ <http://cida.usgs.gov/>
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo