I'm having issues expanding a puppet-server deployment beyond ten nodes. Specifically, the issue appears to be authentication related, but I cannot track down what would cause it. **Deployment** I have a puppet-server deployment with an external CA. The master certificate is signed by an intermediate certificate and the agent certificates are signed by another intermediate. Both intermediates are signed by the same root. [This is the deployment as described in the Puppet documentation](https://docs.puppetlabs.com/puppet/latest/reference/config_ssl_external_ca.html#option-3-two-intermediate-cas-issued-by-one-root-ca). Additionally, I have a running Puppet DB deploy connected to Puppet. **The Problem** Before the problem started, I had 10 working nodes. I've had no issues with authentication using the external CA (certificates generated using EJBCA). When I attempted to add three additional nodes, each node has the same kind of errors that suggest authentication problems. These errors are 403 errors when running `puppet agent -t`. As an example, when attempting to run puppet, the following happens: # /opt/puppetlabs/bin/puppet agent -t Info: Retrieving pluginfacts Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': Error 403 on SERVER: Forbidden request: host01.local.test(XXX.XXX.XXX.XXX) access to /puppet/v3/file_metadata/pluginfacts [search] at :124 Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: Error 403 on SERVER: Forbidden request: host01.local.test(XXX.XXX.XXX.XXX) access to /puppet/v3/file_metadata/pluginfacts [find] at :124 Info: Retrieving plugin Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': Error 403 on SERVER: Forbidden request: host01.local.test(XXX.XXX.XXX.XXX) access to /puppet/v3/file_metadata/plugins [search] at :124 Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: Error 403 on SERVER: Forbidden request: host01.local.test(XXX.XXX.XXX.XXX) access to /puppet/v3/file_metadata/plugins [find] at :124 Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, $concat_basedir not defined. Try running again with pluginsync=true on the [master] and/or [main] section of your node's '/etc/puppet/puppet.conf'. at /etc/puppetlabs/code/environments/production/modules/concat/manifests/setup.pp:19:5 on node host01.local.test Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: Error 403 on SERVER: Forbidden request: host01.local.test(XXX.XXX.XXX.XXX) access to /puppet/v3/report/host01.local.test [save] at :124 This happens on only the newest nodes. When I disable the authentication on the puppet server in `/etc/puppetlabs/puppet/auth.conf`, the `puppet agent -t` command completes successfully. So, this suggests that it is tied to the puppet server authentication. Because existing nodes were able to authenticate and run successfully, I revoked an existing certificate and generated a new certificate for one of the known working nodes. When running `puppet agent -t`, the run completes successfully. So, this suggests that the CA - client auth is working as expected since newly generated certificates work for existing nodes. Finally, when I test one of the node's client certificates against the puppet server, it appears to authenticate successfully: [root@host02 /etc/puppetlabs/puppet/ssl]# openssl s_client -host puppetserver.local.test -port 8140 -cert certs/host02.local.test.pem -key private_keys/host02.local.test.pem -CAfile certs/ca.pem CONNECTED(00000003) depth=2 C = US, ST = XXXXXXXX, L = CITY, O = ORGANIZATION, OU = ORGANIZATION Certificate Authority, CN = ORGANIZATION Root CA, emailAddress = email@local.test verify return:1 depth=1 C = US, ST = XXXXXXXX, O = ORGANIZATION, OU = ORGANIZATION Certificate Authority, CN = "ORGANIZATION Intermediate CA, Puppet Master", emailAddress = email@local.test verify return:1 depth=0 CN = puppetserver.local.test, O = "ORGANIZATION", C = US verify return:1 --- Certificate chain 0 s:/CN=puppetserver.local.test/O=ORGANIZATION/C=US i:/C=US/ST=XXXXXXXX/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Intermediate CA, Puppet Master/emailAddress=email@local.test 1 s:/C=US/ST=XXXXXXXX/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Intermediate CA, Puppet Master/emailAddress=email@local.test i:/C=US/ST=XXXXXXXX/L=CITY/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Root CA/emailAddress=email@local.test --- Server certificate -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- subject=/CN=puppetserver.local.test/O=ORGANIZATION/C=US issuer=/C=US/ST=XXXXXXXX/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Intermediate CA, Puppet Master/emailAddress=email@local.test --- Acceptable client certificate CA names /C=US/ST=XXXXXXXX/L=CITY/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Root CA/emailAddress=email@local.test /C=US/ST=XXXXXXXX/O=ORGANIZATION/OU=ORGANIZATION Certificate Authority/CN=ORGANIZATION Intermediate CA, Puppet Agent/emailAddress=email@local.test --- SSL handshake has read 3581 bytes and written 5495 bytes --- New, TLSv1/SSLv3, Cipher is AES256-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : AES256-SHA256 Session-ID: 55DF7610FB120D65B86CF5AACFDA01A865A1922BB8F71B81F375F2BFA3AFC7D8 Session-ID-ctx: Master-Key: 799ED3E825FE614A1CC0D90D8F9CD9A696228C6077A4D25C6A0C80FE54017647B624F1C1D36D880723005E179924B98E Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1440708083 Timeout : 300 (sec) Verify return code: 0 (ok) --- I know Puppet Enterprise has a 10 node limitation, but I'm not running Puppet Enterprise. I'm running `puppetserver-2.1.1-1.el7`. Here are the currently installed puppet-related RPMs: [user@puppetserver]~% rpm -qa | grep puppet puppet-agent-1.2.2-1.el7.x86_64 puppetserver-2.1.1-1.el7.noarch puppetdb-termini-3.1.0-1.el7.noarch puppetlabs-release-pc1-0.9.2-1.el7.noarch **Questions** - Any ideas as to what might be causing these new nodes to fail? - Does Puppet Server have the same 10 node limitation as Puppet Enterprise?

I have a PE Master in AWS which uses the `puppetlabs-aws` module. I have created a user with admin privileges (to be pared down, eventually) for use speicifically for performing Puppet actions on our AWS environment; Creating Instances, routes, DNS entries, etc. Every now and then an action times out and throws the error in the subject. Ex.: # puppet apply --noop --verbose /etc/puppetlabs/code/environments/production/modules/aws_prod/manifests/init.pp Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for puppetmaster.cspops.int in environment production in 0.07 seconds Info: Applying configuration version '1457384334' Error: Failed to apply catalog: execution expired Sometimes this happens each run. Other times it will actually go through and will continue to do so for a brief period of time. # puppet apply --noop --verbose /etc/puppetlabs/code/environments/production/modules/aws_prod/manifests/init.pp Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for puppetmaster.cspops.int in environment production in 0.07 seconds Info: Applying configuration version '1457384457' Notice: /Stage[main]/Main/Ec2_instance[puppet_prov_test]/ensure: current_value absent, should be present (noop) Notice: Class[Main]: Would have triggered 'refresh' from 1 events Notice: Stage[main]: Would have triggered 'refresh' from 1 events Notice: Applied catalog in 11.04 seconds # puppet apply --verbose /etc/puppetlabs/code/environments/production/modules/aws_prod/manifests/init.pp Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for puppetmaster.cspops.int in environment production in 0.07 seconds Info: Applying configuration version '1457384482' Info: Checking if instance puppet_prov_test is running in region us-east-1 Info: Checking if instance puppet_prov_test is stopped in region us-east-1 Info: Starting instance puppet_prov_test in region us-east-1 Notice: /Stage[main]/Main/Ec2_instance[puppet_prov_test]/ensure: changed absent to running Notice: Applied catalog in 11.47 seconds I understand this is likely something on the AWS side, but the error is from the module and doesn't give any further indication as to which direction the troubleshooting should go. Additionally, I never experience this issue when using the `AWS CLI`. I've looked for auth logs in AWS but have not been able to find anything. Has anyone else dealt with this and have something that might shed some light on why this intermittently breaks?

Hi all, Is there any way to make authentication on top of puppetdb api?

This post is awaiting moderation The client error is as follows： notice: Ignoring --listen on onetime run err: Could not retrieve catalog from remote server: SSLconnect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=puppetca.test.com] warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run err: Could not send report: SSLconnect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate signature failure for /CN=puppetca.test.com] At the server, check with the puppet cert list - all that the client has issued the certificate, but failed