Debugging a broken puppet system (without knowing ruby or puppet)

If your working on a modern big data or web deployment in an agile software shop, puppet is likely the secret sauce behind your cluster.  You better learn how to read it before its too late!

Say you're working on a beautiful system that was elegantly designed by a puppet expert, and there is a small bug.  And you are neither a ruby, nor a puppet expert. How do you debug the problem? 

The options are not appealing to someone who isn't paid to code in ruby.

(1) Learn ruby, then learn puppet - just so that you can debug a simple problem.
(2) Go on the puppet mailing list and post specific (possibly proprietary) code snippets to get help
(3) Take the puppet training class and come back in a month.  I'm going to try another option...
(4) Learn the bare minimum  and move on with your life :)


UPDATE: I just realized a cool trick for visualizing puppet configs.   This probably is the first thing you might want to try when entering into a new puppet based config system.

sudo puppet apply -d --confdir=/myapp/puppet/ --modulepath=/myapp/puppet/modules /myapp/puppet/manifests/site.pp --noop --graph --graphdir /tmp/puppetgraphs

And then:

dot -Tjpg /tmp/puppetgraphs/relationships.dot

And then you get something like this:

 
Learn some ruby basics. 

:: 

To access constants defined in the Object class, operator :: without the left hand side operand can be used. So when you see X::Y::Z errors in a puppet script, it means that there is an error in an inner class (Z) defined for outer class (X).   From the ruby syntax page at njit.

=>

In ruby, a => means a "hash rocket", its how you define a key value pair.  In puppet, its similar => is an "attribute", which is (essentially) a key value pair.  Hence, it borrows the syntax of hash elements from a ruby hash map.  From the puppet docs.

Now that you're a ruby expert : lets get to the #1 most important debugging feature in the universe - the print statement.

Puppet modules are, at the end of the day - code.   And like all your other favorite programming languages, they feature a logging feature, which you can leverage really easily.  Its called "notify".
For example, you can inject notify anywhere into a class....From the Puppet Cookbook:

PUPPET VERSIONS

A very important thing to note is that puppet code for 3.0  isn't backwards compatible.  I learned this the hard way.  Make sure you have a 2.x version of puppet installed if you have a 2.x code base.  Sometimes, the warnings may not be there, because puppet 3.x will try to run anyways, but conventions (like smart lookup of unqualified variable names) won't work anymore in 3.x, even though they worked in 2.x.

Confdir : No match found

1) I recently , for example, ran into an issue where there were no parameters being found by my puppet apply run.  The error looks like this:

Error: No match found for 'hadoop_head_node' in any data file during extlookup() at .............

So what was the error?  The issue was that my configuration files were not on the path, because puppet wanted a /config/ directory UNDERNEATH the --confdir option.  

The lesson here is that, if you get the "no match found" error, then either (1) puppet isn't reading your conf files or (2) it IS reading your conf files, but you dont have the parameter defined.  And more importantly : In some cases, you may need to put your files in a subdirectory of the --confdir.   

In general, you can pass in the value of $confidir when you apply a puppet module.  To confirm that its getting picked up - 

Could not retrieve facts for xxxxx: no implicit conversion of nil into String

This can occur when you have corrupt modules, or incorrect modules, or something somewhere in between.   It appears that this can actually even occur when your puppet scripts don't depend on any modules at all.  The lesson is ~ keep your modules directory clean , and install modules you trust.

Missing "value" errors : 

In puppet 3.0, versus 2.6 (as mentioned above), there are certain types of variable declarations that don't work anymore.   For example, unqualified variable declarations in template files, which worked in certain flavors of puppet 2.x, don't work in 3.x....

Obviously these are just a few simple tricks to help you get started debugging a puppet environment.  You can also ask around on #puppet (freenode) for help.  
 If comes to worse... You might have to bite the bullet and learn ruby !