Finding Trouble

If there was any one skill I could rank as number 1 for new hires, be they junior or senior, it would be troubleshooting.  If someone out there has a surefire method for identifying that trait up front in potential hires – that would get me past just about any other negative aspect against them; including constant throat clearing and repetitive ball point pen clicking.  Top tier troubleshooters are a rare gem.  When you find one, never ever ever let them go.  More on finding that skill in a minute, let me give an example of one in action.

---

I have this gal that I work with – Josephine.  When I first met her she was finishing up her MCSE but only had about a year of decent experience with SQL Server in any type of professional assignment.  She was so genuine in her interview and so obviously smart that we took a chance and spent a senior requisition on her – even though her resume was more junior to mid level and I really needed a super senior.  Fast forward 6 months and she is now my most trusted and skilled DBA.  She has a lot of great skills, honesty, work ethic, to name a few – but way up there at number one…  she can troubleshoot like no one I’ve ever met.  She just has a brain for figuring crap out – even if she has no prior knowledge or experience in a specific area.

Case in point:

We have a custom log shipping solution that I actually blogged about awhile back.  We upgraded that group of servers and the PowerShell script that runs it, stopped working.  The step in the script that moved the log backup file from one server to another wouldn’t connect.  We checked rights first, all good.  We dug into the firewall and splunk.  Found unrelated blocking, fixed it, but didn’t fix the original problem.  The OS had been rebuilt underneath as well so we started looking into group policy issues, firewall issues again, run as a service settings.  The script worked fine when run manually, just not when run under the agent.  We created a temporary credential under my own login, built a proxy against it, then told the agent job to run PowerShell under that.  Still no go.

Out of the blue Josephine says “fixed it’.  Turns out the PowerShell script is running under the SQL Server provider.  So, the UNC paths we were using had no meaning.  We dropped a C: at the top of the script to put the provider under the operating system provider and BAM, everything started working again.  More on that specific issue can be found under this connect item.

She figured all this out in about 15 minutes after a couple others of us had all been banging our heads against it for a couple days.  By the way, ask Josephine what her knowledge level of PowerShell is and her response is: beginner – copy and paste only.  Ask me what her troubleshooting skills are, and I say – up there with agent Romanoff.  Black Widow good.

Back to my point:

I want feedback.  When hiring, how do you discover good troubleshooting skills?  The age old light-bulb went out scenario?  Do you give them a test.  Do you just ask open ended, tell me about your process to troubleshooting.  Do you just roll the bones and pray?  I want to know.  Because I need a few more people like Josephine.  Oh, and if you value your life, you won’t try to hire Josephine away from me.  That wasn’t a joke.

Image Credits:  Fair use – satire – Lego Black Widow minifgure.

Advertisement