The Art of SQL Server Database Administration, Development, and Career Skills for the Technically Minded
TSQL2sday topic number 51 is outstanding. Hosting Adam Machanic‘s rotating blog party this month is Jason Brimhall aka Twitter @SQLRnnr who asks “when has somebody (a CTO, Developer, Business User) placed a bet that was far too risky in your opinion?”
I’m really looking forward to reading the comedy / tragedy this topic should spark among my peers.
Every decision of significance entails some form of risk. That’s life. It’s the follow up decisions that tend to compound the original risk. This is when accurate information is critical.
We’ve all seen horrible decisions based on bad ones. Success or failure of the first isn’t even a good indicator of what happens next. On one side you have a positive outcome. Emboldened, the decision maker raises the bar with a far riskier play. On the other side you have a negative outcome. The motive this time is to make up for, or hide the results of the first.
The really costly decisions – the ones with the kind of exponential risk that I’m talking about however – are those where follow up decisions are made before the outcome of the first is even known. Decisions made without facts.
I witnessed this risky decision chain in practice several years ago.
A business need was identified. A vendor solicited. A sales pitch offered. A promise to deliver an application made. This promise contained a specific feature set made by a group of developers who had little to no true database expertise in the group. Talented in language of choice I’m sure – but not relational theorists.
Millions changed hands while the business began putting into place significant dependencies on the successful delivery of the feature.
The problem? The vendors were building a freaky complex process that database administrators know as merge replication. The vendor apparently didn’t know that Microsoft already made the solution they were looking for. Neither did the business.
Several months later the solution was delivered and the DBA team was asked to implement and support. It was the first they’d heard of it – any of it – seriously. This new feature was a closely held secret by those near top; they didn’t want internal IT getting in the way of progress.
I’m sure you are aware that replication is complex – especially the merge variety. When the ins and outs are buried in code that is out of control of the administrators it becomes a nightmare. The solution was rife with problems but long before delivery the business had already made promises, contracts, and commitments that would not be easily undone – especially with pride on the line.
Root cause analysis
Why would a group of individuals risk millions of dollars and reputation on an approach they had literally no experience to assess? I know why a vendor would. Profit and survival. But why would a business?
In this case it came down to trust. The folks making the decisions didn’t trust the people who could advise them on the complexity of the solutions being considered. Past experience had ruined relationships between the two teams. The business was tired of being told no by internal IT with no additional information or options.
The root cause analysis identified not a technical issue but one of personnel dysfunction. This was not an isolated incident. This was a situation in the final throws of long term decline. Shortly major changes began to take shape, attitudes adjusted, chains of command re-established, and more than a couple people let go on both floors.
There is no stopping risk. As a friend of mine was fond of saying… sometimes you just have to grab the scissors and run. Our primary job as database professionals is to protect the data but an extremely close second is to establish relationships of trust with those in a position to make decisions affecting that data.
It’s important to know that with or without you those decisions are going to be made and most of them will entail some amount of risk so grab the scissors and run.
You must not only be an administrator but a subject matter expert. An SME must also naturally be a mentor. It’s a delicate balancing act at times to be both the protector of the kingdom without also being the obstacle of progress – but there you have it. That’s the job.
I challenge you to attempt to understand the needs of your business almost as much as you understand the needs of your database server. Accept that risk exists and the mitigating factor is communication and understanding.
Keep this in mind and hopefully you’ll never find yourself troubleshooting a subquery … with a function in the where clause … inside a cursor … inside an update trigger … trying to keep two unique instances in sync. Because that sucks.