Developers Need To Be Trusted
I worked at a client in the insurance business a few years ago where in addition to .Net development, I was also responsible for keeping their online .Net applications up and running. They were mission critical systems supplying quotes to the backend of many other companies. When they went down, which was often as they were badly written in out of date versions and doing bizarre things such as using SQL Server tables without any keys defined, there would soon be a conference call with a dozen or more managers from various business and technical departments on it, and I would get invited into the call.
It would start off by a senior business managers banging on about how downtime was costing them £10,000 an hour (or whatever they dreamt up that day), purely to pile the pressure on. Next, all the other parties, such as network admins, DBAs, etc., would insist the problem wasn't theirs (without any proof) and it was an application problem, and soon drop off the call, leaving me to have to do all the work. I would be asked what the problem is, and I would tell them I don't know, I was only just told about it. They would start throwing random suggestions at me, and I would say I couldn't say without looking, they would ask if we had changed anything? No. Had we deployed anything? No. It would always end up with them asking how long will it take to resolve it? Again, I would say I don't know without looking, and I would point out I didn't actually have any access to production to look, so how would they like me to proceed? In response there would always be a long silence, followed by suggestions such as could someone (who was trusted) share their screen with me (so they could monitor what I did)?
They didn't trust developers, none were allowed any access to production. My conclusion was they either thought most developers were malicious and untrustworthy and might sabotage the production environment, or were incompetent and afraid they would break something. The irony of it was that the support staff, especially the junior ones, who were trusted with access to production were the ones most likely to break something through incompetence, and the developers were the ones actually keeping the systems working. Developers are often only ones in an organisation that actually understand how an application should be or is deployed, how it should be configured, and able to trouble shoot and diagnose problems. Their logic is that it is perfectly ok to give full access to staff who have no idea how the systems work, no idea how they are configured and take no responsibility for them, but not to developers.
On a number of occassions I was asked if I could look for the problem in the development environment, or one of the test environments, without any error information to go on. Did they think I was psychic? On one occassion a manager asked if I could look through the source code and see if I code see the possible cause of the production problem, an application with 200,000 lines of code. Absolute idiocracy. Sometimes the problem would be in the data, or only triggered by a specifc set of data, and could only be identified by looking at the production data involved. On another occasion they had all agreed it was a problem with the application code (how I don't know), only for Microsoft to eventually analyse a dump and identify the cause was a directory in production containing over 1 million PDFs choking the server CPU every time an Index scan ran. I had spent weeks trying to find the cause in development and test environments where the problem didn't even exist, while the network admins who had allowed the directory to grow to such a size had washed their hands of it and walked away from the issue on day 1.
Developers need access to prodution to be able to sort out deployment issues, configuration issues, and ongoing production support. They don't need update access (I prefer not to have it), but should at least have read only access so they can see files deployed, configuration settings, and view data. When I build a system I build in certain pro-active error checking, I often design in an Error table in the database, and wrap areas of code in Try/Catch blocks and write the technical details of the error, inclduing things that only developers need such as the source code location, Call Stack, etc., to the table. If developers have read access to this table they can monitor it for unreported issues or developing problems, but its pointless in organisations where they refuse you any access at all. Network managers need to wake up and realise developers are like anyone else in the company and just as trustworthy and at least give them read access and appreciate the fact that they are the ones more than anyone else who keeps system running.