Change Control: Overcoming the Compulsion to Repeat the Past
Many businesses are deploying software through cloud or hosted service providers, by one estimate by a factor of 4 to 1. They may not realize that the cloud provider implements partner services, such as security, on a particular cycle. The result is that Web application customers – that would be you, the developer – may need to synchronize development cycles with their maintenance cycles. That effectively re-creates the problems you thought you were dodging with local hosting and your own IT department.
Change management can be perceived in two ways: as a means of delivering change, or as a means of resisting it. It is the singular task that joins the IT and development departments, for better or worse. For administrators and IT professionals, change management tools are often sold as policy enforcement tools: reactive mechanisms for resisting change when employees try to impose it on systems with their own installations and uploads. For developers, “change management” refers to the care and feeding of the application lifecycle. Their efforts are devoted to maintaining the evolving needs of customers, and ensuring that the development team is keeping pace.
One of the greatest transitions in the past five years is the move to cloud-based deployment. Conceptually, virtual machines are now accessible from any device and cloud-based apps are now addressable from any browser. As a result, apps are now easier and more cost-effective to produce, and their development models now sync splendidly with the new model of iterative development common with Agile teams. Development teams need change management more urgently now (not the resistance kind), to keep better track of where they are and where they’re going.
But the whole change management process can actually fall apart, and break down into truly psychological chaos. Because we’re talking about more than recording who added which code to the current project’s build scripts. This is about the management of change in organizations.
Some weeks ago, I spoke with an executive of a security tools provider that is in the process of transitioning many of its on-premise services into cloud-based services. I’m concealing the identity of this company and its products because the nature of what we discussed became quite sensitive.
As this security provider told me, when Web apps move to the cloud, often their security processes move with them. Many cloud service providers offer premium tier hosting services: for instance, Acquia’s “shared responsibility model” built on Amazon’s AWS platform, Rackspace Intensive Hosting, Oracle’s On Demand hosted services for the public sector, and Sungard’s AvantGard premium tier. These premium services include the assignment of dedicated lead technicians: essentially, IT managers who work for the host but who respond to the customer. These managers have their own security policies, their own work cycles, and their own change control models.
Historically, security functions take place during downtime. But now that apps are in someone else’s cloud, along with thousands of other customers worldwide — where “daytime” is any time at all — “downtime” is impossible to fathom. This creates a quandary for the security provider, which now must follow two sets of policies: those of its customers and those of their hosts.
Optimally, these policies should be merged, making the development cycle, the management cycle, and the security cycle compatible, like wheels in a single machine. As was explained to me, usually the cloud provider takes charge of this mission. It begins by enforcing a new discipline, with a new style of change control. From the provider’s perspective, this discipline is simply a way to ensure smooth operation, by providing what it considers to be basic, required service. Depending on the data’s nature, classification, and sensitivity, change control may also be a federal requirement (for example, the SSAE 16 auditing requirement for certified accountants).
The job of enforcing this new discipline falls to the cloud provider because, let’s face it, that’s what the company is paid to do. But the task of interpreting the details of what’s being enforced falls to someone employed by the provider’s customer. Cloud providers typically do a thorough job of providing these details; I’m told, including in-depth analytics reports of traffic patterns, usually with heuristic charts. Optimally, security processes for a data center are scheduled for the regular “valleys” in those traffic charts. But with a public cloud data center, which simultaneously serves customers spanning the globe, collectively everyone’s “peaks and valleys” iron each other out.
So the cloud provider may provide some abstract view of its own change model, which may include a peek at its own engineering change logs (ECLs).
As a developer, you probably deal with change on an hourly basis. When you use software, such as a team-oriented IDE, to help follow a prescribed itinerary, that software usually generates an audit trail. Some IT managers call the documents that form this audit trail engineering change logs (ECLs).
Some businesses have adopted a formal methodology for managing changes to their information systems. For the last decade or so, some of these may follow an engineering change model (ECM), whose associated files are managed on a server. (In 2003, a pair of University of Hong Kong professors suggested that the ECM be maintained on a completely separate machine, but in this age of virtualization that’s probably impractical.) Most modern Web apps developers follow a more dynamic approach called change log management.
I was intrigued by what this security tools provider told me, so I contacted someone who’s an expert in performance management. Good performance management takes place when every losing NFL team studies, analyzes, and re-analyzes the tapes the next day.
I wondered: Can the success of change management projects be measured through performance management tools? Can an analysis of the ECLs show us success or failure plainly on a chart? That might help development teams isolate the points when companies fall back upon familiar patterns, and start resisting change rather than embracing it.
Web applications are complex systems that are subject to failure. Regularizing and normalizing the development cycle should not create opportunities for teams to fall back into familiar patterns. Instead, we should look for patterns of guidance: signals that may not mean much by themselves, but through repetition and regularity form a clearer picture of how change is being implemented and managed. A Web application is constantly shifting, and patterns of those shifts are sometimes only detectable by studying past performance.
Many businesses — perhaps yours included — have moved or are moving to the organizational model called DevOps. It’s not particularly a new idea, at least not in the world where actual work is being done, though recently it’s become a buzzword among marketing folks. In the enterprise, though, to one extent or another, administrators and IT personnel have always been coders. Their tools have been different and the languages used to be different.
But now that software development is taking after the Web apps model, and Web apps are based on scripting languages like PHP and Python — languages more similar to the ones admins have been using for decades — both IT executives and vendors see an opportunity to merge two business units, and thus two work cycles, into one.
Often, the IT department and the development team have two separate mindsets. So a measure of their workflow in terms of performance would look as different from one another as two sets of EKGs from separate species. When one mindset is devoted to the upkeep of the systems, and the other to the upkeep of the applications, unifying the two under one banner – whatever you call it – inevitably places the two at odds with one another.
So the performance tools expert advised me that organizations should select someone who can serve as the change leader: a kind of liaison between departments, which have their own itineraries, workflows, and cycles of change. Someone needs to sell both departments, but especially IT, that change is how everyone grows.
Having a vision is wonderful. Having a plan is better. Having an aching, unavoidable desire for change... is rare.
As we’ve seen too often recently, you can plaster the walls with posters of visionaries whose faces are adorned with “HOPE” or “CHANGE,” but in reality, hope is not the seed of change, and vision doesn’t come from a talking head. When you entrust one person with the job of “change visionary,” optimally that person should have a positive stake in the outcome.
Here is what can happen: Someone in the organization is chosen as the liaison officer for the new public or hybrid cloud provider. This is the person who was responsible for implementing change control, or whatever passed for it, for the business’ on-premise data center. This is also the person who was uniquely capable of reading the ECLs and making adjustments to the schedule that kept work humming along smoothly. His former job was necessitated in part on the security of the systems he managed.
And now all that’s gone. Security is now being coordinated with the cloud provider based on its own change control policies, which are now out of this person’s control. Along comes someone from outside who assumes the role of “discipline enforcer,” and hopefully this other person is accessible via e-mail, or has business hours in roughly the same time zone. Meanwhile, the whole process of apps deployment appears to have been simplified so that it works through a self-service kiosk no more difficult to use than an ATM.
If everything works right, and hope and change are successful in their mission of transforming and even simplifying the enterprise, what guarantee does this person have of a job going forward? Who or what depends on him anymore? And even if he stays employed, will he be respected for the role he plays? What incentive is he given to succeed that's stronger than the incentive to resurrect the one-time status quo?
This may not yet be an epidemic, though it could very well be a phenomenon: Businesses whose development teams, four or five years ago, were bound to rigid itineraries where point release cycles were only feasible every 18 months, may yet find themselves bound to itineraries whose cycles are every six weeks. There are, after all, so many variables in play: The cloud technicians have maintenance cycles, the security providers must follow those cycles, and the viability of the virtual test environments depends on how well these third parties have followed their respective itineraries. It takes a dependable, reliable, veteran IT technician to be able to sort through all the layers of bureaucracy, and to distill the truth from all those ECLs.
Or, at least, this is how the situation can be characterized. After all, the new job description does call for a “visionary.”
I asked Rackspace Senior Engineer Rob Collazzo, point blank, does his firm dictate the change control cycles for its customers, and enforce some new discipline? While his answer did provide some insight, on its face, it spelled out, “No.”
“We stay out of the way,” Collazzo says. “But if they ask, we’ll give them best practices for deployment, using
and using the cloud to deploy and test when they go live in their production environments, so they can take advantage of the cost savings that the cloud provides. We can offer advice and suggestions, but in the end, it all comes down to what the customer wants.”
Rackspace’s top tier service level, he admits, is intended to reduce the customer’s IT burden. By this, he absolutely does mean his service intends to change the role of IT managers from engineers to visionaries. He acknowledges the existence of the Two Camps of IT: sysadmins and developers; and he sees the emergence of DevOps as amalgams of members of both older camps. But he also admits that, even in shops that have formally embraced the DevOps structure, there ends up being two poles that can, when left to themselves, work against one another. So in Collazzo’s model, Rackspace service alleviates the workers at one of those poles from “fighting fires all the time, or just doing simple, day-to-day administration tasks — ensuring backups are working, checking logs” and enables them to be reassigned to something more “strategic.”
“‘Grunt work’ is kind of a bad term, but that’s what it is,” says Collazzo. “If I’m working with Rackspace, I don’t have to worry about that stuff anymore. I can assume that it’s all going to work.”
Automation vs. Motivation
It’s easy, when you’re dealing with technology and the processes that drive and maintain it, to mentally disassemble systems into shining shards of virtual components and reassemble them into better functioning models. When we center our discussions of efficiency in the workplace solely around abstract processes and flexible itineraries, it’s easy to perceive Agile as a goal, to see ourselves as practitioners of iterative processes. If we pattern our work the way we pattern our software, then conceivably, our needs will be minimized and our goals will be incremental, gradual, building. It’s like dance steps... isn’t it?
If the 20th century taught us anything, it’s that human beings cannot be automated. Despite the most ambitious goals of clinical researchers, pseudo-psychologists, totalitarians, and TV programming executives, people cannot be made to follow purely mechanical patterns of existence. Oh yes, they can follow repetitive cycles. They can, and generally do, create and follow patterns of work that can be less than complex, perhaps even mundane. But these are psychological tendencies, named so because they come with one thing that automated procedures inherently lack: defense mechanisms.
The new wave of cloud technology would have an entire department full of veteran engineers magically condense itself into one person. (Corporations that have grown dependent on downsizing prefer to see this process as magic, rather than what it is.) And that person would manifest himself as the on-premise equivalent of a civil rights leader, whose message would be that the very definition of his own role, and the functions and tasks for which he was depended on for years, have been outmoded in favor of someone else’s brand name.
This isn’t rational. And what we may be seeing instead in some organizations is not only a breakdown of the change control process, but a resumption of the old cycles of dependence that help assure certain people of a regular paycheck.
The danger in resuming these old work patterns is not that the magic of the cloud will never be able to manifest itself. It’s that the defense mechanisms that help entrench those patterns will grow stronger, and even destructive.
My point is this: Change management should be a job description, not just a product category. But job security — such as it is today — depends upon the skills you have today being necessary tomorrow. And if you know in advance that they’re not, especially by virtue of the fact that someone has just been appointed in charge of change management, then the one skill you may have to depend upon for your own job security is to ensure change does not happen too quickly. So the reaction to change, and thus the nature of change management in many organizations, are defensive and reactionary. The nature of those defenses, as well as the antidote for their effects on the organization, is not technological but psychological.
Id vs. Ego
Psychologists and psychoanalysts have a theory; employees build personal dependencies upon their everyday work routines. They defend these routines against alteration or even adaptation because, over time, workers come to rely upon routines for their personal comfort, self-assurance, and security.
The psychologists base their ideas on two sources: Psychoanalyst Dr. John Bowlby originated the “Theory of Attachment,” the notion that people develop styles of interpersonal behavior based on how they received nurturing and care from their families or caregivers. Bowlby’s work, in turn, was based on Dr. Alfred Adler, who believed employees actively condition their work environments to maximize their return on trust and respect. With their work as foundation, the new theory makes this argument: Employees project their own patterns of upbringing upon their workplace, borrowing patterns from their old home life to help make them feel they belong. In so doing, they protect and defend their tasks and work cycles — especially the reliably repetitive ones — as they would their own families.
So when changes in the nature of work itself remodel customary processes and routines, employees and entire businesses may take steps — perhaps covertly, maybe subconsciously — to recreate the substance of those tasks, and to resume their old work patterns as best they can. And here’s the clincher: Because some families are dysfunctional, and because employees find comfort in their own family-like routines whether or not they are dysfunctional, they defend and, if necessary, recreate dysfunctional routines in their workplace even when it makes no logical sense for them to do so.
Dr. Michael A. Diamond, professor at the Truman School of Public Affairs at the University of Missouri, has researched patterns of work in various high-stress workplaces (PDF available here). He writes:
“Configurations of authority and interpersonal relationships at work produce and perpetuate collective identities and ideologies in the form of organizational cultures. These institutional cultures are shaped by repetitive thematic and patterned narratives signifying experientially shared organizational stories, metaphors, and histories. Due in part, however, to the institutionalization of repetition and routine in the service of production, organizational members become oblivious to their workplace culture and practices. Defensive routines become automatic and commonplace... These structured and sometimes mechanistic environments grow to be comfortingly familiar, taken for granted, and unconscious to their participants... The sense of familiarity and sameness that comes from routine may be experienced as comforting to some, if only because eventually the taken for granted is rendered unconscious.”
Diamond’s study centered not on apps development teams but on police departments. In one case, the upper echelons of the department gave themselves all the mundane paperwork. But in so doing, they built themselves up as the center of gravity for the entire organization. When patrol officers complained that their work was ignored, they were ignored. Some officers who sought respect, or at least parity, actively sabotaged police operations, and even damaged equipment in an effort to get their superiors’ attention. Of course, those people were excised. Over a quarter century, the department systematically replaced them with officers who complained less that their superiors didn’t care, because they, in turned, cared not a whit for their superiors.
One organization with two magnetic poles: one that produces the work and the other that processes it. Both jobs are critical and neither is less vital than the other. Both sides are dependent on each other. Neither is acutely aware of the other’s presence. But in the minds of people on both sides, the weight of the world rests on their shoulders. You don’t have to carry a firearm and handcuffs to identify with this scenario.
In Dr. Diamond’s work, he leaves this message for us:
“Organizational researchers and change practitioners would be well served by a theoretical framework and comprehensive theory of human personality that informs their understanding of the human compulsion to repeat and not to learn from experience. Such a theory should provide insight into the human tendency to engage in what appears at the surface to be worthless and futile, self-defeating, acts of repetition, and acts that reinforce the status quo with maladaptive strategies and structures.”
At some point, we need to re-learn that departments and organizations are not methodologies. They’re people. The only way to influence behavior that has ever worked is through communication, which can only take place after we’ve realized and accepted that people are not processes that can be reprogrammed. Their work is their lives, and by providing them with their environment for work, what we normally call “the business” becomes a part of their psyches, of their identities, of who they are. You can’t flip a switch and change them from one role to another like an operating system. But you can give them the means to adapt, and the assurance that once the change is made, they will be valued, appreciated, and respected.
From the beginning of life, once she’s fed, warm, and sheltered, all a person truly needs is to belong.