Lessons learned - Transforming the Cage Monkey.

Monday, June 13, 2016

Photo_022007_001

As anyone who has ran hosted infrastructure knows, there is a particular species of IT administrator, almost never seen out of their native environment, often present in meetings as a disembodied voice over a noisy phone call.  Some organizations call them, Server Support Specialists, others call them Junior Systems Administrators, they call themselves, “Cage Monkeys”.

Charged with the smooth operation of the usually remotely located hosting facility, these folks rack and stack new servers, cable and label new switches, monitor the operation for failed components, and assist with deployment of new hardware, software and applications.  As a former Operations Manager and CIO, I can tell you a good cage man is worth his weight in bananas.

Sadly, with my exposure to AWS and the DevOps methodolgies they use, I can see that the days of the cage monkey are at an end.  Not merely because of the trend to migrate to AWS, but due to the way that AWS operations are becoming the “GOLD STANDARD”, in properly managed IT Operations.

Let us investigate some of the functions a Junior Administrator would serve in a hosted environment.

The “rack and stack”.  (The function of slotting a new server and applying power.)

Yes, some presence will be required to accomplish the initial install, but past that using hyper-converged infrastructure, and virtualization, a pair of beefy servers can be used to replace tens if not multiple tens of individual servers.

The “floppy shuffle”. (The function of installing, configuring, and patching a new server.)

Once the rack and stack is done,  the use of PXE-Boot, vmware’s auto deploy, or the various windows automated deployment tools, to install and configure the base OS / virtualization layer, remove the need for an “in-cage” presence.  At the worst use of IP-KVM’s or iLOM’s, will allow the remote administrator to install the base software.

The “cable and label”. (The function of cabling new servers and managing the cable plant.)

With the inclusion of high bandwidth connections from said virtualization hosts to a properly designed network infrastructure, network changes to accommodate  new services is a software driven event.

The “disk swap”. (Swapping out failing / failed disks.)

Properly redundant, and properly managed storage subsystems, mean that a failing / failed disk replacement can either be a scheduled event, can be handed off to the storage vendors techs, or at worst handed off to the hosting facility techs.

The “What’s that sound”. (AKA, What the hell just happened?)

Call it situational awareness, many times I have been in a data center and the ambient noise level has changed.  Fans going into high rpm mode, indicating a server reboot, disk arrays alarms beeping indicating a possible failure, all of which produce a “What the hell just happened?”

Proper monitoring, trending and capacity planning will allow the remote admin to have notice of critical events, without the need for a “biological auditory analysis engine”.

All of the above functions are handled very well by AWS, but the methods used can be applied hosted / on premise environments by using some of the suggestions I’ve provided.  Other on site functions can be similarly modified by a careful examination of the AWS practices.

The new “cage-monkey” 

As I spoke prior; “I can see that the days of the cage monkey are at an end.”  This does not mean that the folks filling this function will soon be filling the unemployment lines, it means that they will be learning new skills:

The “rack and stack” will be replaced with a Fully Automated Install,Pxe-boot, FAI, cobbler, kickstart, auto deploy, etc.

The “floppy shuffle” will be replaced with expertise in “Automated Deployments”, Chef, Puppet, Ansible, Salt, etc.

The “cable and label” will be replaced with expertise in “hyper-converged infrastructure”, and network configuration tools, cisco works, Chef, Puppet, etc.

The “disk swap” will be replaced with proper monitoring, remote hands and onsite support from the storage vendor

The “What’s that sound”, will be replaced with proper monitoring, alerting and in many cases auto correction methods tied to the monitoring systems.

The new cage monkey, will be the master of many Infrastructure as Code methods, and have the capacity to deploy large numbers of new servers / services, in a very short time.  One could call these folks, DevOps engineers, I am sure they will devise their own deprecatory name, probably “code monkey”.

Unfortunately,  the old hand cage monkey, will not appreciate this.  Once they do not need to be off site at the cage, they will need to conform to corporate dress standards, arrive at the office on time, take reasonable length lunches, and not leave early, and not play the latest video game while waiting for operations to finish.  (I did say I functioned as a remote admin for a number of years).

I see it is time for lunch, and I have a nice fresh banana, so I will close this ….

Leave a Reply

Your email address will not be published. Required fields are marked *