While untangling some of the messy parts, I questioned how the system ended up so inconsistent. The answer appeared to be related to different consultants being brought in over time to perform tasks and there was no internal guidance for the overall system. So each consultant performed their task in isolation of any overall design guidance, completed their specific task and moved on and would most likely never touch the system again.
After learning how changes were being introduced to the system, it became clear how things were so inconsistent and why it was extremely complex to maintain. The system was in chaos and heading further into chaos. Today, we are well onto our way to a manageable and maintainable platform and we have tools in place to audit for consistency and accuracy. Records that were inconsistently maintained have been replaced with automated reports that query the information directly from the elements of the platform. The elimination of many manual tasks have helped to gain the "buy in" of the design direction and discipline in the current design processes.
Hopefully one can learn something from reading about how the system mentioned above was able to get into such a chaotic and unmaintainable system. Allowing changes to occur by people without accountability or guidelines was the largest contributor. It has taken 12 months to get the system into a manageable state and I estimate another 6 months to get some of the networking atrocities changed to make for a more scalable system. Other things that have been instilled are daily backups for all devices, consistent ntp server configuration, consistent snmp trap configuration and alarming.
I expect to be taking proactive actions with the system by mid-2016 instead of the reactive mode that we've been working in out of pure necessity.
No comments:
Post a Comment