Data Analysis by Unsophisticated People

I have been involved on a project recently where my role was more peripheral and I'd usually just spot check some of the changes for people who were deeply involved with the project. I'm referring to them as unsophisticated people since they failed to use a method that is repeatable and accurate.

I gave it little attention until I was sent a huge spreadsheet which contained provisioning codes for thousands of different product options and asked to verify that they were all correct. I realized at that point that the people deeply involved in the project have been doing things very manually and there were errors which were being discovered.

There's no way in hell that I was going to take the manual approach which would be extremely mind numbing and still error prone. So I wrote a perl script that simply parsed every row and cell which I decode with a hash table into human readable text. This was step 1.

I extended my parser to become a validator. I looked at all of the products and recognized that certain products should have certain provisioning codes. So I created templates using an array of the provisioning values and sorted. So now my parser reads each row of the spreadsheet, pushes each provisioning value into an array, sorts the array when all cells have been obtained, sorts the array and then compares with all of the templates (@current_row_array ~~ @certain_product_template). If it matches a template, then I simply indicate which template it matched and it is very likely fine. The ones that do not match any template have some type of error and need to be investigated. I was amazed that I have never seen or used the Perl '~~' operator.

Taking 2 hours to write this simple perl script and create the templates will pay off largely by saving everybody time and reducing errors. There will certainly be changes and I can simply keep running my validator until everything matches the templates. So we now only need to concentrate on the templates being valid (about 40 of them) instead of thousands of rows of provisioning codes.

Awful GUI Interfaces

About 2.5 years ago, I was pulled onto a cable television platform for some companies we purchased so that I could work some magic and get things manageable. Over the years, I've seen many different interfaces for managing devices. CLI, web applications, custom applications, and Java applications of varying degrees. I have a wide range of experience for comparison.

I've found some of the most egregious user interfaces for cable platform devices. Some things are just horrendous.

Specific example:  There's a guide server that we use for about 9 different markets. If a change is made and you need to push out the modified files to each market. (The GUI is simply ftp'ing files to a different device.) When you click the button in the GUI, it blocks and prevents you from sending anything to another market for 20 minutes. It is insane! Granted it is a Windoze "server", but blocking for 20 minutes because a ftp transfer is in progress is just crazy. I thought I'd prod their support and ask if there was any mechanism where one could tell the GUI to push files to all markets instead of waiting 20 minutes, then moving to the next market. Seems like it'd be a desirable feature instead of making a user wait 180 minutes for 9 markets to complete updates. I was not surprised with the response. Their support indicated there was no such thing within the GUI or externally of the GUI.

Managing Engineering Data

Remember the days of having engineering data books for all kinds of devices. I had a bookshelf full of books for processors, companion devices, and specifications. Semiconductor device manufacturers would publish and send books for distribution to our engineering teams along with application notes.

I recently reorganized my electronic "bookshelf" of data books, data sheets, and specifications. I decided to use my free 50GB box.com account to maintain synchronization with a folder on my desktop. This has worked out better than I imagined it would. I found that renaming the files using the box.com interface is actually quite nice. I've had this box.com account sitting there for several years and really never used it. I always imagined them having some fiscal issue and closing down like ubuntuone did. Granted, I did not lose anything when ubuntuone dissolved. I simply made sure everything was on my local filesystem at the time of termination. I'm also apprehensive about the type of data that I put on such a remote site. I refrain from putting any personal or financial information on such a site. Files related to devices or areas of engineering are perfect to put on such a site. Really, if it gets compromised or somebody downloads the ARM instruction set... Would it really even matter?

I had Sparkleshare running at some point in the past on my Linux server. I really did not spend much time organizing my files back then. I think using box.com is definitely easier to manage and organize things.

Conclusion:  I'd recommend using box.com to act as your virtual bookshelf. It will allow you to quickly organize your files and rename them nicely with very little time and effort.