Data Analysis by Unsophisticated People

I have been involved on a project recently where my role was more peripheral and I'd usually just spot check some of the changes for people who were deeply involved with the project. I'm referring to them as unsophisticated people since they failed to use a method that is repeatable and accurate.

I gave it little attention until I was sent a huge spreadsheet which contained provisioning codes for thousands of different product options and asked to verify that they were all correct. I realized at that point that the people deeply involved in the project have been doing things very manually and there were errors which were being discovered.

There's no way in hell that I was going to take the manual approach which would be extremely mind numbing and still error prone. So I wrote a perl script that simply parsed every row and cell which I decode with a hash table into human readable text. This was step 1.

I extended my parser to become a validator. I looked at all of the products and recognized that certain products should have certain provisioning codes. So I created templates using an array of the provisioning values and sorted. So now my parser reads each row of the spreadsheet, pushes each provisioning value into an array, sorts the array when all cells have been obtained, sorts the array and then compares with all of the templates (@current_row_array ~~ @certain_product_template). If it matches a template, then I simply indicate which template it matched and it is very likely fine. The ones that do not match any template have some type of error and need to be investigated. I was amazed that I have never seen or used the Perl '~~' operator.

Taking 2 hours to write this simple perl script and create the templates will pay off largely by saving everybody time and reducing errors. There will certainly be changes and I can simply keep running my validator until everything matches the templates. So we now only need to concentrate on the templates being valid (about 40 of them) instead of thousands of rows of provisioning codes.

Awful GUI Interfaces

About 2.5 years ago, I was pulled onto a cable television platform for some companies we purchased so that I could work some magic and get things manageable. Over the years, I've seen many different interfaces for managing devices. CLI, web applications, custom applications, and Java applications of varying degrees. I have a wide range of experience for comparison.

I've found some of the most egregious user interfaces for cable platform devices. Some things are just horrendous.

Specific example:  There's a guide server that we use for about 9 different markets. If a change is made and you need to push out the modified files to each market. (The GUI is simply ftp'ing files to a different device.) When you click the button in the GUI, it blocks and prevents you from sending anything to another market for 20 minutes. It is insane! Granted it is a Windoze "server", but blocking for 20 minutes because a ftp transfer is in progress is just crazy. I thought I'd prod their support and ask if there was any mechanism where one could tell the GUI to push files to all markets instead of waiting 20 minutes, then moving to the next market. Seems like it'd be a desirable feature instead of making a user wait 180 minutes for 9 markets to complete updates. I was not surprised with the response. Their support indicated there was no such thing within the GUI or externally of the GUI.

Managing Engineering Data

Remember the days of having engineering data books for all kinds of devices. I had a bookshelf full of books for processors, companion devices, and specifications. Semiconductor device manufacturers would publish and send books for distribution to our engineering teams along with application notes.

I recently reorganized my electronic "bookshelf" of data books, data sheets, and specifications. I decided to use my free 50GB box.com account to maintain synchronization with a folder on my desktop. This has worked out better than I imagined it would. I found that renaming the files using the box.com interface is actually quite nice. I've had this box.com account sitting there for several years and really never used it. I always imagined them having some fiscal issue and closing down like ubuntuone did. Granted, I did not lose anything when ubuntuone dissolved. I simply made sure everything was on my local filesystem at the time of termination. I'm also apprehensive about the type of data that I put on such a remote site. I refrain from putting any personal or financial information on such a site. Files related to devices or areas of engineering are perfect to put on such a site. Really, if it gets compromised or somebody downloads the ARM instruction set... Would it really even matter?

I had Sparkleshare running at some point in the past on my Linux server. I really did not spend much time organizing my files back then. I think using box.com is definitely easier to manage and organize things.

Conclusion:  I'd recommend using box.com to act as your virtual bookshelf. It will allow you to quickly organize your files and rename them nicely with very little time and effort.

Password Management

Almost everybody has a plethora of usernames and passwords to keep track of for all the different sites, machines, and applications. A few years ago, I started using an algorithm instead of a memorized password or a slight variant when necessary. This worked nicely for most things. Another thing that I used for a period was chemical formula combined with some name variant of the chemical. Benzene ring compounds provided numerous combinations. I probably would have continued using this method, but I'm an electrical engineer and there is a reason I did not become a chemical engineer.

Today, I just tell my password manager to generate a random password and it takes a good part of a day for the new password to be stored in my overworked brain synapses. 

So what password manager can one use? I use one called 'pass' which is totally command line based on one of my Linux machines. It can be downloaded from https://www.passwordstore.org and took me about 10 minutes to get completely functional.

The https://www.passwordstore.org web site gives example usage. This seems to work for me, but it probably won't be for everybody. The one thing that is almost certain... Today, people have so many different usernames and passwords to maintain with different requirements that one definitely needs a safe place to store all their credentials.

Mail Server Access

There are probably not too many people that experience this issue, but if you have a mail server or even a web server on your LAN then you have probably encountered this issue.

You have a domain that allows you to connect from external locations via the WAN IP address. Now your device is on your LAN and you need to connect via a LAN IP instead of the resolved WAN IP. When you think about it, this becomes clear that the role of DNS needs to provide the answer. In my case, I simply added an entry for my domain to point to a LAN IP on my router. My router's IP is the first DNS entry for connected devices. Problem solved...

Yikes! My Apple ® Lightning Cable Failed

This past weekend, I was walking through the kitchen to make my morning coffee and I noticed a discoloration on the shroud of the USB cable we use to charge our Apple ® mobile devices.

I disconnected the cable from the wall outlet. Then I touched the cable approximately 1 inch from the short and it was quite hot to the touch. Approximately 30 seconds post disconnect, I touched the connector that a device would be connected to and it was still very hot to the touch.

Fortunately, no device was connected and nothing flammable was in proximity of the short. It is not known if there was enough heat produced to ignite a Lithium battery of a device, but I'm glad that a device was not connected for me to find out.

This is most likely the oldest and most used Apple lightning cable that we use. I suspect that the insulation surrounding Vcc and GND must have failed within the shroud, shorted producing the damaged cable. I would consider the usage for this cable to be quite normal. We do not bend it into acute angles or twist it. Simple angles and rotations to get connected to a device (iPod, iPad, or iPhone).

As an Electrical Engineer, I felt the responsibility to report this as an incident to UL since it is clearly a safety issue that could have resulted in a house fire. I'll be shipping the cable to UL in the morning for them to analyze.

If this is a failure mode that occurs after 2 to 3 years of usage, it would be very bad given the number of these cables in use with some of the first devices using these cable hitting this timeframe. If you notice a difference in the shape of the cable shroud, failure for devices to charge, or discoloration of the shroud, then stop using that cable and error on the side of safety.

iptables and denyhost

If you have a Linux server exposed to the Internet, then you may be aware of the constant authorization attempts from all over the world. If you are not aware, then you just have not looked at your log files to see all the failed attempts.

This is where denyhost comes into play. It will monitor these attempts and after a number of failed attempts add an iptables entry to block that IP. I typically allow 3 attempts before adding the IP to be blacklisted. This has worked nicely for me. I proactively block huge blocks of IP addresses with a simple script that gets all the IP addresses for certain countries and blocks them all. This has been the single most effective measure to eliminate a huge percentage of the auth guessing attempts.