Dealing With an Emergency

Dealing With an Emergency

This virus situation is very scary.  The unknown is always scary.  Routine practices are just not always an option.  Currently it seems our leaders, federal and local are making decisions that are not necessarily tested, or backed up by statistics or science.  Heck, statistics are a moving target, science takes time to establish definitive effects, and we may not have time for thorough testing.

The situation reminds me of an emergency encountered by Merrill Lynch, the Wall Street firm I was employed at many years ago.

 

Our department processed security transactions on the New York Stock Exchange.  At a time when we were only a few years into sophisticated computer processing and away from a vastly manual operation, we had a sudden, totally unexpected emergency.

An unusual surge of activity in the pharmaceutical firm, Pfizer (Symbol PFE,) created an emergency in our processing system. 

In short, the computer sorted transactions by stock symbols alphabetically.  A bit of a no-brainer for early computer minds.  People just thought alphabetically.  The problem was, an unexpected surge in one symbol did not allow a random distribution of data over multiple computer lines.
 

Activity in PFE became a huge traffic jam.

Imagine cars traveled on the highway in lanes reserved for specific models.  If everyone suddenly started driving Fords, that lane would grind to a halt, and eventually no cars could get off the exit ramp.

PFE did that to our system.  The delay was threatening to hold up other computer processing beyond 10 PM that evening.  If that happened other processes would be delayed, the firm might not be ready for the next day’s business and our customers would not be able to access their securities or money.  It would not have been good!

 

For those behind the scene it was an emergency, for the company it was a potential catastrophe.

Ordinarily, reprogramming the computer system to better handle this traffic issue would have taken several days, several more days to test, and one big weekend test to confirm it would work.

In this case, one of the programmers had been working on an idea to process business by local office designation.   This would insure a completely random data distribution.

 

Think cars being routed to lanes by the first letters on their license plate.

Around 7 PM a program was produced to change the processing.  At 7:30, a courageous decision was made to change the program, basically untested, and rerun the whole days business through this new program.  At 8 PM  we began to reprocess the days traffic. (a day’s processing was generally completed by 4:15, 15 minutes after trading was done)

 

The program worked.  Traffic processed smoothly with no backups and we were done processing by 9:30.  No other computer processes were delayed.

The CEO of Merrill Lynch never knew there was an issue, customers never knew there was an issue, it was as if there was never a problem.

That night several of us had a few stiff drinks before calling it a day.  We should have carried the programmer who spoke up with an idea and the manager who said “Do it” home on our shoulders.

The next day all was well in the world.

I hope we have creative people today working behind the scene to mitigate the pain we are experiencing with this Coronavirus.