‘Twas the night before Christmas and all through the house, not a creature was stirring, except perhaps a mouse…with a hammer…
One of our Project Managers, Paula, was telling a story the other day I thought you should hear. Take it away Paula:
Remember sitting in junior high, watching the film of the mice attempting to navigate through the maze to get to the chunk of cheese at the end? The narrator would drone on about the smart mouse, the one who successfully made it and was enjoying a well-earned snack. We recently helped out an employer client on a new Online Enrollment system with one of our more trusted vendors. Going in, everyone was confident this would be a successful project. We had a great team, a realistic timeline and many, many smart mice on the project.
As we approached the Go Live date, when the system would be available to thousands of employees, concern arose as to whether the online tool was robust enough to handle numerous web hits and transactions on the site. PlanSource has been around for some time and they handle extremely high volumes of employees; however, our client was concerned and asked for 200 test cases to be set up so they could stress test the system. Back in the days of Interactive Voice Response (IVR) systems, where employees called an 800 number and enrolled via the telephone key pad (yes I have been around that long), there was a program called the Hammer. This software would literally hammer the phone lines with calls to be certain the system could withstand the volume. We would all stand around one computer nervously waiting for the test to commence and we often launched the program by shouting, “Now lowering the Hammer!” We then ran to the servers and watched all the pretty lights blinking and knew we had a solid system. Stress testing for web applications is not as consistent these days, as there are often stories in the media about major websites being shut down due to unusually high traffic. (See HealthCare.gov’s recent fiasco.) It’s a valid concern for any business and certainly for any employer using Online Enrollment for the first time. (For all its limitations, a paper enrollment never had a 404 error.)
Back to present day: our client gets its 200 test cases and they “lower the hammer.” Unbeknownst to any of us at Lockton, they had a smart mouse– a very smart mouse. He wrote a program which used these 200 test cases and created additional logins which rose to about 50,000 ghost users “hammering” the site. Our main contact at PlanSource receives a call at home from the President of her company asking what is happening. They must have thought they were being attacked by a hacker! One of their smart house cats traced one of the test accounts and found an email with the client’s name, determined what was happening and terminated all the test cases– all 50,200 of them– in a matter of minutes.
As you can imagine, the following day we had a call to discuss what is allowed in testing and what is not allowed. The system, at that moment for the client, was configured to accept 200 logins. While the traffic did not come close to crippling PlanSource’s live clients, it did create some alarm. Needless to say that’s fantastic validation of a vendor’s durability and awareness. It was a somewhat entertaining call as far as implementation-type calls go and I was kind of proud of the client, to be honest. Our PlanSource contact described the issues it caused and the levels of upper management which ended up involved and pleaded there be no recurrence. The developer who created the program which created the ghost accounts joined late and was apologetic, yet not contrite. When asked why he felt the need to hit the system so hard his response was simple: “We want to be certain that what happened with healthcare.gov does not happen here.”
Enough said, we get it, call over!!
Isn’t that a great HR tech version of a holiday tale? Thanks to PlanSource for being such good sports with this well-intentioned example of risk management. We sing your praises here so other prospective clients can leave their hammers at home.