Why is Testing Important?
The History of Bugs
Thoroughly testing software is one of the most critical aspects of the software development process. One minor or rarely seen glitch can have extreme and lasting consequences. Historically, since computers and software came into existence there have been bugs plaguing these systems. In fact most companies now not only rely on their engineers to thoroughly test their own code but actually have entire teams solely dedicated to testing for and flushing out these pesky bugs. Bugs are inevitable and impossible to fully eradicate from your programs. But through the careful implementation of some known methods they can be reduced in quantity and severity. To demonstrate the importance of testing, below I will talk about some of the most famous and consequential bugs from recent history. Then I will list a few methods that individuals or teams can use to help fight back.
Boeing 737 MAX
In late 2018 through early 2019 Boeing lost two commercial passenger jets due to a software problem. The issue stemmed from Boeing, in their efforts to stay competitive in the market using cheaper software to fix an airframe design issue that was actually a hardware problem. For many years the 737 model planes sat relatively close to the ground, but as time went on and the planes were adapted to use bigger and better engines. Rather than using taller landing gear or accommodating the engines better, they were just mounting the engines much more forward of the wing. This allowed the engines to fit but at the cost of throwing off the precise balance of the airframe. When thrust was applied by pilots the engines in result would pitch the aircraft up raising its nose. To counter this effect, they implemented a software package called MCAS or Maneuvering Characteristics Augmentation System. MCAS was responsible for countering this pitching effect automatically. Part of the problem was that engineers failed to use multiple sensors for redundancy in case of a part damage or failure during flight. Second the MCAS software was sloppy and did not include logical checks to verify input validity. Basically, a situation of GIGO or garbage in garbage out, but on a tragically big scale. One plane crashed due to this angle of attack sensor failing and causing it to output a reading change of 12 degrees to 70 degrees in under a second. This condition was impossible and the software could have easily included features to recognize and ignore this data or go into a failsafe mode. Instead, the MCAS software made wildly inaccurate pitch corrections to the plane ultimately leading to its demise.
Consequences
- The loss of 346 lives over 2 separate crashes.
- The 737 MAX line of planes were grounded for extensive investigation and refitting of proper systems. This cost the company and its customers loads of money and delays.
- Consumer trust took a serious hit here as well, with lack of trust in the engineering and or leadership coupled with the loss of sales revenue during investigations Boeing stock took a sizable loss during this time.
- Loads of costly lawsuit payouts and legal fees.
- Big shakeups in upper management.
Source
Mars Climate Orbiter
In 1998 a robotic space probe was launched by NASA to study the atmospheric and climate characteristics of Mars. However, after nearly a year of travel the orbiter was lost as its trajectory placed it too close to the planet. This trajectory miscalculation either caused the orbiter to burn up in the atmosphere or escape the planets orbit, either of which being a total loss of the craft. While this bug was not technically a breakdown of software necessarily, I feel it definitely highlights the importance of testing. The cause was a mismatch of units of measurements between two different software systems. Systems are often outsourced on large projects or at least developed by other teams or individuals and having a clear and detailed outline could have helped prevent this mistake. One system built and supplied by Lockheed Martin produced results in a United States customary unit instead of Lockheed Martin’s own SIS or Software Interface System. Another system supplied in house by NASA expected the Lockheed system to output in SI units in accordance with the SIS. The software made calculations about impulse forces produced by thruster firings. Lockheed Martin’s software produced these calculations in pound-force seconds where the NASA systems expected readings in newton-seconds. This led to incorrect thruster usage by a factor of 4.45 and ultimately placed the craft into the wrong trajectory. The software packages themselves worked correctly with the data they were given but better planning, double checking, testing and results modelling could have saved this mission from going sideways quite literally with faulty data.
Consequences
- The cost of this mission was $327.6 million. The orbiter was a total loss.
- Significant time was lost. Years of planning and development was rendered useless including nearly a year during the travel phase of the orbiter.
- Research loss. Scientists were without doubt eager to dive into the new data planned to be coming back from this orbiter. Possibly giving humanity insights into many different facets of our solar system and universe. This was a major setback for researchers, likely in many different fields.
Source
Testing Methods
Black White and Gray
Black, White, and Gray-Box testing refer to how much you as a tester know about the inner workings of what you’re testing. Black-Box tests are done by someone who may know what the program is supposed to do but not necessarily how. They start testing by entering various inputs to see what come out the other side. In White-Box testing, you are doing the opposite. You test from the perspective of someone knowing exactly how the method operates. This gives you the unique ability to think about and create specific targeted tests to see if you can cause a failure. Gray-Box testing is a mix of both, where the tester has some but limited knowledge of how the program or method works.
Integration Testing
Integration tests are done when a program or method is implemented for the first time. These tests determine how well the new software integrate with existing systems.
Exhaustive System Testing
This is a full system test from beginning to end and tests every possible path of interaction throughout a program. As you could imagine, this is a very time consuming and thorough process but does yield great results for catching bugs.
Acceptance Testing
Not only do programs need to be tested for bugs and functionality but also for whether or not they accomplish what they were meant to do. Acceptance tests determine whether the finished application was able to successfully meet the customers requirements.
Accessibility Testing
Whether a large program or a website, this form of testing is important for making sure everything is accessible to those with disabilities or impairments.
Performance Testing
This form of testing checks how a system works under varying usage demands. A system may work fine until faced with high user loads or limited resources such as memory. These tests will check that the infrastructure put in place can handle whatever may be expected by the system.
Security Testing
Security is very import in todays world. Systems need to be checked and updated often to maintain their integrity against malicious attacks. These tests verify the security of login processes, communication and data within a system.
Best Practices
There are many more forms of testing that could be used to verify the functionality of new or existing software. Its also very important to take a proactive approach and maintain good habits as a developer. These habits include coding and testing your own code as you write it, and not only that but do so when you are most alert and likely to find problems. Have other people test your code. You may gloss over or ignore something in your own code where as an outside opinion may reveal a vulnerability or a more efficient way to accomplish the task. You should also keep in mind the consequences of changes in an existing system. A small change to correct a small bug may accidentally introduce a much more consequential issue. There are many more testing methods and best practices to keep in mind when developing new software. These are just a few that stuck with me, and the stories above go to show just how critically important it is to test your work.
Follow me to keep in touch and see what I’ve been working on.
Patrick Jeffers is a Charlotte, NC native and current student working on his second associates degree in Information Technology – Full Stack Programming. As a student of Central Piedmont Community College and Wake Tech, he aims to continue growing his skills for web and application development. As an Automotive Technician for nearly 15 years, he plans to use this new education as he transition careers into the tech industry.