Satellites and the Importance of Testing GROUP 6 Background Satellite's are a very important part to our everyday life. ,000 miles above us No Maintenance No margin of error Need a set of quality testing guidelines. Quality aspects Different Test types and their procedures Test Design and Environment Case testing Integration Testing Component Testing Failures and the Consequences Clean Room Purpose: to provide a clear understanding of already verified and tested components by adapting the environment to limit defects during integration. Basic Strategy: clean an entire area (room/code) that will become the clean environment that you will use. To integrate any new components within the clean room, it must pass rigorous tests to ensure its integrity and verify it's functionality By using a clean environment during development, the team can determine where/when defects are injected; this can guaranty that clean components are not only validated but also verified through the clean room tests that have been precompiled Predicate Calculus's use for guaranteeing correctness Predicate calculus can be used to verify that functions/methods correctly map all possible set defined inputs to their true respective output These tests are more rigorous than simple random testing or boundary testing, but has an increased overhead to complete the process of verification An example: z(P(z) Q(x)) y Q(y) By forcing all clean code to meet specific testing standards, we can effectively guaranty that the entire system should work flawlessly under it's normal operational conditions; this is the most reasonable level of completeness that is obtainable. You can only protect against threats that you can predict. If applying predicate calculus, you can try to identify and prove the purely logical aspects within the code to begin as axioms, "a statement or proposition that is regarded as being established, accepted, or self-evidently true". Results of clean room use By using a test driven development practice, the early establishment of clean code tests can be used to validate that the functions in components adhere closely with the requirements being specified in user's story cards The precompiled tests will also verify with absolute completeness regardless of equivalence classes and boundary conditions; this ensures that the code will behave as anticipated under normal operational conditions and environments Clean room testing does not validate that they system will be able to handle unexpected or external influences including environmental drawbacks The Planning Game Now take out a piece of paper and write down what you think there is a picture of on the next page. By the way you better get it right... Now pay a dollar to each Instagram user because you didn't guess this picture correctly Cost of Failure One satellite can cost from million. If the satellite doesn't work then the company doesn't get paid. Cost of Failure - includes costs of failures that occurred because of failure to prevent and detect software errors. Internal Failure costs- costs of correcting errors that have been detected by design reviews, software tests, and acceptance tests and completed before the software is installed at customer sites. External failure costs - include all costs of correcting failures detected by customers of the maintenance team after the software system has been installed Internal Failure Costs These should have all been found by the time the product reaches the customer. costs of redesign or design corrections subsequent to design review and test finding costs of reprogramming or correcting programs in response to test findings. cost of repeated design review and re-testing. Occur from poor design and code quality If you can't fix these issues the product is doomed. The software is also been developed along side the hardware components so this is devastating to the project. External Failure Costs These are the ones you need to plan extensively for. Normally found by the customer after delivery, they are hidden. Cannot have these in space programs When the product is delivered it is in space and there is no fixing it. Customer needs to be reimbursed if these are found ( millions of dollars ) Effects of a Failure Loss of live Millions of dollars in damage or loss of payment Damage to company's reputation Loss of investment Job loss Severe reduction in sales Component Testing What is a component? Anything you want it to be Single class Cluster of classes that are tightly coupled Definition: conceptually related unit Is it worth the effort to test detailed components? Select a component for testing when the penalty for the component not working is greater than the effort required to test it. - John D. McGregor Not all components are large, important, or complex enough to test independently Types of Components Test these ones no matter the size or complexity Reusable components Tested over wider range of values Domain components Tested for correctness and faithfulness of the representation Commercial components Individually sold products Tested as reusable components Tested as potential sources of liability Number of Tests Based on risk analysis If component is part of a use case that is high risk then the component is classified as high risk also Ex: The satellite moves according to the authorized command High Risk Security authorization unit is part of this use case so it too is high risk Satellites 8000 test steps on one unit (component) software for radio transmitter End-to-End testing takes about 2-3 months Why So Many Tests? You can't mess up - mission critical Goes back to the cost of failure Won't get paid Investors won't be satisfied Rather disappointed Won't come back There's no fixing it after it is in space Testing Satellites About Building a satellite requires a lot of time. Satellites require a very technical process which requires testing in clean environments. Clean Room Clean room conditions needed for testing. Low levels of environmental pollutants like dust, microbes, particles, and vapors. Tests are performed in house or by the prime contractor building the satellite. National Technical Systems (NTS) provides testing, certifications, and supply chain management solutions to ensure the satellite is built correctly, safe, and reliable. The Tests Tests for atmosphere and dusty environments to mimic the conditions the satellite will endure. Intense vibration levels and shock environments testing. Temperature tests are also done to ensure the electronics wont freeze or overheat. After Testing Physically inspected and functional testing to ensure it is ready to launch. Similar to SE testing. Once these components have undergone proper inspection, they are finally added to the satellite. All tests are completed in a cost effective way. Similarities Between Satellites and Software Testing Every component is tested and then integrated. Similar to bottom up testing. Problems easily identified using integration testing. Robustness testing is also relevant because it focuses on deliberately breaking the system. Combining Large System Modules Multiple Modules all attached as one Service Module Communications Module Solar Arrays Antennas All aspects have separate system tests Now need full system test Kind of like unit testing Integration For satellites: 6 hours (Big Bang) Modules all on different levels Working on Airplane Electric Systems Example Multiple modules: Takes input as binary file, takes input as C file, takes input as assembly file Top Down and Bottom Up approach (created files) Catastrophy DO-178B-Level A failures Level A is a catastrophic failure Catastrophic failure means the plane is going down (recovery is impossible) Or does it? 1.0 * 10 ^ -9 per hour is 1.0 per year for 114,155 continuous flying planes. Even further safety Backups Multiple systems running How much testing? 1.0 * 10 ^ -9 per hour means you need to know that there will be less than 1 failure every 360 billion seconds. Requires almost complete code coverage MC/DC Coverage MC/DC coverage gives near complete coverage as far as data flow is concerned independently influence the decision What about more than one module? Low Coupling, High Cohesion Coupling: is the manner and degree of interdependence between software modules Systems usually share little with each other But to reach Level A, more than just Simple Integration testing needs to be performed MC/DC should be used on the whole system as well MC/DC with multiple modules int x, y, z; If( x > y ) z = x + 5; Else z = y + 3; int z; z = _z; If (z == 6) output(Good); Else output(bad); Any Questions?