Title: Phase 6 Test and Evaluate
Course: ATEP - MMT: Module A: > Section: PHASE 6 TEST AND EVALUATE

6.1.1 Determine the Appropriate Tests

Use existing testing procedures when possible or develop a new test, or several tests, to assess the performance of the design solution. Test the design solution, collect performance data, and analyze the data to show how well the design satisfies constraints and specifications. Physical testing has been the main type of testing in the past while current computer aided drawing programs now offer a virtual testing capability that saves both time and money.  When possible, both types of testing are used to verify the accuracy of each type.  For instance, a wind tunnel test for wing shapes will be used after a computer simulation is completed to verify the results.  A computer simulation is only as good as the program that was written to create the test.  An example of virtual testing for velocity and pressure with a Computer Aided Drawing program is shown on the right.  The change in colors show the velocity and the pressures created by the air flow over the fuselage and wings to help determine the minimal amount of drag and stress.

Determine the appropriate tests:

1. Destructive testing:  Tests that cause the product to fail in order to understand a specimen's structural performance or material behavior under different loads. Examples of several different types are shown here from Sheffield Hallam University's Materials Department.
2. Non-destructive testing: A wide group of analysis techniques used in science and industry to evaluate the properties of a material, component or system without causing damage. It does not permanently alter the item being inspected which makes it a highly valuable technique that saves both money and time in product evaluation, troubleshooting, and research.
3. Functional analysis:  Does it resolve the problem?  Pop-tops on cans solved the problem of forgetting to carry a can opener and the desire of not wanting to carry a tool to open cans.  Air foils are specifically designed for a variety of reasons such as weight, altitude and speed.
4. Failure analysis:   Why does a product fail?  Studying the reasons behind a failure will assist in creating better and safer products in the future.  By studying the failure of the Tacoma Narrows Bridge in Washington, engineers have improved the designs of all bridges built since 1942.  In the same way, engineers that studied the destruction of a Rolls Royce engine verified the safety devices added to their engines can save lives.
Tacoma Narrows Bridge Collapse
5. Value analysis: Review the functionality of a product, find the estimated cost for each function and show each function as compared to that of a competitor’s. What value is it to the end user?  Pass your cursor over the paper clip below to  show the added serrations which help hold papers together as an added value.
6. End user testing: Alpha testing takes place at developers' sites, and involves testing of the operational product by internal staff.  It is done to assess whether the product works as intended before it is released to external customers.
7. Beta testing takes place at customers' sites, and involves testing by a group of customers who use the system at their own locations and provide feedback.  It is done to assess the reliability and to identify any "bugs" in the product before it is released to other customers. The latter is often called “field testing."  An example is shown here:
Felix Baumgartner's supersonic freefall from 128k' - Mission Highlights

6.1.2 Existing Test Procedures

Over the years consumers have given suggestions and required better products from manufacturers. Engineers have also examined how a product should react under specific conditions to prevent accidents and reduce liability.  As a result, standardized test procedures have been developed to determine if a product will meet both safety standards and longevity standards.  Governments have created and catalogued many of these tests that pertain to their own laws.  Two other organizations that have catalogued these tests are:

The American Society for Testing and Materials (ASTM) (http://www.astm.org)  is an international standards organization that develops and publishes voluntary consensus technical standards for a wide range of materials, products, systems, and services.  They publish the Annual Book of ASTM Standards each year in print, Compact Disc (CD) and have an online version. The online version was available by subscription and cost was based upon usage. For 2010, the complete set of books or CDs cost almost $9700 and included 82 volumes.

Underwriters Laboratories (UL) (http://www.ul.com) is a safety consulting and certification company.  It has participated in the safety analysis of many of the last century's new technologies, most notably electrical products and creating safety standards for electrical devices and components.  Underwriters Laboratories provides safety-related certification, validation, testing, inspection, auditing, advising and training services to a wide range of clients, including manufacturers, retailers, policymakers, regulators, service companies, and consumers.

6.1.3 Develop New Tests and Procedures

Advances in technology, materials and engineering have created the need to develop new methods of testing products to meet a variety of new standards. These tend to be aimed at the safety of the consumer and the liability of the manufacturer.  If an existing test can not sufficiently prove that the reliability or safety of a product is consistent with current expectations, then a new test must be created.  One example is shown here:

A380 Engine Blade Off Test

6.1.4 Recording Test Results

Observing (monitoring) and recording the results of the tests in the engineering notebook will suggest how you can improve the design or the construction of the solution. Repeating each test many times will give a more reliable result.

Include the rationale for the testing methods chosen by explaining what is being tested and what each test is expected to determine. Do not assume that one type of test will be sufficient to determine the viability of your product. Several different types of testing might be required.

- Explain the conditions that exist for each test.
- State the set up requirements  to include the parts, tools and miscellaneous mounting brackets.
- Pre-determine the pass or fail requirements.  How will you know if the test was successful?
- List the procedural methods used to run the test.
- State the safety procedures taken to prevent injuries or damage.

Testing should also be done scientifically by changing only one factor at a time and verifying the results. For example, the factor could be the material, the weight, the finish or the shape. The two charts below show two variables each and how changing one affects the other.

Again, create a matrix that allows you to easily visualize how the changes affect the product. Make note of the factors (variables) that affect the performance of your design in the notebook.

All the information from your testing is then gathered and analyzed to see if the design solves the problem. Converting the information to graphs and charts is a valuable method to visually show the results of the testing and should be included in both the engineering notebook and the final reports.

Continue to next lesson