Key Takeaways
Discover core strategies used to validate and optimize deep learning systems in manufacturing:
- Statistical testing: Collect thousands of data points to validate defect detection and calculate ROI, ensuring robust system performance.
- Two-tier inspections: Combine deep learning with human review to reduce false positives, improve accuracy, and enhance training datasets.
- Continuous improvement: Regularly update training sets with new data to adapt to variations, ensuring long-term system reliability and efficiency.
In the final phase of getting a deep learning project started in a manufacturing setting — once the system has performed well in testing environments — the team must carry out factory acceptance testing (FAT) procedures.
In the FAT phase, the vision system and a human inspector continue to classify parts as either good or bad, and an expert reviews any ambiguities and applies the correct labels. All statistics are then tracked to help determine human versus vision system performance.
Statistical and repeatability testing
Once a deep learning system has made its way to the factory floor for acceptance testing, the team should perform statistical testing, which involves collecting thousands of data points over a long period of time to capture the full range of defect types and frequency. This approach can be combined with manual inspection validation to help calculate an accurate project ROI. In practice, this means letting the deep learning system run in parallel to the manual inspection process and, after a month, collecting the images and comparing results. Any ambiguities or discrepancies can then go to the assigned expert, who can make the correct determination to produce accurate statistical data.
This testing method allows the team to perform end-to-end validation of multistep inspections, composite testing of part and defect types, cost measurements, and accurate ROI calculations. The only downside is that statistical testing requires a large and representative dataset, which isn’t always immediately available. On top of that, changes to image formation, product appearance, or ground truth labels can invalidate weeks’ or months’ worth of data, which is very important to keep in mind.
Ultimately, repeatability testing is not representative of true defect distribution or true defect appearance, which prevents companies from obtaining accurate estimates on overkill, underkill, and ROI. Keep this in mind if you’re forced to deploy repeatability testing methods.
Adding an inspection layer
Another common method deployed during the FAT phase is two-tier inspection. Here, a deep learning system performs the first inspection, and any uncertainties are sent to a human inspector for a secondary check. This helps lower false positives and false negatives. The results can also improve the deep learning training process.
A two-tiered approach lets companies reduce underkill and overkill whenever parts that fall into the ambiguous category can be reworked. In addition, this method helps reduce scrap by limiting the production of additional bad parts while improving overall system confidence, which saves on labor costs and enables continuous improvement by identifying challenging part images to add to the training set.
Continuous improvement
Even after obtaining positive FAT results, the deep learning team must continuously improve the system by gathering data and adding it to the training set. This data includes images of good parts, bad parts, borderline parts, and any new part variations, as well as discrepancies between the vision system and the human inspector. A team member can manually select these images and add them to the training set to validate the new model. A team sets itself up for success by doing this, since the system can more easily adapt to variations over time, such as those caused by lighting changes, parts handling adjustments, and new components. And whenever new images containing rare defect cases become available, the team should add these images to the system to further finetune it.
When a manufacturing company carries out all project phases — including initial planning, data collection and ground truthing, optimization, and FAT — deep learning systems can deliver tremendous value. It doesn’t end there, however. The team must continue to gather data and improve the model on an ongoing basis. The end results will get better and better over time.
If you're just starting your deep learning journey, be sure to check out the earlier parts of this series. Begin with Part 1: Planning, move on to Part 2: Collecting Data and Establishing Ground Truth, and then explore Part 3: Optimization to refine your approach.