How to Make Embedded Computer Vision Work in the Real World
Computer vision models can deliver impressive results in controlled environments. Turning them into reliable embedded products is mostly a matter of engineering the camera and the hardware around the model.
Yalantis is a compliance-first, full-cycle tech partner with deep expertise in IoT and computer vision, covering software, embedded, hardware and AI development. This article draws on four Yalantis projects to show how that work is done in practice - from optics and image correction to power design and field validation.
Why vision models that work in the lab struggle on real devices
A computer vision model that performs well in a lab often struggles on a real device because the lab hides the problems a physical system creates. In the lab, the images are consistent and the lighting is controlled. There's usually enough processing power to run inference without worrying about the rest of the system.
Then the model moves into the real world.
On a factory floor, reflections from metal surfaces can look like defects, and parts on a conveyor leave only a short window for inspection. These problems are easy to underestimate during model development.
At Yalantis, we've run into them across very different embedded computer vision projects: inspecting automotive electronics, detecting fractures in brake calipers, monitoring stock inside commercial refrigerators and analyzing microplastic particles in flowing water.
The constraints differed from project to project. Some systems had to keep pace with a fast production line. Others had to run for years on batteries or produce measurements that were hard to validate at all.
All four projects led to the same realization: a reliable vision system depends on engineering everything around the model, from the first image the camera captures to the decision the device delivers.
That work often starts at the camera.
Why fixing the image comes before changing the model
Many vision errors start before the image ever reaches the model, so the camera and image processing are the first places to look. When a vision system starts making mistakes, it's tempting to blame the model and reach for more training data or a bigger network.
We saw this on an automotive electronics line. The manufacturer used automated optical inspection (AOI) to check printed circuit boards for engine and airbag controllers, and the AOI was flagging too many good boards.
Most of the trouble came from lead-free solder joints. Their uneven, reflective surfaces created patterns the AOI struggled to tell apart from real defects, so operators spent hours each shift re-checking boards that were often fine.
Replacing the AOI equipment would have been expensive, so we developed a secondary inspection system to examine the boards the original machine flagged. Before selecting a model, our engineers spent two weeks on the factory floor analyzing flagged images. Most false alarms turned out to come from changing lighting and solder reflections, so the images had to be normalized before inference.
A similar problem appeared in another automotive project, this time with cast aluminum brake calipers. The manufacturer needed to detect micro-fractures in polished metal parts, and a generic vision system had already failed because reflections and shifting ambient light hid the cracks.
The team started with how the images were captured. Multi-angle dome lighting with polarizing filters suppressed specular reflections, and high-resolution monochrome sensors with telecentric lenses produced consistent images without perspective distortion.
With the optics in place, a custom U-Net segmentation model running on NVIDIA Jetson Orin could outline potential cracks. Real examples of rare defects were scarce, so the training set was extended with synthetic images.
The two projects reached the same goal with different techniques. The PCB system fixed image consistency in preprocessing, while the caliper station fixed it with lighting and optics. Both approaches took load off the neural network.
This is why image acquisition deserves as much engineering attention as inference. A model cannot recover details the camera never captured, and it cannot reliably separate a crack from the glare that hides it.
Practical tip: before changing the model, study a sample of real failure images. If the errors come from glare or uneven illumination, they are usually cheaper to fix at the camera or in image processing.
A good image is only the first step. It still has to be processed and turned into a decision that reaches the right destination, and on a production line all of that has to happen fast.
How to keep embedded vision fast enough for a production line
To keep up with a production line, every stage from image capture to the result reaching the controller has to fit inside the line's timing budget. The brake caliper project shows how quickly computer vision becomes a timing problem.
The parts moved along the line at 1.2 meters per second. Once a caliper reached the inspection point, the system had a short window to find a fracture and send the command that removes the part. A correct prediction that arrives after the part has passed the reject mechanism has no practical value.
So the system was designed around the entire path from image capture to physical action. The camera captures the part, and the Jetson Orin runs the segmentation model. The application then reports the result to the line's PLCs over OPC UA, and an air-powered arm diverts any part with a crack.
The project reported a 15% increase in line speed with the new inspection approach, which shows why vision performance has to be evaluated as part of the production process.
The PCB project hit a timing limit too, at a different point in the chain. Once the team knew the images had to be normalized, software-based correction proved too slow for the line, so that work moved into dedicated hardware.
The system runs on an NXP i.MX 8M Plus processor. Its integrated image signal processors handle the corrections, and its neural processing unit runs the defect-detection model. That model is based on MobileNet SSD and trained on the line's defect classes: solder bridges, tombstoning, insufficient wetting, and cold joints. NXP's eIQ toolkit optimized it for the target hardware.
With dedicated resources for image preparation and inference, the system keeps up with the conveyor. Each verdict goes to the PLC as a GPIO pass-or-fail signal, so only confirmed defects are routed to manual review.
The project reported a reduction of more than 35% in false rejects without any changes to the existing AOI. No real defects had been reported as reaching customers since go-live.
Both production-line projects show that fast inference is only part of the timing budget. The rest of the system adds its own delays: camera interfaces, memory transfers, frame queues and the link to external equipment.
Those delays also drift as the device heats up and frames pile up in buffers.
For industrial computer vision, the measurement that matters is the complete response time, from the moment the camera triggers to the moment the result reaches the controller. Where a physical action follows, the actuator's response has to fit inside the same window.
This changes how hardware should be selected. Next to advertised AI performance, engineers need to weigh camera interfaces, ISP capabilities, memory bandwidth and thermal behavior, along with the link to the surrounding equipment.
Some embedded vision systems face the opposite challenge: they have to use as little energy as possible while still doing their job.
How to design a vision device to run for years on batteries
Running for years on batteries means spending energy only on the images that matter and handing part of the workload to a low-power processor. Consider a small camera installed inside a commercial refrigerator. It photographs the shelves when the door opens and sends useful images to a cloud platform, where grocery and convenience stores monitor stock remotely (Figure 1).
Here the hard constraint is power. The device has to run for at least three years on a single set of non-rechargeable batteries, and that requirement changes the whole architecture.
Yalantis developed the camera hardware and firmware to work with the customer's existing cloud platform.
The first prototype captured and uploaded an image every time the door opened, and many of those images were useless. Some were blurred by movement, and others were blocked by a shopper's hand. Uploading all of them wasted energy and filled cloud storage with images nobody needed.
The team changed the capture workflow. Each door opening now triggers two captures one second apart, and an on-device model selects the better frame. Only that frame is uploaded.
The model's only job is to pick the frame worth sending. That makes it part of the power design.
Fewer uploads solved only part of the power problem. Cold and humidity reduce battery capacity, and measurements from the first manufactured units showed higher consumption than the main processor's datasheet suggested.
The team reworked the hardware architecture. A second, low-power processor became the main controller and took over part of the workload. The enclosure also went through several iterations to fit long-life batteries while keeping the camera small enough to stay unobtrusive.
Preliminary measurements indicated that the revised architecture could meet the three-year target, although testing was still ongoing.
Image quality needed work as well. The model had been trained on images from production refrigerators, and in a test refrigerator with different lighting it occasionally missed obstructions. That made validation in the target stores essential. At the reported stage, one unit was running in a street café, and another 20 prototypes were being prepared for field tests before mass production.
The comparison with the automotive projects is revealing. On the production line, the work centered on processing images fast enough to act before parts passed the reject point. In the refrigerator, it centered on deciding when to capture and what was worth sending, all while preserving battery life.
That is one of the most important lessons in embedded development: the best architecture is the one that fits the application's hardest constraint.
Sometimes the hardest constraint is proving that the system's measurements can be trusted at all.
How to validate an embedded vision system in the field
Field validation means testing the system under the conditions where it will actually run and finding a way to check results when no reference method exists. The microplastics monitoring project shows how hard that can be. The goal was to build devices that count and size microplastic particles for utilities and industrial facilities.
Two devices were involved: an in-line module that imaged water as it flowed past a viewing window in a pipe, and a benchtop analyzer for detailed sample testing. Both sent their results to a web dashboard designed to give early warnings when particle levels rose.
Yalantis developed the electronics and software for both devices, working around enclosures designed by the manufacturer's engineers.
Here the camera looks at a medium that keeps changing. Water turbidity and color affected image quality, and the host pipeline set the flow rate. Illumination was the main imaging parameter the device could control, so the optical setup and the processing pipeline had to be designed together.
The system combined classical image processing with a machine learning model trained on a large set of manually labeled particle images. A fixed live-demonstration deadline left no time for a separate prototype phase, so the imaging chain was specified as one integrated system from the start.
The hardest problem was validation. The project had no reference method for directly counting microplastics in flowing water, and laboratory techniques work with filtered samples that are hard to compare with continuous in-line measurements.
The team therefore evaluated results with theoretical calculations and expert reviews of the captured images. Detection and rejection thresholds stayed configurable while formal acceptance criteria were still being defined.
Seven months after development began, both devices ran in a live demonstration at a major industry trade show. Long-term measurement accuracy in production conditions still had to be proven.
Reliability means something different for each kind of device. For an inspection system, it covers false rejects and missed defects under sustained production load. For a measurement instrument, it depends on validating results against meaningful reference criteria.
Testing has to reflect how the device will actually be used. A model that performs well on a test set may still fail when the lighting changes, and a processor that meets its timing budget in a short benchmark may throttle after a few hours.
Engineers also need to plan for failure. What should an inspection system do when a verdict arrives too late, or when an image is too poor for a reliable decision? The answers decide whether the surrounding equipment can depend on the system.
What it takes to turn a vision demo into an embedded product
Turning a vision demo into an embedded product takes engineering the whole system around its hardest constraint. In these projects, the most important decisions happened outside the neural network: in the optics, the image correction, the power design and the validation method.
Together, these projects point to one way of approaching embedded computer vision (Figure 2). Start by identifying the constraint that matters most for what the device has to do in the real world. Then design the whole chain around it, from the camera to the system interfaces.
On a fast production line, that means minimizing end-to-end latency so every decision reaches the controller on time. On a battery-powered camera, it means cutting every capture and upload the product doesn't need.
Building these products also takes several disciplines working together. The PCB inspection project alone involved a solution architect, an ML engineer, embedded and backend developers, a QA engineer, and a project manager. The other projects added optics and power work to the usual stages of embedded product development.
That combination of disciplines turns a promising computer vision demo into a working embedded product. A deployable system has to capture the right information and deliver a result another system can rely on, within real hardware limits.
The real engineering challenge is making computer vision work under the conditions where the decision actually matters.
About the author
Mykhailo Maidan is CTO at Yalantis, an engineering company that develops embedded and edge AI products. The four projects discussed in this article are based on work carried out by Yalantis engineering teams.
The post How to Make Embedded Computer Vision Work in the Real World appeared first on Embedded.