Five Questions Every Manufacturer Should Ask Their AI Vendor (From Someone Who Sells AI)

Most articles telling manufacturers how to evaluate AI vendors are written by analysts who have never deployed a model on a production line. This one is written by a vendor. This means – full disclosure – that I have an interest in what manufacturers buy.

However, it also means I’ve spent the last 6 years watching manufacturers evaluate AI vendors, and I’ve noticed a pattern: the questions that come up in meetings with a manufacturing AI vendor are often not the questions that determine whether a deployment succeeds. Buyers ask about accuracy percentages, demo videos, and feature lists.

The deployments that never get off the ground or fail later do so because of things the buyer never thought to ask about — how the integration with a specific PLC works, who retrains the model when a new SKU is introduced, or who owns the trained model.

This article lists questions I want a manufacturer interested in AI deployment to ask me. Some of them will sound obvious, but they don’t get asked enough.

If I were sitting on the buyer’s side of the table, these are the five things I would make my AI vendor discuss in detail before I signed anything:

1. That the model works at my line speed, on my parts, in my plant
2. That someone other than the vendor can keep it running
3. That it integrates with my specific PLC, MES, and ERP
4. That it knows what to do when it sees something it has never seen before
5. That I own what I build with them, and I can walk away with it if I need to.

Sounds obvious, but each one is critical and harder to answer than it might sound.

So let’s dive in.

1. “Show me your model running at my line speed, on my parts, in my plant.”

This is the biggest gap between a demo and a deployment, and it’s easy to skip past in an evaluation because the demo can look so persuasive. The vendor brings a laptop and a few sample parts and the model performs beautifully, but some months later, the same model is missing defects and nobody can quite explain why.

The reason is almost always the same. Demos run on cherry-picked parts at whatever speed the vendor’s laptop is comfortable with and ideal environmental conditions, like there isn’t a speck of dust around. Production runs on varied parts you actually make, under challenging environmental conditions and at the speed your line actually moves.

The model that works at ten parts per minute on a folding table is not the same model that has to make a decision on a high-speed line in under fifty milliseconds.

What you actually need is proof. A proof-of-concept on your line is one way to get it, and for a genuinely novel or tricky problem, that’s the right way. But it isn’t the only way, and for problems the vendor has solved before, it may not be the best way.

A solution that has been running for years in another plant at your throughput, on similar parts, solving similar problems, can offer proof that things work in the real world.

A vendor can share performance data from similar deployments, they can put you in contact with a customer running a similar problem at similar scale or offer a performance guarantee tied to specific, measurable outcomes on your line. And when a POC genuinely is warranted e.g., in case of novel or unusual product geometries, a defect type the vendor hasn’t seen before, or a fundamentally new application — the POC can be structured in phases that naturally roll into production.

The deeper point: a statement like “Our accuracy is 99.x%” without context about the parts, the speed, and the conditions it was measured under, is a marketing claim not a performance guarantee. A vendor worth working with understands this and will offer to prove it, in whatever form makes sense for you.

2. “Tell me who maintains this model after deployment — and what happens when parts change.”

This is a critical question to ask. The pitch of AI vendors often focuses on getting the system live and tends to skip over what happens when something genuinely new shows up on the line.

Let’s define what “something new” actually means. A well-built and -trained AI system handles a lot of the variation that traditional vision systems can’t. Lighting drifts as bulbs age, a layer of dust on the camera, changing temperatures or a nudge to the camera that slightly shifts its position – these are the failure modes in rule-based vision that AI can deal with.

But AI isn’t magic, either; it needs more training to deal with a genuinely new “thing”, like a new SKU, a new defect class, or geometry. These changes happen regularly in manufacturing and the model needs to be able to deal with them.

In an ideal scenario your existing quality engineer or technician can retrain the model, using tools designed for them, in a reasonable amount of time. That answer has three important components:

1. Your existing team — not a new hire or expensive consultant
2. Tools designed for them — a GUI an engineer can use without learning Python or touching the model architecture.
3. A reasonable amount of time – maybe a day or two, not a week or month-long process, and, importantly, not a vendor engagement with its own SOW.

Any deployment that doesn’t meet all three criteria will become a dependency that can surface at any time.

A good vendor will have proof that they have actually done this, e.g. they show you an interface or put you in touch with a customer who is already operating.

The deeper point: a manufacturing AI system that requires the vendor’s involvement for every adjustment is a system that runs at the vendor’s speed, not yours. The whole reason to bring AI on your shop floor is to give your existing team a smarter tool. If the tool can only be operated by the vendor, it isn’t really yours, but a service contract with a model attached.

3. “Walk me through how this integrates with my PLC, MES and ERP.”

This is the question to ask, if you want to know if a vendor has actually deployed their solutions or just demoed them.

Think about it that way: an AI model that detects defects but can’t trigger a reject arm is a research project, a model that can’t log its decisions to your MES is a glorified spreadsheet, a model that can’t push quality data into your ERP is a data silo. It’s the integration of the model into your existing system that makes it useful.

Our own conversations with manufacturers show that integration is where the majority of pilots stall. The reason is: integration is hard, every plant uses a different combination of equipment types and models, vendor stacks, and all have institutional and undocumented knowledge built up over decades.

Look for a vendor who has done this before and answers your integration questions specifically. Ask them which PLCs they’ve integrated with, which protocols they support — OPC-UA, PROFINET, EtherNet/IP, Modbus TCP — and which they have actually used in production. Ask them which MES platforms they’ve pushed data into and what that integration looked like at the field level.

A good answer to these questions sounds like a conversation about your specific stack. If the vendor can name protocols, data flows, point out challenges they encountered before and their solution, then they have done this before. An answer like: “We integrate with all major systems” should make you double down and ask for specifics.

You can find a real example of this kind of integration here.

The deeper point: integration is where AI stops being a model and starts being a system. You need a vendor who hasn’t just developed the coolest model, but one who knows how to make it work on your shop floor.

4. “What happens when the model sees something it has never seen before?”

Every production line eventually shows the AI something it wasn’t trained on, like a new defect type, a bit of debris on the conveyor or a reflection nobody anticipated. The question is what the system will do then.

The answer to this question separates vendors who know production environments (and AI) from those who know AI only. How these edge cases are handled is critical in production and a good answer to this question has several pieces:

• There needs to be an escalation path in place, usually involving one or more experienced team members who review, if the model isn’t sure. This happens mostly in the beginning, when the system is still learning and later in rare cases.
• Critically, a feedback loop needs to be set up so the model can learn from these cases and get better at detecting rare events and new products.
• Explicit rules need to be established what happens when issues are detected. One of our customers established that three defects in a row are grounds to stop the line and investigate the root cause.
• Drift detection needs to be put in place so when the model starts seeing a higher rate of low-confidence outputs than usual operators are alerted and can take action before the line is producing scrap.

For one of our customers, a high-throughput manufacturer in the CPG space – after initial training the model – was capable of identifying completely new classes of defects, that human inspection of samples had missed so far. That’s the level of performance you’d ideally like to achieve.

When you talk to vendors be aware of statements like “Our accuracy is very high.” Or worse: “The model is so well-trained, we don’t really see that case.” Both answers are telling you that the model wasn’t built to deal with unknowns.

The deeper point: the model is not the product. The system around the model is the product. Demos hide that distinction, but production reveals it.

5. “Who owns the data — and what happens to my models if I leave?”

This is the question every buyer should ask during the first meeting, but rarely does.

Over the months and years a system runs on your line, you accumulate something genuinely valuable: thousands, sometimes many millions of labeled images of your specific parts, defects, and edge cases. The trained models built from that data are a treasure trove of insights and information about your particular production environment. That asset is probably the most valuable thing the deployment produces, next to the inspection itself.

The question who owns it has to be: “you”.

The answer matters when you want to change vendors, renegotiate terms, or move a line to a different facility. If your training data lives in the vendor’s cloud and the contract ends, what happens to it? If your trained models can only run inside the vendor’s runtime, can you take them with you?

A good vendor has a clear answer to these questions: you own your data, you own the models built from it, and the system is designed so you can walk away with both. That means the data should live on your hardware, not on a server you don’t control. The export formats shouldn’t be locked to the vendor’s runtime. There might be explicit terms in your contract about data ownership, model portability, and exit procedures.

Be aware of an answer like “We retain the trained models, but you keep your data.” You need both and the vendor has no business keeping models trained on your data.

Addressing what looks like a legal question might not always be obvious to an engineer, but this is a critical aspect: without the trained models, the data is just a folder of pictures.

The deeper point: an AI deployment isn’t software you license. It’s an asset you build, with the vendor as a partner. Who owns the asset at the end of the partnership is a question worth settling before the partnership begins.

Conclusion for evaluating a manufacturing AI vendor

None of these five questions is about AI.

They’re about engineering discipline, integration, maintenance, accountability, and ownership — the same things manufacturing has always cared about when buying plant equipment.

AI doesn’t change any of those concerns; it just adds a model in the middle. The vendors who treat AI as another piece of industrial equipment – with the same level of scrutiny, the same integration standards and expectations about long-term ownership – tend to produce the deployments that succeed. The answers to the five questions above reveal whether you are dealing with a vendor that understands this.

Here is a structured way to evaluate where you are in your AI journey. If you’re considering manufacturing AI vendors right now and want a sounding board, I am happy to connect and talk through what good looks like, even if we end up not being the right fit. That’s how I’d want a vendor to behave if I were sitting on your side of the table.

Click here for a deeper look at how we deployed AI in manufacturing.

FAQs

How is AI inspection different from traditional machine vision?

Traditional machine vision uses rule-based algorithms — fixed parameters that look for specific features at specific positions. It’s fast and reliable when conditions are stable, but it breaks when anything changes: a lighting shift, a new product variant, a slightly different surface finish. AI vision uses models that learn from examples and adapt to variation. AI-based models can handle the kinds of subtle environmental drift that would force you to constantly recalibrate a rule-based system. The trade-off is that AI requires good training data and a more thoughtful evaluation process — which is what most of this article is about.

Do I need a data scientist on staff to deploy manufacturing AI?

You shouldn’t. The first generation of manufacturing AI required deep ML expertise to deploy and maintain, which is one of the reasons the industry has a reputation for stalled pilots — most manufacturers don’t have data science teams, and the ones that do can’t dedicate them full-time to a single inspection line. Modern systems are designed to be operated by the same quality engineers and technicians who already manage your inspection equipment. If a vendor’s pitch requires you to hire or train ML specialists in-house, that’s worth questioning.

How much labeled data do I need to get started?

It depends on the application — the complexity of the defects you’re catching, the variation in your parts, the difficulty of the visual problem. For relatively standard defect detection on consistent parts, modern systems can produce useful results with a few hundred labeled images per defect class, especially when transfer learning from pre-trained models is involved. For genuinely novel applications or unusual geometries, the requirement can be substantially higher. A good vendor will give you a realistic estimate based on your specific problem rather than a generic answer.

How long does a typical deployment take?

Anywhere from a few weeks to a few months, depending on three things: the complexity of the inspection problem, the state of your existing infrastructure (cameras, lighting, network), and the integration work required to connect the system to your PLCs and MES. A vendor who promises a fixed timeline without asking about your specific situation either doesn’t know what they’re talking about or is going to be unpleasantly surprised after getting started. The right answer is a phased plan with realistic milestones for each phase.

What if my plant runs older equipment — does AI still work?

Yes, in most cases. Manufacturing AI is designed to layer on top of existing equipment rather than replace it. A reasonably modern industrial PC and standard cameras are usually enough to run inference at the edge, and integration with older PLCs — including those running PROFINET, EtherNet/IP, or Modbus TCP — is well-established. The bigger constraints are usually network infrastructure (do you have a path to get data out of the cell when you need to?) and lighting (consistent enough to give the model a clean view?). A good vendor will survey your plant before quoting and tell you specifically where you’ll need to invest beyond the AI itself.