Annotations don’t scale. Managing an annotation data engine is costly, time-consuming, and problematic. We chatted about this in our earlier post here. The takeaway? Making datasets interpretable is essential to preventing critical model failures and avoiding harmful content generation.
So then why are annotations so critical to most AI companies?
When Something is better than Nothing
For any Harry Potter fans out there, building an AI system can feel like Potions class. Sometimes, the right mix of data, training algorithms, and scale creates an incredible model. But figuring out the exact recipe - what data to use, which model to try, what training algorithms to employ, and which hyperparameters to tune - is incredibly difficult.
For most companies deploying a new AI system, the most straightforward approach is to collect labels for the task at hand.
- Training a new robot? Collect and label trajectories (bounding boxes, segmentation, etc.).
- Aligning an LLM with healthcare or legal standards? Collect multiple model outputs and have humans rank them by preference to train a reward model for RLHF or to fine-tune an LLM directly with DPO.
- Finding people in videos? Collect bounding boxes.
- Classifying defective parts? Label images as defective or not.
This approach works well in the beginning because something is always better than nothing. From startups with no data to enterprises with exabytes of it, most teams start by gathering labels because getting a working prototype is the top priority. This manual approach continues until it stops working. It’s usually only when the gains from adding more labels diminish that companies explore other methods like pre-training, larger models, or dataset introspection. Data introspection is still necessary, but it typically comes after a baseline model is already functional.
Move fast, annotate faster
Companies wisely decide to ship a rough MVP and iterate. To that, we say your labeling workflow should move as fast as your AI development.
That’s why at Interpret AI, we’re building Agentic Annotations. It's simple:
- Provide a single prompt describing how you want your data labeled.
- Iterate on a few diverse examples selected by our foundation model.
- When you’re ready, pre-annotate the rest of your massive dataset automatically—no more human bottleneck.
Traditional annotation services like Scale AI, Surge, and Labelbox all share the same fundamental problem: a human must review and label every single data sample. This is the bottleneck that others try to solve by throwing more people at the problem. We believe that after a few examples are labeled, the rest should be automated. Companies trying to get from 0 to 1 just need good-enough labels to jumpstart the flywheel.
What’s the catch?
It goes without saying that high-quality manual annotations are generally better than automated pre-annotations. However, manual annotation can take months, while pre-annotations can be completed in days or even hours. ML teams often try to avoid costly manual labels by using judge-LLMs, foundation models like SAM, or other techniques to get "good-enough" results. Our belief, however, is that teams building new products benefit most from focusing on their model development, not on building internal annotation platforms.
Four steps to a working AI product
Companies that want their AI systems to solve real customer needs should ask four fundamental questions:
- Define the task: What should your AI model do? (e.g., determine policies for robots, detect cybersecurity threats, generate customer support text).
- Identify the data: What data is needed to train your model? (e.g., robot trajectories, images of defective merchandise, examples of good and bad customer support).
- Establish an evaluation process: How will you evaluate your model on benchmarks and with customers? While manual annotation makes sense for small, critical datasets, Agentic Annotations offer a valid alternative for achieving market-ready results much faster.
- Diagnose underperformance: What is the root cause when your model fails? Is it a data problem or a modeling problem? A data introspection platform can identify data gaps or anomalies.
While all these questions are essential, at Interpret, we help with (3) and (4). For evaluation (3), developers can use our Agentic Annotations to rapidly label data and test models. For diagnosing underperformance (4), when a model isn't working as expected, our data introspection platform makes your data interactive so you can understand what's causing the issue.
Prioritizing data understanding, model evaluation, and rapid labeling iterations will put your team on the fastest path to a working AI product.