The AI Skill Factory

Turning local experiments into org-wide capabilities

With the mass adoption of agentic coding tools, it’s still a little crazy to me that almost anyone can build almost anything.

Some gems will emerge from this process, no doubt. So will a lot of slop and waste.

But amidst all the tokenmaxxing, the issue I keep coming back to is: how do companies get the most value from AI as a systematic, org-wide capability, not just individual experimentation?

More specifically, I think leaders and AI operators should be asking these questions:

1. How do you identify the unique-to-your-org applications of AI that provide the most leverage?

2. How do you scale those solutions from siloed innovations to reliable organizational infrastructure, distributed to everyone who faces similar problems?

It’s great if your best salesperson develops a workflow that saves them 10 hours a week or increases their win rate by 20%. But the value is limited unless you can rapidly scale that innovation across your entire field.

So this is the challenge: identify the use cases with proven value, industrialize them, and speed their adoption across the team.

Think of this like an innovation pipeline or skill factory designed to continually upgrade your people with the most impactful capabilities for their role.

What are agent skills?

The skill is emerging as the atomic unit of portable AI capability. They’re like Standard Operating Procedures (SOPs) for agents.

Just like human employees following an SOP, an agent executing a command on my computer leveraging a skill should follow a similar process and achieve similar results as an agent executing the same command on yours.

A packaged skill can contain some combination of:

• Instructions

• Organizational or personal context

• Instructions for using available tools and knowledge sources, often through MCP

• Templates and examples

• Rules and guardrails

• A defined job to be done

The possibilities for what a skill can do are endless. Examples might include:

• Marketing: visual brand guidelines (design), voice guidelines (copywriting), metric definitions (reporting).

• Sales: call feedback (post-call coaching), email generation (post-call follow-up), competitive strategy, pipeline review.

• CS: business review generator, usage analysis, portfolio review.

• Support: research past tickets, debug user account, log product feedback.

• Product: synthesize bug reports, draft a feature spec.

• etc.

What is a “skill factory?”

I made this up, but it’s a useful way to describe the organizational machinery that turns a successful experiment into a shared capability. It includes everything from discovering ideas to deciding what should be industrialized to how skills are distributed.

This process has a strong conceptual connection with the work traditionally done by Training, L&D, or Enablement professionals: identify target capabilities and best practices for a given segment of the workforce, train workers on those skills, then measure outcomes—ideally in the form of repeatable results and improved performance.

The difference now is that, in addition to the strictly human process of upskilling (which will continue), we have a technical process for instantly distributing skills to each worker’s AI assistant. It’s a bit like Neo in The Matrix learning martial arts in real time as software is uploaded to his brain.

In many cases, the user doesn’t even need to be aware of the skill contents in order to use it. An agent simply invokes it on the user’s behalf in the right context and follows the process.

For example, when a “brand guidelines” skill is distributed across a marketing team, marketers don’t need to memorize fonts and colors when generating assets. The agent simply knows.

The skill production line (end-to-end process)

This is the mature process I envision. I can already see it in emergent form in my own organization and others.

Step 1: Ideation / Experimentation

Every skill starts with an idea for improving the quality or efficiency of how work is done.

There are two scenarios:

• Bottom up: an individual user in a role identifies a need or opportunity to improve their own work.

• Top down: A central function such as Ops or AI Enablement identifies an area for optimization and leads a pilot effort.

Both are important. Both should feed into the same pipeline.

Bottom-Up

In the bottom-up scenario, the individual user first builds a local experimental workflow with help from their LLM assistant.

Pros

Individual contributors are closest to the work, feel their own pain points most keenly, and can identify solutions that might never occur to someone outside the role.

They can also move quickly when building within existing governed infrastructure, bypassing committees and approvals.

Cons

Local experimentation can be duplicative, wasteful, or too shaped by an individual’s idiosyncratic preferences to be widely relevant. Individual users are also limited in the scope of what they can re-engineer.

Top-Down

Many companies now have people or teams with formal AI transformation mandates. These teams have the political capital and resources to take on ambitious projects that change how work is done.

Pros

Program leaders can look at metrics and see macro issues or opportunities that a single IC lacks visibility into or can’t feasibly address. Some opportunities span departments or require coordinated changes to data, systems, and processes.

These top-down interventions are best for “big bets” that target high-leverage changes or reinvent workflows completely.

Cons

AI program leaders need to be thoughtful, empathetic, and truly good listeners to gain adoption with the field. Solutions imposed top-down without user enthusiasm seldom land well.

Step 2: Pilot

Before rolling out a new workflow broadly, use a pilot to validate that it works.

It should be large enough to reveal genuine value but contained enough to stop without major sunk costs.

An enthusiastic group of pilot users is gold. They can provide feedback to enable rapid iteration and also suggest new ideas you may never have thought of.

(I’ve seen this firsthand when working with beta testers of agents I’ve built. Some of the best features came from watching how users ACTUALLY used the agent, which revealed behaviors I didn’t expect.)

Step 3: Prove Value

This is currently one of the most challenging parts of the process, as many personal AI assistants have little observability. It’s also difficult to measure the ROI of things like “work I did that I wouldn’t have done without AI” or “quality improvement as a result of using AI”.

This being said, here are a few things to consider:

• Problem scope: is the underlying problem frequent and important?

• Adoption: if one or more people use something voluntarily, it suggests value is being delivered.

• Task-based efficiency: Task A used to take us one hour (measurable) and now takes us 5 minutes. The delta is new capacity created. BUT this is only valuable if you can redeploy that capacity productively (i.e., it's not wasted) or you save on costs.

• Unit volume efficiency: We used to produce x widgets for a given input. Now we produce x+y for the same input at equal or better quality. The surplus is likely due to AI.

• Quality improvement: This is AI helping a person work on the right things, a manager provide better coaching, a seller execute better, or a leader have a better strategy. The return here can be vastly more valuable, but attributing improvement to AI is difficult and expensive (e.g., running a cohorted incrementality test).

• Capability creation: AI enables work that wasn't happening before because it was too expensive or operationally impractical. E.g., we used to hand review a few dozen closed opportunities or prospect calls. Now we use AI to systematically review hundreds or thousands. This value can be incredibly strategic but is difficult to attribute.

• Scalability: What would have to be true for it to scale?

This isn’t going to be perfect. But using the best measures available, and a degree of intuition and common sense, you can identify workflows that deserve broad distribution as a skill.

Step 4: Industrialization

This is where personal experiments or bounded pilots become robust, hardened organizational capabilities.

It’s squarely the domain of the Systems expert, because here the focus is no longer on WHAT to build but on ensuring the resulting infrastructure is scalable, maintainable, secure, and resilient.

Industrialization addresses things like:

• Defining intended behavior and failure conditions

• Replacing real data with safe templates or fixtures

• Designing permissions and safety boundaries

• Handling unavailable tools and degraded states

• Establishing ownership

• Writing setup and usage documentation

• Packaging the workflow using a standard convention

• Deciding how changes will be reviewed and released

The way you execute on this depends on your platform and tooling. I’ll do a separate post on how to set this up specifically in Claude.

Behavioral Evals

As you move from experimental free-for-all to a more professional distribution system, evaluations (“evals”) become critical.

Just like you have regression testing for your other systems to make sure new deployments don’t break things, evals provide a clear definition of what success looks like and validate that changes don’t break existing behavior.

Evals should be maintained alongside each skill and executed automatically on every proposed change. Evals can be code-based/deterministic (preferred when possible) or use a second model acting as judge when success is qualitative.

This is a big topic, so I’ll cover the step-by-step in part 2 as well.

Step 5: Learning

Learning is what transforms this process into a continuous loop.

It involves activities like:

• Tracking usage of specific skills

• Getting qualitative feedback from users

• Collecting bug reports or feedback requests and feeding them back into the top of the pipeline

• Measuring value realized (see “Prove Value” section above)

Conclusion

I predict that over the next 6-12 months, we’ll see a decreasing emphasis on isolated AI heroics and increasing focus on value engineering across the organization.

A few things will drive this change—chief among them, the changing token economy. It will get too expensive to waste time on ineffective initiatives, and the pressure to show tangible value (not just adoption) will steadily increase.

“How do I build some cool wizardry in Claude Code?” is a fundamentally different question (and requires a different skill set) than “How do I identify the most impactful AI use cases and distribute them across the organization?”

The latter is a larger, more strategic, and ultimately more important question for companies that need to evolve into AI-native workforces.

Stay tuned for part 2 of this post, which will deep dive into the specifics of configuring a plugin marketplace and eval harness for Claude.