There is a question every CIO eventually faces — not which AI model to run, but which organizational model to run. Should AI capability live centrally, in a dedicated team with cross-enterprise authority? Or should it be embedded in the business units closest to the problems? Or some combination of both?
This is not an abstract design exercise. The answer shapes how fast your organization can move, how consistently risk is managed, and whether AI becomes a strategic capability or a scattered collection of experiments that never compound.
:::kicker Module 6: Operating Model · Article 17 of 20 :::
This article maps the three primary AI organizational archetypes — centralized, federated, and hybrid hub-and-spoke — against the real trade-offs enterprises face when scaling from pilot programs to production AI portfolios. It draws on the talent and roles foundation laid in Building an AI-Ready Organization and connects directly to the process embedding work covered in Embedding AI into Business Processes.
Why Org Design Is the Hardest AI Decision
Most AI strategy discussions focus on models, platforms, and data. Organizational structure gets treated as an implementation detail — something to sort out after the technology decisions are made.
This is backwards.
:::pullQuote "The bottleneck in enterprise AI is rarely the model. It's the organizational seam between the people who build AI and the people who use it." :::
Technology can be purchased or built. Talent can be hired. But organizational structure determines whether capability is coordinated or fragmented, whether governance is enforced or ignored, and whether learning compounds across teams or gets trapped in silos. Getting the structure wrong is expensive — not just in wasted investment, but in the opportunity cost of AI that works locally but never scales.
The challenge is that there is no universally correct answer. The right model depends on your enterprise's size, business unit diversity, existing data and platform maturity, and the degree of AI risk in your domain. What follows is a framework for reasoning through these trade-offs — not a prescription.
The Three Archetypes
Archetype 1: Centralized AI (The Center of Excellence Model)
In the centralized model, all significant AI capability — data science, ML engineering, platform engineering, governance — lives in a single team, typically called an AI Center of Excellence (CoE). Business units submit requests or work with the CoE as an internal service provider.
:::inset Typical CoE headcount at large enterprises: 25–120 people, depending on scope and maturity. :::
What the CoE owns:
- AI platform infrastructure (compute, MLOps tooling, vector stores, model registries)
- Foundation model selection and vendor relationships
- Enterprise AI governance policies and risk review processes
- Reusable assets: fine-tuned models, prompt libraries, evaluation frameworks
- AI talent hiring, standards, and career development
Where centralization works well:
- Organizations with high regulatory exposure requiring consistent governance (financial services, healthcare, insurance)
- Early-stage AI programs where shared infrastructure prevents duplicated investment
- Enterprises where domain knowledge is relatively uniform (e.g., professional services with similar client engagement patterns)
- Companies where data is centrally owned and platform fragmentation is already a problem
:::callout The governance dividend: Centralized models excel at enforcement. When a risk control or compliance requirement changes, a CoE can update policies and tooling once, with enterprise-wide effect. Federated models require propagation — and propagation is slow and imperfect. :::
Where centralization breaks down:
The CoE becomes a bottleneck. Business units with urgent AI needs wait in queue behind other priorities. Domain expertise — the deep understanding of supply chain logistics, customer service patterns, or manufacturing processes — doesn't live in the CoE; it lives in the business units. Centralized teams build generic solutions that don't quite fit the specifics of any domain.
There is also a cultural failure mode: the CoE becomes an ivory tower. It pursues technically elegant projects that are difficult to operationalize, while business units work around it with unauthorized tools and shadow AI.
Archetype 2: Federated AI (Embedded Domain Teams)
In the federated model, AI capability is distributed across business units. Each unit builds and owns its own AI talent, tools, and initiatives. A light central function may exist for policy guidance, but execution authority sits with the business.
:::didYouKnow McKinsey research found that companies with distributed, domain-embedded AI teams reported higher rates of production deployment — but also higher rates of governance incidents and technical debt accumulation. :::
What federated teams own:
- Domain-specific model development and deployment
- Workflow integration within their business context
- Their own tooling choices (within enterprise guardrails, where they exist)
- Hiring for domain-relevant AI skills (e.g., NLP for customer experience; computer vision for manufacturing QC)
Where federation works well:
- Large, diversified conglomerates where business units operate essentially as independent companies
- Organizations where domain expertise is the primary differentiator (e.g., specialized manufacturing, research-intensive industries)
- High-velocity environments where speed of iteration matters more than cross-unit consistency
- Mature organizations that already have strong data and engineering capability in business units
Where federation breaks down:
Fragmentation. Every business unit reinvents the same infrastructure, negotiates its own vendor contracts, and builds its own governance process — or skips governance entirely. When something goes wrong in one unit, there is no mechanism to identify whether the same risk exists elsewhere.
Talent concentration is another failure mode: the best AI engineers cluster in the most technically sophisticated or best-funded business unit, leaving others without meaningful capability. You end up with wide variance in AI maturity across the enterprise.
Archetype 3: Hybrid Hub-and-Spoke (The Mature Model)
Most enterprises that have been operating AI programs for more than two years converge on a hybrid model. The hub provides shared infrastructure, governance, and strategic coordination. The spokes are embedded domain teams with deep business context and execution authority.
:::comparisonTable
| Dimension | Centralized (CoE) | Federated (Domain) | Hybrid Hub-and-Spoke |
|---|---|---|---|
| Governance consistency | High | Low | High (hub-enforced) |
| Domain relevance | Low | High | High (spoke-owned) |
| Speed to deploy | Slow | Fast | Medium-fast |
| Infrastructure efficiency | High | Low (duplication) | High (shared platform) |
| Talent leverage | Concentrated | Distributed | Distributed with CoE support |
| Innovation surface | Narrow | Wide | Wide |
| Risk of fragmentation | Low | High | Medium |
| Best fit | Regulated, early-stage | Diversified, mature | Most large enterprises |
| ::: |
The hub-and-spoke model is not a compromise — it is a deliberate architecture. The hub does not do everything; it does only what benefits from being centralized. The spokes do not operate in isolation; they build on shared foundations.
What the Hub Owns (and What It Doesn't)
The failure mode of hub-and-spoke is that the hub tries to own too much and becomes the same bottleneck as a pure CoE. The hub's scope should be ruthlessly limited to what genuinely benefits from centralization.
:::checklist Hub responsibilities (non-negotiable):
- Enterprise AI platform: compute, MLOps tooling, model registry, vector infrastructure
- Foundation model governance: approved model roster, vendor contracts, security review
- AI risk framework: risk tiers, review triggers, incident escalation paths
- Data access governance: AI-specific data classification, consent management, lineage standards
- Shared evaluation infrastructure: red-teaming capability, bias testing frameworks, output monitoring
- Community of practice: cross-spoke knowledge sharing, pattern libraries, shared playbooks
Hub responsibilities (avoid):
- Owning every AI use case (creates bottleneck)
- Approving every prompt or model configuration change (creates friction without safety value)
- Managing business unit AI roadmaps (removes domain accountability)
- Building domain-specific models for business units (misaligns expertise) :::
What the Spokes Own
Domain AI teams embedded in business units should have genuine autonomy within the guardrails the hub establishes. The spoke model fails when domain teams are dependent on hub approval for everything — at that point, federation is an illusion.
Effective spokes own:
Use case prioritization: Domain leaders understand their workflow ROI better than any central team. Spoke teams work with business stakeholders to identify, evaluate, and sequence AI initiatives — without waiting for central approval.
Prompt engineering and application configuration: Within approved models and platforms, spoke teams configure, tune, and evaluate AI behavior for their specific context. A customer service team configuring tone, escalation logic, and topic restrictions for a support copilot is exercising domain expertise the hub cannot replicate.
Workflow integration: The spoke team owns the integration between AI output and business process — the human-in-the-loop design, exception handling, and performance monitoring that makes AI useful in practice, not just technically functional.
Local performance monitoring: Spoke teams monitor their AI applications for quality, relevance, and business outcome alignment. They escalate to the hub when they detect risk patterns that may have enterprise-wide implications.
:::pullQuote "The hub sets the rules of the road. The spokes decide where to drive." :::
Staffing the Model: What Each Layer Needs
Hub Staffing
A functional hub for a large enterprise (10,000+ employees) typically requires:
| Role | Function |
|---|---|
| VP/Director of AI Platform | Hub leadership; enterprise AI strategy |
| ML Platform Engineers (3–5) | Infrastructure, MLOps, tooling |
| AI Governance Lead | Risk framework, policy, audit readiness |
| AI Security Specialist | Model security, red-teaming, access controls |
| Data Governance Liaison | AI-specific data classification and lineage |
| AI Enablement Lead | Community of practice, training, playbooks |
| Evaluation Engineer | Benchmark design, bias testing, output quality |
The hub does not need to be large. Fifteen to twenty-five people running shared infrastructure and governance for an enterprise of 25,000 is reasonable. The mistake is staffing the hub as if it will do all the work.
Spoke Staffing
Spoke teams vary significantly by domain complexity and AI investment level. A minimum viable spoke — one that can own a genuine AI capability rather than just consume the hub's outputs — needs:
- AI Product Manager: Translates domain problems into AI use case specs; owns the business case and adoption roadmap
- ML Engineer or AI Engineer (1–2): Configures, fine-tunes, and integrates models; owns the spoke's technical execution
- Domain Subject Matter Expert: Provides evaluation judgment; the person who knows what "good" looks like in context
- Data Steward (often part-time): Manages spoke-level data quality and access for AI workflows
:::inset Rule of thumb: One dedicated AI engineer per major workflow automation initiative. Trying to spread one engineer across five concurrent AI projects produces five mediocre implementations. :::
The Community of Practice Layer
Between hub governance and spoke execution, there is a third layer that is underappreciated: the AI community of practice (CoP). This is the connective tissue that prevents federated teams from drifting into isolation.
An effective AI CoP is not a meeting. It is an operational structure:
:::timeline
- Weekly: Shared Slack/Teams channel for real-time questions, pattern sharing, and vendor news
- Bi-weekly: Technical sync — spoke teams share implementation challenges; hub presents new platform capabilities
- Monthly: AI review — business outcomes, risk incidents, lessons learned across spokes
- Quarterly: Strategy alignment — AI portfolio review against enterprise priorities; roadmap updates
- Annually: AI summit — external speakers, cross-enterprise showcase, capability benchmarking :::
The CoP is where federated learning actually happens. When a spoke team solves a novel prompt engineering problem, the CoP is the mechanism by which every other spoke team benefits. Without it, the same problem gets solved independently — and inconsistently — seven times.
Navigating the Transition: From CoE to Hub-and-Spoke
Most enterprises start with a centralized CoE (because it is easier to govern a new capability from the center) and eventually need to distribute. This transition is organizationally sensitive — it requires giving up central control while maintaining central accountability.
:::callout Common transition mistake: Announcing the federated model before the platform is ready. If business units are expected to run their own AI programs but the shared infrastructure doesn't exist yet, they will build their own — and you will have created the fragmentation you were trying to avoid. :::
A structured transition follows this sequence:
Build the platform first. Before distributing execution authority, establish the hub infrastructure that spokes will depend on: MLOps tooling, approved model registry, governance framework, data access controls.
Stand up two or three pilot spokes. Work with early-adopter business units to embed AI teams and validate the hub-spoke interface — what the hub provides, what the spoke owns, and where the handoffs are.
Document the patterns. Turn pilot learnings into spoke playbooks: how to configure a new AI application, how to request platform resources, how to escalate governance questions.
Scale the model. Extend to additional business units using the playbook. The hub's role shifts from doing to enabling.
Revisit hub scope annually. As spokes mature, some hub services become unnecessary (spokes develop their own capability); new hub responsibilities emerge (e.g., agentic systems governance, as covered in The Rise of Agentic Systems).
Governance in a Federated World
The most common objection to the federated or hybrid model is governance: "If every business unit is running its own AI, how do we ensure consistency, safety, and compliance?"
The answer is that governance in a federated model is not weaker — it is differently structured. Instead of approval-based governance (everything passes through a central gate), it is standards-based governance (the hub defines what is required; spokes demonstrate compliance).
:::callout The difference matters. Approval-based governance is a bottleneck. Standards-based governance is a framework. Spokes can move fast and be compliant if the standards are clear, tooling enforces them, and auditing is automated rather than manual. :::
This requires the hub to invest in governance infrastructure — automated policy checks embedded in CI/CD pipelines, output monitoring dashboards accessible to spoke leads, risk tier definitions that spoke teams can apply themselves without escalating every decision. The AI Governance in Practice article covers this infrastructure in detail.
Vendor Ecosystem Considerations
Organizational model decisions interact with vendor relationships. In a centralized model, the enterprise typically negotiates one or two primary AI platform relationships (e.g., Azure AI with Microsoft Copilot stack, or Google Vertex AI with Gemini). In a federated model, different business units may end up with different vendors — Salesforce Einstein for CRM AI, ServiceNow AI for IT operations, and a foundation model API for custom development.
:::comparisonTable
| Vendor Pattern | Organizational Fit | Risk |
|---|---|---|
| Single-platform (e.g., Azure AI + Copilot) | Centralized CoE | Vendor lock-in; less flexibility for domain-specific needs |
| Multi-platform with hub coordination | Hub-and-spoke | Platform sprawl if hub governance is weak |
| Fully decentralized vendor selection | Federated | Contract fragmentation; security and compliance inconsistency |
| Foundation model API + domain orchestration | Mature hybrid | High — requires significant internal platform engineering capability |
| ::: |
The most common pattern at Fortune 500 scale: a primary enterprise AI platform relationship managed by the hub (often Microsoft or Google), with hub approval required for any spoke team adopting a different foundation model or specialized AI vendor. This preserves flexibility while maintaining contractual and security governance.
Signals That Your Current Model Isn't Working
Organizational structures fail gradually, then obviously. These are the early warning signs:
Signs a centralized CoE is failing:
- Business units have built their own unofficial AI tools using personal API keys or shadow SaaS subscriptions
- The CoE's project backlog is 6+ months long; business urgency can't wait
- CoE outputs are technically sophisticated but don't get adopted
- Business unit leaders describe AI as "something IT is doing"
Signs a federated model is failing:
- Multiple business units have built near-identical AI tools independently
- A governance incident in one unit reveals the same risk exists in three others — and nobody knew
- AI talent is leaving domain teams because they feel isolated from the broader AI community
- No one can answer the question "what AI is running in production across the enterprise?"
Signs a hub-and-spoke model is drifting:
- Spokes are waiting on hub approval for changes that should be within spoke authority
- The hub has expanded its scope to include use case development (CoE failure mode returning)
- The community of practice has become a status-update meeting rather than a learning exchange
The Organizational Model as a Living System
The right AI organizational model is not a permanent decision. As AI capability matures, as the enterprise's portfolio of AI initiatives grows, and as the risk profile of AI applications increases — particularly with agentic systems covered in The Enterprise of Agents — the organizational model needs to evolve.
The CIO's role is not to find the right answer once. It is to build an organizational system that can adapt: that surfaces problems before they become crises, that distributes learning rather than hoarding it, and that maintains accountability even as authority is distributed.
The hub-and-spoke model, done well, is not a compromise between centralization and federation. It is a deliberate acknowledgment that different decisions belong at different levels — and that the job of leadership is to be clear about which is which.
Key Takeaways
- Centralized CoE models offer governance consistency and infrastructure efficiency but create bottlenecks and distance AI from domain expertise.
- Federated models enable speed and domain relevance but risk fragmentation, duplication, and governance gaps.
- Hybrid hub-and-spoke is the mature enterprise pattern: hub owns platform, governance, and enablement; spokes own use case prioritization, workflow integration, and domain configuration.
- The community of practice is the connective tissue that prevents federation from becoming isolation.
- Build the platform before distributing authority — federated teams without shared infrastructure create the fragmentation you were trying to prevent.
- Governance in a federated model is standards-based, not approval-based — enforcement happens through tooling and monitoring, not central gates.
Next in the series: Embedding AI into Business Processes — how AI moves from standalone capability to integrated workflow.
Related reading: Building an AI-Ready Organization · AI Governance in Practice · CI/CD Pipelines That Deliver