AI and Generative AI Foundations
1. Purpose
This page establishes the terminology used throughout the Site Application AI architecture. It prevents product names, protocols, UI applications, and AI concepts from being treated as equivalent.
2. Concept Stack
flowchart TD
AI(["Artificial Intelligence"]):::actor --> ML["Predictive / Analytical ML"]:::model
AI --> GEN["Generative AI"]:::model
GEN --> LLM["Language Models"]:::model
GEN --> MM["Multimodal Models"]:::model
GEN --> MEDIA["Image / Audio / Video Generation"]:::model
GEN --> EMB["Embedding Models"]:::model
LLM --> CAP["Application Capabilities"]:::process
MM --> CAP
EMB --> CAP
CAP --> AS[["Assistant"]]:::model
CAP --> AG[["Agent"]]:::model
CAP --> AUT[["AI-enabled Automation"]]:::process
AS --> RES[("Tools + Knowledge + Memory")]:::store
AG --> RES
AUT --> RES
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef store fill:#CCFBF1,stroke:#0F766E,stroke-width:1px,color:#0B1220
3. AI vs Generative AI
Artificial Intelligence (AI) is the broad category of systems that perform tasks associated with perception, prediction, classification, reasoning, planning, or decision support.
Generative AI (GenAI) is a subset focused on creating or transforming content such as text, code, images, audio, and other media from prompts and context.
The Site architecture should therefore avoid using AI, GenAI, and LLM as synonyms.
4. Model Types Relevant to Site
| Model Type | Typical Use |
|---|---|
| Language model | Chat, summarization, extraction, reasoning, code generation |
| Multimodal model | Combined text/image/audio understanding |
| Embedding model | Convert content into vectors for semantic retrieval/search |
| Speech-to-text model | Transcription and voice input |
| Text-to-speech model | Voice output |
| Vision model | Image/document/scene understanding |
| Image generation model | Create or edit images |
| Video generation/understanding model | Generate or analyze video where supported |
| Domain/predictive model | Forecasting, anomaly detection, classification, scoring |
Not every model belongs in the Assistant/Agent runtime. Predictive models can also be invoked as ordinary tools or analytics services.
5. Model Provider vs Model vs Deployment
Keep these concepts separate:
flowchart TD
P(["Provider / Runtime"]):::actor --> F["Model Family"]:::model
F --> M["Model / Version"]:::model
M --> D[["Deployment / Endpoint"]]:::external
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef external fill:#F1F5F9,stroke:#64748B,stroke-width:1px,color:#0B1220,stroke-dasharray:4 3
Example configuration pattern:
provider_ref: provider-profile
model_ref: model-profile
deployment_ref: production-deployment
This allows the Site to change endpoint, region, provider, or model version without rewriting every Assistant or Agent definition.
6. Inference
Inference is the runtime use of a trained model to produce an output from input.
flowchart LR
I(["Input + Instructions + Context"]):::actor --> M[["Model"]]:::model
M --> O(["Output"]):::output
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef output fill:#DCFCE7,stroke:#15803D,stroke-width:1px,color:#0B1220
The input may include user messages, retrieved knowledge, tool results, images, audio, or structured business data depending on model capabilities.
7. Prompts and Instructions
A prompt is runtime input to a model. Site prompt configuration should distinguish:
- system/developer instructions;
- task template;
- user input;
- retrieved knowledge/context;
- tool definitions/results;
- output schema.
flowchart TD
S(["System / Policy Instructions"]):::govern --> A["Prompt Assembly"]:::process
T(["Task Template + Variables"]):::actor --> A
C(["Authorized Context"]):::store --> A
U(["User Input"]):::actor --> A
A --> M[["Model"]]:::model
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef store fill:#CCFBF1,stroke:#0F766E,stroke-width:1px,color:#0B1220
classDef govern fill:#FFE4E6,stroke:#BE123C,stroke-width:1px,color:#0B1220
Prompt templates must be versioned and must not contain secrets.
8. Context Window
A model can only process a bounded amount of input context per request. The Site should not solve this by inserting every available document into every prompt.
Use:
- retrieval;
- summarization;
- filtering;
- chunk selection;
- state compaction;
- structured business context.
9. Embeddings
Embeddings are floating-point vectors representing semantic characteristics of content. Similar content tends to be closer in embedding space according to the chosen similarity/distance metric.
flowchart TD
C(["Text / Content"]):::actor --> E[["Embedding Model"]]:::model
E --> V["Floating-point Vector<br/>[0.018, -0.224, 0.071, ...]"]:::store
V --> I[("Vector Index")]:::store
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef store fill:#CCFBF1,stroke:#0F766E,stroke-width:1px,color:#0B1220
Embeddings support semantic search and retrieval; they do not themselves generate natural-language answers.
10. RAG
Retrieval-Augmented Generation (RAG) combines external retrieval with model generation.
flowchart TD
Q(["User Query"]):::actor --> R["Retrieve authorized relevant context"]:::store
R --> P["Prompt + Retrieved Context"]:::process
P --> M[["Generative Model"]]:::model
M --> O(["Grounded Response"]):::output
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef store fill:#CCFBF1,stroke:#0F766E,stroke-width:1px,color:#0B1220
classDef output fill:#DCFCE7,stroke:#15803D,stroke-width:1px,color:#0B1220
RAG is an application architecture pattern, not a new type of model.
11. Tools / Function Calling
A model or agent can be given descriptions of approved operations that it may request or invoke.
Examples:
- search Site knowledge;
- retrieve an asset;
- call a calculation service;
- invoke an n8n automation;
- create a draft task;
- query Microsoft Graph.
Tool access must remain permission- and policy-controlled. A tool description does not grant authorization by itself.
12. Memory and State
Different state types must remain separate:
| State | Example | Scope |
|---|---|---|
| Request context | Current object, current page | One request/turn |
| Conversation/session state | Prior messages and tool results | Session |
| Execution state | Agent plan, pending approval, workflow step | Execution |
| Long-term memory | Durable learned/user/task information | Cross-session |
| Knowledge | Documents/specifications/reference data | Shared authoritative source |
13. Assistant and Agent
An Assistant is primarily user-led. An Agent is goal-led and can dynamically choose tools or steps.
flowchart LR
subgraph AssistantFlow["Assistant"]
U(["User"]):::actor --> Q["Ask"]:::process
Q --> H["AI helps"]:::model
H --> D(["User decides next"]):::actor
end
subgraph AgentFlow["Agent"]
G(["Goal"]):::actor --> R["Reason"]:::model
R --> T["Tool"]:::process
T --> O["Observe"]:::process
O --> C{"Continue / Stop"}:::gate
end
style AssistantFlow fill:#94A3B81A,stroke:#94A3B8,stroke-width:1px,stroke-dasharray:3 3
style AgentFlow fill:#94A3B81A,stroke:#94A3B8,stroke-width:1px,stroke-dasharray:3 3
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef gate fill:#FEF3C7,stroke:#B45309,stroke-width:1px,color:#0B1220
An Assistant may use tools. Tool use alone does not make it an Agent.
14. Automation and AI
AI can be inserted into a deterministic workflow without turning the workflow into an Agent.
flowchart LR
E(["Event"]):::actor --> X["Extract"]:::process
X --> L["LLM classify"]:::model
L --> R["Rule"]:::gate
R --> A[["API"]]:::external
A --> C(["Complete"]):::output
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef gate fill:#FEF3C7,stroke:#B45309,stroke-width:1px,color:#0B1220
classDef output fill:#DCFCE7,stroke:#15803D,stroke-width:1px,color:#0B1220
classDef external fill:#F1F5F9,stroke:#64748B,stroke-width:1px,color:#0B1220,stroke-dasharray:4 3
This remains Automation if the designer predetermined the control flow.
15. Autonomous Operation
Autonomous operation is a control/governance mode in which an Agent or capability composition continues toward a bounded objective with limited human intervention.
Required boundaries include:
- permissions;
- allowed actions;
- approval policy;
- stop conditions;
- time/cost/iteration limits;
- escalation;
- audit/traceability.
16. Multimodal Processing
The Site can support multimodal content without forcing all sources into one processing path.
flowchart LR
D(["Document"]):::actor --> DP["Text / layout extraction"]:::process
DP --> CTX[("Context")]:::store
I(["Image"]):::actor --> IP["Vision / OCR when needed"]:::model
IP --> CTX
A(["Audio"]):::actor --> AP["Transcription or audio model"]:::model
AP --> CTX
V(["Video"]):::actor --> VP["Transcript + scene / frame sampling"]:::model
VP --> CTX
T(["Telemetry"]):::actor --> TP["Structured query / analytics"]:::process
TP --> CTX
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef model fill:#EDE9FE,stroke:#6D28D9,stroke-width:1px,color:#0B1220
classDef store fill:#CCFBF1,stroke:#0F766E,stroke-width:1px,color:#0B1220
Processing should be selected by modality and use case. Processing every frame of every video by default is wasteful and usually unnecessary.
17. Product Roles
| Product / Technology | Correct Role in Architecture |
|---|---|
| Open WebUI | AI user interaction/interface |
| Flowise | Visual AI flow/agent orchestration |
| n8n | Workflow/process automation with optional AI features |
| Model provider/runtime | Model inference/embedding capability |
| Vector database/store | Semantic retrieval/index storage |
| MCP | Protocol for AI tool/context integration |
| OpenAPI | Description contract for HTTP APIs |
| Microsoft Graph | Microsoft 365 API surface |
| Microsoft Agent 365 SDK | Optional enterprise identity, observability, notification, and governed-tool extensions for agents |
| Agent SDK | Agent execution/runtime abstraction |
18. Design Rule
The Site capability model should describe what the capability does. Provider, SDK, workflow engine, model, UI, and protocol are implementation references beneath that capability.
flowchart TD
B(["Business Capability"]):::actor --> C{"4 A Classification"}:::gate
C --> I[["Implementation Runtime"]]:::process
I --> R["Provider / Knowledge / Tools / UI"]:::process
classDef actor fill:#DBEAFE,stroke:#1D4ED8,stroke-width:1px,color:#0B1220
classDef process fill:#EEF2F7,stroke:#475569,stroke-width:1px,color:#0B1220
classDef gate fill:#FEF3C7,stroke:#B45309,stroke-width:1px,color:#0B1220