AI Workflow
Orchestrator

1.1 What is XteVision Prodigy?

XteVision Prodigy is a self-hosted, visual node-based AI workflow builder designed for engineers, researchers, and teams who want to compose complex AI pipelines through an intuitive drag-and-drop interface. It provides multi-modal AI support, long-term memory (Wisdom), agentic tool calling, RAG embeddings, and integration with multiple AI backends—all running on a local network with full data ownership.

1.2 Key Capabilities

• Visual Node Editor — drag-and-drop canvas for composing AI workflows
• Session Authentication — user registration and login for all AI actions
• Long-Term Memory (Wisdom) — PostgreSQL-backed knowledge base with auto-recall and auto-store
• Agentic Tool Calling — web search, browser automation, file I/O, database queries, and more
• RAG / Embeddings — semantic chunk selection using nomic-embed-text via Ollama
• Multi-Modal AI — native vision support (Qwen-VL, LLaVA, etc.) via image nodes
• Multiple AI Backends — MLX, LM Studio, Ollama, and GitHub Copilot (bridge)
• Document Extraction — PDF/DOCX/XLSX/PPTX/TXT/CSV parsing
• Database Integration — direct MySQL/MariaDB queries from workflow nodes
• Web Search — SearXNG metasearch with configurable engines and language
• Trigger System — boolean gate any AI node for conditional routing
• Plugin-Driven Architecture — extensible node catalog with manifest-based discovery
• HUD Console — real-time heads-up display for tool calls, memory events, errors
• Workflow Templates — ready-made JSON workflows for rapid deployment

1.3 Target Users

• AI engineers building multi-step pipelines
• Research teams needing reproducible AI workflows
• Enterprises requiring data-local AI orchestration
• Developers prototyping agentic systems
• Teams integrating multiple AI backends (local + cloud)

2.1 High-Level Architecture

┌─────────────────────────────────────────────────────────┐
│              Browser (Any Device on LAN)                  │
│                    HTTPS / LAN                            │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│           Kali Linux Server (192.168.1.4)                │
│  ┌──────────────────────────────────────────────────┐  │
│  │        proxy_server.py (Port 9999)                │  │
│  │  • Serves UI (HTML/CSS/JS)                        │  │
│  │  • CORS proxy → AI backends                       │  │
│  │  • Auth (session tokens in PostgreSQL)            │  │
│  │  • Wisdom long-term memory (PostgreSQL)           │  │
│  │  • DB query (MySQL/MariaDB)                       │  │
│  │  • Document extraction (PDF/DOCX/XLSX/etc.)      │  │
│  │  • Tool execution (web search, browser, etc.)     │  │
│  └──────────────────────────────────────────────────┘  │
│  PostgreSQL (Port 5432)    SearXNG (Port 8888)         │
└──────────────────────┬─────────────────────────────────┘
                       │ LAN (192.168.10.3)
┌──────────────────────▼─────────────────────────────────┐
│              Apple Silicon Mac                          │
│  ┌────────────┐ ┌──────────┐ ┌──────┐ ┌──────────┐   │
│  │ MLX Server │ │LM Studio │ │Ollama│ │ Copilot  │   │
│  │ Port 5010  │ │Port 1234 │ │Port  │ │ Bridge   │   │
│  │            │ │          │ │11434 │ │Port 5015 │   │
│  └────────────┘ └──────────┘ └──────┘ └──────────┘   │
└───────────────────────────────────────────────────────┘

2.2 Communication Flow

1. Browser → Proxy Server: HTTPS requests with session authentication (X-Session-ID header)
2. Proxy Server → AI Backends: CORS proxy forwarding to MLX, LM Studio, Ollama, or Copilot Bridge
3. Proxy Server → PostgreSQL: Wisdom memory, user sessions, knowledge base queries
4. Proxy Server → SearXNG: Web search metasearch with configurable engines
5. Proxy Server → Playwright: Browser automation for agentic web interactions
6. Proxy Server → Browser: JSON responses with workflow execution results, SSE streaming

2.3 Deployment Model

Prodigy is designed for local-first, network-accessible deployment:

• Proxy server runs on a dedicated Kali Linux machine (192.168.1.4)
• AI backends run on Apple Silicon Mac (192.168.10.3) for optimal GPU/Neural Engine utilization
• Accessible from any device on the LAN via HTTPS
• Optional reverse proxy for public access (prodigy.xtevision.net)
• All data stays on local infrastructure (except optional Copilot cloud access)

4.1 Prerequisites

• Kali Linux / Debian server (192.168.1.4 recommended)
• Apple Silicon Mac (192.168.10.3 for AI backends)
• PostgreSQL (port 5432, for Wisdom memory and auth)
• SearXNG (port 8888, optional for web search)
• Python 3.8+ (for proxy server)

4.2 Server Setup (Kali Linux)

Step 1: Install Python Dependencies

pip install psycopg2-binary pymysql pypdf python-docx openpyxl \
            python-pptx chardet bcrypt playwright

# Optional: Playwright browsers (for playwright_browse tool)
playwright install chromium

# Optional: agent-browser CLI for richer manual browser automation
npm install -g agent-browser

# Optional: OCR support inside Document nodes
sudo apt install tesseract-ocr poppler-utils

Step 2: Database Setup

# Create the Prodigy PostgreSQL database and tables
psql -U postgres -f postgres.sql

# Apply auth session tables
psql -U postgres -d prodigy -f sql_auth_setup.sql

Step 3: Start the Server

cd /home/xteve/prodigy
./start.sh
# or directly:
python3 proxy_server.py

Step 4: Access the Application

LAN:     http://192.168.1.4:9999
Public:  https://prodigy.xtevision.net
Direct:  http://192.168.1.4:9999

4.3 AI Backend Setup (Apple Silicon Mac)

# MLX Server
mlx_lm.server --model <model-path> --port 5010

# LM Studio (GUI - starts automatically)

# Ollama
ollama serve  # port 11434

# GitHub Copilot Bridge
cd /path/to/copilot
node copilot_bridge.mjs

4.4 First-Time Setup

1. Register Account: Click "Register" on the login screen to create a user
2. Log In: Authenticate with your credentials (session token stored in sessionStorage)
3. Configure AI Backends: Edit config.json to set backend URLs and default models
4. Import a Workflow Template: Use the Import button to load a ready-made workflow from workflows/
5. Test a Node: Add an Input node → AI node → Exit node, then click Run

5.1 Directory Layout

prodigy/
├── index.html                    # Main UI entry point
├── styles.css                    # Application styling (dark/light modes)
├── config.json                   # Runtime config (backends, SearXNG, embedding, Wisdom DB)
├── proxy_server.py               # Python HTTP server (port 9999)
├── copilot_bridge.mjs            # GitHub Copilot ↔ OpenAI bridge (port 5015)
├── start.sh                      # Convenience launch script
├── postgres.sql                  # PostgreSQL schema (knowledge_base, users, sessions)
├── sql_auth_setup.sql            # Auth table setup
├── js/
│   ├── main.js                   # Entry point
│   ├── state.js                  # Global state & backend config
│   ├── constants.js              # NODE_TYPES enum, throttle values
│   ├── utils.js                  # fetchWithTimeout, abort controllers, URL helpers
│   ├── logic/
│   │   ├── execution.js          # Core workflow engine — all node execution logic
│   │   └── workflow.js           # Save / load / import / export
│   └── ui/
│       ├── auth.js               # Login / register UI
│       ├── connections.js        # SVG connection drawing
│       ├── dashboard.js          # Dashboard & workflow generation
│       ├── node_manager.js       # Node creation, settings panels
│       ├── playground.js         # Canvas, drag, pan, resize
│       └── priority_manager.js   # Execution order drag-and-drop
├── workflows/                    # Ready-made workflow templates
│   ├── incident_triage_escalation.json
│   ├── customer_support_triage.json
│   ├── competitive_intel_brief.json
│   ├── sql_ops_assistant.json
│   ├── deep_research_sweep.json
│   ├── schema_migration_checklist.json
│   ├── db_incident_response.json
│   ├── xlsx2wisdom.json
│   ├── marketing.json
│   ├── software.json
│   ├── agency.json
│   └── developer/                # Software delivery agency templates
│       ├── software_development_agency_sota.json
│       ├── software_development_agency_enterprise_saas.json
│       ├── software_development_agency_ai_agent_platform.json
│       ├── software_development_agency_regulated_internal_tool.json
│       └── software_development_agency_parallel_specialists.json
├── plugins/                      # Node plugins (built-in + manifest)
└── logs/
    └── app.log                   # Application log

5.2 Module Dependencies

main.js (bootstrap)
  ├── state.js
  ├── constants.js
  ├── utils.js
  ├── logic/execution.js → state.js, constants.js, utils.js
  ├── logic/workflow.js → state.js, utils.js
  ├── ui/auth.js → state.js, utils.js
  ├── ui/connections.js → state.js, playground.js
  ├── ui/dashboard.js → state.js, workflow.js, node_manager.js
  ├── ui/node_manager.js → state.js, constants.js, execution.js
  ├── ui/playground.js → state.js, connections.js, constants.js
  └── ui/priority_manager.js → state.js, workflow.js

7.1 Input / Output Nodes

7.2 Database Nodes

7.3 Web / Search Nodes

7.4 AI / Reasoning Nodes

7.5 Role / Agency Nodes

7.6 Logic / Control Nodes

7.7 Project / Communication Nodes

7.8 Industrial / Specialized Nodes

8.1 Plugin Interface (IProdigyNode)

All node types are defined through the plugin registry. Each node plugin provides metadata such as:

• id — unique identifier for the node type
• nodeType — enum value for execution routing
• title — display name in the toolbar and node panels
• description — used by the dashboard architect when telling the AI which node types are available
• ports — input and output port definitions for connection validation
• runtime capabilities — tool calling, Wisdom integration, RAG support, etc.

8.2 Standard Capabilities

class IProdigyNode:
    def Initialize(self, config): ...
    def Execute(self, input_data, ctx): ...
    def Report(self): ...
    def Hibernate(self): ...
    def UpdateConfig(self, new_config): ...

8.3 Plugin Lifecycle

1. Discovery: Plugins are loaded from the plugins/ directory at server startup
2. Registration: Each plugin registers itself with the central node catalog
3. Execution: The workflow engine routes execution to the appropriate plugin based on nodeType
4. Hot-Swapping: Plugins can be reloaded without restarting the server (WIP)

8.4 Dashboard Integration

The dashboard workflow architect reads the live plugin registry and uses each plugin's description field when generating workflows. This keeps generated workflows aligned with the actual node catalog, including newer role nodes and any future manifest-loaded plugins.

9.1 Backend Configuration

Backends are configured in config.json under the backends key. Each backend has a URL and default model.

9.2 Dashboard Routing Policy

• Default for workflow generation: ollama with gemini-3-flash-preview:latest (low-cost)
• Private sources (uploaded documents, DB nodes, Wisdom, project work): routed to local backends first
• Public-safe coding tasks: prefer GitHub Copilot free models before premium models
• Mixed sensitivity workflows: Prodigy inserts a local AI Decider + Logic Switch before cloud nodes to gate unsafe content
• Local heavy AI work: sequenced rather than fanned out in parallel to respect shared VRAM budget on Apple Silicon

9.3 GitHub Copilot Bridge

The bridge translates OpenAI-compatible requests into GitHub Copilot SDK calls and runs separately on the Apple Silicon Mac:

cd /path/to/copilot
node copilot_bridge.mjs

# Override config path when needed:
COPILOT_BRIDGE_CONFIG_PATH=/path/to/copilot_bridge.config.json node copilot_bridge.mjs

Available Copilot Models

The bridge supports both streaming (stream: true) and non-streaming requests, and reuses sessions per model to avoid per-request handshake overhead.

Bridge-Local Browser Tools

• playwright_browse — high-level rendered-page browse/read tool over the Apple Silicon Playwright HTTP bridge at /browse
• agent_browser — lower-level structured browser automation for open/click/fill/press/wait/snapshot/get/is actions

10.1 Basic Flow

1. Add Nodes: Click toolbar buttons to add nodes to the canvas
2. Connect Nodes: Drag from an output port (bottom of a node) to an input port (top) to connect nodes
3. Run: Click Run or enable Auto-Run (re-runs every 2 seconds on input changes)

10.2 Example Workflows

Web Research → AI Summary

[Input: topic] → [SearXNG] → [AI: summarise findings] → [Exit]

Vision Analysis

[Image upload] → [AI (vision model)] → [Exit]

Document Q&A with Long-Term Memory

[Document] → [AI (with Wisdom recall on)] → [Wisdom node: store result] → [Exit]

Conditional Routing

[Input] → [AI Decider] → true port  → [AI: handle success] → [Exit]
                       → false port → [AI: handle failure] → [Exit]

10.3 Trigger Gates

Connect any node's output to the trigger port of an AI / Brainstorm / Translator / Decider node. The downstream node will only execute when the trigger signal evaluates to true.

10.4 Workflow Persistence

• Auto-Save: Workflows save to localStorage on every change
• Export: Download current workflow as a JSON file
• Import: Load a workflow from any JSON file

10.5 Workflow Templates

Ready-made templates are in workflows/. Import via the Import button in the toolbar.

10.6 Developer Agency Templates

The workflows/developer/ category includes software-delivery agency templates. They all start by locking a project codebase with a Project node before interpreting the client brief, and use role nodes plus governance gates to reduce ambiguity on complex client requests.

11.1 Overview

AI nodes can automatically recall relevant entries from the PostgreSQL knowledge_base table before inference, and store new facts after a successful run.

11.2 How It Works

• Recall: Uses a parameterised ILIKE full-text search — injection-safe
• Storage: Triggered automatically by AI nodes when "Auto-store to Wisdom" is enabled
• Direct Access: You can also use a dedicated Wisdom node to read or write directly
• User Isolation: Wisdom rows are owned per authenticated user; the proxy scopes knowledge_base and messages queries to the current user
• Security: users, sessions, pg_*, and information_schema are not exposed through the Wisdom query endpoint

11.3 Configuration

Wisdom is configured in config.json under the wisdom key:

{
  "wisdom": {
    "host": "192.168.1.4",
    "port": 5432,
    "database": "prodigy",
    "user": "prodigy_user",
    "password": "your-password"
  }
}

11.4 Legacy Shared Rows

If you already have legacy shared Wisdom rows, run docs/sql_wisdom_user_isolation.sql and decide whether to backfill those old rows to a specific owner or leave them inaccessible to normal users.

12.1 Available Tools

12.2 Tool Execution Flow

1. AI node requests a tool call during inference
2. Proxy server validates the tool name and parameters
3. Tool executes on the server side (Playwright, filesystem, database, etc.)
4. Results return to the AI node for continued reasoning
5. All tool events are logged to the HUD Console

12.3 Browser Automation

Two levels of browser automation are available on the Apple Silicon host:

• playwright_browse: Uses the local HTTP browser bridge configured in copilot_bridge.config.json. High-level API for navigating and reading rendered pages.
• agent_browser: Uses the agent-browser CLI. Lower-level structured automation for open/click/fill/press/wait/snapshot/get/is actions. Optionally uses CDP when playwrightCdpTarget or PLAYWRIGHT_CDP_TARGET is configured.

13.1 Session Authentication

• User registration and login via the login screen
• Session token stored in sessionStorage (cleared on tab close)
• Token sent as X-Session-ID header on every request
• Passwords hashed with bcrypt in PostgreSQL

13.2 Data Isolation

• Wisdom rows are scoped per authenticated user
• Knowledge base and message queries are automatically filtered by user ID
• System tables (users, sessions, pg_*, information_schema) are never exposed through query endpoints

13.3 Backend Routing Privacy

• Prodigy treats MLX, LM Studio, and Ollama as local/private backends
• GitHub Copilot is treated as third-party and must only receive public-safe content
• Private sources (uploaded documents, DB nodes, Wisdom, project work) are routed to local backends first
• When a workflow mixes sensitive inputs with third-party AI steps, Prodigy inserts a local AI Decider + Logic Switch to gate unsafe content

13.4 Multi-Threading Safety

The server is multi-threaded — each request runs in its own thread, so long-running AI inference calls do not block the UI or other requests. Request size limit: 200 MB (for document uploads).

14.1 Endpoints

14.2 Authentication Header

GET /health
X-Session-ID: your-session-token

14.3 AI Request Format

POST /ai
Content-Type: application/json
X-Session-ID: your-session-token

{
  "action": "ai_inference",
  "backend": "mlx",
  "model": "your-model",
  "prompt": "Your prompt here",
  "tools": ["web_search", "fetch_webpage"],
  "wisdom_recall": true,
  "wisdom_store": true
}

14.4 Proxy Request Format

POST /proxy
Content-Type: application/json
X-Session-ID: your-session-token

{
  "url": "https://api.example.com/data",
  "method": "GET",
  "headers": {}
}

15.1 Main Configuration Sections

config.json carries the administrable defaults for:

• backends.*: AI URLs and default models
• embedding: embedding endpoint, model, and chunking settings
• wisdom: PostgreSQL/Wisdom connection defaults
• defaults.mysql: default MySQL/MariaDB node values
• defaults.wisdom: default Wisdom node values
• searxng: URL plus search engines and language
• proxy: proxy port, sandbox root, Playwright CDP base, and allowlists

15.2 SearXNG Configuration

Each SearXNG node has two optional dataset attributes you can set directly in the node panel:

These fall back to config.searxng.engines / config.searxng.language if not set on the node.

15.3 Embedding Configuration

{
  "embedding": {
    "endpoint": "http://192.168.10.3:11434/api/embeddings",
    "model": "nomic-embed-text",
    "chunkSize": 500,
    "chunkOverlap": 50
  }
}

16.1 Configuration

The Communicator node reads defaults from an optional notifications block in config.json. Node-level fields can override these values at runtime.

{
  "notifications": {
    "defaults": {
      "channel": "email",
      "emailTo": "team@example.com",
      "emailSubject": "Prodigy Update",
      "telegramChatId": "123456789"
    },
    "smtp": {
      "host": "smtp.example.com",
      "port": 587,
      "user": "mailer@example.com",
      "password": "",
      "fromAddress": "mailer@example.com",
      "starttls": true,
      "ssl": false,
      "timeout": 30
    },
    "telegram": {
      "botToken": "<telegram-bot-token>",
      "chatId": "123456789",
      "parseMode": "Markdown",
      "disablePreview": false
    }
  }
}

16.2 Supported Channels

• Email: SMTP with STARTTLS support, configurable timeout
• Telegram: Bot token-based messaging with Markdown parsing

17.1 Overview

The built-in PLC / SPS node supports a second-stage runtime model aimed at industrial prototyping:

• Dynamic named sensor and actuator pins rendered directly as graph connectors
• Retained state persisted inside the workflow snapshot
• Multiple execution engines

17.2 Execution Engines

17.3 PLC Bridge Config

For Structured Text deployment and Modbus/OpenPLC bridging, add an optional plc block to config.json:

{
  "plc": {
    "openplcApiUrl": "http://192.168.1.50:8080/api",
    "modbusHost": "192.168.1.50",
    "modbusPort": 502,
    "inputCoilOffset": 0,
    "outputCoilOffset": 100
  }
}

The backend proxy uses this config for ST upload and Modbus I/O synchronization. This is intended for the remote Linux/OpenPLC side; the browser never talks to Modbus directly.

17.4 Toradex Board Access

The VPN-connected Torizon board is configured separately from the PLC bridge so SSH access can be tested before camera or LiDAR hardware is present:

{
  "toradex": {
    "name": "Toradex Verdin VPN",
    "host": "192.168.190.71",
    "port": 22,
    "user": "",
    "sshKeyPath": "",
    "remoteProjectPath": "/home/torizon",
    "pythonPath": "python3",
    "connectTimeoutSec": 8,
    "probeCommand": "uname -a"
  }
}

The Admin Config overlay exposes these fields and includes a Test Board Access action. That probe first checks raw TCP reachability to the SSH port, then attempts a non-interactive SSH login when a user is configured.

18.1 Adding a Custom Node

1. Create a new plugin file in plugins/
2. Implement the IProdigyNode interface
3. Register the plugin in the catalog
4. Restart the server (or hot-reload if enabled)

18.2 Plugin Template

class MyCustomNode:
    id = "my_custom_node"
    nodeType = "MY_CUSTOM"
    title = "My Custom Node"
    description = "Does something awesome"
    
    def Initialize(self, config):
        self.config = config
        
    def Execute(self, input_data, ctx):
        # Process input
        result = "Output based on: " + input_data
        return result
        
    def Report(self):
        return {"status": "ok"}
        
    def Hibernate(self):
        pass
        
    def UpdateConfig(self, new_config):
        self.config = new_config

18.3 Frontend Extensions

The frontend is modular. Key extension points:

• js/constants.js: Add new node types to the enum
• js/ui/node_manager.js: Add settings panels for new nodes
• js/logic/execution.js: Add execution logic for new node types
• js/ui/playground.js: Add custom rendering for new node visuals

18.4 Keyboard Shortcuts

19.1 Server Won't Start

• Port conflict: Edit PORT = 9999 in proxy_server.py
• Missing packages: Run the pip install command from the Prerequisites section

19.2 Login / Auth Errors

• Ensure postgres.sql and sql_auth_setup.sql have been applied to the prodigy database
• Verify PostgreSQL is running on 192.168.1.4:5432 and the prodigy DB exists

19.3 AI Node Not Responding

• Check the backend is reachable: curl http://192.168.10.3:5010/v1/models
• Verify the model name matches what the backend reports
• Check the HUD console (bottom of screen) for error messages

19.4 Copilot Bridge Not Connecting

• Make sure you are logged in to the copilot CLI on the Mac: copilot auth login
• Restart the bridge: node copilot_bridge.mjs
• Bridge health check: curl http://192.168.10.3:5015/health
• Verify browser automation is installed on the Mac when using Copilot browser tools: agent-browser --help
• If needed, point the bridge at a different browser endpoint with PLAYWRIGHT_CDP_TARGET or a different CLI path with AGENT_BROWSER_BIN

19.5 Document Loader Errors

• Install PDF support: pip install pypdf
• Legacy formats (.doc, .xls, .ppt) are not supported — convert to .docx/.xlsx/.pptx
• Scanned PDFs require OCR: sudo apt install tesseract-ocr poppler-utils
• Set OCR language: export TESSERACT_LANG=eng (or eng+chi_sim) before starting the server

19.6 Wisdom Recall Returns Nothing

• Check the knowledge_base table exists in the prodigy PostgreSQL database
• Verify the Wisdom config (host/port/db/user) in config.json

19.7 SearXNG Returns No Results

• Confirm SearXNG is running: curl http://192.168.1.4:8888/search?q=test&format=json
• Adjust engines/language in the SearXNG node settings panel

19.8 Garbled Text (Chinese / Russian / etc.)

The proxy server auto-detects encodings via chardet. If issues persist check the source site's declared charset.