Media understanding

OpenClaw can summarize inbound media (image/audio/video) before the reply pipeline runs. It auto-detects when local tools or provider keys are available, and can be disabled or customized. If understanding is off, models still receive the original files/URLs as usual.

Vendor-specific media behavior is registered by vendor plugins, while OpenClaw core owns the shared tools.media config, fallback order, and reply-pipeline integration.

# Goals

• Optional: pre-digest inbound media into short text for faster routing + better command parsing.
• Preserve original media delivery to the model (always).
• Support provider APIs and CLI fallbacks.
• Allow multiple models with ordered fallback (error/size/timeout).

# High-level behavior

If understanding fails or is disabled, the reply flow continues with the original body + attachments.

# Config overview

tools.media supports shared models plus per-capability overrides:

{
  tools: {
    media: {
      models: [
        /* shared list */
      ],
      image: {
        /* optional overrides */
      },
      audio: {
        /* optional overrides */
        echoTranscript: true,
        echoFormat: '📝 "{transcript}"',
      },
      video: {
        /* optional overrides */
      },
    },
  },
}

# Model entries

Each models[] entry can be provider or CLI:

{
  type: "provider", // default if omitted
  provider: "openai",
  model: "gpt-5.5",
  prompt: "Describe the image in <= 500 chars.",
  maxChars: 500,
  maxBytes: 10485760,
  timeoutSeconds: 60,
  capabilities: ["image"], // optional, used for multi-modal entries
  profile: "vision-profile",
  preferredProfile: "vision-fallback",
}

{
  type: "cli",
  command: "gemini",
  args: [
    "-m",
    "gemini-3-flash",
    "--allowed-tools",
    "read_file",
    "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
  ],
  maxChars: 500,
  maxBytes: 52428800,
  timeoutSeconds: 120,
  capabilities: ["video", "image"],
}

# Defaults and limits

Recommended defaults:

• maxChars: 500 for image/video (short, command-friendly)
• maxChars: unset for audio (full transcript unless you set a limit)
• maxBytes:
  • image: 10MB
  • audio: 20MB
  • video: 50MB

# Auto-detect media understanding (default)

If tools.media.<capability>.enabled is not set to false and you haven't configured models, OpenClaw auto-detects in this order and stops at the first working option:

To disable auto-detection, set:

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

# Proxy environment support (provider models)

When provider-based audio and video media understanding is enabled, OpenClaw honors standard outbound proxy environment variables for provider HTTP calls:

• HTTPS_PROXY
• HTTP_PROXY
• ALL_PROXY
• https_proxy
• http_proxy
• all_proxy

If no proxy env vars are set, media understanding uses direct egress. If the proxy value is malformed, OpenClaw logs a warning and falls back to direct fetch.

# Capabilities (optional)

If you set capabilities, the entry only runs for those media types. For shared lists, OpenClaw can infer defaults:

• openai, anthropic, minimax: image
• minimax-portal: image
• moonshot: image + video
• openrouter: image
• google (Gemini API): image + audio + video
• qwen: image + video
• mistral: audio
• zai: image
• groq: audio
• xai: audio
• deepgram: audio
• Any models.providers.<id>.models[] catalog with an image-capable model: image

For CLI entries, set capabilities explicitly to avoid surprising matches. If you omit capabilities, the entry is eligible for the list it appears in.

# Provider support matrix (OpenClaw integrations)

# Model selection guidance

• Prefer the strongest latest-generation model available for each media capability when quality and safety matter.
• For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
• Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
• CLI fallbacks (whisper-cli, whisper, gemini) are useful when provider APIs are unavailable.
• parakeet-mlx note: with --output-dir, OpenClaw reads <output-dir>/<media-basename>.txt when output format is txt (or unspecified); non-txt formats fall back to stdout.

# Attachment policy

Per-capability attachments controls which attachments are processed:

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

# Config examples

{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-5.5", capabilities: ["image"] },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["image", "audio", "video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image", "video"],
        },
      ],
      audio: {
        attachments: { mode: "all", maxAttachments: 2 },
      },
      video: {
        maxChars: 500,
      },
    },
  },
}

{
  tools: {
    media: {
      audio: {
        enabled: true,
        models: [
          { provider: "openai", model: "gpt-4o-mini-transcribe" },
          {
            type: "cli",
            command: "whisper",
            args: ["--model", "base", "{{MediaPath}}"],
          },
        ],
      },
      video: {
        enabled: true,
        maxChars: 500,
        models: [
          { provider: "google", model: "gemini-3-flash-preview" },
          {
            type: "cli",
            command: "gemini",
            args: [
              "-m",
              "gemini-3-flash",
              "--allowed-tools",
              "read_file",
              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
            ],
          },
        ],
      },
    },
  },
}

{
  tools: {
    media: {
      image: {
        enabled: true,
        maxBytes: 10485760,
        maxChars: 500,
        models: [
          { provider: "openai", model: "gpt-5.5" },
          { provider: "anthropic", model: "claude-opus-4-6" },
          {
            type: "cli",
            command: "gemini",
            args: [
              "-m",
              "gemini-3-flash",
              "--allowed-tools",
              "read_file",
              "Read the media at {{MediaPath}} and describe it in <= {{MaxChars}} characters.",
            ],
          },
        ],
      },
    },
  },
}

{
  tools: {
    media: {
      image: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
      audio: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
      video: {
        models: [
          {
            provider: "google",
            model: "gemini-3.1-pro-preview",
            capabilities: ["image", "video", "audio"],
          },
        ],
      },
    },
  },
}

# Status output

When media understanding runs, /status includes a short summary line:

📎 Media: image ok (openai/gpt-5.4) · audio skipped (maxBytes)

This shows per-capability outcomes and the chosen provider/model when applicable.

# Notes

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (e.g. only DMs).

# Related

• Configuration
• Image & media support