Articles
What can the latest AI do? 600+ capabilities comprehensively mapped from Gemini, Claude, OpenAI, and other major platforms
Automatically translated from the Japanese original.
Introduction
AI is advancing at a remarkable pace, and the list of things it can do keeps growing. Yet most of what circulates on social media and in tech news consists of isolated use cases; it is hard to find anything that offers a bird's-eye view of what AI can do as a whole. In this article, we take roughly 15 of the major AI products with large global user bases and, drawing primarily on their official documentation, compile a comprehensive list of their capabilities. It covers features for both general users and developers. We hope it serves as a useful reference for understanding what AI can do today.
We classify AI capabilities into six categories: understanding, retrieval, reasoning, generation, action, and other. The rationale is that today's AI agents can be seen as integrating concepts such as the ReAct (Reason + Act) framework, the RAG (Retrieve + Generate) model, and the Retrieve-Reason-Act loop. We therefore treat understanding, retrieval, reasoning, generation, and action as the representative building blocks of an AI agent and organize the capabilities accordingly.
AI products covered and reference documentation
ChatGPT: The flagship generative AI, with unmatched name recognition and user numbers.
Gemini: Tightly integrated with Google Search, Android devices, and Google Workspace, it is embedded in the everyday digital infrastructure of users worldwide.
Claude: Hugely popular and enthusiastically backed by professionals for its natural writing, sophisticated logical reasoning, and programming ability.
Microsoft Copilot: Built on OpenAI technology under the hood, but because it ships as standard with Windows and Office products such as Word and Excel, adoption among business users is exceptionally high.
OpenClaw: Runs on your own PC (a local environment). Simply issue instructions from a chat app such as WhatsApp or Discord, and the AI autonomously reads and writes files and operates your browser.
Perplexity AI: An AI search engine dedicated to reading multiple websites and synthesizing a direct answer rather than returning a list of links. Its user base is expanding rapidly around the world.
Character.ai: An AI that lets you converse with a wide range of user-created characters (anime characters, celebrities, and more). It draws enormous traffic, particularly among younger users.
DeepSeek: An AI from China that delivers performance on par with OpenAI and Anthropic at an extremely low cost, rapidly winning over developers and companies worldwide.
Llama: Meta's open-source model. It is the most widely used engine behind AI applications built by startups around the world.
Mistral: An AI from France. Centered on the European market, it enjoys strong popularity as a lightweight yet high-performance open model.
Grok: Developed by xAI, led by Elon Musk. Its greatest strength is its connection to real-time information from X (formerly Twitter).
Kimi: Originating in China, it can ingest ultra-long documents of several million characters in one pass, and its user base is growing rapidly, especially across Asia.
Midjourney: The leading AI for generating high-quality images from text. It runs on Discord and is a favorite of creators and designers worldwide.
Stable Diffusion: The de facto standard among open-source image generation AIs. It can be installed locally on a PC and customized to your liking.
DALL·E: An image generation AI developed by OpenAI that produces high-quality images exactly as described in your text prompt.
Genspark: A next-generation platform for search, information gathering, and document creation powered by AI agents.
"Understanding" capabilities
These capabilities let the AI grasp, in a structured way, the user's intent, ambiguous natural language, and multimodal inputs such as images and audio.
Text understanding, summarization, and information extraction
Text summarization: A core capability that understands the context of long texts or multiple documents and extracts the key points and conclusions [OpenAI, Google/Gemini, Anthropic/Claude, Microsoft Copilot, Mistral, Meta/Llama, Kimi, DeepSeek]
Structured data extraction (Structured output / JSON schema): Accurately extracts and interprets entities and specific values from unstructured text according to a defined JSON schema [OpenAI, Google/Gemini, Anthropic/Claude, Meta/Llama, xAI, DeepSeek]
Entity extraction: Identifies and extracts proper nouns and keywords such as personal names, company names, dates, and monetary amounts from long texts [OpenAI, Google/Gemini, Anthropic/Claude, Microsoft Copilot]
Topic and user intent interpretation (Intent recognition / Topic triggering): Interprets the purpose (intent) behind a user's utterance or ambiguous question and triggers the appropriate agent flow [Microsoft Copilot, OpenClaw, Google/Gemini]
Multilingual translation and contextual understanding (Multilingual support): Natively interprets text written in multiple languages and translates or summarizes it into a specified language [OpenAI, Google/Gemini, Anthropic/Claude, Meta/Llama, xAI, DeepSeek]
Content moderation and safety interpretation (Moderation / Safety checks): Analyzes and understands whether text contains policy violations, harmful content, or other inappropriate elements [OpenAI, Meta/Llama, Google/Gemini, xAI]
Conversational context tracking (Context management / Threads): Remembers and understands past message history and the flow of a session in order to respond in context [OpenAI, Anthropic/Claude, Google/Gemini, xAI, OpenClaw, Perplexity]
Sentiment analysis and tone interpretation (Sentiment analysis): Understands and extracts the user's emotional state and tone from the wording and expressions in the text [Google/Gemini, OpenAI, Microsoft Copilot]
Whole-picture comprehension of large contexts (Long-context understanding): Ingests extremely long inputs—1 to 5 million tokens, or 256K, for example—in a single pass and understands them as a whole [Google/Gemini, Anthropic/Claude, Meta/Llama, Kimi, DeepSeek]
Automatic extraction and interpretation of prompt variables (Templatize a prompt): Automatically identifies and extracts variables from user input and understands them as a template structure that can easily be integrated into a system [Anthropic/Claude]
Character psychology inference and relationship extraction (Character mind map): Interprets and organizes the thought processes and interpersonal relationships of characters in books and stories [Meta/Llama]
Persona and behavioral rule interpretation (Persona understanding): Autonomously understands and adapts to the role, tone, and rules specified in system prompts or configuration files (such as AGENTS.md) [OpenClaw, Anthropic/Claude, Mistral, OpenAI]
In-depth analysis of research papers and academic data (Analyze research papers): Reads and interprets academic papers and research data in complex formats to extract scientific insights [Meta/Llama, Google/Gemini, Anthropic/Claude]
Cross-document understanding (Multi-document analysis): Interprets the relationships and contradictions among multiple uploaded files and understands them as an integrated whole [OpenAI, Anthropic/Claude, Google/Gemini, Mistral, Kimi]
Understanding organizational data in its business context (Data grounding): Reads internal SharePoint sites, email, chat, and the like, and understands them within the company's specific business context [Microsoft Copilot, Google/Gemini]
Context reuse through prompt caching (Prompt caching): Caches lengthy contexts so information can be interpreted and extracted efficiently without recomputation [OpenAI, Anthropic/Claude, xAI, DeepSeek]
Continued contextual understanding via chat prefixes (Chat Prefix Completion): Accurately understands a partially written text prefix and predicts and extracts the context that follows [DeepSeek, OpenAI]
Interpretation of clauses in legal documents and contracts (Legal document analysis): Accurately understands and extracts risk clauses and obligations from lengthy legal documents [Anthropic/Claude, OpenAI, Google/Gemini, Kimi]
Extraction of work history and skills from résumés (Resume/Profile extraction): Extracts and understands skill sets and career histories from HR data and résumés in structured form [OpenAI, Google/Gemini, Anthropic/Claude]
Contextual extraction from medical and healthcare records (Healthcare data extraction): Interprets and extracts key medical insights and symptoms from clinical records and patient data [Google/Gemini]
Action item extraction from meeting records (Action item extraction): Interprets and extracts decisions and tasks from meeting transcripts and support chat histories [OpenAI, Microsoft Copilot, Anthropic/Claude]
Identification and extraction of personal and confidential information (Sensitive Data Protection / PII extraction): Understands personal information in input data and extracts it for protection and masking [Google/Gemini, OpenAI, Microsoft Copilot]
Multimodal and visual understanding (images, video, UI)
Understanding objects and scenes in images (Image understanding / Vision): Interprets the objects, situations, and spatial relationships depicted in an image as visual context [OpenAI, Google/Gemini, Anthropic/Claude, xAI, Meta/Llama, DeepSeek, Kimi]
Optical character recognition and text extraction (OCR / DeepSeek-OCR-2): Extracts textual information from scanned documents and photographs and interprets it as structured data [OpenAI, Google/Gemini, DeepSeek, Anthropic/Claude]
Interpreting UI mockups as code (Image mock up to HTML): Understands the intent behind handwritten notes or UI designs shown in an image and interprets them as a code structure [Google/Gemini, OpenAI, Anthropic/Claude]
Visual analysis of complex charts and graphs (Chart & Graph analysis): Reads numerical insights and logical trends from data graphs rendered as images [Anthropic/Claude, Meta/Llama, OpenAI, Google/Gemini]
Understanding visual causality and logical structure (Visual Causal Flow): Interprets and extracts information flows and logical cause-and-effect relationships from complex diagrams and flowcharts [DeepSeek]
Understanding video frames and context (Video understanding): Takes video files as input and analyzes scene transitions, content, and the context of motion to extract information [Google/Gemini, xAI, Kimi]
Integrated analysis of text and visual content in PDFs (PDF support / Document AI): Interprets and extracts not only the text in a PDF but also its figures, tables, and layout structure at the same time [Anthropic/Claude, Google/Gemini, OpenAI, Kimi, Mistral]
Multimodal interpretation of social media posts (Enable Image/Video Understanding): Specializes in extracting and understanding the context of images and videos embedded in social media posts [xAI]
Interpreting the UI structure of computer screens (Computer use vision): Visually understands UI layouts and interactive elements from desktop screenshots [Anthropic/Claude, OpenAI]
Interpreting transparent regions in image-editing masks (Masking understanding): Reads the alpha channel of a supplied mask image to determine which regions should be edited or regenerated [OpenAI]
Visual interpretation of the physical world through a camera (Device Camera Vision): Interprets real-world footage fed directly from a camera, such as in a mobile app, and understands the situation [OpenClaw, Google/Gemini]
Maintaining context across multi-turn image generation (Multi-turn image context): Carries forward and interprets the context of an image generated earlier in the conversation (via the previous response ID) [OpenAI]
Visual inference of age and biometric information from facial photos (Age Assurance / Persona Verification): Analyzes and interprets biometric information from a selfie to estimate the user's age range [C.AI, OpenAI]
Multimodal mixture-of-experts understanding (Mixture-of-Experts Vision-Language): Interprets sophisticated inputs that mix images and text by routing them to the most suitable expert model [DeepSeek]
Visual interpretation of spatial lighting and 3D structure (ReLi3D / SViM3D): Physically interprets the 3D shape of objects and the lighting conditions of the environment from a single 2D image [Stability AI]
Synchronized interpretation of audio and visuals in video (Video audio analysis): Understands and extracts information by jointly analyzing the audio track and the on-screen motion of video content [Google/Gemini]
Interpreting geospatial data (Geospatial Analytics / Earth Engine): Extracts and understands terrain changes and spatial insights from satellite imagery and map data [Google/Gemini]
Understanding code, systems, and logs
Understanding project-wide code structure and dependencies (Codebase understanding): Understands the logic and dependencies spanning multiple files across an entire project, rather than a single file [Google/Gemini, Anthropic/Claude, OpenAI, Kimi]
Interpreting error logs and stack traces (Error trace analysis): Extracts and understands the root cause of system errors from terminal output and crash logs [Anthropic/Claude, Google/Gemini, OpenAI]
Interpreting the intent of Git commits and PR diffs (Diff understanding): Analyzes code diffs to interpret the intent behind changes and identify review points [Anthropic/Claude, Google/Gemini, Microsoft Copilot]
Interpreting database schemas and SQL intent (Database Assistance): Understands the logic of complex existing SQL queries and table metadata and explains them in plain language [Google/Gemini, Microsoft Copilot]
Extracting insights from spreadsheet data (Code Interpreter data analysis): Loads CSV or Excel data, interprets the meaning of the numbers, and extracts charts and trends [Mistral, OpenAI, Microsoft Copilot]
Autonomous understanding of terminal environments and directory structures (CLI Workspace understanding): Lets a CLI tool autonomously grasp and interpret the current project environment and directory layout [Anthropic/Claude, Mistral, Google/Gemini]
Interpreting API specifications and policy code (Apigee / API policy understanding): Reads enterprise API specifications and security policy code to understand the intent behind their configuration [Google/Gemini]
Interpreting data skew and drift in MLOps (Model Monitoring): Analyzes and understands skew in the input data fed to machine learning models and trends toward degradation [Google/Gemini]
Interpreting security logs and threat intelligence (Security Copilot / Threat Analysis): Extracts signs of cyberattacks and the context of threats from massive volumes of system logs and network traffic data [Microsoft Copilot, Google/Gemini]
Interpreting cloud infrastructure configurations and resources (Cloud Assist / Infrastructure understanding): Infers the optimal cloud resources and deployment configuration from the context of source code and the editor [Google/Gemini]
Understanding the context of session history and agent state (Session history retrieval): Allows an agent to read information from past sessions and trace back task progress and context [OpenClaw, OpenAI]
Understanding the structure of external data sources such as Jira and Google Drive (MCP server resources): Understands and extracts the structure of tickets and documents in external SaaS tools connected via MCP [Anthropic/Claude, OpenAI, Microsoft Copilot]
Interpreting surrounding context for FIM completion (Fill-in-the-Middle Completion): Infers the missing logic that belongs in the middle of a code block from the code that precedes and follows it [DeepSeek, OpenAI]
Understanding requirements for code modernization (Code modernization analysis): Interprets and understands the original business logic when migrating legacy code to modern frameworks [Google/Gemini, Anthropic/Claude]
Extracting troubleshooting steps (Troubleshooting extraction): Pulls the necessary resolution steps from manuals and similar sources based on a user's description of an error [Google/Gemini, Anthropic/Claude]
Interpreting the structure of webhook payloads (Webhook event parsing): Understands the JSON data in webhooks sent from external systems and extracts the relevant event information [OpenAI, OpenClaw]
Interpreting the state and sensor data of mobile devices (Mobile node context): Enables an agent to treat information sent from a mobile node, such as battery level and system status, as context [OpenClaw]
Understanding the independent context of Git worktrees (Worktrees context): Understands the context specific to the current directory when separate tasks are progressing in multiple working environments [Anthropic/Claude]
Extracting meaning from developer trace logs (Trace / Debug log understanding): Interprets logs of the model's internal processing and network requests to pinpoint bottlenecks and error locations [OpenAI, OpenClaw]
Interpreting cloud cost and FinOps data trends (Cost Management / Billing analysis): Reads API usage charges and infrastructure cost data to surface abnormal spending patterns and optimization opportunities [Google/Gemini, OpenAI]
Autonomous interpretation of project instruction files (e.g. CLAUDE.md): Lets an agent read instruction files placed in a directory on its own and understand the project's requirements [Anthropic/Claude, OpenClaw]
Interpreting prompts for third-party model compatibility (OpenAI Compatibility): Interprets and converts requests sent in OpenAI's API format into a form that other models can understand [Meta/Llama, Perplexity, DeepSeek]
Understanding speech and audio
Speech recognition and transcription (Speech-to-text / Whisper): Takes audio data as input and interprets and extracts what was said as accurate text [OpenAI, Google/Gemini, xAI]
Interpreting emotion and tone in real-time speech (Realtime API / Voice API): Understands not just the meaning of the words but the speaker's emotion and tone in real time from ultra-low-latency audio input [OpenAI, xAI, Microsoft Copilot]
Speaker separation and conversation structure understanding (Diarization): Identifies "who spoke when" in audio data and understands the flow and structure of multi-party conversations [OpenAI]
Interpreting user intent from speech in IVR (Interactive voice response): In voice-based customer support and similar settings, interprets what the customer wants and decides whether to hand off to a human operator [Microsoft Copilot]
Contextual understanding of recorded voice consent (Voice Consents interpretation): Extracts and understands the user consent information required when using speech synthesis APIs and the like, from text or audio [OpenAI]
Understanding search results, external knowledge, and tools
Integrated summarization of web search results (Web-grounded summarization): Consolidates information gathered from multiple web pages, understanding how it relates and where it conflicts, and produces a summary [Perplexity, OpenAI, Google/Gemini, Anthropic/Claude, Mistral, Kimi]
Deep understanding of complex topics (Deep research / Sonar): Works through information on the web in multiple stages to build a deep understanding of a complex subject and summarize it [OpenAI, Google/Gemini, Perplexity]
Interpreting the reliability and relevance of sources (Search API filters): Extracts the semantic relevance of information from search result titles and snippets, without any LLM generation step [Perplexity]
Interpreting domain-specific context in search results (Domain controls): Understands specialized context from search results restricted to particular domains such as healthcare or finance [Perplexity]
Context-preserving information extraction for RAG (Contextual retrieval): When splitting information into chunks, interprets and extracts it in a way that keeps the context of the full document intact [Anthropic/Claude, Perplexity]
Semantic vectorization and interpretation of text (Embeddings): Interprets the meaning and nuance of text numerically as multidimensional vectors in order to identify similarity [OpenAI, Anthropic/Claude, Perplexity, Google/Gemini]
Understanding real-time trends and thread flow on X (Thread fetch & Trend analysis): Interprets the conversational flow of an entire social media thread and the latest topics as context [xAI]
Interpreting evidence for fact-checking (Grounding verification): Cross-checks the model's answer against web search results to understand whether the stated facts hold up [Google/Gemini, Microsoft Copilot]
Extracting precise source context for citations (Citation extraction): Accurately extracts the passages from search results that support an answer and links them as citations [Perplexity, OpenAI, Google/Gemini]
Semantic understanding of a user's own collections (Collections / Knowledge base): Extracts metadata from a set of documents uploaded by the user and understands their overall context [xAI, Mistral]
Understanding tool selection intent and arguments (Tool choice / Function calling interpretation): Autonomously interprets from a user's request which tool should be used and with what parameters [OpenAI, Anthropic/Claude, Google/Gemini, Meta/Llama, xAI, DeepSeek, Mistral]
Extracting legal evidence for eDiscovery (Legal hold retrieval): Semantically searches, extracts, and understands conversation histories and files from organizational data that are relevant to litigation or audits [Microsoft Copilot]
Learning and interpreting a user's past prompt patterns (Personalization / Memory): Remembers a user's past instructions and preferences to interpret the intent of the current prompt more accurately [Google/Gemini, OpenAI, OpenClaw]
Interpreting telecom network traffic data (Telecom Data Fabric / Subscriber Insights): Analyzes carrier traffic data and subscriber information to understand and extract behavioral patterns [Google/Gemini]
Interpreting product search intent for retail (Vertex AI Search for commerce): On e-commerce sites, understands the characteristics of the product a user is looking for from a vague search query [Google/Gemini]
Real-time extraction of news and stock data (Real-time data extraction): Extracts and understands the latest news headlines and financial market figures from search results in structured form [Perplexity]
Structural understanding of diverse document formats (Markup interpretation): Interprets the meaning of Markdown, HTML, DOCX, and similar documents while preserving markup structures such as headings and tables [OpenAI, Anthropic/Claude, Google/Gemini, Mistral, Kimi]
Deep understanding through reasoning models and analytical approaches
Deep interpretation of complex problems via reasoning models (Reasoning models): Understands the true nature of a problem through a step-by-step logical thinking process rather than intuitive pattern matching [OpenAI, DeepSeek, xAI, Google/Gemini, Kimi]
Interpreting multi-stage context through extended thinking (Extended thinking): Takes time to thoroughly work through the background and constraints of a problem and interpret its logical context [Anthropic/Claude]
Interpreting task difficulty through adaptive thinking (Adaptive thinking): Autonomously understands the complexity of an incoming task and adjusts how much reasoning to allocate to it [Anthropic/Claude]
Interpreting requested reasoning effort (Reasoning effort): Understands the depth of thinking specified by the user and extracts information at a corresponding level of granularity [OpenAI, OpenClaw]
Logical interpretation of hard mathematical and scientific problems (Deep thought / Math reasoning): Interprets rigorous logical arguments and the context of theorems from text containing equations and physical laws [DeepSeek, xAI, Google/Gemini]
Automatic extraction of data insights (Data Insights): Autonomously understands and extracts correlations and business meaning from database tables and log data [Google/Gemini, Microsoft Copilot]
Extracting context from marketing results (Campaign summarization): Understands the factors behind success or failure from campaign delivery data and summarizes them [Google/Gemini]
Interpreting agent health and performance metrics (Agent analytics): Understands user frustration and resolution rates from bot conversation logs and extracts areas for improvement [Microsoft Copilot]
Self-evaluation of prompts and understanding optimization intent (Prompt optimizer analysis): Has the model analyze its own responses and reinterpret the prompt so that it better satisfies the user's original intent [OpenAI]
Interpreting correct answers in automated test sets (Evaluations grading): Has the LLM itself read test responses and interpret, against set criteria, the context of what counts as correct or incorrect [OpenAI, Anthropic/Claude, Google/Gemini]
Real-time chat interpretation for live streaming (Twitch chat analysis): Extracts and understands the overall sentiment and questions of viewers from stream chat comments in real time [OpenClaw]
Understanding mentions and addressees in chat apps (Mention / Routing interpretation): In Slack, Discord, and similar platforms, interprets who a message is directed at and whether the bot should respond [OpenClaw, Anthropic/Claude, Microsoft Copilot]
Interpreting task goals in non-interactive mode (Non-interactive mode analysis): Fully understanding the end goal from the initial prompt alone and carrying the task through to completion, without any human feedback along the way [Anthropic/Claude]
Interpreting API rate limits and metadata (Rate limit / Meta info extraction): Programmatically reading response headers from an API to determine remaining token quotas, request IDs, and similar details [OpenAI, Perplexity]
Sharing and understanding handoff context between agents (Handoff context understanding): Interpreting the conversation history so far when a task is handed off from a bot to a human operator or to another agent [OpenAI, Microsoft Copilot]
Contextual detection of ad insertion points in video (Video Stitcher API): Understanding the context of video content and identifying the right moments to dynamically insert ads on the server side [Google/Gemini]
Inferring age from user activity signals (Age assurance signals): Holistically analyzing signals such as login information and activity history on the platform to determine a user's age bracket [C.AI]
"Search" capabilities
These capabilities retrieve information the AI was never trained on—such as the latest news or confidential internal data—from external databases and the web.
Web search and deep research
Autonomous web search (Web search tool): The model independently searches the internet for up-to-date information and facts, then incorporates them into its answer [OpenAI, Anthropic/Claude, xAI/Grok, Kimi].
Deep research (Deep research): Investigates a complex topic in depth by autonomously working through multiple web sources over several stages [OpenAI, Google/Gemini].
Web grounding (Web Grounding): Uses real-time search engine results to ground answers in evidence and pull in the latest information [Google/Gemini, Microsoft Copilot, Perplexity].
Raw web search results (Search API): Returns raw, ranked web search results (title, URL, snippet) without any intervening LLM reasoning [Perplexity].
Web fetch tool (Web fetch tool): Directly accesses a web page at a specified URL, loads its content, and extracts the information [Anthropic/Claude].
Le Chat web search (Le Chat Web search): Runs web searches from within the chat interface to retrieve current information for research in Canvas [Mistral].
M365 web grounding (Microsoft 365 Copilot Chat): Uses web search results to research the latest information and perform secure fact-checking [Microsoft Copilot].
AI answers with citations (Sonar API): Researches knowledge on the web in real time and generates "AI answers with citations" [Perplexity].
Fact-checking and double-checking (Double-check): Searches web sources to verify the factual accuracy of a generated answer [Google/Gemini].
Viewing related sources (View related sources): Finds and displays the web sources behind an answer so users can check them directly [Google/Gemini].
Domain-controlled and filtered search (Search Filters / Domain Controls): Restricts web search results to specific domains or sites [Perplexity].
Parallel multi-query search (Multi-query support): Runs multiple search queries in parallel within a single request to return composite web search results [Perplexity].
Real-time data extraction (Real-time data extraction): Pulls real-time data such as news and stock prices out of search results [Perplexity].
Agent API search (Agent API): Conducts research using multiple third-party models together with an integrated web search tool and presets [Perplexity].
Search crawler index retrieval (Perplexity Crawlers): Retrieves information based on web index data collected by the official crawlers [Perplexity].
RAG, vector search, and embeddings
File search (File search): Searches a set of uploaded files, such as PDFs, and extracts the passages relevant to the user's question [OpenAI, Anthropic/Claude].
Retrieval-augmented generation (Retrieval / RAG): Semantically locates information within your own document collection and uses it as the basis for answers [OpenAI, Mistral, Meta/Llama, Anthropic/Claude].
Vector store management (Vector Stores): Stores, manages, and searches document vector data to make retrieval fast and accurate [OpenAI].
Text embeddings (Embeddings API): Converts the meaning of text into numerical vectors to improve the accuracy of semantic search and custom RAG pipelines [OpenAI, Perplexity, Anthropic/Claude].
Context-preserving retrieval (Contextual retrieval): Retains the overall document context when splitting it into chunks, dramatically improving RAG retrieval accuracy [Anthropic/Claude].
Contextualized embeddings (Contextualized Embeddings): Generates context-aware vector data to raise the retrieval accuracy of RAG systems [Perplexity].
Collection search (Collections Search): Performs semantic search within user-created collections (sets of documents) to find the information needed [xAI/Grok].
Collection metadata retrieval (Collection Metadata): Retrieves metadata and property information for RAG collections you have created [xAI/Grok].
Files API retrieval (Files API Retrieval): Reads text files and documents uploaded via the API and searches them for information [OpenAI, Anthropic/Claude, xAI/Grok].
Integrated PDF search (PDF support): Searches and retrieves not only the text in a PDF but also its visual content, such as charts and graphs [Anthropic/Claude].
Direct search over ultra-long context (Ultra-long Context Retrieval): Leverages very large context windows (e.g., 256K) to locate information directly across entire documents, without RAG [Kimi].
Custom agent knowledge search (Custom agent knowledge base): Searches the proprietary knowledge base linked to a custom agent to retrieve business-specific information [Mistral].
Enterprise document search (Vertex AI Search): Instantly searches large volumes of unstructured corporate data with an understanding of meaning [Google/Gemini].
Vector search infrastructure (Vector Search): Infrastructure that delivers scalable semantic search tailored to your data size and requirements [Google/Gemini].
Llama RAG integration (Llama RAG Integration): Connects the model to a vector database, via resources such as the Cookbook, to search external knowledge [Meta/Llama].
Searching internal data and SaaS integrations
M365 business data grounding (Microsoft 365 Data Grounding): Searches across an organization's documents, emails, Teams chats, and other business data to ground answers [Microsoft Copilot].
Google Workspace data search (Connected Apps): Searches and retrieves personal work data and files from Gmail, Google Drive, Docs, and more [Google/Gemini].
MCP server resource search (MCP and Connectors): Uses the Model Context Protocol to retrieve data from external databases and SaaS tools on remote servers via a standardized protocol [OpenAI, Anthropic/Claude, Microsoft Copilot].
Jira/Slack data retrieval (MCP SaaS Retrieval): Searches and extracts Jira ticket information and Slack message history directly via MCP [Anthropic/Claude, OpenClaw].
Google Drive document retrieval (Google Drive Retrieval): Searches for and reads specifications and design documents stored in Google Drive via MCP [Anthropic/Claude].
Direct data retrieval from external APIs (Data retrieval): Works with ChatGPT Actions and similar tools to search and pull the required data directly from specific endpoints, such as internal databases [OpenAI].
Connector-extended search (Connectors): Connect external SaaS systems and proprietary databases to expand the scope of what the agent can research and the data it can retrieve [Microsoft Copilot].
Public websites as knowledge sources (Knowledge sources - Public website): Point the agent at specific public websites and have it research information from them [Microsoft Copilot].
SharePoint knowledge search (Knowledge sources - SharePoint): Connect your organization's SharePoint as a knowledge source and search internal documents within a secure environment [Microsoft Copilot].
Unstructured data extraction (Document AI): Search and process unstructured documents such as forms and invoices, automatically extracting and retrieving the data you need [Google/Gemini].
Unified data catalog search (Dataplex Universal Catalog): Search, manage and retrieve all of your cloud-hosted data across sources from a single place [Google/Gemini].
eDiscovery / legal hold search (eDiscovery / Legal hold): Search and retrieve stored Copilot conversation histories and system logs in preparation for litigation or audits [Microsoft Copilot].
Power Platform connector retrieval (Power Platform Connector Retrieval): Within flows such as Power Automate, search data from connected connectors and pass it to Copilot [Microsoft Copilot].
Code, database and development research
GitHub repository search (GitHub Repository Search): Import or connect GitHub repositories and search directly across the codebase and its history [Google/Gemini, Microsoft Copilot].
Cross-repository private codebase search (Custom Codebase Search): Connect your company's GitHub, GitLab or similar and search for code across a massive private codebase [Google/Gemini].
Code review context retrieval (Code Review Context Retrieval): During code reviews on GitHub and similar platforms, search for and retrieve the context of related PRs and code [Google/Gemini, Anthropic/Claude].
Local codebase search (Claude Code File Search): From a CLI environment, autonomously read an entire local codebase and locate the files and dependencies you need [Anthropic/Claude].
Error trace search (Error Trace Retrieval): Starting from error logs or symptoms in the terminal, trace and search the codebase for the root cause of a bug [Anthropic/Claude].
BigQuery metadata retrieval (BigQuery Data Insights): Search table metadata within a database and retrieve useful queries and information that yield insights [Google/Gemini].
Database query search (Database Search): Search and retrieve information from SQL databases using natural-language instructions [Google/Gemini, Microsoft Copilot].
Code interpreter data retrieval (Code Interpreter Data Retrieval): Load data such as spreadsheets, then run code to search for and extract the information and insights you need from it [Mistral, OpenAI, Microsoft Copilot].
Smart search for API specifications (Smart Search in API hub): A Vertex AI-powered feature that lets developers efficiently search for and retrieve internal API specifications and similar resources [Google/Gemini].
Cloud response log / trace retrieval (Cloud Logging / Trace Retrieval): Search and retrieve system logs and error traces to gain visibility into performance [Google/Gemini].
Autonomous tool search (Tool search): From a large pool of available tools, search for and identify the tool best suited to the task at hand [OpenAI].
File retrieval from sandboxes (Docker Sandbox File Retrieval): Safely search and retrieve, from the outside, execution results and files stored inside a secure container (sandbox) environment [OpenClaw].
Llama Stack retrieval (Llama Stack Retrieval): Search and retrieve proprietary data from self-hosted models through the Llama Stack API layer [Meta/Llama].
Platform, social media and specialized search
X post search (X Search): Search real-time posts on X (formerly Twitter) by keyword or user to pick up trends and facts [xAI/Grok].
Search within specific X accounts (allowed_x_handles): Narrow post searches to specific X accounts (up to 10) and retrieve results from them alone [xAI/Grok].
Search excluding specific X accounts (excluded_x_handles): Exclude specific X accounts from search results when retrieving information [xAI/Grok].
Date-range X search (Date Range): Specify a date range (ISO 8601) to search X posts from a particular period [xAI/Grok].
X thread retrieval (Thread fetch): Fetch the entire conversation flow of a specific thread or post on X and use it as context [xAI/Grok].
X image/video content search (Media Understanding): When searching X, search and interpret not just text but also the content of images and videos [xAI/Grok].
Security threat intelligence search (Security Copilot Threat Retrieval): Search vast volumes of security data and logs to retrieve indicators of cyberattacks and threat intelligence [Microsoft Copilot, Google/Gemini].
Financial data search (Copilot for Finance): Search ERP systems and financial data to retrieve the financial insights and related data the business needs [Microsoft Copilot].
CRM sales data search (Copilot for Sales): Search and retrieve customer information and deal histories from CRM systems such as Dynamics 365 [Microsoft Copilot].
Support knowledge search (Copilot for Service): During customer support, search and retrieve relevant knowledge base articles and past case histories [Microsoft Copilot].
Unified search across the analytics platform (Copilot in Microsoft Fabric): Search and retrieve the data you need from multiple unrelated data sources and consolidate it in the analytics platform [Microsoft Copilot].
Azure cloud resource search (Copilot in Azure): Search and retrieve the status of Azure cloud resources, their operation history and troubleshooting guides [Microsoft Copilot].
Product search for e-commerce (Vertex AI Search for commerce): For retailers, retrieve advanced product search results and recommendations from enterprise data [Google/Gemini].
Telecom subscriber data retrieval (Telecom Subscriber Insights): For the telecommunications industry, search, ingest and retrieve subscriber data [Google/Gemini].
Geospatial data retrieval (Geospatial Data Retrieval): Search and retrieve Earth observation data and satellite imagery from sources such as Google Earth Engine [Google/Gemini].
Conversation history, context and memory search
Prompt cache search (Prompt caching): Quickly search and pull past input files and context from a cached long context, without reprocessing it [OpenAI, Anthropic/Claude, xAI/Grok].
Chat thread history retrieval (Conversation History / Threads): An API feature for searching and retrieving context and past message information from previous chat threads [OpenAI].
Local session history search (sessions_history): The agent autonomously looks back through past exchanges and other session histories to search for and retrieve the context it needs [OpenClaw].
Local session list retrieval (sessions_list): The agent searches for and retrieves a list of all active sessions that exist in its own workspace [OpenClaw].
Custom instruction and memory search (Personalization retrieval): Search and extract user preferences and context from past chat history and custom instructions, and reflect them in responses [Google/Gemini].
Visual retrieval of desktop screen information (Computer Use Desktop Retrieval): Visually inspect the desktop environment and autonomously search for and retrieve information from on-screen apps and browsers [Anthropic/Claude].
Moderation data retrieval (Moderation Retrieval): Use a safety model to retrieve verdicts and flag data indicating whether text violates policies [OpenAI, Meta/Llama].
File and local system search
Direct local file reading (read tool): The agent reads files and documents stored on the local device directly to retrieve information [OpenClaw, Anthropic/Claude].
Process and system state retrieval (Process / Bash Retrieval): Runs shell commands to look up and retrieve the state of the local system and information about running processes [OpenClaw, Anthropic/Claude].
Mobile node state retrieval (Mobile Node Retrieval): Looks up and retrieves device sensor data and system state from a paired iOS/Android node [OpenClaw].
File listing (List files): Retrieves a list of all files uploaded to the API or workspace [OpenAI, xAI/Grok, Anthropic/Claude].
File content retrieval (Retrieve file content): Downloads and retrieves the actual contents of a specific stored file [OpenAI, xAI/Grok].
Upload status retrieval (Files Upload Status): Looks up and retrieves status information for large files whose upload is still in progress [OpenAI, xAI/Grok].
Vector store file retrieval (Retrieve vector store file): Looks up and retrieves a specific file, or its status, stored within a vector store [OpenAI].
File batch retrieval (Retrieve a file batch): Looks up and retrieves the batch-processing status of a set of files added to a vector store in bulk [OpenAI].
Retrieving metadata, administrative logs, and evaluation data
Run log search (Retrieve runs): Searches for and retrieves the execution logs (Runs) of threads executed via the Assistants API and similar services [OpenAI].
Batch result retrieval (Retrieve a batch): Looks up and retrieves the results and status information of asynchronous batch jobs [OpenAI, xAI/Grok].
Message batch result retrieval (Message Batches Retrieval): Looks up and retrieves the status and results of messages processed in bulk [Anthropic/Claude].
Workspace audit log retrieval (Audit Logs Retrieval): Lets administrators search for and retrieve audit logs covering API key usage history, user actions, and more [OpenAI, xAI/Grok].
Administrator data retrieval (Admin API Data Retrieval): Lets administrators look up and retrieve workspace permission settings and API key usage data [Anthropic/Claude, OpenAI].
Agent performance analytics retrieval (Agent Analytics Retrieval): Looks up and retrieves information for analyzing agent health, conversation logs, and performance data [Microsoft Copilot].
Billing and usage retrieval (Billing & Usage Retrieval): Looks up and retrieves API usage, costs, and billing information for each workspace [Mistral, xAI/Grok, OpenAI].
API quota and rate limit retrieval (Rate Limits Retrieval): Retrieves the status of a user's current API quota and rate limits [Perplexity, OpenAI].
System status retrieval (System Status Retrieval): Retrieves the overall operational status of the API system via the API itself [Perplexity, Anthropic/Claude].
Webhook event retrieval (Webhook event retrieval): Retrieves and parses event information and payloads sent from external systems via webhooks [OpenAI, OpenClaw].
Source metadata retrieval for generated images (Citations Retrieval): Retrieves the citation metadata for the images that served as the basis for an answer produced through web search [Perplexity, xAI/Grok].
Model evaluation report retrieval (Vertex AI Model Evaluation Retrieval): Looks up and retrieves analysis results for objective evaluation metrics of generative AI models (such as BLEU and ROUGE) [Google/Gemini].
Evaluation data search (Retrieve an eval): Searches for and retrieves past test sets and evaluation data (Evals) used for objective model evaluation [OpenAI].
Cloud quota retrieval (Cloud Quotas Retrieval): Looks up and retrieves allocation (quota) information for all services running in the cloud [Google/Gemini].
Model listing (List models): Retrieves a list of available AI models and their details via the API [OpenAI, xAI/Grok, DeepSeek].
Capabilities related to "reasoning"
These are capabilities for breaking complex tasks down into smaller steps and carrying out logical thinking and planning, such as "because A, therefore B."
Reasoning models and thinking processes
Reasoning-specialized models (Reasoning models): Solve complex tasks using a step-by-step logical thinking process (Chain of Thought) rather than an intuitive, immediate response [OpenAI, DeepSeek, xAI/Grok, Kimi]
Reasoning effort control (Reasoning effort): Lets you specify the depth of thinking ("low", "medium", "high", etc.) in the API request to control how much compute is devoted to reasoning [OpenAI, OpenClaw]
Extended thinking and reasoning (Extended thinking): Performs deep logical reasoning on complex problems by working through a step-by-step thinking process over an extended period [Anthropic/Claude]
Adaptive thinking (Adaptive thinking): The model autonomously interprets the complexity and difficulty of a task and dynamically adjusts its level of reasoning (thinking) [Anthropic/Claude]
Managing reasoning effort and task budgets (Task budgets / Effort): Lets you set the effort and compute budget devoted to reasoning, controlling the depth and cost of the analysis process [Anthropic/Claude]
Thinking mode (Thinking Mode / deepseek-reasoner): Unfolds an explicit thinking process in response to complex prompts to perform deeper logical analysis [DeepSeek]
Fast reasoning models (Fast-reasoning models): Optimize the balance between tool-calling accuracy and reasoning speed to perform analysis and logical reasoning with low latency [xAI/Grok]
Explicit generation of the thinking process (Reasoning Trace generation): Outputs, as text, the trail of thought showing how the model reasoned "in its head" before producing the final answer [OpenAI, DeepSeek]
Multi-step reasoning models (Thinking Models): Reason through logical solution steps, in multiple stages, for complex business problems and data analysis tasks [Kimi]
Dynamic command control of thinking level (/think command): Lets you dynamically adjust the depth of the agent's reasoning from the local environment using chat commands and similar controls [OpenClaw]
Data analysis and computational reasoning
Code interpreter (Code interpreter): The AI autonomously generates and executes Python or other code to carry out mathematical calculations, dataset analysis, and chart creation in a logical manner [OpenAI, Microsoft Copilot, Mistral]
Local code execution for reasoning (Code execution tool): The AI itself executes code in a local or sandboxed environment to reason through numerical aggregation and statistical analysis [Anthropic/Claude, xAI/Grok]
Automated database insight reasoning (BigQuery Data Insights): Interprets database metadata to automatically reason out and generate insights and SQL queries useful for data analysis [Google/Gemini]
SQL query intent and performance analysis (Database Development Assistance): Reasons through the logic of complex SQL, explains it in plain language, and analyzes optimization proposals to improve performance [Google/Gemini]
Spreadsheet data analysis and reasoning (Le Chat Code Interpreter): Loads spreadsheet data and executes code to analyze trends, generate charts, and derive insights [Mistral]
Integrated insight reasoning across multiple data sources (Copilot in Microsoft Fabric): Uncovers connections between multiple, seemingly unrelated data sources to automatically reason out and generate valuable business insights [Microsoft Copilot]
Reasoning about optimal data visualization (Power BI Copilot): Based on the results of data analysis, reasons out the dashboard layouts and chart representations that users will find easiest to understand [Microsoft Copilot]
Insight reasoning from customer data (Dynamics 365 Copilot): Analyzes customer data and deal histories in the CRM to infer the best sales actions and customers' latent needs [Microsoft Copilot]
Reasoning in notebooks for data science (Colab Enterprise): Seamlessly supports the reasoning and execution of Python code in data analysis and ML model training workloads [Google/Gemini]
Batch reasoning over large datasets (Batch processing reasoning): Process large volumes of data-analysis and reasoning tasks asynchronously in bulk, extracting insights while keeping costs down [OpenAI, Anthropic/Claude, xAI/Grok]
Reasoning for coding, development, and debugging
Inferring latent bugs across an entire codebase (Codebase understanding): Reason about potential bugs and vulnerabilities using the context of the whole project rather than a single file [Google/Gemini, Anthropic/Claude]
Inferring missing logic (FIM Completion): Understand the surrounding code on both sides (Fill-in-the-Middle) and infer and generate the logical block of code that is missing in between [DeepSeek, OpenAI]
Inferring subsequent logic from a prefix (Chat Prefix Completion): Take a supplied code prefix, interpret the context so far, and infer how the logic should continue [DeepSeek]
Inferring dependencies across a project (Cross-file reasoning): Read a codebase from a CLI or similar environment and infer the logical dependencies between multiple files and the overall system structure [Anthropic/Claude, Kimi]
Reasoning about and designing architecture (Architecture reasoning): Infer the optimal system architecture from a user's requirements and design the foundation for complex program code [OpenAI, Google/Gemini]
Inferring root causes from error traces (Error trace reasoning): Logically trace terminal error logs and stack traces back to the root cause of a bug in the code [Anthropic/Claude, Google/Gemini, OpenAI]
Inferring the intent behind commit and PR diffs (PR diff reasoning): Analyze the diff of changed code, infer the logical intent behind why the change was made, and turn it into a message [Anthropic/Claude, Google/Gemini]
Analyzing the causes of app crashes (App crash analysis): Summarize and analyze crash logs from Firebase and similar services, and logically infer the root cause of the bug and the troubleshooting steps [Google/Gemini]
Inferring structure from a project overview (Scaffolding reasoning): Given nothing more than a natural-language overview typed into the terminal, logically infer the directory structure and initial files the project needs [Mistral]
Inferring the next step while programming (Code completion reasoning): As a developer writes code, read the context within the project and automatically infer the code logic needed next [Microsoft Copilot, Google/Gemini]
Reasoning for research and complex problem-solving
Integrative reasoning for deep research (Deep research reasoning): Run web searches repeatedly across multiple stages, logically synthesize multiple pieces of information, and reason toward deep insights and conclusions [OpenAI, Google/Gemini]
Factual reasoning from synthesized search results (Web-grounded reasoning): Gather the latest knowledge from the web, reason about the relationships and contradictions among multiple sources, and derive accurate answers [Perplexity, OpenAI]
Analyzing the reliability and relevance of sources (Search API filtering): Without generating text with an LLM, score and reason about the semantic relevance and trustworthiness of raw search results [Perplexity]
Deep logical reasoning on hard scientific and mathematical problems (Scientific/Math reasoning): From text involving advanced mathematical computation (e.g., DeepSeek-Math) or physical laws, reason through rigorous logical derivations and theorem-proving processes [DeepSeek, xAI/Grok, Google/Gemini]
In-depth analysis of research papers and academic data (Analyze research papers): Read and interpret research papers and academic data in complex formats, and logically extract and analyze scientific insights [Meta/Llama, Anthropic/Claude]
Big-picture analysis of extremely long contexts (Long-context analysis): Analyze an entire context spanning millions of tokens to reason from a macro perspective and produce comprehensive summaries [Meta/Llama, Google/Gemini, Kimi]
Planning agent workflows (Agent API workflow planning): Reason about and plan the optimal execution steps for complex research workflows that combine external models and web search tools [Perplexity, OpenAI, Anthropic/Claude]
Integrated multi-document analysis and report structuring (Canvas analysis): Analyze search results and multiple uploaded documents side by side, and infer the logical structure of a report or proposal [Mistral, OpenAI]
Inferring character psychology and analyzing relationships (Character mind map reasoning): From a novel or other long-form context, infer characters' thought processes and interpersonal relationships and organize them into a mind map [Meta/Llama]
Fusing predictive analytics with generative AI (Predictive AI integration): Combine numerical predictions from conventional machine learning with the logical reasoning capabilities of generative AI to perform advanced business reasoning [Google/Gemini]
Visual and multimodal reasoning
Logically reasoning about relationships between objects in an image (Visual reasoning): Visually infer and interpret how the objects in an image relate to one another and the context behind the scene [OpenAI, Anthropic/Claude, Google/Gemini, xAI/Grok, Kimi]
Inferring insights from charts and graphs (Chart/Graph analysis): Visually infer numerical insights, logical trends, and correlations from graph data presented as images [Anthropic/Claude, Meta/Llama, OpenAI]
Multimodal mixture-of-experts reasoning (Mixture-of-Experts Vision-Language): For inputs that mix images and text, infer and select the most appropriate experts within the model to achieve advanced understanding [DeepSeek]
Inferring visual causality and logical structure (Visual Causal Flow): Infer information flows and logical cause-and-effect relationships from complex diagrams, flowcharts, and optical character recognition (OCR) output [DeepSeek]
Physically reasoning about 3D structure and lighting (ReLi3D / 3D reasoning): From a single 2D image, physically infer and reconstruct an object's 3D shape, the lighting conditions of its environment, and the reflectance properties of its materials [Stability AI]
Visually inferring motion and space from an image (SViM3D / Video generation reasoning): Take a single still image as input, visually infer the natural motion and 3D material structure that should exist within it, and turn it into video [Stability AI]
Multimodal reasoning over social media posts (Social media multimodal reasoning): When searching X, reason logically not only over the text of posts but also over the visual context of any images and videos they contain [xAI/Grok]
Visually reasoning about desktop UI structure (Computer use vision): Visually understand a PC screen and infer which UI elements to operate, and in what sequence, to accomplish a goal [Anthropic/Claude, OpenAI]
Inferring code structure from UI mockups (UI to HTML reasoning): Visually interpret hand-drawn sketches or UI mockup images and infer the corresponding logical front-end code structure [Google/Gemini, OpenAI]
Inferring video context and temporal causality (Video analysis reasoning): Analyze the frame transitions and audio of an uploaded video to infer temporal cause-and-effect relationships and the intent behind actions [Google/Gemini, xAI/Grok]
Logically reasoning about mask regions for image editing (Masking understanding): In image editing, compare the region defined by a supplied mask image against the intent of the user's prompt to infer the optimal edit [OpenAI]
Inferring biometrics and age from selfies (Age assurance vision): Analyze facial biometric information in a selfie to visually infer and determine a user's age (e.g., whether they are 18 or older) [C.AI, OpenAI]
Integrated reasoning across multimodal information (Integrated multimodal reasoning): Seamlessly combine inputs from different modalities—text, images, video, audio, and more—to perform unified logical reasoning [Google/Gemini, Kimi, OpenAI]
Reasoning for systems, security, and operations
Machine-speed security threat reasoning (Security threat reasoning): Rapidly analyze vast volumes of security logs to logically infer signs of threats such as cyberattacks and their attack patterns [Microsoft Copilot]
Diagnostic reasoning about cloud operations (Cloud infrastructure diagnostics): Analyze cloud infrastructure logs and metrics to diagnose the operational state of a system and infer the steps to resolve bottlenecks [Microsoft Copilot, Google/Gemini]
Reasoning about harmfulness for content moderation (Content moderation reasoning): Analyze the content of text and images to logically infer and score whether it violates policy or carries potential harm [OpenAI, Meta/Llama]
Monitoring for data skew and model degradation (MLOps monitoring): Analyzes skew and drift in the input data fed to machine learning models and objectively assesses how far a model has degraded [Google/Gemini]
Business-context reasoning across organizational data (Enterprise context reasoning): Reads across internal SharePoint sites, email, chat and other sources, organizes the information logically, and infers the current state of work or the answer to a question [Microsoft Copilot]
Code-level interpretation of API policies (API policy reasoning): Interprets API specifications and security-policy code in enterprise environments and infers the intent behind their configuration [Google/Gemini]
Architecture reasoning from source-editor context (Editor context reasoning): Infers the optimal deployment configuration and resources from the code written in a cloud infrastructure source editor [Google/Gemini]
Holistic age inference from user activity signals (Age assurance signals): Analyzes signals such as login information and on-platform activity history in combination to infer a user's age bracket [C.AI]
Guardrail-compliant safe-action reasoning (Guardrails reasoning): When an agent makes tool calls, it autonomously checks them against predefined safety and ethics policies and reasons out a safe course of action [OpenAI, Microsoft Copilot, Anthropic/Claude]
Visualizing debugging reasoning (Developer mode trace / Trace logs): Outputs the agent's internal network requests and reasoning logs so developers can analyze and pinpoint system bottlenecks [OpenAI, OpenClaw]
Tool selection and agent-behavior reasoning
Autonomous reasoning over tool choice and parameters (Tool choice reasoning): Analyzes the user's instruction and autonomously decides which of several supplied external tools to call, and with what arguments (JSON) [OpenAI, Anthropic/Claude, Google/Gemini, xAI/Grok, Meta/Llama, Mistral, DeepSeek]
Logical design of agent flows (Agent flow reasoning): From natural-language instructions, logically designs the complex processing steps and conditional branches of the workflow an agent should execute [Microsoft Copilot]
Task delegation to sub-agents (Orchestration reasoning): A main agent breaks a complex task into fine-grained steps and plans which specialized sub-agent each should be delegated to [OpenAI, Anthropic/Claude]
Dynamic agent routing (Multi-agent routing): Determines which workspace or agent context should apply to each incoming channel or user and routes traffic accordingly [OpenClaw]
Autonomous reasoning over persona and action plans (Persona/Workspace reasoning): Reads configuration files in the workspace (such as AGENTS.md), analyzes project requirements and user preferences, and works out a plan of action [OpenClaw, Anthropic/Claude]
Schema reasoning for structured data output (Structured output reasoning): Maps results inferred from unstructured data onto a strict JSON schema, outputs them reliably, and hands them off to a data analytics platform [OpenAI, Anthropic/Claude, Google/Gemini, Meta/Llama, DeepSeek]
Intent inference from conversation context and history (Context management reasoning): Logically takes into account past exchanges and the content of previously generated outputs when reasoning about subsequent responses and actions [OpenAI, Anthropic/Claude, Google/Gemini]
Interpreting retrieved data to reach a final answer (Action retrieval reasoning): Interprets raw database information retrieved from external APIs and reasons out the final answer to the user's question [OpenAI, Microsoft Copilot]
Topic triggering from user utterances (Topic trigger reasoning): Infers the true intent (topic) behind a user's vague utterance and launches the appropriate agent workflow [Microsoft Copilot, Google/Gemini]
Retrospective reasoning over session history (Session history reasoning): The agent autonomously looks back through past exchanges and the history of other sessions to infer and extract the information needed for the current task [OpenClaw]
Learning user-specific instructions and preferences (Personalization reasoning): Infers a user's preferences and work context from past chat history and custom instructions to generate personalized responses [Google/Gemini, OpenClaw]
Emotion and context inference from real-time voice (Voice/Emotion reasoning): From ultra-low-latency voice input, infers not just the meaning of the words but also the user's emotion, tone and conversational context in real time, and responds accordingly [OpenAI, xAI/Grok, Microsoft Copilot]
Optimizing, evaluating and self-improving reasoning
Fine-tuning open models (Fine-tuning recipes): Fine-tunes models such as Llama for specific use cases [Meta/Llama]
Prompt caching (Prompt caching): Caches very long prompts to reduce API latency and cost [OpenAI, Anthropic, grok]
Self-reflective automatic prompt optimization (Prompt optimizer reasoning): The model objectively analyzes its own reasoning process and autonomously works out and rebuilds the prompt structure that performs best for solving the task [OpenAI, Anthropic/Claude]
Automatic inference of prompt variables (Prompt templating): Automatically identifies and extracts variables from the user's input text and converts it into a prompt template structure that is easy to embed in a system [Anthropic/Claude]
Objective evaluation of model reasoning (Model evaluation / Evals): Uses a defined test set to analyze and score the accuracy and logical soundness of an AI model's reasoning in a data-driven way [OpenAI, Google/Gemini, Anthropic/Claude]
Analytical reasoning over agent performance (Agent analytics reasoning): Analyzes an autonomous agent's conversation logs and health, and infers improvement tips for lowering error rates and raising user satisfaction [Microsoft Copilot]
Comparing how parameter changes alter reasoning (Playground analysis): Adjusts prompts and reasoning parameters in a playground to compare and analyze the logical changes in model output [Mistral, OpenAI]
Learning and adapting reasoning patterns (Reasoning fine-tuning / SFT): Teaches a model domain-specific reasoning patterns tailored to particular business data through supervised fine-tuning [OpenAI, Google/Gemini]
Semantic-distance reasoning over text (Embeddings reasoning): Converts the meaning of text into multidimensional vectors and mathematically infers and analyzes semantic similarity and distance between pieces of information, for use in RAG [OpenAI, Anthropic/Claude, Perplexity, Google/Gemini]
Reasoning about semantic connections in long-context retrieval (Contextual retrieval reasoning): When documents are split into chunks for retrieval, preserves the context of the whole document and the connections between pieces of information during inference and extraction [Anthropic/Claude, Perplexity]
Skipping redundant reasoning via context caching (Context Caching): Caches large input contexts on the system side, avoiding unnecessary logical computation so that subsequent reasoning proceeds efficiently [DeepSeek]
Fast reasoning through predicted outputs (Predicted Outputs reasoning): Leverages cached or already-known output content and has the model predict it, optimizing inference latency in arriving at results [OpenAI]
Reasoning about correctness criteria for test results (Evaluations grading): The LLM itself reads the answers from automated tests and logically interprets whether they are correct or incorrect against human-defined criteria [OpenAI, Anthropic/Claude]
Infrastructure and hardware inference optimization
Quantization and memory optimization for inference (Quantization reasoning): Runs and offloads advanced inference even on memory-constrained devices by using models optimized through quantization [Stability AI/Diffusers]
Inference speed boost via PyTorch compilation (torch.compile reasoning): Compiles and optimizes the model's inference graph to dramatically improve execution speed at the software level [Stability AI/Diffusers]
NVIDIA TensorRT-optimized inference (TensorRT optimized reasoning): Pushes the inference speed of image-generation and language models to the limit for NVIDIA GPU architectures when running computations [Stability AI/Diffusers]
AMD GPU-optimized inference (AMD Optimized reasoning): Applies model tuning specialized for inference on AMD hardware to maximize hardware performance [Stability AI/Diffusers]
Eliminating storage-bandwidth bottlenecks in agentic inference (DualPath inference): An advanced inference computation technique that removes storage-bandwidth bottlenecks during inference in agentic LLMs [DeepSeek]
Kernel-optimized inference with FP8 and latent attention (DeepGEMM / FlashMLA): Accelerates model inference using FP8-precision GEMM kernels and efficient multi-head latent attention [DeepSeek]
Domain-specific, search, and information reasoning
Real-time trend reasoning over social data (Social trend reasoning): Infers and extracts trending topics and social insights in real time from the latest posts on social media platforms such as X (Twitter) [xAI/Grok]
Reasoning about the drivers of marketing results (Campaign summarization reasoning): Analyzes performance data from messaging campaigns and similar initiatives to infer the factors behind success or failure, along with concrete actions for improvement [Google/Gemini]
Semantic reasoning over custom collections (RAG data reasoning): Extracts metadata from user-uploaded document sets (collections) and infers semantic insights from them [xAI/Grok, Mistral]
Specialized reasoning over domain-specific data (Domain controls reasoning): Works from search results restricted to a specific domain, such as healthcare or finance, to infer context and extract information where deep expertise is required [Perplexity]
Reasoning about source reliability (Citation reliability reasoning): When generating answers from search results, infers which sources are most reliable and relevant and attaches them as citations [Perplexity, xAI/Grok, OpenAI]
Reasoning over the conversational flow of an entire thread (Thread fetch reasoning): Traces back through a specific thread or post on social media to build a holistic understanding of the context of the whole conversation [xAI/Grok]
Semantic reasoning over enterprise documents (Enterprise Search reasoning): Infers context and semantic relationships across vast volumes of internal unstructured data to search and summarize it instantly [Google/Gemini, Microsoft Copilot]
Traffic reasoning for telecommunications networks (Telecom Data Fabric reasoning): Analyzes carrier traffic data and subscriber information to infer behavioral patterns and opportunities for network optimization [Google/Gemini]
Spatial reasoning over geospatial data (Geospatial Analytics reasoning): Infers and extracts terrain changes and spatial insights from satellite imagery and map data such as Google Earth Engine [Google/Gemini]
Capabilities related to "generation"
These are the capabilities that build on the results of search and reasoning to ultimately produce output that is valuable to people: text, images, code, and more.
Text, document, and content generation
Natural-language text generation (Text Generation): Generates essays, articles, reports, and other text based on a prompt [OpenAI, Google/Gemini, Anthropic/Claude, Meta/Llama, xAI/Grok, Mistral, DeepSeek, Kimi, Perplexity]
Research report generation with citations (Citation-backed Generation): Generates reports grounded in the latest web search results and backed by accurate, fact-based citations [Perplexity, OpenAI, Google/Gemini]
Summary generation from multiple files (Summary Generation): Reads across large sets of documents or uploaded files and generates a summary that extracts the key points [OpenAI, Anthropic/Claude, Google/Gemini, Mistral, Kimi]
Multi-turn dialogue generation (Multi-round Conversation): Remembers the conversation history and generates coherent, natural dialogue that maintains context [OpenAI, Anthropic/Claude, Google/Gemini, xAI/Grok, DeepSeek, Kimi]
Quiz and study material generation (Quiz/Flashcards Generation): Automatically generates study quizzes and memorization flashcards from the contents of documents or notebooks [Google/Gemini]
Illustrated picture book generation (Illustrated Storybook): Generates picture book content by pairing story text with illustrations (images) that match the narrative [Google/Gemini]
Marketing copy generation (Campaign generation): Generates compelling, effective ad text and copy based on campaign data and product information [Google/Gemini, Microsoft Copilot]
Insight and recommendation generation (Actionable Recommendations): Analyzes crash logs or campaign results and generates concrete improvement measures and troubleshooting steps [Google/Gemini]
Code explanation generation (Code Explanation): Deciphers the logic and intent of complex, hard-to-read program code and generates a clear explanation in natural language [Google/Gemini, Anthropic/Claude, OpenAI, Microsoft Copilot]
Multilingual translation generation (Translation Generation): Understands the input text and generates accurate translations into a wide range of specified languages [OpenAI, Google/Gemini, Anthropic/Claude, Meta/Llama, DeepSeek]
Persona- and tone-matched text generation (Persona generation): Generates text that mimics a specified tone or character (a pirate, an expert, and so on) [OpenAI, Anthropic/Claude, OpenClaw]
Meeting minutes and action item generation (Meeting summaries): Generates organized decisions and to-do lists from audio recordings or transcripts [Microsoft Copilot, OpenAI, Anthropic/Claude]
Customer support reply generation (Customer support reply): Generates the most appropriate support reply email based on the inquiry and the knowledge base [Microsoft Copilot, Anthropic/Claude]
Explicit generation of the thinking process (Reasoning Trace generation): Before delivering a final answer, generates the thinking process itself as text, showing how the model worked through its logic [OpenAI, DeepSeek]
Resume and cover letter generation (Resume/Cover letter generation): Generates customized resumes and cover letters based on the user's employment history and information about the target company [OpenAI, Anthropic/Claude]
Idea and brainstorming generation (Ideation): Generates a diverse list of ideas for project planning or problem solving from a user's abstract request [OpenAI, Google/Gemini, Anthropic/Claude, Meta/Llama]
Domain-specific expert answer generation (Domain-controlled Generation): Generates highly specialized answers based solely on search results restricted to a specific domain, such as healthcare or finance [Perplexity]
Image generation, design, and editing
New image generation from text (Text-to-Image): Generates high-quality images from scratch based on natural-language prompts [OpenAI, Google/Gemini, xAI/Grok, Stability AI, Midjourney]
Image-to-image generation (Image-to-Image): Combines a source image with text instructions to generate a new image with a transformed style or altered content [Stability AI/Diffusers, OpenAI]
Partial image editing and inpainting (Inpainting / Masking): Uses a specified mask image to regenerate and edit only a particular region of an image according to a new prompt [OpenAI, Stability AI/Diffusers]
Image variation generation (Create Variation): Generates multiple alternative versions of an image while preserving the composition and style of the original [OpenAI, Stability AI]
Image outpainting (Outpainting / Extend): Infers the background or scenery beyond the borders of an existing image and extends it naturally [OpenAI, Stability AI]
Super-resolution and upscaling (Super Resolution): Infers the missing detail in low-resolution or generated images and enlarges them to high resolution [Stability AI/Diffusers]
Latent-space upscaling (Latent Upscale): A generation process that increases image resolution within the latent space [Stability AI/Diffusers]
Lighting recalculation and generation (ReLi3D): Recalculates the lighting of a 3D model or image and regenerates the image under different lighting conditions [Stability AI]
Color correction and color guidance generation (ReSWD): Regenerates an image's color correction and tonal balance using distribution-matching techniques [Stability AI]
Image generation guided by edges and pose skeletons (ControlNet): Uses control images such as outline sketches or human poses to generate images with a locked-in composition [Stability AI/Diffusers]
Unconditional image generation (Unconditional Image Generation): Generates random images without any conditioning input such as a text prompt [Stability AI/Diffusers]
Character-consistent generation (Character Reference): Uses reference images of a specific character to generate that character in different scenes while preserving its defining traits [Midjourney, OpenAI]
Style reference generation (Style Reference): Generates new images that mimic the visual style of a reference image [Midjourney]
Omni reference generation (Omni Reference): Generates the same character or object consistently across multiple images [Midjourney]
Multi-turn image generation (Multi-turn image generation): Carries over the context of previous image generations (via the Previous response ID) to refine and generate images conversationally [OpenAI]
Image generation with format and compression settings (Image formatting): Produces image files in a specified format such as PNG, JPEG, or WebP, with a chosen compression level applied [OpenAI]
Video, 3D, and multimodal generation
Text-to-video generation (Text-to-Video): Generates high-quality video footage with motion from a text prompt [OpenAI, Google/Gemini, xAI/Grok]
Image-to-video generation (Image-to-Video): Takes a single still image as input and generates a video that brings it to life with natural motion [Stability AI, OpenAI]
Video generation with synced audio (Video with Synced Audio): Generates video from text that includes not only motion but also audio synchronized to the footage [OpenAI]
Video remixing and editing (Remix / Edit Video): Alters the style or elements of existing video content and regenerates it as a new, remixed video [OpenAI]
3D footage generation (Stable Virtual Camera): Generates a 3D visual representation with camera movement from a single input image [Stability AI]
3D material generation (SViM3D): Uses a diffusion model to generate materials and natural motion for 3D objects from a single image [Stability AI]
Character insertion into video (Create Character / Video): Generates a character within a video based on specified character information [OpenAI]
Mixed-modality output generation (Mixture of Modalities / Janus): Generates output in which images and text are seamlessly interleaved within a single model [DeepSeek]
Speech and audio generation
Text-to-speech / speech synthesis (Text-to-Speech / TTS): Converts text into natural, expressive human-sounding speech [OpenAI, Google/Gemini, xAI/Grok]
Real-time voice conversation (Realtime Audio Generation): Accepts text and audio input with extremely low latency and generates natural spoken responses in real time [OpenAI, xAI/Grok, Microsoft Copilot]
Podcast-style audio generation (Audio Overviews): Turns documents and other source material into podcast-style audio featuring a conversation between multiple hosts [Google/Gemini]
Sound effects synced to video (Foley Control): Analyzes the movement and scenes in a video and generates perfectly matched, natural sound effects in sync with it [Stability AI]
Speech translation output (Speech Translation): Translates input speech into another language and synthesizes the result as spoken audio [OpenAI]
Music and audio generation (Audio generation): Generates music, melodies, ambient sounds, and more from text [Google/Gemini, Stability AI/Diffusers]
Speaker-diarized timestamps (Diarize Output): Generates a timestamped record of who spoke when from conversational audio data [OpenAI]
Interactive voice agent responses (IVR Generation): Generates the utterances an agent uses to converse with customers over voice channels such as the telephone [Microsoft Copilot]
Voice consent records (Voice Consents): Creates the user consent records required when using certain speech synthesis APIs [OpenAI]
Programming and code generation
Code generation from natural language (Code Generation): Generates program code such as functions and classes based on a developer's prompt [OpenAI, Google/Gemini, Anthropic/Claude, Mistral, Microsoft Copilot]
Project-wide file editing and generation (Multi-file Code Generation): Reads the entire codebase and generates implementation code for new features that spans multiple files [Anthropic/Claude, Google/Gemini]
Real-time code completion (Code Completion): Reads the surrounding context as a developer writes in the editor and automatically suggests and generates the next piece of code [Microsoft Copilot, Google/Gemini]
Fill-in-the-middle completion (Fill-in-the-Middle Completion): Understands the code before and after a gap and infers and generates the missing logic in between [DeepSeek, OpenAI]
Chat prefix completion (Chat Prefix Completion): Picks up from a partially written prefix of code or text supplied by the user and generates what follows [DeepSeek, OpenAI]
Project scaffolding (Project Scaffolding): Generates a complete directory structure and initial set of files from nothing more than a natural-language outline [Mistral]
SQL query generation (SQL Generation): Generates complex SQL queries from natural-language instructions with an understanding of the database schema [Google/Gemini, Microsoft Copilot]
GraphQL schema and mutation generation (Firebase Data Connect): Generates data schemas and GraphQL queries for applications [Google/Gemini]
Automatic commit message generation (Commit message generation): Understands the diff of changed code and generates an appropriate Git commit message [Anthropic/Claude, Google/Gemini]
Pull request (PR) description generation (PR description generation): Summarizes the implemented features or fixes and autonomously generates a PR title and description [Anthropic/Claude, Google/Gemini]
Python script generation for code-based analysis (Code Interpreter generation): The AI writes and runs its own Python code to perform data analysis and computations [OpenAI, Microsoft Copilot, Mistral]
Patch (diff) generation (Apply Patch): The agent generates a patch for a specific part of the codebase and applies it autonomously [OpenAI, Google/Gemini]
Shell command generation (Shell Command Generation): Generates the appropriate bash or shell commands to run in the terminal based on the user's goal [Anthropic/Claude, OpenClaw, OpenAI]
Unit test generation (Unit Test Generation): Generates appropriate test code, including mocks and assertions, for untested source code [Anthropic/Claude, Google/Gemini]
API specification (OpenAPI/Swagger) generation (API specification generation): Generates API specifications from existing backend code [Google/Gemini]
Direct local file generation (File Write / Create): The agent autonomously uses write and edit tools to generate scripts and documents directly on the user's device [OpenClaw, Anthropic/Claude]
Structured data, vector, and metadata generation
Structured JSON output (Structured Output): Generates precisely formatted data from unstructured text, strictly following a defined JSON schema [OpenAI, Anthropic/Claude, Google/Gemini, Meta/Llama, DeepSeek, xAI/Grok]
Text embedding generation (Embeddings Generation): Encodes the meaning of text as a high-dimensional numerical vector, forming the foundation for RAG and semantic search [OpenAI, Anthropic/Claude, Perplexity, Google/Gemini]
Contextualized Embeddings: Vectorizes text chunks while retaining the context of the entire document, producing higher-precision embeddings [Perplexity, Anthropic/Claude]
Batch Generation: Uses asynchronous batch APIs to produce large-scale text generation and structured-data extraction in a single bulk run [OpenAI, Anthropic/Claude, xAI/Grok]
Tool / Function Calling Generation: Interprets user instructions logically and generates the arguments (such as JSON data) needed to invoke external APIs and tools [OpenAI, Anthropic/Claude, Google/Gemini, Meta/Llama, xAI/Grok, DeepSeek, Mistral]
Vector Store Indexing: Automatically generates chunks and vectors from a set of uploaded files and builds a searchable index from them [OpenAI]
Predicted Outputs: Lets the model anticipate output that is already known (such as the bulk of a source file), dramatically reducing latency and speeding up response generation [OpenAI]
Moderation Output: Evaluates the content of inputs and model outputs and generates scores or flag data indicating policy violations or harmful content [OpenAI, Meta/Llama, Google/Gemini]
Usage/Tokens Output: Generates usage data for each API request, including the number of input and output tokens consumed by the prompt and generation, as well as cache hit rates [OpenAI, Anthropic/Claude, DeepSeek, xAI/Grok]
Bounding box generation: Detects the position of specific objects or people within an image and outputs their coordinate data as text or JSON [OpenAI, Google/Gemini]
UI, design, and workspace generation
Image Mock up to HTML: Generates the corresponding HTML/CSS and similar code from hand-drawn sketches or images of UI designs [Google/Gemini, OpenAI, Anthropic/Claude]
Artifacts / Claude Cowork: Generates React components, SVG, HTML, and more directly within the chat interface and renders them as live, dynamic previews [Anthropic/Claude, OpenAI]
Live Canvas / A2UI: Lets the agent autonomously generate and lay out interactive, operable UI on screen based on the user's instructions [OpenClaw]
Canvas: Generates proposals, planning documents, and app scaffolds inside a dedicated collaborative workspace (Canvas) [Mistral, OpenAI]
Visualizations: Automatically generates easy-to-understand graphs and charts from analyzed data [Microsoft Copilot, Mistral]
Power BI dashboards: Consolidates information from enterprise data warehouses and generates the design and layout of business dashboards [Microsoft Copilot]
Character Mind Map: Interprets the thoughts and relationships of characters in a long narrative and generates a visual mind-map structure [Meta/Llama]
Agent, automation, and prompt generation
Fine-tuning recipes: Fine-tunes open models such as Llama for specific use cases [Meta/Llama]
Prompt caching: Caches lengthy prompts to reduce API latency and cost [OpenAI, Anthropic, grok]
Prompt Generator: Takes a user's brief requirements and generates a well-structured prompt designed to get the best performance out of the AI model [OpenAI, Anthropic/Claude]
Templatize a Prompt: Automatically identifies and extracts variables from text and turns it into a prompt template that is easy to embed in a system [Anthropic/Claude]
Create / Build an Agent: Generates and builds a bot's behavior and knowledge configuration from scratch based on natural-language instructions [Microsoft Copilot, OpenAI, Google/Gemini, Mistral]
Build an Agent Flow: Automatically generates agent action flows involving complex business procedures and conditional branching from natural-language instructions [Microsoft Copilot]
Automation Flow Generation: Has the AI generate and document API integrations and automation flows across multiple systems within a visual editor [Google/Gemini]
Subagent Spawning: The main agent breaks a complex task into parts and dynamically spawns and invokes child agents to handle specific pieces of work [OpenAI, Anthropic/Claude, OpenClaw]
Custom agent knowledge base: Generates a dedicated knowledge base or index from a specific set of documents for an agent to draw on [Mistral]
Infrastructure, system configuration, and test generation
Evaluation Test Sets: Automatically generates diverse test cases and ground-truth data for objectively evaluating model performance [Microsoft Copilot, OpenAI]
CI/CD integration generation: Generates CI/CD configuration files, such as GitHub Actions, for running code reviews and automated tests [Anthropic/Claude, GitHub]
Workspace config generation: Generates configuration files such as AGENTS.md and SOUL.md that define an agent's personality and tool permissions [OpenClaw]
Shell execution logs: Runs code or shell scripts in a sandboxed environment and returns the resulting standard output and error logs as text [OpenAI, OpenClaw]
Webhook payload generation: Dynamically generates JSON payloads for sending event notifications to external systems or server-side controls [OpenAI, OpenClaw]
Cron Configuration: Generates cron schedule settings so that agents can run tasks on a recurring basis [Google/Gemini, Anthropic/Claude, OpenClaw]
Streaming and other generation
Streaming Events: Streams text tokens and partial image data in real time via SSE as they are being generated [OpenAI, Anthropic/Claude, Perplexity]
Fallback response generation: When the primary API returns an error, calls an alternative model or local capability to safely generate a fallback response [OpenClaw]
"Action" capabilities
These are capabilities that call external tools (APIs, browsers, calculators, internal systems, and so on) to intervene directly in the real world or in other systems.
Agent building and orchestration foundations
Create / Build agents: Autonomously builds an agent's behavior and structure from natural-language instructions [OpenAI, Google/Gemini, Microsoft Copilot, Mistral]
Agents SDK / Agent API: Builds agent orchestration, handoffs, and tool execution at the code level [OpenAI, Anthropic/Claude, Perplexity]
Managed Agents: Manages and deploys agents on hosted infrastructure, complete with stateful sessions and a persistent event history [Anthropic/Claude, Google/Gemini]
Subagents / Child agents: The main agent splits up a task and invokes child agents to handle specific pieces of work in parallel [OpenAI, Anthropic/Claude, Microsoft Copilot, OpenClaw]
Custom workflows / Gems: Turns custom mini-apps or flows into agents and runs them to automate specific tasks [Google/Gemini, Mistral]
Background agent execution (Background mode / Routines): Lets an agent keep running its tasks autonomously in the cloud or in the background even after the user has stepped away [OpenAI, Anthropic/Claude]
Scheduled and cron-based automation (Schedule actions / Cron): Has an agent carry out actions automatically at a set time or on a recurring schedule, such as a morning PR review [Google/Gemini, Anthropic/Claude, OpenClaw]
Tool calling and API integration
Function and tool calling foundation (Function / Tool calling): A shared foundation that lets the AI autonomously generate execution commands (e.g., JSON) for external tools and request that they be run, based on the user's instructions [OpenAI, Google/Gemini, Anthropic/Claude, xAI, Meta/Llama, Mistral, DeepSeek]
Model Context Protocol integration (MCP and Connectors): Uses the standardized MCP protocol to connect to and operate local or remote external data sources and tools [OpenAI, Anthropic/Claude, Microsoft Copilot, GitHub]
Custom action execution (Custom Actions): Calls external APIs (databases, SaaS services) directly from the AI chat and autonomously sends data to or updates those systems [OpenAI, Microsoft Copilot, Google/Gemini]
Hook-triggered commands (Hooks): Automatically triggers and runs arbitrary shell commands, such as formatters or linters, at defined moments like after a file edit or before a commit [Anthropic/Claude]
Platform extensions (Extensions / Plugins): Gives an existing platform environment the ability to execute actions in external tools [Google/Gemini, Microsoft Copilot]
Desktop and local system operation
Visual, autonomous computer operation (Computer use tool): A hands-on action capability in which the AI visually understands the desktop screen and autonomously moves the mouse, clicks, and types [Anthropic/Claude, OpenAI]
Command execution in a local shell (Local shell / Bash tool): Lets an agent autonomously run commands in the system's terminal or shell environment [OpenAI, Anthropic/Claude, OpenClaw]
Direct text file editing (Text editor / Edit tools): Lets an agent open local text files directly and edit or overwrite their contents [Anthropic/Claude, OpenClaw]
Autonomous file system access (read / write tools): Lets an agent browse files on the user's device and read from and write to them directly [OpenClaw, Anthropic/Claude]
Autonomous process management (Process tool): Lets an agent manage processes running on the local system, including starting and stopping them [OpenClaw]
Autonomous management of local worktrees (Worktrees): Creates an isolated local environment so an agent can work on a separate task without polluting the main working environment [Anthropic/Claude]
Autonomous troubleshooting in the terminal (CLI Agents): Reads errors and system state in a CLI environment, autonomously reasons out a fix, and then executes and repairs it [Google/Gemini, Anthropic/Claude, OpenClaw]
Coding and development automation
Code interpreter execution (Code interpreter): Lets the agent write and run code such as Python inside a secure sandbox to complete tasks like data processing on its own [OpenAI, Microsoft Copilot, Mistral]
Autonomous multi-file code editing (Agent Mode): Autonomously implements new features or fixes bugs across multiple files within an IDE or CLI [Google/Gemini, Anthropic/Claude]
Automatic patch (diff) generation and application (Apply Patch): A tool with which an agent generates a fix patch for a specific part of the codebase and applies it autonomously [OpenAI]
Automatic project scaffolding (Scaffolding): Autonomously generates, in one pass, the directory structure and initial set of files a project needs from a natural-language outline [Mistral]
Git repository automation (Git automation): Works directly with git to autonomously stage changes, write commit messages, and go from creating a branch all the way to opening a PR [Anthropic/Claude]
Autonomous pull request code review (PR Review): Has the AI automatically review PRs submitted on GitHub and similar platforms, then propose and carry out code fixes [Google/Gemini, Anthropic/Claude, Microsoft Copilot]
AI automation within CI/CD pipelines (CI/CD integration): Triggers the AI inside pipelines such as GitHub Actions or GitLab to automate issue triage and reviews [Anthropic/Claude, GitHub]
GitHub issue management workflow automation (GitHub Actions): Has the AI autonomously run the workflow of classifying and handling issues in a repository [Microsoft Copilot, GitHub]
Automatic generation and updating of database schemas and queries (Firebase Data Connect): Generates schemas and mutations in GraphQL and similar languages from natural-language instructions, automating database operations [Google/Gemini]
Autonomous support for API specification authoring (Apigee / API Management): Draws on enterprise context to assist with and generate actions for writing API specifications and configuring policies [Google/Gemini]
External app and service integration
Google Workspace app operation (Connected Apps): Gives the AI access to the user's Gmail and Google Drive so it can draft emails and create files on their behalf [Google/Gemini]
Autonomous Google Calendar event creation (Calendar control): Interprets scheduling details from the user's instructions and creates or adjusts calendar events directly [Google/Gemini]
Voice control of smart home devices (Smart home control): Directly controls the power and settings of connected smart home devices through the AI app [Google/Gemini]
Autonomous CRM record updates (Dynamics 365 actions): Has the AI autonomously update data in CRM systems (such as Salesforce) based on the content of sales and support conversations [Microsoft Copilot]
Automation through Power Platform integration (Power Platform AI actions): Embeds AI into automation workflows such as Power Automate, making cross-system actions AI-driven [Microsoft Copilot]
Visual generation of SaaS-to-SaaS workflows (Application Integration): Builds and generates automation flows that link multiple third-party apps and APIs visually, with AI assistance [Google/Gemini]
Task integration and updates in Jira, Slack, and similar tools (MCP SaaS tools): Autonomously updates Jira tickets and posts notifications to Slack via MCP and similar protocols [Anthropic/Claude, OpenClaw]
Automated calls and messages via WhatsApp (WhatsApp actions): An action capability that places calls and sends messages on the user's behalf through WhatsApp and similar services from a mobile AI app [Google/Gemini, OpenClaw]
Security, sandboxing, and governance
Guardrail-based behavior restrictions (Guardrails): Safely limits the scope of an agent's actions and permissions to prevent actions that violate policy [OpenAI, Microsoft Copilot]
Secure tool execution in a sandbox (Sandbox execution): Runs tools inside an isolated environment such as Docker so that an agent's actions cannot damage the system [OpenAI, OpenClaw]
Remote actions via SSH/OpenShell (Remote Sandbox): Lets an agent securely access a remote or separate system over SSH or similar and execute actions there [OpenClaw]
Access-permission controls for actions (Data policies / Allowlist): Lets administrators control and restrict which actions an agent may perform and which data it may access [Microsoft Copilot, OpenClaw]
Automated defensive actions against security incidents (Security Copilot Plugins): When a cyber threat is detected, the AI automatically executes defensive and containment actions [Microsoft Copilot]
Autonomous diagnosis and optimization of cloud infrastructure (Cloud Assist): Analyzes the operational state of a cloud environment and autonomously proposes and executes infrastructure-optimization and cost-reduction actions [Google/Gemini]
Automated orchestration of MLOps pipelines (MLOps automation): Automatically runs and manages pipeline processes for AI models, such as monitoring, evaluation, and retraining [Google/Gemini]
Access-approval controls for DMs from unknown senders (DM pairing): For security, requires a pairing code for messages from unknown users and restricts the actions they can trigger [OpenClaw]
Messaging and chat-platform integrations
Autonomous responses across a multi-channel inbox (Multi-channel gateway): Connects an agent simultaneously to numerous platforms—WhatsApp, Slack, Discord, Telegram, and more—so it can respond autonomously on each [OpenClaw]
Dynamic per-message agent routing (Multi-agent routing): Routing that automatically dispatches each incoming message to a different agent (workspace) based on the channel or user it came from [OpenClaw]
Execution of Discord- and Slack-specific actions (Platform native actions): Performs system actions tailored to the Discord and Slack APIs, such as creating channels and managing permissions [OpenClaw]
Triggering tasks directly from Slack (AI for Slack): An integration in which a mention in Slack has the agent carry out a task directly (e.g., opening a PR for a bug fix) and post the result back [Anthropic/Claude, OpenClaw]
Launching autonomous actions from webhook events (Webhooks): Receives webhooks from external systems and uses them as triggers to kick off autonomous agent actions [OpenAI, OpenClaw]
Actions triggered by incoming email (Gmail Pub/Sub): Hooks into the receipt of specific emails in Gmail to have the agent perform autonomous actions [OpenClaw]
Action integration via custom chat widgets (ChatKit / UI Actions): Embeds custom widgets within the chat UI so that system actions can be executed through button presses and similar interactions [OpenAI]
Autonomous spawning of new sessions (sessions_spawn): Lets the agent itself launch new sessions as needed and process subtasks in parallel [OpenClaw]
Sending messages to other sessions (sessions_send): Lets the agent autonomously send messages to other running sessions and coordinate with them [OpenClaw]
Dynamic state control through operator commands (Operator commands): Lets you type commands such as /reset or /think in the chat to instantly change the agent's reasoning level or behavior mode [OpenClaw]
Voice, real-time, UI, and device control
Autonomous responses from real-time voice agents (Voice agents / Realtime API): Processes spoken input in real time and carries on a natural-sounding voice conversation while simultaneously executing system actions [OpenAI, xAI, Microsoft Copilot]
Handoff control for IVR (Interactive voice response): Responds autonomously on voice channels such as the telephone and, when necessary, hands the conversation off to a human operator with full context preserved [Microsoft Copilot]
Wake-word agent activation (Voice Wake): Listens for a wake word on the device and instantly triggers agent actions from spoken instructions [OpenClaw]
Real-time rendering and manipulation of dynamic UI (Live Canvas / A2UI): Based on the user's instructions, the agent autonomously generates and lays out interactive UI components on screens such as a mobile app [OpenClaw]
Autonomous remote control of mobile devices (Mobile nodes): Pairs with a mobile app so the agent can remotely operate a smartphone's camera, capture its screen, and run device commands [OpenClaw]
Autonomous search and information-gathering actions
Autonomous real-time data search on X (Twitter) (X Search tool): The agent autonomously searches X posts for specified keywords and collects the data [xAI]
Account-scoped search actions (X Search - handles): Allows or excludes specific X handles (up to 10) to run noise-free search actions [xAI]
Automated multi-provider search workflows (Agent API): Combines multiple third-party models with web-search tools to run autonomous research workflows [Perplexity]
Autonomous operation of an official web-browser tool (Web Search / Browser tool): The agent autonomously drives a web browser, navigating to the pages it needs and retrieving information [OpenAI, Anthropic/Claude, DeepSeek, Kimi, OpenClaw]
Networking, deployment, and other operational actions
One-click integrated agent deployment (Teams / M365 deployment): An action that deploys a built agent directly into Teams or Microsoft 365 work environments, making it available across the entire organization [Microsoft Copilot]
Secure remote action control via Tailscale integration (Tailnet integration): Uses a VPN (Tailnet) to control agent actions and gateways within a secure remote environment [OpenClaw] 67. Autonomous setup via CLI onboarding (CLI Onboarding): Automatically runs and walks you through the agent's initial configuration, channel integrations, and daemon installation, all from the terminal [OpenClaw]
Automated orchestration of cloud infrastructure for agents (Enterprise Agent Platform): For enterprises, centrally automates the deployment, scaling, and management of AI agents [Google/Gemini, Microsoft Copilot]
Autonomous in-app browser rendering (In-app browser): Lets the agent render web pages directly within the CLI or IDE and display its research findings there [Anthropic/Claude]
Building self-hosted APIs for AI models on your own infrastructure (Llama Stack): Uses Llama models to automatically build APIs and agent-integration systems end to end on your own infrastructure [Meta/Llama]
Task optimization and error handling
Event delivery via real-time streaming (Streaming events): Streams the agent's process of calling tools and taking actions to external systems as real-time events [OpenAI]
Persistent management of conversation and execution state (Conversation state): Saves the agent's conversation history and in-progress state so that complex, long-running actions can be continued [OpenAI]
Automated testing and evaluation of agent workflows (Evaluate workflows): Automatically tests and evaluates whether a built agent's action flows function correctly [OpenAI]
Capability extension from a skills registry (Skills / ClawHub): Dynamically adds "skills"—packaged sets of custom procedures and capabilities—to a workspace, expanding the range of actions the agent can take [Anthropic/Claude, OpenClaw]
Self-diagnosis and repair of endpoint status (Self-diagnostics / Doctor): Automatically detects misconfigurations and environment problems and runs troubleshooting actions [OpenClaw]
Server-side call control (Server-side controls): While something like the Realtime API is running, lets the server interject to control the agent's speech output and actions [OpenAI]
Autonomous troubleshooting of app crashes (Firebase error analysis): Pinpoints the root cause from an app's crash logs and autonomously proposes code-fix actions such as patches [Google/Gemini]
Autonomous application code modernization (Code modernization): An action in which the agent reads a codebase written in legacy languages or frameworks and autonomously rewrites it for a modern architecture [Anthropic/Claude]
Automatic code formatting in CI/CD (Auto-formatting in CI): Uses hooks so that a formatter (such as Prettier) runs autonomously immediately after the agent edits code [Anthropic/Claude]
Automated lint checks (Auto-linting): The agent autonomously runs lint tools before committing or opening a PR and fixes any errors they surface [Anthropic/Claude]
System Integration and Automation
Automatic dependency updates via package managers (Dependency update): The agent autonomously operates package managers such as npm or pip to update libraries [Anthropic/Claude]
Automated compile and build execution (Build automation): The agent autonomously runs the system's build commands (make, build, and the like) and resolves any errors that occur [Anthropic/Claude]
Autonomous attachment to running containers (Container manipulation): An action in which the agent attaches to a running Docker container and executes commands inside it [OpenClaw]
Automatic tool generation from custom prompts (Prompt as a tool): Wraps a user-written custom prompt as a tool in its own right so that other agents can invoke it [Microsoft Copilot]
Simulated conversation testing for voice agents (Test voice-enabled agents): Automatically tests the actions and responses of a voice-call agent you have built in a simulator [Microsoft Copilot]
Automated single sign-on (SSO) actions (SSO Integration): When an agent is used in Teams or similar environments, autonomously handles authentication actions via Entra ID and other identity providers [Microsoft Copilot]
Autonomous data processing from uploaded files (File inputs processing): The agent receives files uploaded by the user and autonomously analyzes and transforms them [OpenAI]
Text-to-speech actions (TTS execution): The agent sends the text it generates to an API to produce and play back audio [OpenAI, xAI, OpenClaw]
Speech-to-text actions (STT execution): The agent receives audio data and calls an external transcription tool to convert it into text [OpenAI, xAI]
Autonomous RAG pipeline orchestration (RAG integrations): A chain of actions in which the agent takes a user's question, autonomously queries a vector database, and synthesizes the retrieved information into an answer [Anthropic/Claude, Mistral]
Chat integration with live-streaming platforms (Twitch integration): The agent connects to the chat of a live-streaming platform such as Twitch and converses autonomously with viewers [OpenClaw]
Automated task execution in non-interactive mode (Non-interactive Mode): A mode in which the CLI agent runs fully automatically to complete tasks in CI/CD pipelines and similar settings, with no human intervention [Anthropic/Claude]
Automated infrastructure-as-code deployment (Terraform/IaC execution): An action in which the agent generates code for infrastructure management tools such as Terraform and autonomously applies it to provision the environment [Google/Gemini]
Fallback actions on system errors (Fallback routing): When the primary agent or API goes down, automatically routes to a subsystem or lightweight model so the task can continue [OpenClaw]
Automatic permission switching between main and non-main sessions (Permission switching): Autonomously switches tool execution permissions and sandboxing requirements depending on whether the request comes from the user's own operations or from external access [OpenClaw]
Inline actions within the developer's own environment (IDE integrations): The agent performs code insertion and refactoring actions directly inside editors such as VS Code and JetBrains [Anthropic/Claude, Google/Gemini]
Autonomous in-browser web research and data extraction (Web fetch / scraping): An action in which the agent, given a URL, autonomously fetches the web page, parses the DOM, and extracts the required data [Anthropic/Claude]
Parallel autonomous execution of multiple queries (Multi-query support): The agent breaks a complex task into multiple search queries and runs them against the API simultaneously to gather information [Perplexity]
Autonomous retrieval of entire X threads (Thread fetch): An action in which the agent traces back through a specific thread on X and autonomously retrieves the full context of the conversation [xAI]
End-to-end integration with third-party platforms (WhatsApp/Cloud Provider integration): Uses Llama models and similar to autonomously chain a series of actions from apps such as WhatsApp through to cloud infrastructure [Meta/Llama]
Other Capabilities
These are capabilities that fall outside the categories of understanding, search, reasoning, generation, and action covered above. They relate to areas such as learning and optimization, evaluation and monitoring, management and operations, security and governance, infrastructure and deployment, and developer tooling.
Learning, Customization, and Performance Optimization
Fine-tuning (Supervised fine-tuning): Fine-tunes a model on your own dataset to improve accuracy on specific tasks [OpenAI]
Vision fine-tuning (Vision fine-tuning): Multimodal fine-tuning that incorporates image data [OpenAI]
Direct preference optimization (Direct preference optimization / DPO): Optimizes model outputs to align with human preferences [OpenAI]
Reinforcement fine-tuning (Reinforcement fine-tuning): Optimizes model behavior based on a specific reward signal [OpenAI]
Custom model training (Custom training): Trains open models and others independently within your own environment [Google/Vertex AI]
Fine-tuning open models (Fine-tuning recipes): Fine-tunes Llama and similar models for specific use cases [Meta/Llama]
Prompt caching (Prompt caching): Caches lengthy prompts to reduce API latency and cost [OpenAI, Anthropic, xAI]
Context caching (Context Caching): Caches large input contexts on the system side [DeepSeek]
Batch processing API (Batch API): Processes large volumes of API requests asynchronously in bulk, cutting costs in half [OpenAI, Anthropic, xAI]
Latency optimization / predicted outputs (Predicted Outputs): Dramatically speeds up generation by having the model predict outputs that are already known [OpenAI]
Priority and flex processing (Priority / Flex processing): Optimizes cost by lowering the processing priority of API requests [OpenAI]
Provisioned throughput (Provisioned Throughput): Reserves dedicated compute resources to guarantee stable throughput [xAI]
Inference quantization and memory offloading (Quantization & Offloading): Optimizations for running large models on resource-constrained devices [Diffusers]
TensorRT-optimized models (NVIDIA TensorRT optimized): Pushes inference speed to its limits on NVIDIA GPUs [Stability AI]
AMD GPU-optimized models (AMD Optimized): Model tuning specialized for image generation on AMD hardware [Stability AI]
PyTorch compilation optimization (torch.compile): Compiles code to boost inference speed [Diffusers]
Prompt self-optimization (Prompt optimizer): The model analyzes its own prompts and automatically revises them into the most effective structure [OpenAI]
Microsoft 365 Copilot Tuning (preview): Tunes Copilot to your organization's own data and terminology [Microsoft]
Evaluation, Monitoring, and Observability
Automated model evaluation tools (Evals / Graders): Evaluates and scores model performance and accuracy in a data-driven way [OpenAI]
Building evaluation test sets (Create test sets): Create your own test data for evaluating agents [Copilot Studio]
Evaluation tool UI (Evaluation Tool): Compare prompt outputs side by side in the console for visual evaluation [Anthropic]
Automated agent workflow evaluation (Evaluate workflows): Test whether complex agent behavior flows work as intended [OpenAI]
Model playground (Studio playground): Compare and test model outputs in real time while adjusting parameters [Mistral AI]
Generative AI model evaluation service (Gen AI Evaluation): An enterprise-grade platform for objective model evaluation [Google/Vertex AI]
MLOps platform (MLOps): Automate model monitoring, evaluation, and pipeline orchestration [Google Cloud]
Monitoring model input skew and drift (Model Monitoring): Watch for shifts in data trends and model degradation in production [Google/Vertex AI]
Agent performance analytics (Analytics): Analyze metrics such as conversation success rates and user satisfaction [Copilot Studio]
Autonomous agent health analysis (Agent health): Monitor error rates and operational status of autonomously acting agents [Copilot Studio]
Cloud system observability (Cloud Logging / Monitoring): Unified monitoring of infrastructure and API logs and metrics [Google Cloud]
Application tracing (Error Reporting / Trace): Trace and pinpoint the source of API call latency and errors [Google Cloud]
System status API (System Status API): Monitor service availability programmatically [Perplexity]
Detailed token usage tracking (Counting tokens / Usage): Precisely monitor the number of tokens consumed by prompts and caching [OpenAI, Anthropic]
Administration, operations, and billing
Organization and workspace management (Manage workspaces): Create and manage separate workspaces and environments for each team [Mistral AI, Anthropic]
Project management (Projects): Separate API keys and settings by application using projects [OpenAI]
User invitations and management (User Management / Invites): Invite members to an organization and manage the account lifecycle [OpenAI, Mistral AI]
Role assignment (Roles / Permissions): Grant viewer, editor, administrator, and other permissions on a per-user basis [OpenAI, Mistral AI]
API key creation and rotation (API Key Management): Issue and revoke API keys for each workspace [General]
Administrator API (Admin API): Automate workspace and key management programmatically [OpenAI, Anthropic]
Admin tracking API keys (Admin API Keys): Special API keys for tracking usage across the entire organization [OpenAI]
Billing and FinOps dashboard (Billing Management): Visualize and manage API spend and costs by team [Mistral AI, xAI]
Quota and rate limit management (Rate limits & Tiers): Check current API rate limits (RPM/TPM) and tiers, and manage caps [OpenAI, Perplexity]
Cloud cost management tools (Cost Management): Optimize costs across your infrastructure and set budget alerts [Google Cloud]
Audit logging (Audit Logs): Record a history of who changed what in the system and when, for use in security audits [OpenAI, xAI, Microsoft]
License assignment (Assign licenses): Grant users licenses to use Copilot and agents [Copilot Studio]
Model registry (Model Registry): Centrally manage versions and metadata of trained models [Google/Vertex AI]
Feature store (Feature Store): Share and manage features for reuse across ML models [Google/Vertex AI]
Subscription auto-renewal management (Managing Subscription): Upgrade or downgrade paid plans and manage automatic renewal [Midjourney]
Organization tax status settings (Tax / VAT): Configure VAT exemption and tax information for businesses [Midjourney]
Credit consumption management (Credits management): Track consumption of credits (the platform currency) for image generation and similar tasks [Stability AI]
Security, governance, and access control
Single sign-on integration (SSO / SAML): Secure login integration using your corporate IdP (such as Okta) [Mistral AI, xAI]
SSO integration with Microsoft Entra ID: Seamless authentication when using agents in Teams and other Microsoft apps [Microsoft]
Role-based access control (RBAC): Control access to APIs and resources through fine-grained permission settings [xAI]
Agent data policy configuration (Data policies): Restrict the internal data agents can access and the actions they can perform [Copilot Studio]
Safety filters and moderation (Moderation): APIs that evaluate and block harmful prompts and outputs [OpenAI, Meta]
Cybersecurity verification (Cybersecurity checks): Check whether an application meets security standards [OpenAI]
Age verification and minor protection (Under 18 API / Age Assurance): Determine whether a user is 18 or older based on biometric data and usage signals [OpenAI, C.AI]
Compliance management (Compliance): Features for adhering to strict regulatory standards such as SOC 2, GDPR, and CCPA [xAI]
Data residency controls (Data residency): Strictly specify the geographic region (country) where data is stored and processed [xAI]
Data encryption and separation (Encryption / Separation): Keep data excluded from training stored separately, and encrypt data in transit and at rest [Meta]
mTLS authentication (mTLS Authentication): Strict API authentication using certificates on both the client and server sides [xAI]
Security Command Center (Security Command Center): Unified monitoring of threats and vulnerabilities across your cloud environment [Google Cloud]
Sensitive data protection (Sensitive Data Protection): Automatically discover, mask, and protect sensitive data such as PII (personally identifiable information) [Google Cloud]
VPC Service Controls (VPC-SC): Prevent data from being taken out of (exfiltrated from) the corporate network [Google Cloud]
eDiscovery and data retention (eDiscovery / Legal hold): Retain and search Copilot conversation histories in preparation for litigation or audits [Microsoft 365]
API fingerprint management (Fingerprint): Manage identifiers for tracking changes to the backend model [xAI]
DM pairing and access control (DM pairing / Allowlist): Block DMs from unknown users and require approval before they can get through [OpenClaw]
Secure sandbox environments (Docker/SSH Sandbox): Confine agents to isolated container environments when letting them operate on the system [OpenClaw]
Infrastructure, deployment, and scaling
Llama Stack (self-hosted API layer): A foundation for hosting Llama models as an API on your own infrastructure [Meta]
Llama Everywhere (edge deployment): Deploy models directly to desktops and Linux-based on-premises environments [Meta]
AI edge deployment (Google AI Edge): Deploy lightweight AI models to mobile and embedded devices [Google]
Cloud model deployment (Prediction / Serving): Expose trained models as APIs for batch inference and online prediction [Google/Vertex AI]
Serverless container deployment (Cloud Run / Cloud Functions): Deploy AI application backends with automatic scaling and no infrastructure management [Google Cloud]
Kubernetes cluster integration (Google Kubernetes Engine): Orchestrate large-scale containerized AI workloads [Google Cloud]
Enterprise agent deployment (Enterprise Agent Platform): Deploy the agents you have built to production at scale [Google Cloud]
Deployment on third-party clouds: Run Claude models on Amazon Bedrock or Vertex AI infrastructure [Anthropic]
Deployment on Microsoft Foundry: Deploy AI models and agents in an enterprise-grade PaaS environment [Microsoft]
API gateway and integration (Apigee / Application Integration): A routing layer for securely exposing your own APIs and connecting them to AI [Google Cloud]
Tailscale networking integration: Securely reach a local AI assistant from remote environments over a Tailnet [OpenClaw]
Air-gapped and on-premises deployment (Google Distributed Cloud): Stand up AI infrastructure in closed networks isolated from the internet [Google Cloud]
CI/CD pipelines (Cloud Build / Cloud Deploy): Automate the build and deployment of AI applications [Google Cloud]
Infrastructure as code (Infrastructure Manager): Provision and manage cloud infrastructure for AI as code using tools such as Terraform [Google Cloud]
Batch and task scheduling (Cloud Scheduler / Batch): Run heavy asynchronous data-processing and inference jobs on a schedule [Google Cloud]
Dedicated inference endpoints (Inference Endpoints): Deploy custom models on Hugging Face as dedicated API infrastructure [Hugging Face]
Developer experience, interfaces, and ecosystem
Web Control UI dashboard: Manage a local agent's chats, configuration, and sessions from the browser [OpenClaw]
CLI onboarding (openclaw onboard): Install and configure the agent automatically through an interactive terminal session [OpenClaw]
macOS system management app (OpenClaw.app): Start the gateway and monitor its status from the menu bar [OpenClaw]
System self-diagnostics (openclaw doctor): Automatically detect misconfigurations and environment problems to aid troubleshooting [OpenClaw]
Release channel switching (Update channels): Switch between Stable, Beta, Dev, and other release channels with a single command [OpenClaw]
IDE extensions (VS Code / JetBrains extension): Bring the AI assistant directly into the editor of your development environment [Anthropic, Google]
Local worktree environments (Worktrees): Create isolated working branches for the agent without cluttering your main workspace [Anthropic]
Project-specific instruction files (CLAUDE.md / AGENTS.md): Place a file at the repository root so the agent automatically loads the project's conventions [Anthropic, OpenClaw]
Non-interactive automation mode (Non-interactive Mode): Run the CLI agent automatically without human intervention, for example in CI/CD [Anthropic]
Developer mode / tracing (/trace): Output detailed logs of the agent's internal processing and network requests [OpenAI, OpenClaw]
OpenAI-compatible API endpoints (OpenAI Compatibility): Call third-party models (such as Llama and DeepSeek) using your existing OpenAI SDKs unchanged [Meta, Perplexity, DeepSeek]
Model Context Protocol (MCP) ecosystem: A protocol layer that standardizes context so different AI models and tools can work together [Anthropic, OpenAI, Microsoft]
Custom widget embedding (ChatKit Widgets): A developer feature for embedding third-party custom widgets inside the ChatGPT UI [OpenAI]
Prompt comparison and management (GitHub Models): A developer tool for comparing the outputs of multiple prompts side by side and managing them [GitHub]
GitHub Actions automation: Run AI models inside CI/CD workflows, triggered by repository events [GitHub]
Notebook environment integration (Colab Enterprise): Bring AI model development and data science together in a Jupyter Notebook-style environment [Google Cloud]
In-app browser integration (In-app browser): Let the agent render web pages directly within the CLI or IDE and display its research findings [Anthropic]
Closing thoughts
What AI can do keeps expanding, and we will continue to update this list over time. For my part, I am developing AI agents specialized for the real estate and construction industries, and I intend to focus our research and development on the areas that general-purpose AI like the tools above will struggle to cover, even in the future.
The end
Read next ↓