Action Space

The Action Space defines the set of executable human-like operations that agents can perform in the sandbox, thereby controlling the scope of agent behavior. This document introduces the types of actions supported in the Lybic sandbox and their usage methods.

Overview

Lybic provides a unified action execution interface with support for two main usage modes:

Computer Use (Desktop Use): Applicable to Windows, Linux, and macOS sandboxes, supporting mouse and keyboard operations
Mobile Use (Mobile Use): Applicable to Android sandboxes, supporting touch, swipe, app management, and other operations

API Interface

Execute Action

Endpoint: POST /api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute

Common Parameters:

action (object, required): The action object to execute (see action types below)
includeScreenShot (boolean, optional): Whether to include a screenshot URL in the response after action execution, defaults to true
includeCursorPosition (boolean, optional): Whether to include cursor/touch position in the response after action execution, defaults to true

Response Format:

{
  "screenShot": "https://...",
  "cursorPosition": {
    "x": 500,
    "y": 300,
    "screenWidth": 1920,
    "screenHeight": 1080,
    "screenIndex": 0
  },
  "actionResult": {}
}

Computer Use Action Space

Desktop operations for Windows and Linux sandboxes.

1. Mouse Operations

Click (mouse:click)

Click the mouse at the specified coordinates.

Parameters:

type: "mouse:click"
x: X coordinate (pixels or fraction)
y: Y coordinate (pixels or fraction)
button: Mouse button flag (1=left, 2=right, 4=middle, 8=back, 16=forward)
holdKey (optional): Modifier keys to hold while clicking, such as "ctrl", "alt", "alt+shift"
relative (optional): Whether coordinates are relative to the current mouse position

Coordinate Formats:

Pixel coordinates:

{
  "type": "px",
  "value": 500
}

Fractional coordinates (recommended, adapts to different resolutions):

{
  "type": "/",
  "numerator": 1,
  "denominator": 2
}

cURL Request Examples:

# Left-click at screen center (50%, 50%)
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:click",
      "x": {"type": "/", "numerator": 1, "denominator": 2},
      "y": {"type": "/", "numerator": 1, "denominator": 2},
      "button": 1
    }
  }'

# Right-click at absolute position (500, 300)
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:click",
      "x": {"type": "px", "value": 500},
      "y": {"type": "px", "value": 300},
      "button": 2
    }
  }'

# Ctrl + Left-click
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:click",
      "x": {"type": "px", "value": 500},
      "y": {"type": "px", "value": 300},
      "button": 1,
      "holdKey": "ctrl"
    }
  }'

Double-click (mouse:doubleClick)

Double-click the mouse at the specified coordinates.

Parameters: Same as click

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:doubleClick",
      "x": {"type": "px", "value": 400},
      "y": {"type": "px", "value": 200},
      "button": 1
    }
  }'

Triple-click (mouse:tripleClick)

Triple-click the mouse at the specified coordinates (typically used to select an entire line of text).

Parameters: Same as click

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:tripleClick",
      "x": {"type": "px", "value": 400},
      "y": {"type": "px", "value": 200},
      "button": 1
    }
  }'

Move (mouse:move)

Move the mouse to the specified coordinates.

Parameters:

type: "mouse:move"
x: X coordinate
y: Y coordinate
holdKey (optional): Modifier keys to hold while moving
relative (optional): Whether to move relative to current position

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:move",
      "x": {"type": "px", "value": 800},
      "y": {"type": "px", "value": 600}
    }
  }'

Drag (mouse:drag)

Drag from start coordinates to end coordinates.

Parameters:

type: "mouse:drag"
startX: Start X coordinate
startY: Start Y coordinate
endX: End X coordinate
endY: End Y coordinate
button (optional): Mouse button, defaults to 1 (left)
holdKey (optional): Modifier keys to hold while dragging
startRelative (optional): Whether start coordinates are relative
endRelative (optional): Whether end coordinates are relative

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:drag",
      "startX": {"type": "px", "value": 100},
      "startY": {"type": "px", "value": 100},
      "endX": {"type": "px", "value": 500},
      "endY": {"type": "px", "value": 500}
    }
  }'

Scroll (mouse:scroll)

Scroll the mouse wheel at the specified position.

Parameters:

type: "mouse:scroll"
x: X coordinate
y: Y coordinate
stepVertical: Vertical scroll steps (positive=up, negative=down)
stepHorizontal: Horizontal scroll steps (positive=right, negative=left)
holdKey (optional): Modifier keys to hold while scrolling
relative (optional): Whether coordinates are relative

cURL Request Example:

# Scroll down 5 steps
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "mouse:scroll",
      "x": {"type": "/", "numerator": 1, "denominator": 2},
      "y": {"type": "/", "numerator": 1, "denominator": 2},
      "stepVertical": -5,
      "stepHorizontal": 0
    }
  }'

2. Keyboard Operations

Type Text (keyboard:type)

Type text content.

Parameters:

type: "keyboard:type"
content: Text content to type
treatNewLineAsEnter: Whether to treat newline characters \n as Enter key, defaults to false

cURL Request Examples:

# Type plain text
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:type",
      "content": "Hello, Lybic!",
      "treatNewLineAsEnter": false
    }
  }'

# Type multi-line text (newlines as Enter)
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:type",
      "content": "First line\nSecond line\nThird line",
      "treatNewLineAsEnter": true
    }
  }'

Hotkey (keyboard:hotkey)

Press a keyboard hotkey combination.

Parameters:

type: "keyboard:hotkey"
keys: Hotkey combination using xdotool key syntax. Examples: "Return", "ctrl+c", "alt+Tab", "ctrl+shift+s"
duration (optional): Duration to hold (milliseconds), range 1-5000

Common Key Names:

Letter keys: "a", "b", "A", "B", etc.
Number keys: "1", "2", "KP_0" (numpad 0), etc.
Function keys: "F1", "F2", ... "F12"
Special keys: "Return" (Enter), "Escape", "Tab", "space", "BackSpace", "Delete"
Arrow keys: "Up", "Down", "Left", "Right"
Modifier keys: "ctrl", "alt", "shift", "super" (Windows/Command key)

cURL Request Examples:

# Press Enter
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:hotkey",
      "keys": "Return"
    }
  }'

# Ctrl+C to copy
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:hotkey",
      "keys": "ctrl+c"
    }
  }'

# Ctrl+Shift+S to save as
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:hotkey",
      "keys": "ctrl+shift+s"
    }
  }'

# Hold Ctrl key for 1 second
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "keyboard:hotkey",
      "keys": "ctrl",
      "duration": 1000
    }
  }'

Key Down (key:down)

Press a single key (without releasing). Use only when keyboard:hotkey is insufficient.

Parameters:

type: "key:down"
key: Key name to press (xdotool syntax)

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "key:down",
      "key": "shift"
    }
  }'

Key Up (key:up)

Release a single key. Use only after key:down.

Parameters:

type: "key:up"
key: Key name to release (xdotool syntax)

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "key:up",
      "key": "shift"
    }
  }'

3. Common Operations

Screenshot (screenshot)

Get the current screen screenshot.

Parameters:

type: "screenshot"

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "screenshot"
    }
  }'

Wait (wait)

Pause for a specified duration.

Parameters:

type: "wait"
duration: Wait duration (milliseconds)

cURL Request Example:

# Wait for 3 seconds
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "wait",
      "duration": 3000
    }
  }'

Finished (finished)

Indicates that the task has completed successfully.

Parameters:

type: "finished"
message (optional): Completion message

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "finished",
      "message": "Task completed successfully"
    }
  }'

Failed (failed)

Indicates that the task has failed.

Parameters:

type: "failed"
message (optional): Failure message

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "failed",
      "message": "Unable to find the target element"
    }
  }'

User Takeover (client:user-takeover)

Indicates that human intervention is needed.

Parameters:

type: "client:user-takeover"

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "client:user-takeover"
    }
  }'

Mobile Use Action Space

Mobile operations for Android sandboxes.

1. Touch Operations

Tap (touch:tap)

Tap the screen at the specified coordinates.

Parameters:

type: "touch:tap"
x: X coordinate (pixels or fraction)
y: Y coordinate (pixels or fraction)

cURL Request Example:

# Tap screen center
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "touch:tap",
      "x": {"type": "/", "numerator": 1, "denominator": 2},
      "y": {"type": "/", "numerator": 1, "denominator": 2}
    }
  }'

Long Press (touch:longPress)

Long-press the screen at the specified coordinates.

Parameters:

type: "touch:longPress"
x: X coordinate
y: Y coordinate
duration: Duration to hold (milliseconds)

cURL Request Example:

# Long-press for 2 seconds
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "touch:longPress",
      "x": {"type": "px", "value": 500},
      "y": {"type": "px", "value": 800},
      "duration": 2000
    }
  }'

Drag (touch:drag)

Drag from start coordinates to end coordinates.

Parameters:

type: "touch:drag"
startX: Start X coordinate
startY: Start Y coordinate
endX: End X coordinate
endY: End Y coordinate

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "touch:drag",
      "startX": {"type": "px", "value": 200},
      "startY": {"type": "px", "value": 500},
      "endX": {"type": "px", "value": 800},
      "endY": {"type": "px", "value": 500}
    }
  }'

Swipe (touch:swipe)

Swipe in a specified direction at the specified position.

Parameters:

type: "touch:swipe"
x: X coordinate
y: Y coordinate
direction: Swipe direction ("up", "down", "left", "right")
distance: Swipe distance (pixels or fraction)

cURL Request Example:

# Swipe up at screen center
curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "touch:swipe",
      "x": {"type": "/", "numerator": 1, "denominator": 2},
      "y": {"type": "/", "numerator": 1, "denominator": 2},
      "direction": "up",
      "distance": {"type": "px", "value": 300}
    }
  }'

2. Android System Keys

Back Button (android:back)

Press the Android back button.

Parameters:

type: "android:back"

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "android:back"
    }
  }'

Home Button (android:home)

Press the Android home button.

Parameters:

type: "android:home"

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "android:home"
    }
  }'

3. App Management

Start App by Package Name (os:startApp)

Start an app by its package name.

Parameters:

type: "os:startApp"
packageName: App package name

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "os:startApp",
      "packageName": "com.android.chrome"
    }
  }'

Start App by Name (os:startAppByName)

Start an app by its name.

Parameters:

type: "os:startAppByName"
name: App name

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "os:startAppByName",
      "name": "Chrome"
    }
  }'

Close App by Package Name (os:closeApp)

Close an app by its package name.

Parameters:

type: "os:closeApp"
packageName: App package name

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "os:closeApp",
      "packageName": "com.android.chrome"
    }
  }'

Close App by Name (os:closeAppByName)

Close an app by its name.

Parameters:

type: "os:closeAppByName"
name: App name

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "os:closeAppByName",
      "name": "Chrome"
    }
  }'

List All Apps (os:listApps)

Get a list of all installed apps on the device.

Parameters:

type: "os:listApps"

cURL Request Example:

curl -X POST "https://api.lybic.cn/api/orgs/{orgId}/sandboxes/{sandboxId}/actions/execute" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "action": {
      "type": "os:listApps"
    }
  }'

4. Keyboard Operations (Mobile)

Mobile Use also supports keyboard operations, same as Computer Use:

keyboard:type - Type text
keyboard:hotkey - Press hotkey (temporarily suspended for security reasons, use keyboard:type for text input instead)

See the Computer Use Keyboard Operations section.

5. Common Operations (Mobile)

Mobile Use also supports the following common operations:

screenshot - Screenshot
wait - Wait
finished - Task completed
failed - Task failed
client:user-takeover - User takeover

See the Computer Use Common Operations section.

Usage Notes

Replace Parameters: In actual use, replace {orgId}, {sandboxId}, YOUR_API_KEY, etc. in the examples with actual values
Coordinate System:
- Recommended to use fractional coordinates {"type": "/", "numerator": x, "denominator": y}, which automatically adapt to different resolutions
- Pixel coordinates {"type": "px", "value": x} are suitable for fixed resolution scenarios
Action Combination: Complex operations can be implemented by combining multiple API calls
Error Handling: Check the actionResult field in the response for execution details

On this page