update docs

version: 1

build:

  os: ubuntu-22.04

  tools:

    python: "3.10"

mkdocs:

  configuration: mkdocs.yml

  fail_on_warning: false

python:

  install:

    - requirements: docs/requirements.txt

 No newline at end of file

# Configuration Explanation

We use the service-based architecture as an enhanced approach for managing environments in VAGEN. This document explains the key configuration parameters for setting up and running experiments with the service architecture.

## Experiment Configuration

### Algorithm

```

algorithm.adv_estimator=bi_level_gae

algorithm.high_level_gamma=1.0

algorithm.kl_ctrl.kl_coef=0.001

```

`algorithm`:

- `adv_estimator`: Sets the advantage estimation method. Set to `bi_level_gae` for TRICO's cross-turn credit assignment.

- `high_level_gamma`: Discount factor for turn-level advantage calculations. Value of 1.0 means no discount across turns.

- `kl_ctrl.kl_coef`: Coefficient for KL divergence penalty. Controls policy deviation from reference model. Default 0.001

### Data

```

data.train_files=data/svg-vision-debug/train.parquet

data.val_files=data/svg-vision-debug/test.parquet

data.train_batch_size=16

data.max_prompt_length=1024

data.max_response_length=648

data.max_trajectory_length=1800

data.image_key=images

data.truncation=error

```

`data`:

- `train_files`: Path to training data in parquet format.

- `val_files`: Path to validation data.

- `train_batch_size`: Number of training examples per batch.

- `max_prompt_length`: Maximum token length for environment prompts.

- `max_response_length`: Maximum token length for model responses.

- `max_trajectory_length`: Maximum combined length of full interaction trajectory.

- `image_key`: Key used to access image data in inputs.

- `truncation`: Behavior when sequences exceed maximum length.

### Actor-Rollout-Reference Model

```

actor_rollout_ref.model.path=Qwen/Qwen2.5-VL-3B-Instruct

actor_rollout_ref.model.use_remove_padding=True

actor_rollout_ref.model.enable_gradient_checkpointing=True

```

`actor_rollout_ref.model`:

- `path`: Base VLM model path.

- `use_remove_padding`: Whether to skip computation on padded tokens.

- `enable_gradient_checkpointing`: Trades computation time for reduced memory usage.

```

actor_rollout_ref.actor.optim.lr=1e-6

actor_rollout_ref.actor.ppo_mini_batch_size=32

actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1

actor_rollout_ref.actor.use_kl_loss=False

actor_rollout_ref.actor.kl_loss_coef=0.001

actor_rollout_ref.actor.kl_loss_type=mse

```

`actor_rollout_ref.actor`:

- `optim.lr`: Actor model learning rate.

- `ppo_mini_batch_size`: Number of samples per PPO update batch.

- `ppo_micro_batch_size_per_gpu`: Micro-batch size per GPU for actor updates.

- `use_kl_loss`: Whether to use KL loss in actor updates.

- `kl_loss_coef`: Weight for KL loss term if enabled.

- `kl_loss_type`: Type of KL loss calculation.

```

actor_rollout_ref.actor.fsdp_config.param_offload=False

actor_rollout_ref.actor.fsdp_config.optimizer_offload=False

```

`actor_rollout_ref.actor.fsdp_config`:

- `param_offload`: Whether to offload parameters to CPU.

- `optimizer_offload`: Whether to offload optimizer states.

```

actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1

actor_rollout_ref.rollout.tensor_model_parallel_size=2

actor_rollout_ref.rollout.name=vllm

actor_rollout_ref.rollout.gpu_memory_utilization=0.2

actor_rollout_ref.rollout.enable_chunked_prefill=False

actor_rollout_ref.rollout.enforce_eager=False

actor_rollout_ref.rollout.free_cache_engine=False

actor_rollout_ref.rollout.n=1

actor_rollout_ref.rollout.top_p=0.95

actor_rollout_ref.rollout.temperature=0.7

```

`actor_rollout_ref.rollout`:

- `log_prob_micro_batch_size_per_gpu`: Micro-batch size for log probability calculations.

- `tensor_model_parallel_size`: Number of GPUs for tensor parallelism.

- `name`: Backend implementation for model deployment.

- `gpu_memory_utilization`: Target GPU memory utilization.

- `enable_chunked_prefill`: Whether to enable chunked context processing.

- `enforce_eager`: Whether to use eager execution mode.

- `free_cache_engine`: Whether to aggressively free cache.

- `n`: Number of rollout sequences per input.

- `top_p`: Nucleus sampling parameter for controlling output diversity.

- `temperature`: Sampling temperature for generation randomness.

```

actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=1

actor_rollout_ref.ref.fsdp_config.param_offload=True

```

`actor_rollout_ref.ref`:

- `log_prob_micro_batch_size_per_gpu`: Micro-batch size for reference model.

- `fsdp_config.param_offload`: Whether to offload reference model parameters.

### Critic Model

```

critic.optim.lr=1e-5

critic.ppo_micro_batch_size_per_gpu=1

```

`critic`:

* `optim.lr`: Critic model learning rate. Typically higher than actor learning rate.

* `ppo_micro_batch_size_per_gpu`: Micro-batch size for critic updates. Controls memory usage during critic training.

```

critic.model.use_remove_padding=True

critic.model.path=Qwen/Qwen2.5-VL-3B-Instruct

critic.model.enable_gradient_checkpointing=True

```

`critic.model`:

* `use_remove_padding`: Whether to remove padding in critic inputs. Improves efficiency in critic computation.

* `path`: Base model for critic. Uses the same model architecture as actor.

* `enable_gradient_checkpointing`: Enables memory-saving for critic. Trades computation time for reduced memory usage.

```

critic.model.fsdp_config.param_offload=False

critic.model.fsdp_config.optimizer_offload=False

```

`critic.model.fsdp_config`:

* `param_offload`: Whether to offload critic parameters. False keeps parameters in GPU memory.

* `optimizer_offload`: Whether to offload critic optimizer states. False keeps optimizer states in GPU memory.

### Trainer

```

trainer.critic_warmup=0

trainer.logger=['console','wandb']

trainer.project_name='vagen_new'

trainer.experiment_name='trico_svg_vision_service'

trainer.n_gpus_per_node=4

trainer.nnodes=1

trainer.save_freq=70

trainer.test_freq=20

trainer.total_training_steps=200

trainer.val_before_train=True

trainer.val_generations_to_log_to_wandb=8

```

`trainer`:

- `critic_warmup`: Number of initial steps for critic-only training.

- `logger`: Logging destinations for experiment tracking.

- `project_name`: Project name for logging organization.

- `experiment_name`: Specific experiment identifier.

- `n_gpus_per_node`: Number of GPUs to use per node.

- `nnodes`: Number of compute nodes for distributed training.

- `save_freq`: Checkpoint saving frequency in steps.

- `test_freq`: Validation frequency in steps.

- `total_training_steps`: Total number of training iterations.

- `val_before_train`: Whether to run validation before training.

- `val_generations_to_log_to_wandb`: Number of generations to log in wandb.

### Rollout Manager

```

rollout_manager.max_turns=2

rollout_manager.window_size=3

rollout_manager.use_multi_turn_reward=False

rollout_manager.use_loss_mask=True

rollout_manager.use_gae_mask=True

rollout_manager.n_trajectory=8

```

`rollout_manager`:

- `max_turns`: Maximum number of interaction turns per episode.

- `window_size`: Context window size for previous interactions.

- `use_multi_turn_reward`: Whether to use turn-level rewards.

- `use_loss_mask`: Enables selective token masking for policy optimization.

- `use_gae_mask`: Applies masking to advantage calculations.

- `n_trajectory`: Number of parallel trajectories for batch processing.

## Service Architecture Configuration

### Environment Configuration File (env_config.yaml)

```yaml

env1:

    env_name: frozenlake   # Name of registered environment service

    env_config:    # Specific configs for your enviornment

        render_mode: text

    train_size: 10000   # Number of training environments 

    test_size: 512   # Number of testing environments

env2:

    env_name: svg   # Name of registered environment service

    env_config:     # Specific configs for your enviornment

        dino_weight: 3.0

        dino_only: False

    train_size: 10000   # Number of training environments 

    test_size: 512   # Number of testing environments

```

**env_name**: Specifies which registered environment service to use. Options include:

- `frozenlake`: Simple grid navigation environment

- `sokoban`: Visual puzzle environment with box pushing

- `svg`: SVG-based environment with reward model integration

- `navigation`: Visual navigation task for embodied AI

**env_config**: Environment-specific configuration (full configs should be defined in `env/your_env/configs`)

**train_size**: Number of training environments to generate

**test_size**: Number of testing environments to generate

### Service Configuration File (service_config.yaml)

```yaml

# UNDER DEVELOPMENT

```

 No newline at end of file

# How to Create New Services

This guide explains how to create new environment services for VAGEN's service-based architecture. The service architecture provides a standardized way to manage multiple environments for VLM agent training with enhanced scalability and flexibility.

## Service Architecture Overview

VAGEN uses a client-server architecture for environment management:

- `BaseService`: Abstract base class that defines the interface all services must implement

- `BatchEnvClient`: Client that communicates with environment servers (fixed)

- `BatchEnvServer`: Server implementation for hosting environments (fixed)

This architecture enables efficient parallel processing across distributed systems and seamless integration of both rule-based rewards and reward models.

## Directory Structure

```

vagen/

├── env/

│   ├── base_service.py        # Abstract base class defining the service interface

│   ├── client.py              # Client for interacting with environment server

│   ├── server.py              # Server implementation for hosting environments

│   └── REGISTERED_ENV         # Registry mapping environment names to services

├── utils/

│   ├── serial.py              # Handle observation serialization from service to client

```

## Component Hierarchy

```

BaseService (ABC)

├── **Batch Methods**

    ├── create_environments_batch()

    ├── reset_batch()

    ├── step_batch()

    ├── compute_reward_batch()

    ├── get_system_prompts_batch()

    └── close_batch()

BatchEnvClient

├── **HTTP Communication**

├── **Batch Methods**

└── **Convenience Methods**

BatchEnvServer

├── **Service Management**

├── **Request Routing**

├── **Batch Method Implementation**

└── **Server Management**

```

## Creating a New Service Step by Step

### Step 1: Inherit from BaseService

Create a new class that inherits from `BaseService`. This class must implement all required methods for interacting with environments:

```python

from vagen.env.base_service import BaseService

class MyNewService(BaseService):

    def __init__(self, **kwargs):

        super().__init__(**kwargs)

        # Initialize your environment-specific components here

    def create_environments_batch(self, env_ids, **kwargs):

        # Initialize batch of environments

        # Return initialization status

    def reset_batch(self, env_ids, **kwargs):

        # Reset environments and return observations

    def step_batch(self, env_ids, actions, **kwargs):

        # Process actions and return (observations, dones)

    def compute_reward_batch(self, env_ids, **kwargs):

        # Calculate rewards for each environment

        # Return rewards and any additional info

    def get_system_prompts_batch(self, env_ids, **kwargs):

        # Return system prompts for each environment

    def close_batch(self, env_ids, **kwargs):

        # Clean up resources for environments

```

### Step 2: Observation Serialization

When passing observations between the service and client, ensure proper serialization:

```

from vagen.utils.serial import serialize_observation, deserialize_observation

# On the service side

## at the end of reset_batch()

serialized_obs = serialize_observation(original_observation)

## at the end of step_batch()

serialized_step = serialize_step_result(observation, reward, done, info)

```

### Step 3: Register your Environment

Register your env in `env/__init__.py`

```

from vagen.env.NEW_service import MyNewService

# Register your service

REGISTERED_ENV["my_new_env"] = MyNewService

```

### Step 4: Define your env config and script

> Please refer to `[Configuration](config.md)`

Define your env config and running script in `examples/`

Please refer to `Frozenlake/service.py` for better service structure understanding

 No newline at end of file

# Welcome to VAGEN Documentation!

VAGEN is a multi-turn reinforcement learning framework designed for training Visual Language Model (VLM) agents efficiently.

## Quick Navigation

- [Run Experiment](run-exp.md)

- [Create your Own Environment](create-env.md)

- [Configuration](config.md)

Use the links above to explore the core functionalities of the project.

## Algorithm Settings

VAGEN supports several advantage estimation algorithms, each with different properties for training VLM agents. The table below summarizes which features are enabled by default with each algorithm:

| Setting           | GRPO | GAE | Bi-Level GAE | Turn-Wise GAE | Masked-GAE |

|-------------------|------|-----|--------------|---------------|------------|

| with_loss_mask    | ✓    | ✓   | ✓            | ✓             | ✓          |

| multi-turn-reward | ✗    | ✓   | ✓            | ✓             | ✓          |

| with_gae_mask     | ✗    | ✗   | ✓            | ✓             | ✓          |

### Algorithm Configurations

#### RICO (Traditional RL)

```

algorithm.adv_estimator=grpo

rollout_manager.use_loss_mask=True

rollout_manager.use_gae_mask=False

rollout_manager.use_multi_turn_reward=False

```

#### AICO (Action-centric Optimization)

```

algorithm.adv_estimator=masked_gae

rollout_manager.use_loss_mask=True

rollout_manager.use_gae_mask=True

rollout_manager.use_multi_turn_reward=False

```

#### TRICO (Turn-aware Optimization)

```

algorithm.adv_estimator=bi_level_gae

algorithm.high_level_gamma=0.95

rollout_manager.use_loss_mask=True

rollout_manager.use_gae_mask=True

rollout_manager.use_multi_turn_reward=True

```

mkdocs>=1.4.0

mkdocs-material>=9.0.0

 No newline at end of file