Each run takes ~4 hours to reach 150 steps on 4 H100s. You can decrease testing frequency to speed up training. Training might be unstable due to loss spikes; we recommend restoring from the latest checkpoint when encountering such cases. We will resolve this issue in future work (see roadmap).
## Service and Environment Architecture
We introduce a new service-based architecture that addresses the challenges of managing multiple embodied environments for VLM agent training. This design enables efficient parallel processing across distributed systems, providing better scalability, standardization, and the ability to seamlessly integrate both rule-based rewards and reward models within the same framework
### Directory Structure
```
vagen/
├── env/
│ ├── base_service.py # Abstract base class defining the service interface
│ ├── client.py # Client for interacting with environment server
│ ├── server.py # Server implementation for hosting environments
│ └── REGISTERED_ENV # Registry mapping environment names to services
├── utils/
| ├── serial.py # Handle observation serilization from service to client
```
### Component Hierarchy
```
BaseService (ABC)
├── **Batch Methods**
├── create_environments_batch()
├── reset_batch()
├── step_batch()
├── compute_reward_batch()
├── get_system_prompts_batch()
└── close_batch()
BatchEnvClient
├── **HTTP Communication**
├── **Batch Methods**
└── **Convenience Methods**
BatchEnvServer
├── **Service Management**
├── **Request Routing**
├── **Batch Method Implementation**
└── **Server Management**
```
### New Service Development
To develop a new environment service, you only need to:
- Create a service class that inherits from `BaseService`
- Register the service with `REGISTERED_ENV`
The FrozenLake service example demonstrates how to:
- Handle batch operations
The SVG service example demonstrates how to:
- Integrate Reward Models (like DINO) directly within the service
- Combine rule-based rewards with model-based rewards
We use the service-based architecture as an enhanced approach for managing environments in VAGEN. This document explains the key configuration parameters for setting up and running experiments with the service architecture.
## Experiment Configuration
### Algorithm
```
algorithm.adv_estimator=bi_level_gae
algorithm.high_level_gamma=1.0
algorithm.kl_ctrl.kl_coef=0.001
```
`algorithm`:
-`adv_estimator`: Sets the advantage estimation method. Set to `bi_level_gae` for TRICO's cross-turn credit assignment.
-`high_level_gamma`: Discount factor for turn-level advantage calculations. Value of 1.0 means no discount across turns.
-`kl_ctrl.kl_coef`: Coefficient for KL divergence penalty. Controls policy deviation from reference model. Default 0.001
This guide explains how to create new environment services for VAGEN's service-based architecture. The service architecture provides a standardized way to manage multiple environments for VLM agent training with enhanced scalability and flexibility.
## Service Architecture Overview
VAGEN uses a client-server architecture for environment management:
-`BaseService`: Abstract base class that defines the interface all services must implement
-`BatchEnvClient`: Client that communicates with environment servers (fixed)
-`BatchEnvServer`: Server implementation for hosting environments (fixed)
This architecture enables efficient parallel processing across distributed systems and seamless integration of both rule-based rewards and reward models.
## Directory Structure
```
vagen/
├── env/
│ ├── base_service.py # Abstract base class defining the service interface
│ ├── client.py # Client for interacting with environment server
│ ├── server.py # Server implementation for hosting environments
│ └── REGISTERED_ENV # Registry mapping environment names to services
├── utils/
│ ├── serial.py # Handle observation serialization from service to client
```
## Component Hierarchy
```
BaseService (ABC)
├── **Batch Methods**
├── create_environments_batch()
├── reset_batch()
├── step_batch()
├── compute_reward_batch()
├── get_system_prompts_batch()
└── close_batch()
BatchEnvClient
├── **HTTP Communication**
├── **Batch Methods**
└── **Convenience Methods**
BatchEnvServer
├── **Service Management**
├── **Request Routing**
├── **Batch Method Implementation**
└── **Server Management**
```
## Creating a New Service Step by Step
### Step 1: Inherit from BaseService
Create a new class that inherits from `BaseService`. This class must implement all required methods for interacting with environments:
```python
fromvagen.env.base_serviceimportBaseService
classMyNewService(BaseService):
def__init__(self,**kwargs):
super().__init__(**kwargs)
# Initialize your environment-specific components here
VAGEN is a multi-turn reinforcement learning framework designed for training Visual Language Model (VLM) agents efficiently.
## Quick Navigation
-[Run Experiment](run-exp.md)
-[Create your Own Environment](create-env.md)
-[Configuration](config.md)
Use the links above to explore the core functionalities of the project.
## Algorithm Settings
VAGEN supports several advantage estimation algorithms, each with different properties for training VLM agents. The table below summarizes which features are enabled by default with each algorithm:
