NetArena Fork

This project forks NetArena (ICLR 2026) to study how AI coding agents perform on network automation tasks. Our goal is to reproduce the existing benchmarks, analyze the framework design, and extend it to wireless network optimization tasks that are currently absent from all existing benchmarks.

NetArena evaluates LLM agents on three network automation tasks. In all three cases, the abstract pattern is the same: the agent receives a natural language task description plus access to a network environment, and must produce an action (code, command, or configuration) that changes the environment to a desired state.

Input:   Natural language task  +  Network environment state
            |                           |
            v                           v
Agent:   Understand the goal    +  Read current state
            |
            v
Output:  Action (code / command / config)
            |
            v
Verify:  Execute action in environment, check desired state reached
         Score on: Correctness + Safety + Latency

All three tasks operate on wired networks and IT infrastructure. Wireless resource allocation, physical layer optimization, UAV trajectory planning, spectrum management, and network slicing are entirely absent. This is the gap we aim to fill.

Agent 拿到一个用 Mininet 搭建的虚拟网络，网络里有一个路由配置错误导致某些主机之间不通。Agent 需要先读取网络状态（路由表、接口配置），判断哪里配错了，然后输出修复命令。评测端执行命令后检查 ping 是否恢复、路径是否正确。

输入：一个有故障的网络 + "H1 无法 ping 通 H3"
输出：修复命令（比如修改某条静态路由的下一跳地址）
验证：修复后 ping 成功 + 路由路径正确

Agent 拿到一个数据中心的拓扑图（用 networkx 图表示，包含交换机、端口、机架的层级关系），以及一条自然语言指令。Agent 需要生成 Python 代码来操作这个图，比如添加设备、删除端口、查询统计信息。评测端执行代码后，对比结果图和标准答案图。

输入：数据中心拓扑图 + "在 ju1.s4.dom 下添加一台交换机"
输出：Python 代码（调用 networkx API 操作图）
验证：执行代码，对比输出图与标准答案

Agent 拿到一个运行中的 K8s 集群，以及一条访问控制需求。Agent 需要生成 NetworkPolicy YAML 配置来实现 Pod 之间的网络隔离。评测端将配置应用到集群后，检查允许的流量能通、禁止的流量被拦截。

输入：K8s 集群 + "禁止 namespace-A 的 Pod 访问 namespace-B 的数据库"
输出：NetworkPolicy YAML 文件
验证：应用策略后，测试流量是否按预期通断

自然语言任务描述 + 网络环境当前状态
        |
        v
Agent 理解目标 + 读取环境
        |
        v
生成动作（代码 / 命令 / 配置文件）
        |
        v
在环境中执行动作
        |
        v
检查：正确性 + 安全性 + 响应时间

三个任务都是有线网络和 IT 基础设施。无线资源分配、UAV、频谱管理等方向完全没有覆盖。

MALT evaluates whether an LLM can act like a network engineer: understand a task instruction in natural language, then write correct Python code to manipulate a data center network topology graph.

"Add new node with name new_EK_PACKET_SWITCH_9 type EK_PACKET_SWITCH, to ju1.s4.dom. Return a graph."

def process_graph(graph_data):
    import networkx as nx
    graph_copy = graph_data.copy()
    target_parent = 'ju1.s4.dom'
    graph_copy.add_node('new_EK_PACKET_SWITCH_9',
                        type=['EK_PACKET_SWITCH'])
    graph_copy.add_edge(target_parent,
                        'new_EK_PACKET_SWITCH_9',
                        type='RK_CONTAINS')
    return {'type': 'graph', 'data': graph_copy}

Benchmark App                    Agent Server                 LLM API
     |                               |                          |
     |-- "Add switch to node X" ---->|                          |
     |                               |-- forward task --------->|
     |                               |                          |
     |                               |<-- return Python code ---|
     |<-- return code ---------------|                          |
     |                               |                          |
     | Execute code in simulator                                |
     | Compare result vs ground truth                           |
     | Output: Correct/Wrong + Safe/Unsafe + Latency            |

Think of it as a coding exam: the question is "use Python to operate a graph database", the student (LLM) writes code, the examiner (benchmark) runs the code and checks against the answer key. NetArena generates exam questions dynamically so the LLM cannot memorize answers.

All three benchmarks focus exclusively on wired network configuration (L3 routing, data center, Kubernetes). No existing benchmark covers wireless network optimization tasks such as power control, UAV trajectory planning, beamforming, spectrum management, or network slicing.

# Clone and setup
git clone git@github.com:tenderzada/NetArena.git
cd NetArena
conda create -n netarena python=3.12 -y
conda activate netarena
pip install -e .
pip install litellm loguru "a2a-sdk[http-server]" uvicorn cattrs tomli httpx jsonlines prototxt_parser scipy

# Terminal 1: Start Agent Server
export DASHSCOPE_API_KEY=sk-xxx
python a2a_llm/litellm_a2a_server.py \
  --model-name "openai/qwen3.5-flash" \
  --api-key "$DASHSCOPE_API_KEY" \
  --api-base-url "https://dashscope.aliyuncs.com/compatible-mode/v1" \
  --host 127.0.0.1 --port 8000

# Terminal 2: Run Benchmark
cd app-malt
cp config.template.toml config.toml
python run.py --config config.toml