Benchmark & Empirical Study · 2026
When LLM agents take on long-horizon work in local workspaces, cloud help is interleaved throughout the trajectory. EC²-Bench evaluates four edge–cloud collaboration strategies on 128 executable tasks, jointly measuring task performance, resource usage, and privacy exposure.
Bringing in the cloud only part of the time recovers substantial capability at a fraction of the cost: with Qwen3.5-2B, Per-step Routing lifts completion from 12.75 to 39.97 using only 22.7% of the cloud-only cost.
The completion gap across strategies narrows from 25.22 points (Qwen3.5-2B) to 4.76 (Qwen3.7-Plus). Exposure does not converge: with Qwen3.7-Plus, strategies differ by only 4.76 points in completion but by 66.23 in privacy exposure.
Weak edges do best when a router hands steps to the cloud to execute (2B: Per-step Routing 39.97 vs. Edge-Ctrl. Delegation 15.86). Stronger edges do best when they call the cloud themselves and only take its advice (Qwen3.7-Plus: Edge-Ctrl. Advisory 76.84).
Knowing when to call the cloud (edge-controlled delegation beats a matched random-invocation baseline by up to +4.4) and following its guidance (instruction-following training raises advisory completion from 24.8 to 28.8).
Edge devices run small models (typically ≤10B parameters) that keep data local but struggle with demanding planning and tool use. Cloud-hosted frontier models are far more capable, but costly, and they see whatever they are sent. Prior edge–cloud systems coordinate the two once per query. In long-horizon agent execution the cloud joins at different points along the trajectory, and each participation decision shapes later steps, accumulates cost, and determines how much workspace information leaves the device.
Edge-only
Every action comes from the on-device model. Nothing leaves the device, but long-horizon workspace tasks quickly hit a capability wall.
Cloud-only
A frontier model runs every step and completes the most tasks, but most sensitive workspace data reaches the cloud, at the highest cost.
Prior benchmarks
RouterBench, M4A1 and others only route a single query or turn. Cost and leakage that build up over a whole agent trajectory go unmeasured.
Does edge–cloud collaboration remain effective for realistic long-horizon agent execution?
How does increasing edge capability impact edge–cloud collaborative execution?
How should edge–cloud collaboration evolve as edge capability advances?
Two single-side baselines plus four representative collaboration strategies. They differ along two axes: who decides when the cloud participates (an external router or the edge agent itself), and what the cloud does once invoked (generate the action, or advise the edge).
Every action is generated by the edge model. No exposure; capability bounded by the on-device model.
Every action is generated by the cloud model. The capability reference point, and the cost / exposure ceiling.
A RouteLLM matrix-factorization router reads the task instruction once and sends the whole trajectory to either the edge or the cloud.
Before every step, the edge model's next-token entropy (GlimpRouter-style) decides whether that step is generated by the edge or the cloud.
Execution stays on the edge. Through a delegation tool, the edge agent hands the current step to the cloud, which generates the next action before control returns.
Execution stays on the edge. The agent asks the cloud for plans, suggestions, or corrections, then generates the action itself.
Edge–Cloud Collaborative Agent Benchmark. 128 executable digital-workspace tasks, adapted from six agent benchmarks plus 7 newly built ones, run in one OpenClaw runtime, and filtered so that edge-only, cloud-only, and collaborative execution can actually differ. Every run records what the cloud saw at each step; 100 tasks carry fine-grained sensitivity annotations, so that exposure can be audited.
Did the agent get the job done? Task-specific weighted checks (rule-based validation, environment-state auditing, and rubric-based LLM-as-a-judge) yield a completion score, and Pass3 requires success in all three runs.
What did it cost? Cloud input, cache-read, and output tokens with their USD price, plus edge-side FLOPs (2·Nact·T). They are reported as a profile, never collapsed into one number.
What reached the cloud? The fraction of annotated sensitivity units (personal and organizational, 12 categories) disclosed in any cloud-visible context. An LLM judge catches paraphrased leaks with 96.9% F1 against human labels.
| Benchmark | Evaluation scenario | Collab. modes | Evaluation dimensions |
|---|---|---|---|
| RouterBench | General LLM tasks | Rq | Task perf. / Cost |
| LLMRouterBench | General LLM & tool-use tasks | Rq | Task perf. / Cost |
| LLMSwitchBench | Single-turn queries | Rq | Task perf. / Cost |
| M4A1 | Multi-turn dialogue | Rs | Task perf. / Cost / Latency |
| EC²-Bench | Long-horizon agent execution | Rq, Rs, D, A | Task perf. / Resource / Privacy |
Rq / Rs: query-level / per-step routing; D / A: edge-controlled cloud delegation / advisory.
Each research question is answered with the corresponding figure from the paper.
Does edge–cloud collaboration remain effective for long-horizon agent execution?
Cloud-only execution performs best but has the highest cloud cost and exposure; edge-only avoids both but loses performance. Collaboration fills the space in between: with Qwen3.5-2B, Per-step Routing lifts completion from 12.75 to 39.97 at 22.7% of cloud-only cost. Several configurations lie above the dashed line joining edge-only and cloud-only, so they beat a proportional mix of the two.
How does increasing edge capability impact collaborative execution?
The best collaborative configuration climbs from 39.97 (2B) to 76.84 (Qwen3.7-Plus), approaching cloud-only's 82.70, and the completion gap across strategies narrows from 25.22 points to 4.76. Privacy exposure does not converge: with Qwen3.7-Plus, strategies differ by only 4.76 points in completion but by 66.23 in exposure.
How should edge–cloud collaboration evolve as edge capability advances?
With Qwen3.5-2B, externally routed strategies win (Per-step Routing 39.97 vs. Edge-Ctrl. Delegation 15.86), and a cloud that executes beats one that advises. As edge capability grows, both orderings reverse: from 9B upward, Edge-Ctrl. Advisory is best, reaching 76.84 with Qwen3.7-Plus. Two edge-side abilities stand behind the shift: deciding when to call the cloud, and following its guidance.
The takeaway: future edge models should be optimized not only for standalone capability but for collaboration itself, deciding when to call the cloud and making good use of what it returns. Because performance converges while exposure does not, privacy becomes the key differentiator among strategies.
Tasks, executable workspaces, verifiers, sensitivity annotations, and the evaluation harness.
github.com/OpenBMB/EC2-BenchFormulation, task construction, privacy annotation and validation, and the complete study.
Read the PDF@misc{huang2026ec2bench,
title = {Empowering Edge Agents: A Systematic
Analysis of Edge-Cloud Collaboration
in LLM Agent Execution},
author = {Huang, Pengcheng and Liu, Zhenghao and
Ren, Shixin and Yu, Shi and Yao, Sijia
and Yan, Yukun and Xiao, Chaojun and
Sun, Maosong},
year = {2026},
url = {https://github.com/OpenBMB/EC2-Bench}
}