docs/infrastructure-plugins.md

Infrastructure plugins

The batch controller does not launch workers directly. It resolves the infrastructure.backend in a batch configuration and delegates admission, launch-command construction, shared-process execution, trace publishing, and final cleanup to that backend.

The built-in aws plugin preserves the existing EC2 + SSM behavior. External packages can provide workers on another cloud, an on-premises Windows pool, or a local virtualization system without changing the benchmark harness or evaluator.

AWS dependencies are optional in packaged installations:

pip install "autocad-bench[aws]"

The base controller, harness, plugin API, and resume planner import without boto3 or botocore; a third-party backend supplies only its own SDK.

Configuration

backend defaults to aws for compatibility, but every AWS resource is operator-supplied. There are no repository-owned resource defaults:

{
  "infrastructure": {
    "backend": "aws",
    "broker_version": "windows-autocad-2019-v10",
    "image_id": "ami-...",
    "subnet_ids": ["subnet-..."],
    "security_group_id": "sg-...",
    "instance_profile_name": "AutoCADBenchWorkerProfile",
    "instance_type": "g4dn.xlarge",
    "aws_region": "us-east-1"
  },
  "evaluation": {
    "bucket": "my-autocad-bench-bucket"
  }
}

aws_profile is optional. When omitted, boto3 and the AWS CLI use their standard credential chain. Automatic AWS evaluation is enabled by default and requires an explicit bucket in the operator's account. Set "evaluation": {"enabled": false} only when deliberately running without the trusted automatic evaluator.

Third-party backends put backend-owned values in settings:

{
  "infrastructure": {
    "backend": "my-windows-pool",
    "broker_version": "windows-autocad-2019-v10",
    "settings": {
      "pool": "autocad-2019",
      "controller_url_env": "WINDOWS_POOL_URL"
    }
  }
}

Do not put credentials in settings; reference environment variables or the backend's native credential provider.

Plugin contract

Implement autocad_bench.infrastructure.InfrastructurePlugin. The controller calls its methods in this order:

1. validate_spec checks configuration without changing external state.
2. preflight performs read-only capacity and worker-image checks.
3. build_rollout_command wraps the infrastructure-neutral harness command in one command that owns the worker lifecycle.
4. run_shared runs that command when the controller is configured for in-process execution. Subprocess mode executes the returned command directly.
5. create_trace_publisher optionally mirrors live artifacts when trace streaming is configured.
6. reap_batch performs idempotent best-effort cleanup after all rollouts.

The complete protocol and request models live in src/autocad_bench/infrastructure/. A backend must:

• isolate simultaneous rollouts from each other;
• make the configured desktop broker reachable by the child harness;
• execute the exact RolloutLaunchRequest.child_command;
• preserve the requested output directory and process exit code;
• never expose evaluator inputs or credentials to the model-visible desktop;
• make cleanup idempotent, including after a partially failed launch.

The plugin receives evaluation arguments separately because the lifecycle wrapper, not the model harness, owns trusted DWG retrieval and evaluation. Trace publishing is enabled with AUTOCAD_BENCH_TRACE_DESTINATION and an optional AUTOCAD_BENCH_TRACE_PREFIX. The AWS plugin also accepts the legacy AUTOCAD_BENCH_TRACE_S3_BUCKET and AUTOCAD_BENCH_TRACE_S3_PREFIX names.

Registration

Publish the implementation as a Python entry point whose name matches InfrastructurePlugin.name:

[project.entry-points."autocad_bench.infrastructure"]
my-windows-pool = "my_autocad_backend:WindowsPoolPlugin"

After installing that package in the same environment as AutoCAD Bench, use "backend": "my-windows-pool" in the batch config. Entry points are loaded only when selected. Applications embedding the controller can instead use register_infrastructure_plugin() for an in-process registration.

The batch state and preflight report record infrastructure_backend, making mixed-backend result directories auditable.

Interrupted-run recovery

Recovery is an optional second protocol. Backends that support autocad-bench-resume implement autocad_bench.infrastructure.InfrastructureRecoveryPlugin in addition to the standard lifecycle contract:

1. validate_recovery_spec validates reconnect-only settings.
2. create_recovery_session creates the recovery control-plane client.
3. list_active_workers maps rollout IDs to backend worker IDs for a batch.
4. worker_id_from_log supports legacy runs that predate durable batch worker indexing.
5. worker_status determines whether a discovered worker can be reconnected.
6. build_recovery_command wraps the infrastructure-neutral harness resume command.
7. terminate_worker idempotently releases a completed or exhausted worker.

New batch states persist the selected infrastructure spec. Recovery plans copy that spec into every entry, use the backend-neutral worker_id, and can contain entries from multiple backends. AWS plans retain instance_id, instance_state, and instance_termination_requested as compatibility aliases for existing controller tooling.

A backend that intentionally does not support interrupted-session recovery can implement only InfrastructurePlugin. Normal batch cleanup still works, while autocad-bench-resume fails explicitly instead of applying another backend's recovery semantics.