Evaluar

Where predictions come from — the three built-in connectors and how to plug in your own.

A connector is the half of a pipeline that decides where predictions come from. Evaluar ships three connectors that cover most cases; pipelines pick one via the PipelineBuilder chain.

The connector classes live in src/evaluar/connectors/ and the BaseConnector interface is in src/evaluar/connectors/base.py.

CallableConnector

The model is a Python callable in-process.

from evaluar.api import detection

def my_model(image_url: str) -> dict:
    return {"prediction": [...]}

pipeline = (
    detection("my_model")
    .callable(my_model)
    .inputs({"sample_001": {"image_url": "..."}})
    ...
)

Argument	Default	Purpose
`fn`	required	A callable taking the per-sample input dict's value (or kwargs derived from it) and returning a raw response.
`model_version`	`None`	Optional version string recorded in run metadata.

Use this when iterating locally on a model that already runs in Python. It's also the connector the test suite uses (tests/e2e/test_code_first_detection_spike.py).

FixtureConnector

The model is a JSON file of pre-recorded responses.

pipeline = (
    detection("my_model")
    .fixture("evaluar/fixtures/my_model_responses.json")
    ...
)

The fixture file is a dict keyed by sample_id, with each value being the raw response the model would have returned. Fixtures are deterministic and fast; the test suite uses them for snapshot-style evaluation.

HttpConnector

The model is behind an HTTP endpoint.

pipeline = (
    detection("my_model")
    .http(
        base_url="http://localhost:8000",
        endpoint="/predict",
        timeout=30.0,
        headers={"Authorization": "Bearer ..."},
        request_transform=lambda sample: {"inputs": sample},
    )
    ...
)

Argument	Default	Purpose
`base_url`	required	Origin to call.
`endpoint`	required	Path appended to `base_url`.
`timeout`	`30.0`	Per-request timeout (seconds).
`headers`	`None`	Static HTTP headers to send with every request.
`request_transform`	`None`	Optional Python callable that reshapes each per-sample input before POSTing it as JSON.
`model_version`	`None`	Optional version string recorded in prediction source metadata.

The default request body is exactly the per-sample input from .inputs(...). For example, .inputs({"s1": {"image_url": "..."}}) posts {"image_url": "..."} for s1. Use request_transform when a production endpoint needs a wrapper shape such as {"inputs": {"image_url": "..."}}.

Built on httpx. Retries are handled by stamina upstream of the connector. HTTP errors include the status code, URL, and a short response-body excerpt to make endpoint debugging less blind.

Custom connectors

Subclass BaseConnector and pass an instance via .connector(...):

from evaluar.connectors.base import BaseConnector, RawResponse

class MyConnector(BaseConnector):
    def fetch(self, sample_id: str, sample_input: dict) -> RawResponse:
        ...

pipeline = (
    detection("my_model")
    .connector(MyConnector())
    ...
)

The connector returns a RawResponse. The pipeline's normalizer (see Normalization) is what turns that into the canonical prediction schema.

Connectors in YAML

Connectors are also expressible in evaluar.yaml:

models:
  my_model:
    type: detection
    connector:
      type: http              # or "fixture" or "callable"
      base_url: http://localhost:8000
      endpoint: /predict
      timeout: 30.0

See YAML manifests for the manifest schema and where the file is consumed.

Connectors

CallableConnector

FixtureConnector

HttpConnector

Custom connectors

Connectors in YAML

On this page