Product guides
Loading documentation…
Datool DocsDatool DocsConfigure the credentials and execution environments used by prompt previews and scorers.
Project provider settings control work performed by the Datool server. Your connected application keeps its own model-client credentials. Organization owners and admins manage provider settings; members can see configuration status without reading credentials.
Open Project settings → AI providers → Add provider, choose the provider, and save its API key. Supported choices are Vercel AI Gateway, OpenAI, and TypeSafe AI.
| Provider | Use |
|---|---|
| Vercel AI Gateway | Chat models and supported native evaluation models through Gateway. |
| OpenAI | Direct OpenAI model calls for prompt previews and LLM scorers. |
| TypeSafe AI | Native evaluation models with configured numeric score mappings. |
Choose the provider and model in the prompt or scorer editor. A saved key means configuration exists; it does not prove validity, account quota, or access to the chosen model. Test one representative example before evaluating a large dataset. Model calls can incur usage.
Keys remain server-side and project-scoped. Replacing one provider's key does not change another provider. Executions do not silently switch to another key. Older providerless scorer versions retain their legacy server configuration; save a new version with an explicit provider to make selection clear.
Prompt model metadata and displayed prices come from a cached catalog. They are not an account-specific billing quote. A runtime request establishes whether the actual account can use the selected model.
JavaScript and Python scorers use Project settings → Sandbox providers. Configure Local container, Vercel Sandbox, or Modal, then choose the default provider. Other configured providers are fallbacks in the displayed order.
Cloud sandboxes receive scorer code and the evidence being scored. Provider credentials are not injected into the scorer environment. Removing every sandbox provider prevents code scorers from running; there is no fallback to executing code directly on the application host.
Startup, authentication, and transport failures can try the next provider. A completed scorer result, code error, protocol error, or execution limit ends that attempt chain. A failing quality score is not a reason to try another sandbox. Results retain the selected provider and attempt history.
Use check_scorer_runtime to inspect configuration without contacting providers. Explicitly use probe_scorer_runtime for up to three scorers on one representative case. A probe can incur model or sandbox usage and records diagnostic evidence; it does not create a saved evaluation.
The deterministic AutoEvals library scorers do not need a model or hosted sandbox account. Start with Exact match to verify the evaluation workflow, then configure the provider required by your chosen scorer.