Skip to content
View on GitHub

Using psychscanner-primal

Four different things bring people to this repo. Pick the one that matches what you're actually trying to do — they need different tools, and some of them don't even need this repo installed.

1. Just run an existing Hub environment

You don't want to write any code — you want to test a model against a task that's already published, like psychscanner-nback.

You don't need psychscanner-primal at all for this. Install the prime CLI and run:

prime eval run psychscanner-nback -m openai/gpt-4.1-mini -n 8 -r 3

See Prime Intellect Environments Hub for the full command reference, and Hub environment tutorial to see what that environment actually does before you spend money running it.

2. Publish your own task to the Hub

You have a new cognitive task in mind and want anyone to be able to run prime eval run your-task against it.

psychscanner-primal is a dev-time tool here, not a runtime dependency of what you publish. The workflow:

  1. Author your task as a task card (JSON stimuli/trials) and dry-run it locally against the free mock-llm family using ExpCard/ScannerModel — see Quickstart and Write and run your own task. This is the fast iteration loop, no verifiers, no Hub, no cost. If the trial sequence itself should depend on the model's response (retry, adaptive difficulty), see Conditional Next Trial.
  2. Once the task card and scoring logic are right, port them into a verifiers-based environment module (environments/<task_name>/<task_name>.py with a load_environment() function) — see Contributing a task for the exact folder shape, and Hub environment tutorial for a worked example.
  3. prime env push to publish — see Prime Intellect Environments Hub.

One accuracy note: your published environment module does not have to import psychscanner at runtime. The shipped psychscanner_nback.py example doesn't — it only depends on verifiers and datasets, reimplementing its own parsing/scoring in plain Python. You can import psychscanner.parsers or other pieces from this package in your environment module if you want to reuse them, but then you must add psychscanner-primal to your environment's own pyproject.toml dependencies (as a git dependency, since it isn't on PyPI — see §4).

3. Contribute a task card to the task library

This is a different, smaller thing than publishing to the Hub, and it has nothing to do with Prime Intellect. task_library() is a plain filename-based lookup: anyone using the psychscanner Python API can call task_library("your_task_name") and get back whichever your_task_name.json it finds first, searching in order:

  1. A dirs= argument passed to the call.
  2. Each directory in the PSYCHSCANNER_TASK_LIBRARY_DIRS environment variable (os.pathsep-separated).
  3. ./demonstrations (relative to the current working directory).
  4. ./tasks (relative to the current working directory) — where this repo's own bundled task cards live, in examples/tasks/.

To contribute a task card for others to reuse this way, there's no PR, no validator, no ledger — you just drop <your_task_name>.json into a demonstrations/ folder (your own, or one shared with a team via PSYCHSCANNER_TASK_LIBRARY_DIRS), and it's immediately fetchable by that filename. This is purely about sharing raw task data for local/programmatic use — it carries no reward signal or Hub packaging on its own. If you want the task scored and runnable by the wider world via prime eval run, that's §2, not this.

Fetching the vetted card index (download_lib)

The workflow above assumes you already have a directory of cards on disk (a checkout of this repo, or your own project's tasks/). For the separate, versioned index of vetted cards — psyscan-library — download_lib() clones/updates a checkout for you and hands back a path ready to pass to task_library's dirs=:

from psychscanner import download_lib, task_library

paths = download_lib()  # library="primal" (this package)
card = task_library("rm_singleturn_demo", dirs=paths["tasks"])

Unlike full psychscanner's download_lib(), this distro has no experiment_library — there's no kind= parameter here, it only ever hands back {"tasks": Path}.

By default download_lib() checks that library= (default "primal") matches the package actually installed in this environment — cards aren't portable between psychscanner and psychscanner-primal — and raises rather than handing back cards that won't run here. Pass library="psychscanner" or library="all" to opt into fetching the other distro's cards anyway (inspecting/porting, or CI that covers both). See the docstring in src/psychscanner/library_download.py for the full parameter reference (dest, ref).

See Pulling Cards from the Library Ecosystem for a runnable, side-by-side worked example that also fetches a card from psychscanner-cog-atlas — the other public card index, which has no download_lib()-style helper yet.

Running a fetched card in one call (run_card)

Fetching a card is only half the work — running it still means building an ExpCardInit, wrapping it in ExpCard, and calling ScannerModel(...).run() by hand. run_card() chains task_library() + all three of those into one call:

from psychscanner import run_card

results = run_card("rm_singleturn_demo", dirs="examples/tasks")

Equivalent to the ExpCardInit/ExpCard/ScannerModel chain shown in Quickstart. run_card's keyword defaults (model, family, memory, cogtype, nsim) match ExpCardInit's own defaults — override any of them, or pass through any other ExpCardInit field (parser, parameters, ...) as extra keyword arguments. proj_dir defaults to ExpCardInit's own default (~/psychscanner) when omitted. Reach for the longer form instead when you need the ExpCard/ScannerModel object itself, e.g. to call to_csv(scanner, ...) afterward. Full parameter reference in the docstring: src/psychscanner/run_card.py.

4. Install as a developer

psychscanner-primal isn't on PyPI — the only supported install path is an editable install from a clone:

uv venv psyscan-primal --python 3.11
source psyscan-primal/bin/activate

git clone https://github.com/saurabhr/psychscanner-primal.git
cd psychscanner-primal
uv pip install -e .

Optional extras, same pattern as any pyproject.toml extras group:

uv pip install -e ".[tests]"       # pytest
uv pip install -e ".[multimodal]"  # beautifulsoup4, httpx
uv pip install -e ".[docs]"        # marimo-book, verifiers, datasets — to build this site
uv pip install -e ".[dev]"         # tests + multimodal together

This also installs a psychscanner-primal console command ([project.scripts] in pyproject.toml, wired to psychscanner.cli:cli) — see CLI reference for every flag — and, since it's editable, any change you make to src/psychscanner/ is live immediately — no reinstall needed. Run the test suite with pytest tests/ from the repo root.