Skip to content
Open in molab View on GitHub Download .py

Hub environment tutorial

The Prime Intellect Environments Hub page covers the prime commands. This page is the hands-on complement: it runs the actual psychscanner-nback environment module, end to end, entirely locally — no prime account, no API key, no cost. This is exactly what prime env push uploads and what prime eval run executes on the Hub; here you can see it work before publishing anything.

1. Load the pieces load_environment() builds

psychscanner_nback.load_environment() wraps a dataset and a parser in a verifiers.SingleTurnEnv — the object prime eval run actually drives. That wrapper installs its own SIGINT/SIGTERM handlers on construction, which only works on a notebook kernel's main thread; this page's cells don't run there, so this tutorial builds the same two pieces directly instead, the exact same way load_environment() does internally. Nothing about the dataset or the scoring logic differs — only the final SingleTurnEnv wrapping step is skipped here.

import sys
from pathlib import Path

sys.path.insert(0, str(Path("../environments/psychscanner_nback").resolve()))
import psychscanner_nback as nback
import verifiers as vf

dataset = nback._build_dataset()
parser = vf.XMLParser(fields=["answer"], answer_field="answer")

Built a dataset with 132 rows — the same Dataset object load_environment() would hand to SingleTurnEnv, straight from nback_demo.json.

2. Look at the dataset

dataset.to_pandas()
promptanswerinfo
0You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 1 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nLast 1 letters shown (oldest to newest): C.\nCurrent letter: Kno-match{'trcode': 'n1_conversation_1', 'n': 1, 'memory_mode': 'conversation'}
1You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 1 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nLast 2 letters shown (oldest to newest): C, K.\nCurrent letter: Dno-match{'trcode': 'n1_conversation_2', 'n': 1, 'memory_mode': 'conversation'}
2You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 1 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nLast 3 letters shown (oldest to newest): C, K, D.\nCurrent letter: Rno-match{'trcode': 'n1_conversation_3', 'n': 1, 'memory_mode': 'conversation'}
3You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 1 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nLast 4 letters shown (oldest to newest): C, K, D, R.\nCurrent letter: Sno-match{'trcode': 'n1_conversation_4', 'n': 1, 'memory_mode': 'conversation'}
4You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 1 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nLast 5 letters shown (oldest to newest): C, K, D, R, S.\nCurrent letter: Yno-match{'trcode': 'n1_conversation_5', 'n': 1, 'memory_mode': 'conversation'}
............
127You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 3 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nSummary of the 9 earlier letters (counts): Bx1, Dx1, Nx1, Px1, Sx1, Tx2, Wx1, Xx1. Most recent 10 letters (oldest to newest): Y, C, X, N, C, B, N, C, R, N.\nCurrent letter: Hno-match{'trcode': 'n3_summary_19', 'n': 3, 'memory_mode': 'summary'}
128You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 3 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nSummary of the 10 earlier letters (counts): Bx1, Dx1, Nx1, Px1, Sx1, Tx2, Wx1, Xx1, Yx1. Most recent 10 letters (oldest to newest): C, X, N, C, B, N, C, R, N, H.\nCurrent letter: Lno-match{'trcode': 'n3_summary_20', 'n': 3, 'memory_mode': 'summary'}
129You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 3 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nSummary of the 11 earlier letters (counts): Bx1, Cx1, Dx1, Nx1, Px1, Sx1, Tx2, Wx1, Xx1, Yx1. Most recent 10 letters (oldest to newest): X, N, C, B, N, C, R, N, H, L.\nCurrent letter: Jno-match{'trcode': 'n3_summary_21', 'n': 3, 'memory_mode': 'summary'}
130You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 3 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nSummary of the 12 earlier letters (counts): Bx1, Cx1, Dx1, Nx1, Px1, Sx1, Tx2, Wx1, Xx2, Yx1. Most recent 10 letters (oldest to newest): N, C, B, N, C, R, N, H, L, J.\nCurrent letter: Hmatch{'trcode': 'n3_summary_22', 'n': 3, 'memory_mode': 'summary'}
131You are taking part in an n-back working-memory task. Letters are shown one at a time. For each new letter, judge whether it is identical to the letter shown exactly 3 letter(s) earlier in the sequence. Respond with only your answer, wrapped in <answer>match</answer> or <answer>no-match</answer> tags.\n\nSummary of the 13 earlier letters (counts): Bx1, Cx1, Dx1, Nx2, Px1, Sx1, Tx2, Wx1, Xx2, Yx1. Most recent 10 letters (oldest to newest): C, B, N, C, R, N, H, L, J, H.\nCurrent letter: Lmatch{'trcode': 'n3_summary_23', 'n': 3, 'memory_mode': 'summary'}

132 rows × 3 columns

Each row is one trial: a letter sequence position, tagged with its n-back level (n) and history condition (memory_mode, either conversation — a raw trailing window — or summary — older letters folded into counts). answer carries the ground-truth match / no-match judgment the rubric checks against — it's never shown to the model.

3. Score a completion

# Simulating what a model's raw completion would look like for one
# trial, then scoring it exactly the way the Hub rubric does.
match_row = next(r for r in dataset if r["answer"] == "match")
nomatch_row = next(r for r in dataset if r["answer"] == "no-match")

match_completion = "<answer>match</answer>"
nomatch_completion = "<answer>no-match</answer>"

match_score = nback.nback_correct(match_completion, match_row["answer"], parser)
nomatch_score = nback.nback_correct(nomatch_completion, nomatch_row["answer"], parser)
Ground truth Completion nback_correct
match <answer>match</answer> 1.0
no-match <answer>no-match</answer> 1.0

Both score 1.0 — the completion's judgment matches the trial's ground truth. Try swapping match_completion for <answer>no-match</answer> and re-run — the score drops to 0.0.

4. Ship it

Everything above ran from the plain Python module in environments/psychscanner_nback/. Once your own environment (see Contributing a task) does the same locally, publishing it is just:

cd environments/<your_task_name>
prime env push

See Prime Intellect Environments Hub for the full command reference — prime eval run, costs, and local-only alternatives via vf-eval.