d12 chat eval (SFT, all tasks)

Funded Job Failed
UUID:370d9a35-c868-4aa3-b039-f32cb13a8564
Tao Linvia🐎 pentiumCreated 207 days agoUpdated 207 days ago

Chat eval on SFT model

Source: SFT checkpoint (MLflow 2bffbfabb3024458ae2778a12f320163)

Tasks: ARC-Easy, ARC-Challenge, MMLU, GSM8K, HumanEval, SpellingBee

Using skip-dryrun, max_time=1200s (20min).

Status: funded_failed (timeout 1200s)

Partial results: ARC-Easy 25.42%, ARC-Challenge 26.96%, MMLU 27.95% (all near random). Generative tasks timed out. Cost: $4.42.

Fix: need longer max_time (3600s) or run tasks individually.

v2 · Edited 207 days ago
1
Dryrun JobSkipped
Funding 
Funded JobFailed
4
Done 

Finance

Fundings at snapshot$8.68

Individual funding and transaction records are not included in the public archive.

Configuration

Repository
https://github.com/mlpatron/mlpatron-nanochat
Version
master → 65c2c6a930c7ad4c2898d249e36dbcad823463f8
Image
ghcr.io/mlpatron/mlpatron-nanochat:latest
→ ghcr.io/mlpatron/mlpatron-nanochat@sha256:a142e86efdcd41e0355e70d46c341cbf9a86b2da9448459009d7b5e8209ffa61
Entrypoint
h100_d12_chat_eval
Resources
NVIDIA H100 80GB × 1·a3-highgpu-1g (26 vCPU / 234 GiB)
Parameters
eval_source=sft·max_problems=9999·tok_max_chars=2000000000·sft_mlflow_run_id=2bffbfabb3024458ae2778a12f320163
Max Time
0.3h(dryrun skipped)