d6-500iter calibration v2

Completed
UUID:ca31e587-dc18-4084-8f12-34ba915a0940
Tao Linvia🐎 pentiumCreated 213 days agoUpdated 213 days ago

Results: COMPLETED

Validation BPB Curve

Stepval_bpb
03.172
501.777
1001.580
1501.431
2001.289
2501.216
300~1.17
350~1.14
400~1.12
450~1.10
5001.096

Key Metrics

  • Final val_bpb: 1.096 (still decreasing, not converged)
  • Training time: 3.36 min (201.7s)
  • Total wall time: ~15.7 min
  • Actual cost: $1.12
  • tokens/scaling_param: 5.65 (vs Chinchilla 20)
  • MFU: ~40%

Observations

  • Loss still clearly decreasing at step 500 → severely under-trained
  • 500 iters = only 54% of Chinchilla-optimal (929 iters)
  • Confirms d6 at 500 iters is NOT a good baseline
  • Need Chinchilla-optimal (929 iters) or larger depth
Dryrun Job 
Funding 
Funded Job 
Done 

Finance

Fundings at snapshot$4.25

Individual funding and transaction records are not included in the public archive.

Configuration

Repository
https://github.com/mlpatron/mlpatron-nanochat
Version
experiment/baseline-search → d7426893a51c9b71bf739e3ef94c13b456b43731
Image
ghcr.io/nblintao/mlpatron-nanochat:latest
→ ghcr.io/nblintao/mlpatron-nanochat@sha256:28c5544c206237e407241b0a72a8707e63e52750f487b751675b51ff1bce1520
Entrypoint
a100_preset
Resources
NVIDIA A100 40GB × 1·a2-highgpu-1g (12 vCPU / 85 GiB)
Parameters
depth=6·eval_every=50·max_seq_len=2048·num_iterations=500(dryrun: 20)·num_data_shards=8·device_batch_size=16