Evidence.

Sourced from docs/pretraining-results.md, docs/data-pipeline.md, docs/project-overview.md, src/config.py, src/training.py, and scripts/py/TrainSession.py. Items marked Pending (Longleaf pull) require artifacts still on the cluster’s /work filesystem and will be filled in once pulled.

The Run — SLURM receipts

The base model was produced by 10 successful 24-hour SLURM submissions on UNC Longleaf’s l40-gpu partition (NVIDIA L40S 48 GB), preceded on day 1 by 3 OOM crashes and 2 user-cancels while the memory-wall fix was worked out. 15 SLURM job IDs total.

Successful submissions

Sub	SLURM Job	End step	End loss	Δ from prev	Date ended
1	53876531	5,000	2.97	—	2026-06-09
2	54045709	10,000	2.78	−0.19	2026-06-10
3	54161733	15,000	2.69	−0.09	2026-06-11
4	54287786	20,000	2.68	−0.03	2026-06-12
5	54543832	25,000	2.60	−0.08	2026-06-13
6	54615718	30,000	2.58	−0.02	2026-06-14
7	54715359	35,000	2.53	−0.05	2026-06-15
8	54802040	40,000	2.54	+0.01	2026-06-17
9	55153663	45,000	2.50	−0.04	2026-06-18
10	55368282	50,000	2.5186	+0.02	2026-06-19

Pre-flight failures (day 1, 2026-06-08)

SLURM Job	Wall time	Failure mode
53873931	1m 06s	OOM in forward at (q @ k.T) * scale — 1 GiB alloc, 44.10 GiB in use
53874107	1m 13s	OOM in backward — 4.64 GiB alloc, 41.32 GiB in use
53874149	1m 30s	OOM in backward, bf16 autocast alone wasn’t enough — 4.64 GiB alloc, 40.97 GiB in use
53874226	—	User-cancel during memory-wall fix
53874294	—	User-cancel during max_steps=10K → 50K swap

Exit-code breakdown (auto-resume proof)

Submissions 1 – 9 all exited with SLURM exit code FAILED 0:15 — the SIGTERM issued by SLURM at wall-time was caught by the training script, the emergency checkpoint was written, and the script exited cleanly before the SIGKILL landed. Submission 10 ended COMPLETED 0:0 — the natural max_steps exit path, which also wrote checkpoints/final.pt.

Scheduling anomaly

Sub 8 sat in the queue for 18.5 hours before it actually started, breaking the “submit at noon → resume tomorrow at noon” cadence the earlier submissions had settled into. The auto-resume machinery handled it without incident, but the resume window is not the same thing as wall-clock cadence — a lesson for anyone modeling cluster throughput.

Source: docs/pretraining-results.md.

The Numbers — training telemetry

Loss trajectory (waypoints)

Reported at each 24 h submission boundary (every ~5,000 steps).

Step	Training loss	Notes
0 (random init)	11.88	≈ ln(151,936) — uniform over the vocabulary
5,000	2.97	End of Sub 1 — the “easy” drop
10,000	2.78	First coherent multi-sentence prose
15,000	2.69	Cosine decay enters meaningful territory
20,000	2.68	First plateau of the run
25,000	2.60	Plateau broke, steady drop
30,000	2.58	Smallest drop of the run so far
35,000	2.53	First numerical claims in samples (units correct, values wrong)
40,000	2.54	Cosine tail flattens
45,000	2.50	First sub-2.5 readings
50,000	2.5186	Final — perplexity ≈ 12.4

Total drop: 9.36 nats (11.88 → 2.5186) — perplexity dropped from exp(11.88) ≈ 144,350 (essentially uniform over the 151,936-token vocabulary) to exp(2.5186) ≈ 12.4, an ~11,600× improvement (exp(9.3614)) over the random-init baseline.

Sample evolution

All samples are the model’s completion of the prompt “The theory of general relativity”, captured at the sample_every=500 boundary. Reproduced verbatim from docs/pretraining-results.md:142-164.

Step	Completion (truncated at ~200 chars)
500	”The theory of general relativity, and the other of the most important types of the world. The most important, the most important, the most important, the most important, and the most important, the mo…” — mode-collapse on one phrase
6,000	”The theory of general relativity is based on the idea that the universe is a continuous, continuous, and continuous, continuous, and continuous, continuous, and continuous, continuous, and continuous,” — still looping but on rarer words
10,000	”The theory of general relativity was developed by the German physicist Max Planck in 1905. The theory was based on the idea that the speed of light is inversely proportional to the square of the dista…” — first coherent prose. Factually wrong (Einstein, not Planck; 1915, not 1905), but grammatically confident
25,000	”The theory of general relativity is that the universe is expanding. The expansion of the universe is caused by the gravitational attraction of the stars and the planets. The expansion of the universe…” — reasoning across sentences, still factually mixed
40,000	”The theory of general relativity is that the universe is expanding, and that the expansion is accelerating. The expansion is caused by the expansion of space itself. The expansion of space is accelera…” — actually correct cosmology. Metric expansion of space — real GR, not motion-through-space
50,000	”The theory of general relativity is that the universe is expanding, and that the expansion is accelerating. The theory of general relativity is a theory of gravity, and it is based on the idea that th…” — final consolidation

The training loss stayed in a 2.50 – 2.54 band from step 35,000 onward, but the prose continued to improve — an example of the scalar loss being a coarse summary of what the model is actually learning.

The Configuration — reproducibility receipts

Training configuration

{
  "max_steps": 50000,
  "seq_len": 2048,
  "batch_size": 2,
  "grad_accum_steps": 64,
  "effective_batch_size": 128,
  "tokens_per_step": 262144,
  "learning_rate_peak": 3e-4,
  "learning_rate_min": 3e-5,
  "warmup_steps": 500,
  "lr_schedule": "cosine_with_linear_warmup",
  "optimizer": "AdamW",
  "weight_decay": 0.1,
  "weight_decay_scope": "matrices_only (skip norm scales and 1D params)",
  "max_grad_norm": 1.0,
  "precision": {
    "cuda": "bf16 autocast (no GradScaler)",
    "mps_cpu": "fp32 (autocast is a no-op)"
  },
  "attention": "torch.nn.functional.scaled_dot_product_attention (FlashAttention dispatch on Ada Lovelace)",
  "loss_softmax_dtype": "fp32 (kept in fp32 for numerical stability at 151,936 vocab)",
  "dataset": "6-source curated corpus, ~13B tokens, uint32 memmap shards",
  "sample_every": 500,
  "save_every": 1000
}

Values are taken from the production pretrain entry point, longleaf/scripts/py/pretrain.py:90-143 (gitignored — see the CoreConfig note below on why the config values are inlined here instead of linked). The blank test-run main() in scripts/py/TrainSession.py:322-372 shares architecture dims, peak/min LR, weight decay, gradient-clip norm, and micro-batch size, but scales down seq_len (256), grad_accum_steps (2), max_steps (50), warmup_steps (10), and save_every/sample_every (25 each) for a fast smoke run — do not read that file as the source of the production JSON above.

Architecture (RQwen3 = Qwen3-0.6B clone)

{
  "d_model": 1024,
  "n_layer": 28,
  "num_heads": 16,
  "num_kv_heads": 8,
  "gqa_ratio": "2:1",
  "head_dim": 128,
  "intermediate_size": 3072,
  "vocab_size": 151936,
  "max_seq_len": 2048,
  "rope_theta": 1000000.0,
  "rms_norm_eps": 1e-6,
  "normalization": "RMSNorm",
  "position": "RoPE",
  "attention_variant": "Grouped-Query Attention",
  "ffn": "SwiGLU",
  "qk_norm": true,
  "biases": "none (all linear layers bias-free)",
  "embedding_tied": false,
  "total_params": "~751M"
}

Source: src/config.py CoreConfig. Note: CoreConfig’s field defaults (d_model=512, intermediate_size=6144) are not the production values — RQwen3 is always instantiated with the values above at every call site. The same disclaimer applies to src/training.py TrainConfig: its field defaults (batch_size=4, grad_accum_steps=8, max_steps=10000, log_every=10) are dev-scale and are overridden at every production call site by the JSON above.

Parameter budget

Independent per-component check that the ~751M total is honest.

Component	Calculation	Parameters
Token embedding	vocab_size × d_model = 151,936 × 1024	~155.6M
Output head (untied)	d_model × vocab_size = 1024 × 151,936	~155.6M
Per-block: attention (Q, K, V, O)	1024×2048 + 1024×1024 + 1024×1024 + 2048×1024	~6.3M
Per-block: SwiGLU (gate, up, down)	3 × 1024 × 3072	~9.4M
Per-block: 2 × RMSNorm + QK-Norm	small	~few K
Per-block subtotal		~15.7M
28 blocks	28 × 15.7M	~440M
Total	embed + output + blocks	~751M

Embedding + output head together account for ≈ 41 % of the total parameter count — the case study’s Act 2 “the dictionary costs as much as the reasoning” claim, arrived at from arithmetic.

Hardware

• Cluster: UNC Longleaf
• Partition: l40-gpu (NVIDIA L40S 48 GB, Ada Lovelace)
• GPUs per submission: 1
• Requested wall time per submission: 24 h
• Usable VRAM budget: ~44 GiB per submission after CUDA overhead

Software

The Artifact — the checkpoint

state_dict schema

Every checkpoint written by this repo (including final.pt) is a torch.save of a dict with these six keys. Source: src/training.py:161-168.

{
    "step":                  int,          # training step at save time
    "model_state_dict":      OrderedDict,  # every weight tensor
    "optimizer_state_dict":  dict,         # AdamW momentum and variance
    "scheduler_state_dict":  dict,         # cosine-schedule position
    "loss_history":          list[float],  # per-step loss up to `step`
    "data_offset":           int,          # examples consumed so far
}

Load snippet (minimum viable)

import torch
from src.config import CoreConfig
from src.models.rqwen3 import RQwen3

config = CoreConfig(
    d_model=1024, n_layer=28,
    num_heads=16, num_kv_heads=8, head_dim=128,
    intermediate_size=3072, vocab_size=151_936,
    max_seq_len=2048,
)
model = RQwen3(config)

ckpt = torch.load("checkpoints/final.pt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

weights_only=False mirrors src/training.py:185: the checkpoint dict carries the optimizer, scheduler, and Python list for loss_history, so a weights-only load would need torch.serialization.add_safe_globals(...). If you only need the model weights for inference, weights_only=True also works — the loader just pulls ckpt["model_state_dict"] and drops the rest.

Retention

The run wrote a checkpoint every 1,000 steps (50 checkpoints), plus the emergency SIGTERM saves and the final checkpoints/final.pt — 51 files, ~8.5 GB each, ~429 GB total sitting on /work. A checkpoint-pruning pass is in the project backlog to reclaim ~270 GB (keep step 1K, every 5K, and the last 3).

The Data — dataset receipts

Six-source manifest

Reproduced verbatim from docs/data-pipeline.md:45-53.

Source	HuggingFace path	Tokens	Share	Quality filter
FineWeb-Edu	HuggingFaceFW/fineweb-edu	~7.0 B	54 %	score ≥ 3, length 100 – 100,000 chars
Wikipedia	wikimedia/wikipedia config 20231101.en	~2.0 B	15 %	drop stubs < 500 chars, drop disambiguation pages
OpenWebMath	open-web-math/open-web-math	~1.5 B	12 %	length ≥ 200
StackExchange	HuggingFaceH4/stack-exchange-preferences	~1.0 B	8 %	asker-selected answer (fallback: highest pm_score), length ≥ 100
peS2o	MaLA-LM/peS2o-final	~1.0 B	8 %	length ≥ 1,000
Textbooks	HuggingFaceTB/cosmopedia config stanford	~0.5 B	4 %	length ≥ 100
Total		~13 B	100 %

Substitutions worth stating explicitly (both surfaced during the build):

• MaLA-LM/peS2o-final replaces allenai/peS2o v2, which ships as a Python loader script that datasets ≥ 5.0 refuses to execute. Same corpus, parquet-native repackaging.
• HuggingFaceTB/cosmopedia (config stanford) replaces nampdn-ai/tiny-textbooks, which is gated on the Hub. Cosmopedia is the Phi-1.5-style synthetic textbook corpus — apache-2.0, ungated, and the direct open continuation of the Phi-1.5 result the mix leans on.

Deduplication

Exact SHA-256 hash of each document, applied within every source. Cross-source dedup between FineWeb-Edu ↔ Wikipedia handles the highest-overlap pair. MinHash was not used — at 13 B tokens, exact-hash catches the verbatim duplicates that actually waste tokens.

Storage format

Every source is pre-tokenized to uint32 (required by the 151,936-token vocabulary — 16-bit overflows at 65,535) and stored as flat memmap shards. Total on-disk footprint: ~52 GB. Memory-mapping lets the OS page shards in on demand; the loader never allocates all 52 GB into memory.

Sizing rationale

Chinchilla-optimal for 751M params is ~15 B tokens (~20 tokens/param). The 13 B budget is a hair under optimal — close enough to be in the right regime, and the run consumes almost exactly one epoch over the full corpus.

Bugs and fixes — condensed pointers

Cluster bring-up (“Seven Ways SLURM Said No”)

Documented in narrative form in Act 5 of the case study. One-line summary of each:

1. Wrong path root. Miscounted directory levels in the sys.path.insert walk-up — every import failed before a tensor was allocated.
2. Venv not activated. SLURM inherits none of the interactive shell; the activation line was commented out.
3. Wrong GPU resource string. gpu:a100:1 doesn’t exist on this cluster; the real string is gpu:nvidia_a100-pcie-40gb:1. sinfo is the authoritative reference.
4. Wrong partition. Submitting to gpu lands you on ancient GTX 1080s; the A100s live on a separate named partition.
5. Module version mismatch. Setup script installed one Python/CUDA version; job script requested slightly different ones.
6. Missing --qos=gpu_access. The GPU partition requires an explicit QoS flag; without it, submission is rejected outright.
7. Production memory profile OOM. Smoke test ran at seq_len=256; production is seq_len=2048. Fixed by the three stacked memory optimizations below.

The three-fix memory-wall stack (2026-06-08)

Applied in order across three failed submissions:

1. SDPA / FlashAttention. Replaced (q @ k.T) * scale with F.scaled_dot_product_attention. Attention memory drops from O(T²) to O(T).
2. bf16 autocast. Wrapped the forward pass and loss in torch.autocast(dtype=torch.bfloat16). Halves activation memory. No GradScaler — bf16 has fp32’s dynamic range, so the fp16-era scaler is dead weight.
3. Halved the micro-batch. Dropped batch_size from 4 to 2, bumped grad_accum_steps from 32 to 64. Effective batch stayed at 128. The remaining fp32 memory floor is the 151,936-vocab cross-entropy softmax, which autocast can’t shrink.

Other bugs

Documented in docs/pretraining-results.md: CUDA-only smoke-test typo (total_mem vs total_memory), datasets ≥ 5.0 breaking allenai/peS2o, nampdn-ai/tiny-textbooks gated, StackExchange schema drift (kept 0 of 10.8 M docs on the first pass before the fix), trust_remote_code argument removed in datasets 5.0, transient HF Hub 503, stdout block-buffering hiding early progress, and a Mac disk-full episode during a local smoke.

What’s known-pending

The four (possibly six) placeholders above will be filled once the following are pulled from Longleaf:

1. sha256sum checkpoints/final.pt — one-line output, run on /work
2. Verified byte size of checkpoints/final.pt — ls -l on /work
3. Fine-grained per-100-step loss CSV — extracted from the 10 successful SLURM stdout logs on /work
4. Per-submission throughput (tokens/sec) — same source
5. (possibly) pip freeze from the cluster-side venv used for the successful submissions
6. (possibly) manifest.json from the finished dataset build, if not already synced

What’s not in this file (on purpose)

• Eval numbers. No ARC / HellaSwag / MMLU results yet; the case study is explicit that the eval harness has not been run against final.pt. This page will not fabricate scores.
• Instruction-tuned samples. final.pt is a base model — a next-token predictor, not an assistant. SFT is future work.
• A dollar cost. Longleaf is free to UNC students; a cloud-equivalent figure would be misleading. Wall clock (11 days) and effective compute (~240 L40S-GPU-hours across the 10 successful submissions) are the honest anchors.

Self-consistency check

Run this in any Python 3 REPL. It reproduces the perplexity numbers, the loss drop, and the parameter total from first principles, so a skeptical reader can confirm sections above without touching the checkpoint.

import math

# ── perplexity + improvement ratio ──────────────────────────────────────
loss_init, loss_final = 11.88, 2.5186
ppl_init  = math.exp(loss_init)      # ≈ 144,350 (~ vocab_size = 151,936)
ppl_final = math.exp(loss_final)     # ≈ 12.4
ratio     = ppl_init / ppl_final     # ≈ exp(loss_init - loss_final) ≈ 11,630
print(f"init ppl {ppl_init:>10,.0f}   final ppl {ppl_final:.2f}   ratio {ratio:,.0f}x")

# ── parameter total ─────────────────────────────────────────────────────
d, n_layers, n_heads, n_kv, head_dim, ffn, vocab = 1024, 28, 16, 8, 128, 3072, 151_936
embed_out = 2 * vocab * d                                    # untied embed + head
attn_per  = d*(n_heads*head_dim) + 2*d*(n_kv*head_dim) + (n_heads*head_dim)*d
ffn_per   = 3 * d * ffn
per_block = attn_per + ffn_per
total     = embed_out + n_layers * per_block
print(f"embed+head {embed_out/1e6:.1f}M   per-block {per_block/1e6:.1f}M   "
      f"blocks {n_layers*per_block/1e6:.1f}M   total {total/1e6:.1f}M")

Expected output:

init ppl    144,351   final ppl 12.41   ratio 11,631x
embed+head 311.2M   per-block 15.7M   blocks 440.4M   total 751.6M