PinoAI Research

Of 47 incidents in PinoAI's first two days of GPU and research work, 33 were silent (a normal exit and a plausible artefact), and not one silent incident announced itself with an error: of those with a recorded detection, 14 were caught by an independent reviewer, 9 by a number that looked implausible, and 7 by a self-audit or reproduction; the two that reached published papers were corrected without changing any registered verdict.

On 12,956 pooled FilBench items, no base model is favoured in both pairs (Gemma 4 12B vs Qwen3.5-9B p=0.092). The Gemma-vs-Qwen direction flips with the task mix, so the base edge is task-dependent.

Our lighteval harness first scored Qwen2.5-7B-Instruct 44.00 on FilBench Belebele (fil) against a published 68.11; with the chat template off it reproduced all three reference tasks within about one standard error on two platforms (65.00 / 79.80 / 97.96 and 65.00 / 78.79 / 97.96 vs 68.11 / 77.78 / 97.96).

In a pre-registered test against 298 native-speaker gold labels, neither open teacher qualified to label Filipino/Taglish pass/fail training data (Gemma 4 12B-it missed both 15% error bounds; Qwen3.5-9B abstained on about 60% of rows), but Gemma's Taglish naturalness scores passed (weighted kappa 0.784 / 0.719), and Gemma passed its own wrong answers 8 of 13 times against 11.1% for other families' answers.

A zero-shot commercial LLM judge is trustworthy only when it fails Filipino/Taglish answers (5 false passes vs 1 false fail, kappa 0.37); a retuned judge (AUC 0.767) was kept out of the pre-registered test, and its outputs will never train our own judge.

On the 70 automatically scored PH-EVAL v0.1 items, five open instruct models range from 61.4% (Sailor2-8B-Chat) to 90.0% (Gemma 4 12B; corrected from 87.1% after a scoring fix); the other 105 items mostly need a human, and no model drifts to English.

On one H100, full-parameter continued pretraining of Gemma 4 E4B with 8-bit AdamW and bf16 stochastic rounding runs at 4,755 tokens/s, 4% below LoRA r64 at each arm's fastest setting and 10% above it at equal settings (micro-batch 4, checkpointing on), and on one A100-SXM4 it is 4.6% faster than LoRA at the selected settings; the H100 is 33-39% cheaper per token than the A100.

Evaluating ten Qwen3.5, Gemma 4 and Sailor2 checkpoints on free 2x T4 and rented A100 GPUs hit eight inference-layer failure modes; five of them produced plausible but wrong artefacts without an error, and one (a partly stripped Gemma thought-channel marker) understated Gemma 4 12B-it's auto-scored PH-EVAL accuracy by two items until the scorer was fixed (61/70 to 63/70).

On one A100-SXM4-80GB with LoRA r64 at 4,096 tokens and gradient checkpointing, four 8-12B candidates need 245-636 GPU-hours for 2B tokens, none within the gate's budget; a chunked loss cut peak memory by 5-34 GiB without changing speed, a fused DeltaNet kernel sped Qwen3.5-9B up by 84%, and run-to-run spread was about 4%.

Pre-registered H-DEC-001 is supported: Gemma 4 12B beats Qwen3.5-9B (36:8 after a scoring correction; originally 36:10) and Sailor2-8B (79:1; originally 79:3) on 175 paired PH-EVAL items, and has the lowest Filipino tokens per word.

Peers adapt open bases with 10-500B-token continued pretraining plus replay, keep large base tokenizers, and use small native SFT sets. For PinoAI, data (not compute or tokenizer) is the critical path.

Qwen2.5-7B-Instruct passes 0/60 native Taglish prompts (English drift or unnatural code-switching); the LLM judge agrees with a native reviewer at kappa 0.37.

Philippine languages need 1.8-2.0x the tokens of English on all six tokenizers; Gemma 4 is the most efficient by 7-12%.

Applying a reading order fixed before any result, PinoAI selected Gemma 4 E4B for its first continued-pretraining run: Gemma 4 12B won the native Filipino test but needs 635.9 GPU-hours, above the approved budget, and its pre-named fallback E4B (301.4 h) passes.

PinoAI Coder is planned as a fine-tune of an open, Apache-2.0 base chosen by a pre-registered bake-off (English pass@1 primary, a Taglish non-inferiority constraint against Gemma), then trained on execution-verified, cascade-distilled data and test-reward RL, and served as merged weights on the user's own hardware; its claimed contribution is the verified data pipeline, the Taglish and embedded evaluations, and a gain over the base that EXP-CODE-001 must pre-register as a paired, per-problem test with its MDE before any distillation spend, not code pretraining.

PinoAI's first round is a light Filipino continued pretraining of Gemma 4 E4B, followed by a coding SFT ablation (SFT-only vs CPT+SFT) on licence-cleared data, while the other six domains wait on agency approvals, a safety review or later rounds.

Three defects in the evaluation that records the untrained model's before-arm for DEC-005 were caught before any after-arm exists. D1: at the pinned harness commit, the core HellaSwag task scores a generated letter by exact match while its sibling tasks score by log-likelihood, so the first English run mixed two rules; all English tasks now use log-likelihood (DEC-005 Amendment 5), which moved HellaSwag from 393 to 395 of 1,000. D2: the run writer saved a failed run's folder (rc 1) as if it were valid; every check (exit code, never-overwrite, harness commit) now runs in one writer before the first byte is written, covered by a with/without test. D3: an auto-fixing linter removed one unused comment from a hash-registered script; the registered-hash check, run on the staged PR, flagged it and the file was held back before commit; re-adding the comment reproduced the registered hash, and the new hash was registered as DEC-005 Amendment 6. No reported number changed in D2 or D3.

The first contract-format corpus manifest wrote the tokenizer as a dict instead of the "<model>@<revision>" string that the trainer's corpus gate parses, so the gate would have raised AttributeError before step 0; it was caught by reading the gate's source before upload; the existing contract test had passed because it exercised the data module's helpers but never the gate itself. Separately, the first cloud corpus build crash-looped at import on a repository-relative path, with 0 running tasks and no compute billed. Third, a registration shortcut advised by the research analysis left two configs registered with a PROPOSED registration line, and the launch was refused by the launcher before any GPU was allocated. All three were fixed; the first is now covered by a regression test that calls the real gate, and the third by restoring the two-step registration.

A line-by-line review of the continued-pretraining trainer, done before any GPU spend, found six blocking defects (divergence guards after the update, training order = file order, no BOS at position 0, an untested GPU resume path, an un-capped data mixture, a guessed learning rate) and eight smaller ones; all were fixed and covered by 12 new CPU tests in one revision, but the fix of the most serious one was first described as stronger than it is, because the v1 checkpoints were never at risk, only the in-memory state.

Nine process errors made by the research analysis itself in its first days (an optimistic compute estimate, a benchmark pool dominated by one task, an unscoped data filter, hand-typed numbers, an oversized test budget, late timestamps, a misplaced publication gate, two licence misreads and an overstated method comparison) were each caught by a separate check, and none changed a registered decision; most share one root cause: a number or rule entered from memory or applied outside its scope.

A character-length proxy counted 12-24 of 175 answers as cut off at the 512-token cap when each model's own tokenizer shows 53-65 (30-37%), and because its recall differed by model (19% for Qwen3.5-9B, 38% for Gemma 4 12B) it reversed a published comparison; a post-hoc sensitivity analysis showed the base-model verdict still holds (17:6, p = 0.035, where neither answer was cut off).

Reviewing the design and the scoring code of a pre-registered LLM-teacher experiment before any teacher output existed found seven defects, including a missing output that scored as a confident pass and a self-preference disqualifier that could never fire; once fixed, the run was scored once, reproduced byte-for-byte, and measured a 61.5% vs 11.1% self-preference that the original code would have reported as 'ok'.

Three reporting-layer errors could have put wrong statements into published papers or deleted the wrong files: a GPU variant quoted from memory (all three Part C runs were A100 80GB PCIe, not SXM4), memory ranges in source notes that disagreed with the result files (10-25 GiB written, 5.2-33.9 GiB measured), and a deletion step with a possibly unset variable; each was caught by someone reading the artefact, by a paper generator that asserts every number, or by a safety check, and the fix is to write from files and design steps that need no deletion.

Eight infrastructure incidents (a shared two-session GPU limit, local drivers killed by memory pressure or a dropped connection, a full disk, Windows file-name semantics, a warm cloud container that kept the previous model's files, orphaned cloud apps, and an unselectable A100 variant) were contained by separating control on the laptop from execution in the cloud and by making every result self-identifying; no result was lost, and the two silent cases (0-byte files, 69 -> 138 -> 207 mixed details files) were caught by checking file counts and sizes.

Before any model was scored, the evaluation stack failed five times on version and layout drift (a transformers release too old for 8 of 10 checkpoints, gated datasets and refused loading scripts that abort a whole lighteval run, a moved dataset config, and an upstream slicing bug); each was fixed with a pin, an opt-in default or a fail-loud patch, and each fix was verified on a small run before the full evaluations.

With 262K and 248K vocabularies the fp32 logits of the loss, not the model, set the memory limit (one OOM requested exactly 16.00 GiB = 4 x 4,096 x 262,144 x 4 bytes); a chunked, recomputed cross-entropy with the same math (|loss difference| at most 6.4e-4) cut peak memory by 5.2-33.9 GiB at equal micro-batch but changed throughput by only -0.2% to +4.2%, because the runs were compute-bound.

Qwen3.5-9B-Base was being measured on transformers' pure-PyTorch DeltaNet fallback; adding the flash-linear-attention kernel raised its throughput by 83.7% (511.1 to 278.2 projected GPU-hours for 2B tokens) while three non-DeltaNet controls in the same image moved -1.7% to -3.5%, but the first proof that the kernel ran was missing because the engagement check read layer attributes instead of the wrapper closure where transformers 5.17 keeps the resolved function.

Qwen3.5's chat template reasons by default and Gemma 4's does not, so before the fix 20 of 20 Qwen3.5-4B sample answers opened with an English reasoning trace and only 7 reached </think>; separately, a leaked Gemma 4 12B-it thought marker in four forms, two of which the first scorer regex missed, understated its auto-scored PH-EVAL result by two items (61/70 to 63/70) and turned one registered sensitivity check from not significant (p = 0.0654) to significant (p = 0.0039).

Full-vocabulary logits for one multiple-choice forward (about 5.9 to 10.3 GiB at 7 choices x 1,000 tokens) exhausted a 14.56 GiB T4; logit trimming fixed homogeneous batches but its single per-batch window re-grew on mixed-length batches (10.09 GiB requested, about 21.8K kept row-positions), so the full runs moved to one A100 80GB, where all three bases completed 12,956 items each.

The compute-gate probe enabled gradient checkpointing on a model left in eval mode, where Hugging Face skips it, so Gemma 4 E4B's LoRA peaked at 72.51 GiB at micro-batch 1; a first diagnosis (leaked memory from caught OOMs) was falsified by a rerun, and calling model.train() cut the peak to 31.93 GiB at a 20.1% throughput cost, the signature of checkpointing actually running.

lighteval's default chat template turned every FilBench answer letter into the first token of an assistant turn, so Qwen2.5-7B-Instruct scored 44.00 on Belebele (fil) against a published 68.11; a two-arm A/B that switched off one suspect per arm isolated the template (65.00 without it, 44.00 digit for digit with it), and with the template off the harness reproduced all three reference tasks on two platforms.

Two FilBench sample runs on Kaggle exited 0 after 1.7 and 0.1 minutes with the byte-identical 900-byte log of an earlier notebook version, because push, status and output were not bound to one execution; a per-push run_id written by the kernel and checked on download stopped all three refused pushes of the next queue within 0.1 minutes.

2 supported, 1 refuted in part, 1 in test, 7 untested, 1 registered and awaiting results, 2 proposed.

First week: tokenizer cost finding, Taglish failure finding, judge calibration.