Projects | Peizhen Zheng

openPangu-v3: agent-assisted distributed pretraining data pipelines

Sat, 01 Aug 2026 00:00:00 +0000

August 2026–present · Independently developed

arXiv pretraining data: implement main-document discovery, dependency expansion, macro normalization, equation preservation/restoration and figure/reference rewriting; bound long-tail costs with PDF fallback and timeout handling.
Human–agent improvement loop: label parser errors daily, periodically collect failure cases with an agent, and use them to iterate on long-tail templates, equations and figures.
Distributed GitHub processing: use Data-Juicer for files / commits / PRs, separating sample-wise cleaning from aggregation. Combine commit-SHA and content hashes with MinHashLSH and semantic deduplication; reuse syntax-validation signals for scoring, then apply quality/language sampling, packing, decontamination and loss masks.

Mon, 01 Jun 2026 00:00:00 +0000

June 2026–present · Independently developed

Designed for RSI: separate the stable runtime from model adaptations; give agents structured tool results, diagnostics, state capture/replay and layered numerical validation. Track hypotheses and evidence in issues, and validate changes through PR review and CI.
Fused graphs and asynchronous speculation: execute target verification and draft generation in one graph; reuse fixed-address CPU/NPU buffers and update block tables incrementally to overlap device proposals with host scheduling.
Parallel tokenization: implement LoPT with a bounded Rust pool and per-worker BPE caches. With DeepSeek-V4’s tokenizer, 16 threads and a 21.8K-token input, measured latency fell from 13.51 ms to 9.79 ms.

Fri, 01 Aug 2025 00:00:00 +0000

August 2025–June 2026 · Core contributor

Architecture selection: analyze acceptance and trade-offs for three-head MTP with a 2048-token sliding window. Multi-token drafts improve bandwidth-bound decoding, while local attention bounds draft compute and KV-cache costs. Build rapid validation tools for pilot experiments.
Full-graph execution: implement missing multi-MTP calls, dynamic TND boundaries and draft padding so decode inputs as small as 1–3 tokens stay in the static graph under P/D disaggregation.
System outcomes: at 128K input / 1K output and concurrency 1 on a co-located A3 node, Flash / Pro achieve 5.63 / 9.55 ms TPOT. With disaggregated P/D, Pro reaches 1,326 tokens/s/NPU at 18.1 ms TPOT (128K/8K, batch 24 per NPU). These figures cover the combined system optimizations.

Sat, 01 Feb 2025 00:00:00 +0000

February–August 2025 · Core contributor

LoPT: implement overlap splitting, character-position matching and dynamic chunk expansion in vllm-ascend (ascend-vllm/omni-infer). DeepSeek-V3 tokenizer latency falls from 622.4 ms to 108.2 ms on LongBenchV2 with 32 processes, with token IDs identical to serial output.
FlashComm 1/2/3: adapt computation reordering, low-dimensional All2All, weight redundancy and multi-stream overlap. At concurrency 8 and input 2K, FlashComm 1/2 reduce prefill from 951 ms to 699/631 ms. FlashComm3 improves high-concurrency decode throughput by 25%–30%; its shared-expert TP8 configuration saves about 2.08 GB per node. These are separate scheme-specific benchmarks.

Sat, 27 Apr 2024 00:00:00 +0000

Sat, 27 Apr 2024 00:00:00 +0000

Sat, 27 Apr 2024 00:00:00 +0000