Data Collection Pipeline

All commands are run from the project root (syzfix/).

Install

git clone https://github.com/sysec-uic/syzfix.git
cd syzfix
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Collect

# Smoke test (10 bugs)
python -m dataset.main collect --limit 10

# Full collection (~7 000 bugs, 8–10 hours)
python -m dataset.main collect

# Faster: skip patchwork fallback
python -m dataset.main collect --skip-patchwork

Progress is saved to dataset/data/progress.db (SQLite) — simply re-run to resume.

Adding new bugs from syzbot (incremental update)

One command pulls the delta, preps everything locally, and uploads:

# Local-only: incremental crawl + rebuild index + refresh analyzers
python -m dataset.update

# Same, then upload to HuggingFace (skipped automatically when nothing is new)
python -m dataset.update --repo xiaoguangwang/syzfix-dataset

# Useful flags
python -m dataset.update --skip-analysis          # skip the analyzer refresh
python -m dataset.update --repo … --dry-run       # show upload stats, upload nothing
python -m dataset.update --repo … --force-upload  # upload even with 0 new bugs

When no new bugs landed on syzbot, the run costs a single API request and exits — cheap enough to schedule. scripts/cron_update.sh wraps the command for cron (logs to dataset/data/cron_update.log); install it on the machine that holds the data, e.g. weekly on Monday 06:00:

0 6 * * 1 SYZFIX_PYTHON=/path/to/venv/bin/python /path/to/syzfix/scripts/cron_update.sh

Run it on the data-holding machine rather than GitHub Actions: the runner is stateless, so it would have to restore the full corpus (~2 GB download, ~11 GB unpacked) and re-upload it on every run just to fetch a handful of new bugs.

Manual release checklist (maintainers)

A full release has three parts: crawl the new bugs, push the data to HuggingFace, and refresh the website. Run everything from the project root on the machine that holds dataset/data/, with the venv activated.

1. Crawl new fixed bugs from syzbot

python -m dataset.update                  # local only: crawl + index + analyzers

This is the same as the cron job, minus the upload. The run ends with a line like Processed bugs: 7463 total (+49 this run). Check progress, or look for bugs that were missed, with:

python -m dataset.main stats              # pipeline progress per step
python -m dataset.retry_missing stats     # what data is missing and why

2. Push the dataset to HuggingFace

Log in once per machine with a token that has write access to xiaoguangwang/syzfix-dataset, then check who you are logged in as:

hf auth login
hf auth whoami

Then upload. dataset.update uploads only when the crawl found new bugs. --force-upload uploads anyway, for example after a crawl whose upload failed or after retry_missing filled in data:

python -m dataset.update --repo xiaoguangwang/syzfix-dataset --dry-run        # preview only
python -m dataset.update --repo xiaoguangwang/syzfix-dataset
python -m dataset.update --repo xiaoguangwang/syzfix-dataset --force-upload   # upload with 0 new bugs

Every upload sends the whole corpus, so one successful upload also covers any earlier weekly uploads that failed. When a login expires, the cron log prints Not logged in. Run: huggingface-cli login and the upload is skipped without an error, so check dataset/data/cron_update.log for it. After uploading, the new commit shows up at https://huggingface.co/datasets/xiaoguangwang/syzfix-dataset/commits/main.

3. Update the website

The site is built from docs/ by the docs GitHub Actions workflow on every push to main. The landing-page numbers come from the local index, so do not edit them by hand. Instead, regenerate them:

python -m dataset.site_stats --check      # are the numbers stale?
python -m dataset.site_stats              # rewrite the stats block in docs/index.md
git add docs/index.md
git commit -m "docs: refresh landing-page stats"
git push origin main

The site updates at https://sysec-uic.github.io/SyzFix/ a minute or two after the workflow finishes (check the repo's Actions tab). To preview locally:

pip install mkdocs-material
mkdocs serve                              # http://127.0.0.1:8000

What it does under the hood

The crawler is incremental by default: it refetches the current syzbot fixed-bug list, merges it into progress.db via INSERT OR IGNORE, and only processes bugs that have not yet reached step=processed. Existing processed bugs are skipped; only the delta (new bug IDs since the last run) is fetched. Watch for a log line like:

Refreshed bug list: 119 new bugs since last run (7066 total)

dataset.update then rebuilds the downstream artifacts (none of them support per-bug incremental updates — they rebuild from the full corpus). The manual equivalents, if you need a single step:

1. Incremental crawl — python -m dataset.main collect (--no-resume forces a full re-crawl)
2. Rebuild the viewer index — python -m dataset.view build-index
3. Re-run the analyzers (~2–3 min) — python -m analysis.run_all
4. Upload — flat export + full processed data:

   python -m dataset.upload_hf --repo xiaoguangwang/syzfix-dataset
   python -m dataset.upload_hf --repo xiaoguangwang/syzfix-dataset --processed
   

5. Research-repo artifacts — training JSONLs and the memory index are rebuilt from syzfix-research (see its README), then uploaded with --training / --memory from there. Point those at a separate HF repo (e.g. …/syzfix-training): the main dataset repo holds only the raw data, and its card defines only the raw config.

Note the upload of the full processed data repacks the whole corpus (~11 GB → ~2 GB gzipped, streamed with constant RAM); dataset.update skips the upload step entirely when the crawl found no new bugs.

Monitor

python -m dataset.main stats

==================================================
Dataset Statistics  (3 736 / 6 982 bugs collected)
==================================================
With crash report:    3726 (99.9%)
With C reproducer:    2792 (74.9%)
With patch diff:      2813 (75.3%)
With discussion:      2823 (75.7%)
With patch evolution:  676 (18.1%)
==================================================
Pipeline Progress:
  pending  : 3244  █████████████
  processed: 3738  ████████████████

Explore interactively

See exploring.md for the full reference.

Retry failures

python -m dataset.retry_missing stats
python -m dataset.retry_missing patches
python -m dataset.retry_missing patches --limit 100

Export

python -m dataset.main export --format jsonl
python -m dataset.main export --format huggingface

Upload to HuggingFace

hf auth login

# Flat structured export (research / analysis)
python -m dataset.upload_hf --repo YOUR_USERNAME/syzfix-dataset

# Training-format JSONL (5 task configs, ~210 MB, streamed)
python -m dataset.upload_hf --repo YOUR_USERNAME/syzfix-dataset --training

# Full processed data for collaborators (~2 GB)
python -m dataset.upload_hf --repo YOUR_USERNAME/syzfix-dataset --processed

# Dry run
python -m dataset.upload_hf --repo YOUR_USERNAME/syzfix-dataset --training --dry-run

Rate limits

Domain	Rate	Notes
syzkaller.appspot.com	0.25 req/s	1 request per 4 seconds
lore.kernel.org	1 req/s
git.kernel.org	1 req/s
patchwork.kernel.org	1 req/s

Corner cases handled

• Missing patch hash — falls back to lore search by commit title
• Google Groups links — skipped; lore links used instead
• Non-lore discussion URLs — filtered out automatically
• 429 rate limiting — exponential backoff (up to 3 retries)
• Very large threads — truncated to 200 emails max
• Multi-repo commits — tries torvalds/linux, then net, net-next, bpf, bpf-next
• Bugs without fix commits — collected anyway (crash + discussion still useful)
• Both syzbot URL formats — handles ?extid= and ?id= with automatic fallback