Performance study · 18 October 2026

16 GB/s of JSON.
On a laptop.

From Daniel Lemire’s server benchmark to a 24-core Windows laptop: isolated WSL, free-threaded Python, and fewer Python objects.

24.8% higher throughput with field projection versus lazy views, at 24 threads.

Daniel Liezrowice

Engineering Software Lab (ESL)

LinkedIn · eswlab.com

Measured here · 24 threads

16.07 GB/s

Median of seven runs.
7.577 GB counted in 0.472 seconds.

Table 1 · Median throughput, GB/s

Threads	Lazy	Fields
1	1.41	1.67
4	4.38	5.38
8	7.64	8.94
16	11.99	14.37
24	12.87	16.07

In-memory parsing and counting only. Shared laptop, not dedicated hardware. No claim to beat the article’s server.

02 Starting point

Can we take the same idea further?

The inspiration · Daniel Lemire

65.7 GB/s

Reported for lazy JSON event counting on a 64-core dual-Xeon server.

Python 3.14.6t · fastsimdjson 0.6.0
Best of seven runs · data already in memory.

Read the original article, October 12, 2026

The useful question

What can this laptop achieve, and does extracting only three fields outperform lazy object access?

The upstream improvement

fields= already exists in fastsimdjson. It projects selected paths instead of returning document views.

The current upstream documentation reports 78 GB/s vs 61 GB/s on its server comparison. These are not our measurements.

What this study does not claim

We did not invent projection or write a faster parser. We reproduced the workload, compared two existing APIs, and measured the result on different hardware.

Different CPUs, versions and reporting statistics mean the server and laptop figures are not directly comparable.

03 Environment

A separate Python, not a separate machine

System inspected before testing

Machine	ASUS ROG Strix SCAR 16
Processor	Core Ultra 9 275HX 24 cores / 24 threads
SIMD in WSL	AVX2; no AVX-512 exposed
Memory	64 GB host; 48 GB WSL
Runtime	Python 3.14.8t fastsimdjson 0.8.0
Host	Windows 11 Pro + WSL2

The first obstacle

Windows had standard Python 3.14. WSL had Python 3.12. Neither was free-threaded.

Installed a separate 3.14.8t interpreter and virtual environment under the Linux filesystem. Verified the GIL stayed disabled after importing fastsimdjson.

Isolation has a boundary

Dependencies and files were isolated. CPU time was still shared with other Windows and WSL work. No power-plan changes, process termination or exclusive core allocation.

04 Workload

2.25 million events, only three fields matter

7.577 GB

Decompressed NDJSON input

2.25 million

2,247,806 GitHub events

65,269

Opened pull requests

Count two things

Every event, grouped by type.
Only opened pull requests, grouped by the base repository’s language.

GH Archive: March 13, 2024, hours 12–19 UTC. The same eight-hour input used in the article.

The projection

/type
/payload/action
/payload/pull_request/base/repo/language

Input SHA-256 and complete counters are included in results/results.json.

05 Implementation

Change what crosses into Python

Before · lazy document views

fastsimdjson.map_many(
    count_lazy,
    data,
    format="lines",
    threads=24,
    lazy=True,
)

The callback reads event["type"], then traverses nested views for opened pull requests.

After · projected scalar tuple

fastsimdjson.map_many(
    count_fields,
    data,
    format="lines",
    threads=24,
    fields=[
        "/type",
        "/payload/action",
        "/payload/pull_request/base/repo/language",
    ],
)

The callback receives (type, action, language). fields= replaces lazy=True; they cannot be combined.

What stays the same: one Python callback per event, per-thread counters, final counter merging, and the unused map_many return-list overhead.

06 Protocol

One input. Ten configurations. Seventy checks.

Load

Download, decompress and join all eight files before timing.

Establish truth

Decode every event with the independent standard-library implementation.

Warm up

Exercise both APIs at every thread count and verify those counters.

Time and compare

Seven shuffled rounds. Check every result against the complete reference.

Inside the timer

Parsing, selected-value conversion, Python callbacks, counting, disposal of callback results, and merging counters.

Garbage collection enabled; explicit collection runs before each timer. One process, 1 / 4 / 8 / 16 / 24 worker threads.

Outside the timer

Network transfer, storage reads, gzip decompression, reference calculation, warmup, and correctness comparisons.

Report the median, retain every individual timing. GB/s uses decimal input bytes, not physical memory traffic.

07 Results

Field projection wins at every tested thread count

Lazy viewsField projectionMedian GB/s · higher is better

1threads

lazy

1.41

fields

1.67

4threads

lazy

4.38

fields

5.38

8threads

lazy

7.64

fields

8.94

16threads

lazy

11.99

fields

14.37

24threads

lazy

12.87

fields

16.07

Bar widths use a common zero-based 0–18 GB/s scale. Seven runs per configuration.

Full measured sweep

Threads	Lazy	Fields
1	1.41	1.67
4	4.38	5.38
8	7.64	8.94
16	11.99	14.37
24	12.87	16.07

Best tested setting

24 threads

16.07 GB/s median with projection. This is the best of the settings we tested, not a universal optimum.

08 Result at 24 threads

24.8% more throughput. 19.9% less time.

+24.8%

Throughput: 12.87 → 16.07 GB/s

−19.9%

Median elapsed time

117 ms

Saved per 7.577 GB pass

Same data, shorter processing time

Lazy views589 ms

Field projection472 ms

Zero-based relative bar lengths. Medians of seven measurements, 24 threads.

Do not confuse the percentages

Throughput is bytes divided by time. A 24.8% throughput increase corresponds to a 19.9% time reduction, not 24.8% less time.

This is an in-memory workload result, not an application-wide speedup or a disk-ingestion guarantee.

09 Variability

Show every sample, not just the best one

24 threads · individual elapsed times

Round	Lazy, ms	Fields, ms
1	590.1	490.7
2	583.3	523.8
3	588.6	473.5
4	539.9	438.5
5	614.7	446.7
6	646.7	471.6
7	501.8	452.8

Configuration order was shuffled within each round; columns are not simultaneous runs.

Projection’s best observed run

17.28 GB/s

Best is not the headline. The reported median is 16.07 GB/s.

Observed ranges at 24 threads

Lazy: 501.8–646.7 ms
Projection: 438.5–523.8 ms

Seven samples on one shared laptop. These ranges are observations, not confidence intervals or guarantees.

10 Correctness

Faster only counts if the answers still match

Full-dataset verification

70 / 70

Timed results matched every reference counter.

2,247,806 events
65,269 opened pull requests

Also matched the article’s totals and published counts for TypeScript, JavaScript, Python, Java, and unknown language.

Evidence is available

Raw timings, complete counters, input hash, versions and machine metadata are published with the benchmark.

Targeted regression cases

✓Unequal event frequencies

✓Opened versus closed pull requests

✓Null language and non-PR payloads

✓Unicode and escaped strings

✓Small and multi-chunk input

✓Repeated calls at five thread counts

Scope: this verifies our counting workload. It is not a full JSON conformance test or proof that both APIs have identical error semantics.

11 What went wrong

Files were isolated. CPU contention was not.

First session · excluded

A concurrent .NET build started during the benchmark. WSL became heavily CPU-contended, and timing variation increased substantially.

We left the other work untouched. Those performance numbers are not used in this presentation.

The first session’s raw samples were not retained. Its exclusion is documented in the public report.

Second session · published

Waited until three consecutive five-second CPU samples were below 15% busy, then repeated the entire benchmark.

The charts use only that second session’s 70 timings. All counts passed again.

The host remained shared. We did not establish controlled power, thermal or frequency conditions.

Other limits worth keeping beside the headline

One dataset; no orjson/msgspec comparison here; no end-to-end ingestion timing; no confidence interval; no claim that AVX2 on a laptop should match AVX-512 on a 64-core server.

12 Practical use

Use projection when most of the document is irrelevant

A good fit

Large NDJSON streams and only a few known fields.
Free-threaded Python, with compatible extensions.
Per-worker aggregation instead of shared hot counters.
A thread-count sweep on the actual target machine.

On this machine, 24 threads was fastest among the five settings tested.

Check before adopting

fields= returns None for missing paths by default; lazy indexing can raise.
Only request scalars if that is all you need; selected objects are materialized.
Disk or decompression may dominate your real pipeline.
On ordinary GIL-enabled Python, this library does not gain the same threaded parallelism.

The lesson: first reduce unnecessary Python object creation. Then measure parallelism, correctness and the full application separately.

13 Reproduce

The environment, code and evidence are public

git clone https://github.com/zuwasi/python-json-throughput-benchmark.git
cd python-json-throughput-benchmark
bash setup.sh
.venv/bin/python -m unittest -v
.venv/bin/python benchmark.py --rounds 7
.venv/bin/python build_presentation.py

Linux/WSL + uv, curl and gzip. About 1.2 GB of disk; allow at least 20 GiB of available RAM for the full run.

Peak process RSS in our final session: 15.27 GiB, including input loading and reference calculation.

Frozen evidence

results/results.json
70 timings, counters and metadata

results/benchmark.log
Final-session console output

Safe to repeat

New runs write results-local.json. Published measurements stay untouched. Python and packages remain inside the project directory.

For a smaller check: --hours 1 --rounds 1.

14 Next experiment

Native counting is an opportunity, not a result

Measured and complete

16.07 GB/s

Median with existing field projection.

Same input, same answers, 24.8% higher throughput than lazy views at 24 threads.

No custom parser or extension changes were needed.

Proposed, not implemented

Move extraction and per-worker counting into native code, returning only the final counters to Python.

This could remove millions of Python callbacks, temporary tuples and unused callback-result entries.

No speedup is claimed. It needs a prototype, equivalent validation/error behavior, and a fresh benchmark.

Our contribution

A reproducible local experiment, an independent correctness reference, transparent timing samples, and a measured recommendation for this workload on this hardware.

Daniel Liezrowice

Engineering Software Lab (ESL)

linkedin.com/in/liezrowice eswlab.com

Open code · Measured results · Clear limits

Reproduce it.
Inspect it. Discuss it.

7.577 GB, 2,247,806 events, identical counts.
16.07 GB/s median on this laptop.

Code, evidence and this presentation

github.com/zuwasi/python-json-throughput-benchmark

Inspiration: Daniel Lemire’s article Library: fastsimdjson / simdjson Data: GH Archive Design reference: ESL RTK benchmark

Prepared with Amp under Daniel’s direction.
Project code: MIT license. Results are workload-specific.

16 GB/s of JSON.On a laptop.

Measured here · 24 threads

Table 1 · Median throughput, GB/s

The inspiration · Daniel Lemire

The useful question

The upstream improvement

What this study does not claim

System inspected before testing

The first obstacle

Isolation has a boundary

Count two things

The projection

Load

Establish truth

Warm up

Time and compare

Inside the timer

Outside the timer

Full measured sweep

Best tested setting

Same data, shorter processing time

Do not confuse the percentages

24 threads · individual elapsed times

Projection’s best observed run

Observed ranges at 24 threads

Full-dataset verification

Evidence is available

Targeted regression cases

First session · excluded

Second session · published

Other limits worth keeping beside the headline

A good fit

Check before adopting

Frozen evidence

Safe to repeat

Measured and complete

Proposed, not implemented

Our contribution

Daniel Liezrowice

Reproduce it.Inspect it. Discuss it.

Code, evidence and this presentation

16 GB/s of JSON.
On a laptop.

Reproduce it.
Inspect it. Discuss it.