JS/WASM Benchmarks
The primary role of these benchmarks should be to serve as indicators of the absence of performance pitfalls rather than as measures of which project is superior. This is because different projects consistently make different trade-offs. It is inaccurate to claim that Project A is superior to Project B simply because A performs better in certain benchmarks, while Project B may excel in other areas by a significant margin.
The benchmark can be reproduced using the crdt-benchmarks (opens in a new tab) repo.
- The
memUsed
result has been removed from the current benchmark because it is not reported correctly when using WASM.
On 2024-05-05, the benchmark test was performed on a 13-inch M1 MacBook Pro 2020.
The benchmark uses the most recent versions of loro
and yjs
available on that date.
- Note that
parseTime
is significantly higher withautomerge
andloro
when the initial document is not empty (e.g. when syncing content from a remote server). loro
andautomerge
can store a complete DAG of editing history for each keystroke, butyjs
requires additional storage for a Version Vector + Delete Set for each version saved, which incurs significant extra overhead beyond the document size reported.yjs
gc is disabled for fair comparison.loro
has undergone numerous changes since 2023-11-10 (opens in a new tab), particularly in terms of encoding schema (opens in a new tab), shifting from a performance-focused version to one that prioritizes compatibility, resulting in significant performance differences from the original version.
Benchmark setup
B1: No conflicts
Simulate two clients. One client modifies a text object and sends update
messages to the other client. We measure the time to perform the task (time
),
the amount of data exchanged (avgUpdateSize
), the size of the encoded document
after the task is performed (docSize
), the time to parse the encoded document
(parseTime
), and the memory used to hold the decoded document (memUsed
).
B2: Two users producing conflicts
Simulate two clients. Both start with a synced text object containing 100
characters. Both clients modify the text object in a single transaction and then
send their changes to the other client. We measure the time to sync concurrent
changes into a single client (time
), the size of the update messages
(updateSize
), the size of the encoded document after the task is performed
(docSize
), the time to parse the encoded document (parseTime
), and the
memory used to hold the decoded document (memUsed
).
B3: Many conflicts
Simulate √N
concurrent actions. We measure the time to perform the task and
sync all clients (time
), the size of the update messages (updateSize
), the
size of the encoded document after the task is performed (docSize
), the time
to parse the encoded document (parseTime
), and the memory used to hold the
decoded document (memUsed
). The logarithm of N
was chosen because √N
concurrent actions may result in up to √N^2 - 1
conflicts (apply action 1: 0
conflict; apply action2: 1 conflict, apply action 2: 2 conflicts, ..).
B4: Real-world editing dataset
Replay a real-world editing dataset. This dataset contains the character-by-character editing trace of a large-ish text document, the LaTeX source of this paper: https://arxiv.org/abs/1608.03960 (opens in a new tab)
Source: https://github.com/automerge/automerge-perf/tree/master/edit-by-index (opens in a new tab)
- 182,315 single-character insertion operations
- 77,463 single-character deletion operations
- 259,778 operations totally
- 104,852 characters in the final document
We simulate one client replaying all changes and storing each update. We measure
the time to replay the changes and the size of all update messages
(updateSize
), the size of the encoded document after the task is performed
(docSize
), the time to encode the document (encodeTime
), the time to parse
the encoded document (parseTime
), and the memory used to hold the decoded
document in memory (memUsed
).
[B4 x 100] Real-world editing dataset 100 times
Replay the [B4] dataset one hundred times. The final document has a size of over 10 million characters. As comparison, the book "Game of Thrones: A Song of Ice and Fire" is only 1.6 million characters long (including whitespace).
- 18,231,500 single-character insertion operations
- 7,746,300 single-character deletion operations
- 25,977,800 operations totally
- 10,485,200 characters in the final document
N = 6000 | loro | yjs | ywasm | automerge |
---|---|---|---|---|
Version | 0.15.2 | 13.6.15 | 0.17.4 | 2.1.10 |
Bundle size | 1,428,575 bytes | 84,017 bytes | 938,991 bytes | 1,696,176 bytes |
Bundle size (gzipped) | 532,944 bytes | 25,105 bytes | 284,616 bytes | 591,049 bytes |
[B1.1] Append N characters (time) | 147 ms | 153 ms | 171 ms | 279 ms |
[B1.1] Append N characters (avgUpdateSize) | 109 bytes | 27 bytes | 27 bytes | 121 bytes |
[B1.1] Append N characters (encodeTime) | 0 ms | 1 ms | 0 ms | 5 ms |
[B1.1] Append N characters (docSize) | 6,156 bytes | 6,031 bytes | 6,031 bytes | 3,992 bytes |
[B1.1] Append N characters (parseTime) | 27 ms | 27 ms | 65 ms | 59 ms |
[B1.2] Insert string of length N (time) | 0 ms | 6 ms | 1 ms | 18 ms |
[B1.2] Insert string of length N (avgUpdateSize) | 6,104 bytes | 6,031 bytes | 6,031 bytes | 6,201 bytes |
[B1.2] Insert string of length N (encodeTime) | 0 ms | 0 ms | 0 ms | 2 ms |
[B1.2] Insert string of length N (docSize) | 6,111 bytes | 6,031 bytes | 6,031 bytes | 3,974 bytes |
[B1.2] Insert string of length N (parseTime) | 26 ms | 28 ms | 47 ms | 29 ms |
[B1.3] Prepend N characters (time) | 89 ms | 134 ms | 30 ms | 272 ms |
[B1.3] Prepend N characters (avgUpdateSize) | 108 bytes | 27 bytes | 26 bytes | 116 bytes |
[B1.3] Prepend N characters (encodeTime) | 2 ms | 2 ms | 0 ms | 4 ms |
[B1.3] Prepend N characters (docSize) | 12,119 bytes | 6,041 bytes | 6,040 bytes | 3,988 bytes |
[B1.3] Prepend N characters (parseTime) | 33 ms | 46 ms | 38 ms | 73 ms |
[B1.4] Insert N characters at random positions (time) | 94 ms | 140 ms | 101 ms | 268 ms |
[B1.4] Insert N characters at random positions (avgUpdateSize) | 109 bytes | 29 bytes | 29 bytes | 121 bytes |
[B1.4] Insert N characters at random positions (encodeTime) | 3 ms | 2 ms | 0 ms | 6 ms |
[B1.4] Insert N characters at random positions (docSize) | 35,395 bytes | 29,554 bytes | 29,554 bytes | 24,743 bytes |
[B1.4] Insert N characters at random positions (parseTime) | 29 ms | 48 ms | 30 ms | 73 ms |
[B1.5] Insert N words at random positions (time) | 97 ms | 158 ms | 264 ms | 539 ms |
[B1.5] Insert N words at random positions (avgUpdateSize) | 117 bytes | 36 bytes | 36 bytes | 131 bytes |
[B1.5] Insert N words at random positions (encodeTime) | 6 ms | 4 ms | 1 ms | 14 ms |
[B1.5] Insert N words at random positions (docSize) | 94,517 bytes | 87,924 bytes | 87,924 bytes | 96,203 bytes |
[B1.5] Insert N words at random positions (parseTime) | 31 ms | 101 ms | 33 ms | 101 ms |
[B1.6] Insert string, then delete it (time) | 0 ms | 3 ms | 0 ms | 39 ms |
[B1.6] Insert string, then delete it (avgUpdateSize) | 6,218 bytes | 6,053 bytes | 6,052 bytes | 6,338 bytes |
[B1.6] Insert string, then delete it (encodeTime) | 0 ms | 0 ms | 0 ms | 3 ms |
[B1.6] Insert string, then delete it (docSize) | 6,121 bytes | 6,040 bytes | 6,039 bytes | 3,993 bytes |
[B1.6] Insert string, then delete it (parseTime) | 29 ms | 32 ms | 29 ms | 52 ms |
[B1.7] Insert/Delete strings at random positions (time) | 118 ms | 160 ms | 112 ms | 423 ms |
[B1.7] Insert/Delete strings at random positions (avgUpdateSize) | 121 bytes | 31 bytes | 31 bytes | 135 bytes |
[B1.7] Insert/Delete strings at random positions (encodeTime) | 5 ms | 3 ms | 1 ms | 13 ms |
[B1.7] Insert/Delete strings at random positions (docSize) | 81,254 bytes | 41,592 bytes | 41,592 bytes | 59,281 bytes |
[B1.7] Insert/Delete strings at random positions (parseTime) | 28 ms | 49 ms | 41 ms | 80 ms |
[B1.8] Append N numbers (time) | 98 ms | 149 ms | 59 ms | 448 ms |
[B1.8] Append N numbers (avgUpdateSize) | 114 bytes | 32 bytes | 32 bytes | 125 bytes |
[B1.8] Append N numbers (encodeTime) | 2 ms | 3 ms | 0 ms | 6 ms |
[B1.8] Append N numbers (docSize) | 41,725 bytes | 35,634 bytes | 35,634 bytes | 26,985 bytes |
[B1.8] Append N numbers (parseTime) | 29 ms | 31 ms | 43 ms | 76 ms |
[B1.9] Insert Array of N numbers (time) | 16 ms | 7 ms | 3 ms | 47 ms |
[B1.9] Insert Array of N numbers (avgUpdateSize) | 35,732 bytes | 35,657 bytes | 35,657 bytes | 31,199 bytes |
[B1.9] Insert Array of N numbers (encodeTime) | 2 ms | 1 ms | 0 ms | 4 ms |
[B1.9] Insert Array of N numbers (docSize) | 41,748 bytes | 35,657 bytes | 35,657 bytes | 26,953 bytes |
[B1.9] Insert Array of N numbers (parseTime) | 36 ms | 29 ms | 29 ms | 56 ms |
[B1.10] Prepend N numbers (time) | 100 ms | 131 ms | 31 ms | 484 ms |
[B1.10] Prepend N numbers (avgUpdateSize) | 113 bytes | 32 bytes | 32 bytes | 120 bytes |
[B1.10] Prepend N numbers (encodeTime) | 2 ms | 2 ms | 1 ms | 6 ms |
[B1.10] Prepend N numbers (docSize) | 41,745 bytes | 35,665 bytes | 35,665 bytes | 26,987 bytes |
[B1.10] Prepend N numbers (parseTime) | 28 ms | 48 ms | 33 ms | 65 ms |
[B1.11] Insert N numbers at random positions (time) | 99 ms | 141 ms | 121 ms | 517 ms |
[B1.11] Insert N numbers at random positions (avgUpdateSize) | 114 bytes | 34 bytes | 34 bytes | 125 bytes |
[B1.11] Insert N numbers at random positions (encodeTime) | 5 ms | 2 ms | 1 ms | 8 ms |
[B1.11] Insert N numbers at random positions (docSize) | 65,016 bytes | 59,137 bytes | 59,137 bytes | 47,746 bytes |
[B1.11] Insert N numbers at random positions (parseTime) | 44 ms | 48 ms | 50 ms | 428 ms |
[B2.1] Concurrently insert string of length N at index 0 (time) | 1 ms | 1 ms | 0 ms | 75 ms |
[B2.1] Concurrently insert string of length N at index 0 (updateSize) | 9,272 bytes | 6,094 bytes | 6,094 bytes | 9,499 bytes |
[B2.1] Concurrently insert string of length N at index 0 (encodeTime) | 0 ms | 0 ms | 0 ms | 7 ms |
[B2.1] Concurrently insert string of length N at index 0 (docSize) | 12,241 bytes | 12,152 bytes | 12,151 bytes | 8,011 bytes |
[B2.1] Concurrently insert string of length N at index 0 (parseTime) | 27 ms | 27 ms | 34 ms | 60 ms |
[B2.2] Concurrently insert N characters at random positions (time) | 49 ms | 49 ms | 166 ms | 270 ms |
[B2.2] Concurrently insert N characters at random positions (updateSize) | 35,551 bytes | 33,444 bytes | 33,444 bytes | 27,476 bytes |
[B2.2] Concurrently insert N characters at random positions (encodeTime) | 8 ms | 2 ms | 1 ms | 6 ms |
[B2.2] Concurrently insert N characters at random positions (docSize) | 71,851 bytes | 66,852 bytes | 66,852 bytes | 50,686 bytes |
[B2.2] Concurrently insert N characters at random positions (parseTime) | 34 ms | 65 ms | 69 ms | 51 ms |
[B2.3] Concurrently insert N words at random positions (time) | 70 ms | 78 ms | 459 ms | 435 ms |
[B2.3] Concurrently insert N words at random positions (updateSize) | 93,127 bytes | 88,874 bytes | 88,994 bytes | 122,485 bytes |
[B2.3] Concurrently insert N words at random positions (encodeTime) | 14 ms | 8 ms | 2 ms | 28 ms |
[B2.3] Concurrently insert N words at random positions (docSize) | 188,450 bytes | 177,889 bytes | 178,130 bytes | 185,019 bytes |
[B2.3] Concurrently insert N words at random positions (parseTime) | 52 ms | 78 ms | 67 ms | 134 ms |
[B2.4] Concurrently insert & delete (time) | 134 ms | 121 ms | 1,243 ms | 653 ms |
[B2.4] Concurrently insert & delete (updateSize) | 185,994 bytes | 141,122 bytes | 141,122 bytes | 298,810 bytes |
[B2.4] Concurrently insert & delete (encodeTime) | 25 ms | 9 ms | 3 ms | 43 ms |
[B2.4] Concurrently insert & delete (docSize) | 319,143 bytes | 282,349 bytes | 282,358 bytes | 293,831 bytes |
[B2.4] Concurrently insert & delete (parseTime) | 58 ms | 116 ms | 73 ms | 185 ms |
[B3.1] 20√N clients concurrently set number in Map (time) | 21 ms | 56 ms | 60 ms | 1,058 ms |
[B3.1] 20√N clients concurrently set number in Map (updateSize) | 158,556 bytes | 49,169 bytes | 49,162 bytes | 283,296 bytes |
[B3.1] 20√N clients concurrently set number in Map (encodeTime) | 1 ms | 2 ms | 1 ms | 9 ms |
[B3.1] 20√N clients concurrently set number in Map (docSize) | 17,006 bytes | 36,770 bytes | 36,751 bytes | 86,165 bytes |
[B3.1] 20√N clients concurrently set number in Map (parseTime) | 43 ms | 70 ms | 67 ms | 59 ms |
[B3.2] 20√N clients concurrently set Object in Map (time) | 48 ms | 53 ms | 62 ms | 1,126 ms |
[B3.2] 20√N clients concurrently set Object in Map (updateSize) | 197,550 bytes | 85,079 bytes | 85,073 bytes | 398,090 bytes |
[B3.2] 20√N clients concurrently set Object in Map (encodeTime) | 2 ms | 3 ms | 1 ms | 21 ms |
[B3.2] 20√N clients concurrently set Object in Map (docSize) | 35,994 bytes | 72,682 bytes | 72,659 bytes | 112,588 bytes |
[B3.2] 20√N clients concurrently set Object in Map (parseTime) | 50 ms | 81 ms | 68 ms | 68 ms |
[B3.3] 20√N clients concurrently set String in Map (time) | 85 ms | 322 ms | 61 ms | 1,869 ms |
[B3.3] 20√N clients concurrently set String in Map (updateSize) | 7,937,160 bytes | 7,826,246 bytes | 7,826,225 bytes | 8,063,440 bytes |
[B3.3] 20√N clients concurrently set String in Map (encodeTime) | 42 ms | 268 ms | 5 ms | 59 ms |
[B3.3] 20√N clients concurrently set String in Map (docSize) | 7,794,072 bytes | 7,813,860 bytes | 7,813,814 bytes | 97,997 bytes |
[B3.3] 20√N clients concurrently set String in Map (parseTime) | 52 ms | 134 ms | 75 ms | 80 ms |
[B3.4] 20√N clients concurrently insert text in Array (time) | 165 ms | 49 ms | 49 ms | 1,768 ms |
[B3.4] 20√N clients concurrently insert text in Array (updateSize) | 163,670 bytes | 52,744 bytes | 52,735 bytes | 311,830 bytes |
[B3.4] 20√N clients concurrently insert text in Array (encodeTime) | 1 ms | 2 ms | 0 ms | 13 ms |
[B3.4] 20√N clients concurrently insert text in Array (docSize) | 26,610 bytes | 26,589 bytes | 26,580 bytes | 96,423 bytes |
[B3.4] 20√N clients concurrently insert text in Array (parseTime) | 32 ms | 50 ms | 37 ms | 43 ms |
[B4] Apply real-world editing dataset (time) | 1,754 ms | 2,812 ms | 17,556 ms | 7,109 ms |
[B4] Apply real-world editing dataset (encodeTime) | 12 ms | 13 ms | 8 ms | 165 ms |
[B4] Apply real-world editing dataset (docSize) | 273,517 bytes | 226,981 bytes | 226,981 bytes | 129,116 bytes |
[B4] Apply real-world editing dataset (parseTime) | 9 ms | 35 ms | 22 ms | 1,185 ms |
[B4x100] Apply real-world editing dataset 100 times (time) | 175,938 ms | 295,197 ms | skipped | skipped |
[B4x100] Apply real-world editing dataset 100 times (encodeTime) | 1,194 ms | 573 ms | skipped | skipped |
[B4x100] Apply real-world editing dataset 100 times (docSize) | 27,335,558 bytes | 22,694,543 bytes | skipped | skipped |
[B4x100] Apply real-world editing dataset 100 times (parseTime) | 771 ms | 1,185 ms | skipped | skipped |
[B3.5] 20√N clients concurrently insert text (time) | 199 ms | 48 ms | 49 ms | 1,869 ms |
[B3.5] 20√N clients concurrently insert text (updateSize) | 160,590 bytes | 48,115 bytes | 48,135 bytes | 298,020 bytes |
[B3.5] 20√N clients concurrently insert text (encodeTime) | 2 ms | 2 ms | 0 ms | 12 ms |
[B3.5] 20√N clients concurrently insert text (docSize) | 24,969 bytes | 24,310 bytes | 24,325 bytes | 90,768 bytes |
[B3.5] 20√N clients concurrently insert text (parseTime) | 54 ms | 44 ms | 38 ms | 67 ms |