Docs
Performance
JS/WASM Benchmarks

JS/WASM Benchmarks

The primary role of these benchmarks should be to serve as indicators of the absence of performance pitfalls rather than as measures of which project is superior. This is because different projects consistently make different trade-offs. It is inaccurate to claim that Project A is superior to Project B simply because A performs better in certain benchmarks, while Project B may excel in other areas by a significant margin.

The benchmark can be reproduced using the crdt-benchmarks (opens in a new tab) repo.

  • The memUsed result has been removed from the current benchmark because it is not reported correctly when using WASM.

On 2024-05-05, the benchmark test was performed on a 13-inch M1 MacBook Pro 2020. The benchmark uses the most recent versions of loro and yjs available on that date.

  • Note that parseTime is significantly higher with automerge and loro when the initial document is not empty (e.g. when syncing content from a remote server).
  • loro and automerge can store a complete DAG of editing history for each keystroke, but yjs requires additional storage for a Version Vector + Delete Set for each version saved, which incurs significant extra overhead beyond the document size reported.
  • yjs gc is disabled for fair comparison.
  • loro has undergone numerous changes since 2023-11-10 (opens in a new tab), particularly in terms of encoding schema (opens in a new tab), shifting from a performance-focused version to one that prioritizes compatibility, resulting in significant performance differences from the original version.
Benchmark setup

B1: No conflicts

Simulate two clients. One client modifies a text object and sends update messages to the other client. We measure the time to perform the task (time), the amount of data exchanged (avgUpdateSize), the size of the encoded document after the task is performed (docSize), the time to parse the encoded document (parseTime), and the memory used to hold the decoded document (memUsed).

B2: Two users producing conflicts

Simulate two clients. Both start with a synced text object containing 100 characters. Both clients modify the text object in a single transaction and then send their changes to the other client. We measure the time to sync concurrent changes into a single client (time), the size of the update messages (updateSize), the size of the encoded document after the task is performed (docSize), the time to parse the encoded document (parseTime), and the memory used to hold the decoded document (memUsed).

B3: Many conflicts

Simulate √N concurrent actions. We measure the time to perform the task and sync all clients (time), the size of the update messages (updateSize), the size of the encoded document after the task is performed (docSize), the time to parse the encoded document (parseTime), and the memory used to hold the decoded document (memUsed). The logarithm of N was chosen because √N concurrent actions may result in up to √N^2 - 1 conflicts (apply action 1: 0 conlict; apply action2: 1 conflict, apply action 2: 2 conflicts, ..).

B4: Real-world editing dataset

Replay a real-world editing dataset. This dataset contains the character-by-character editing trace of a large-ish text document, the LaTeX source of this paper: https://arxiv.org/abs/1608.03960 (opens in a new tab)

Source: https://github.com/automerge/automerge-perf/tree/master/edit-by-index (opens in a new tab)

  • 182,315 single-character insertion operations
  • 77,463 single-character deletion operations
  • 259,778 operations totally
  • 104,852 characters in the final document

We simulate one client replaying all changes and storing each update. We measure the time to replay the changes and the size of all update messages (updateSize), the size of the encoded document after the task is performed (docSize), the time to encode the document (encodeTime), the time to parse the encoded document (parseTime), and the memory used to hold the decoded document in memory (memUsed).

[B4 x 100] Real-world editing dataset 100 times

Replay the [B4] dataset one hundred times. The final document has a size of over 10 million characters. As comparison, the book "Game of Thrones: A Song of Ice and Fire" is only 1.6 million characters long (including whitespace).

  • 18,231,500 single-character insertion operations
  • 7,746,300 single-character deletion operations
  • 25,977,800 operations totally
  • 10,485,200 characters in the final document
N = 6000loroyjsywasmautomerge
Version0.15.213.6.150.17.42.1.10
Bundle size1,428,575 bytes84,017 bytes938,991 bytes1,696,176 bytes
Bundle size (gzipped)532,944 bytes25,105 bytes284,616 bytes591,049 bytes
[B1.1] Append N characters (time)147 ms153 ms171 ms279 ms
[B1.1] Append N characters (avgUpdateSize)109 bytes27 bytes27 bytes121 bytes
[B1.1] Append N characters (encodeTime)0 ms1 ms0 ms5 ms
[B1.1] Append N characters (docSize)6,156 bytes6,031 bytes6,031 bytes3,992 bytes
[B1.1] Append N characters (parseTime)27 ms27 ms65 ms59 ms
[B1.2] Insert string of length N (time)0 ms6 ms1 ms18 ms
[B1.2] Insert string of length N (avgUpdateSize)6,104 bytes6,031 bytes6,031 bytes6,201 bytes
[B1.2] Insert string of length N (encodeTime)0 ms0 ms0 ms2 ms
[B1.2] Insert string of length N (docSize)6,111 bytes6,031 bytes6,031 bytes3,974 bytes
[B1.2] Insert string of length N (parseTime)26 ms28 ms47 ms29 ms
[B1.3] Prepend N characters (time)89 ms134 ms30 ms272 ms
[B1.3] Prepend N characters (avgUpdateSize)108 bytes27 bytes26 bytes116 bytes
[B1.3] Prepend N characters (encodeTime)2 ms2 ms0 ms4 ms
[B1.3] Prepend N characters (docSize)12,119 bytes6,041 bytes6,040 bytes3,988 bytes
[B1.3] Prepend N characters (parseTime)33 ms46 ms38 ms73 ms
[B1.4] Insert N characters at random positions (time)94 ms140 ms101 ms268 ms
[B1.4] Insert N characters at random positions (avgUpdateSize)109 bytes29 bytes29 bytes121 bytes
[B1.4] Insert N characters at random positions (encodeTime)3 ms2 ms0 ms6 ms
[B1.4] Insert N characters at random positions (docSize)35,395 bytes29,554 bytes29,554 bytes24,743 bytes
[B1.4] Insert N characters at random positions (parseTime)29 ms48 ms30 ms73 ms
[B1.5] Insert N words at random positions (time)97 ms158 ms264 ms539 ms
[B1.5] Insert N words at random positions (avgUpdateSize)117 bytes36 bytes36 bytes131 bytes
[B1.5] Insert N words at random positions (encodeTime)6 ms4 ms1 ms14 ms
[B1.5] Insert N words at random positions (docSize)94,517 bytes87,924 bytes87,924 bytes96,203 bytes
[B1.5] Insert N words at random positions (parseTime)31 ms101 ms33 ms101 ms
[B1.6] Insert string, then delete it (time)0 ms3 ms0 ms39 ms
[B1.6] Insert string, then delete it (avgUpdateSize)6,218 bytes6,053 bytes6,052 bytes6,338 bytes
[B1.6] Insert string, then delete it (encodeTime)0 ms0 ms0 ms3 ms
[B1.6] Insert string, then delete it (docSize)6,121 bytes6,040 bytes6,039 bytes3,993 bytes
[B1.6] Insert string, then delete it (parseTime)29 ms32 ms29 ms52 ms
[B1.7] Insert/Delete strings at random positions (time)118 ms160 ms112 ms423 ms
[B1.7] Insert/Delete strings at random positions (avgUpdateSize)121 bytes31 bytes31 bytes135 bytes
[B1.7] Insert/Delete strings at random positions (encodeTime)5 ms3 ms1 ms13 ms
[B1.7] Insert/Delete strings at random positions (docSize)81,254 bytes41,592 bytes41,592 bytes59,281 bytes
[B1.7] Insert/Delete strings at random positions (parseTime)28 ms49 ms41 ms80 ms
[B1.8] Append N numbers (time)98 ms149 ms59 ms448 ms
[B1.8] Append N numbers (avgUpdateSize)114 bytes32 bytes32 bytes125 bytes
[B1.8] Append N numbers (encodeTime)2 ms3 ms0 ms6 ms
[B1.8] Append N numbers (docSize)41,725 bytes35,634 bytes35,634 bytes26,985 bytes
[B1.8] Append N numbers (parseTime)29 ms31 ms43 ms76 ms
[B1.9] Insert Array of N numbers (time)16 ms7 ms3 ms47 ms
[B1.9] Insert Array of N numbers (avgUpdateSize)35,732 bytes35,657 bytes35,657 bytes31,199 bytes
[B1.9] Insert Array of N numbers (encodeTime)2 ms1 ms0 ms4 ms
[B1.9] Insert Array of N numbers (docSize)41,748 bytes35,657 bytes35,657 bytes26,953 bytes
[B1.9] Insert Array of N numbers (parseTime)36 ms29 ms29 ms56 ms
[B1.10] Prepend N numbers (time)100 ms131 ms31 ms484 ms
[B1.10] Prepend N numbers (avgUpdateSize)113 bytes32 bytes32 bytes120 bytes
[B1.10] Prepend N numbers (encodeTime)2 ms2 ms1 ms6 ms
[B1.10] Prepend N numbers (docSize)41,745 bytes35,665 bytes35,665 bytes26,987 bytes
[B1.10] Prepend N numbers (parseTime)28 ms48 ms33 ms65 ms
[B1.11] Insert N numbers at random positions (time)99 ms141 ms121 ms517 ms
[B1.11] Insert N numbers at random positions (avgUpdateSize)114 bytes34 bytes34 bytes125 bytes
[B1.11] Insert N numbers at random positions (encodeTime)5 ms2 ms1 ms8 ms
[B1.11] Insert N numbers at random positions (docSize)65,016 bytes59,137 bytes59,137 bytes47,746 bytes
[B1.11] Insert N numbers at random positions (parseTime)44 ms48 ms50 ms428 ms
[B2.1] Concurrently insert string of length N at index 0 (time)1 ms1 ms0 ms75 ms
[B2.1] Concurrently insert string of length N at index 0 (updateSize)9,272 bytes6,094 bytes6,094 bytes9,499 bytes
[B2.1] Concurrently insert string of length N at index 0 (encodeTime)0 ms0 ms0 ms7 ms
[B2.1] Concurrently insert string of length N at index 0 (docSize)12,241 bytes12,152 bytes12,151 bytes8,011 bytes
[B2.1] Concurrently insert string of length N at index 0 (parseTime)27 ms27 ms34 ms60 ms
[B2.2] Concurrently insert N characters at random positions (time)49 ms49 ms166 ms270 ms
[B2.2] Concurrently insert N characters at random positions (updateSize)35,551 bytes33,444 bytes33,444 bytes27,476 bytes
[B2.2] Concurrently insert N characters at random positions (encodeTime)8 ms2 ms1 ms6 ms
[B2.2] Concurrently insert N characters at random positions (docSize)71,851 bytes66,852 bytes66,852 bytes50,686 bytes
[B2.2] Concurrently insert N characters at random positions (parseTime)34 ms65 ms69 ms51 ms
[B2.3] Concurrently insert N words at random positions (time)70 ms78 ms459 ms435 ms
[B2.3] Concurrently insert N words at random positions (updateSize)93,127 bytes88,874 bytes88,994 bytes122,485 bytes
[B2.3] Concurrently insert N words at random positions (encodeTime)14 ms8 ms2 ms28 ms
[B2.3] Concurrently insert N words at random positions (docSize)188,450 bytes177,889 bytes178,130 bytes185,019 bytes
[B2.3] Concurrently insert N words at random positions (parseTime)52 ms78 ms67 ms134 ms
[B2.4] Concurrently insert & delete (time)134 ms121 ms1,243 ms653 ms
[B2.4] Concurrently insert & delete (updateSize)185,994 bytes141,122 bytes141,122 bytes298,810 bytes
[B2.4] Concurrently insert & delete (encodeTime)25 ms9 ms3 ms43 ms
[B2.4] Concurrently insert & delete (docSize)319,143 bytes282,349 bytes282,358 bytes293,831 bytes
[B2.4] Concurrently insert & delete (parseTime)58 ms116 ms73 ms185 ms
[B3.1] 20√N clients concurrently set number in Map (time)21 ms56 ms60 ms1,058 ms
[B3.1] 20√N clients concurrently set number in Map (updateSize)158,556 bytes49,169 bytes49,162 bytes283,296 bytes
[B3.1] 20√N clients concurrently set number in Map (encodeTime)1 ms2 ms1 ms9 ms
[B3.1] 20√N clients concurrently set number in Map (docSize)17,006 bytes36,770 bytes36,751 bytes86,165 bytes
[B3.1] 20√N clients concurrently set number in Map (parseTime)43 ms70 ms67 ms59 ms
[B3.2] 20√N clients concurrently set Object in Map (time)48 ms53 ms62 ms1,126 ms
[B3.2] 20√N clients concurrently set Object in Map (updateSize)197,550 bytes85,079 bytes85,073 bytes398,090 bytes
[B3.2] 20√N clients concurrently set Object in Map (encodeTime)2 ms3 ms1 ms21 ms
[B3.2] 20√N clients concurrently set Object in Map (docSize)35,994 bytes72,682 bytes72,659 bytes112,588 bytes
[B3.2] 20√N clients concurrently set Object in Map (parseTime)50 ms81 ms68 ms68 ms
[B3.3] 20√N clients concurrently set String in Map (time)85 ms322 ms61 ms1,869 ms
[B3.3] 20√N clients concurrently set String in Map (updateSize)7,937,160 bytes7,826,246 bytes7,826,225 bytes8,063,440 bytes
[B3.3] 20√N clients concurrently set String in Map (encodeTime)42 ms268 ms5 ms59 ms
[B3.3] 20√N clients concurrently set String in Map (docSize)7,794,072 bytes7,813,860 bytes7,813,814 bytes97,997 bytes
[B3.3] 20√N clients concurrently set String in Map (parseTime)52 ms134 ms75 ms80 ms
[B3.4] 20√N clients concurrently insert text in Array (time)165 ms49 ms49 ms1,768 ms
[B3.4] 20√N clients concurrently insert text in Array (updateSize)163,670 bytes52,744 bytes52,735 bytes311,830 bytes
[B3.4] 20√N clients concurrently insert text in Array (encodeTime)1 ms2 ms0 ms13 ms
[B3.4] 20√N clients concurrently insert text in Array (docSize)26,610 bytes26,589 bytes26,580 bytes96,423 bytes
[B3.4] 20√N clients concurrently insert text in Array (parseTime)32 ms50 ms37 ms43 ms
[B4] Apply real-world editing dataset (time)1,754 ms2,812 ms17,556 ms7,109 ms
[B4] Apply real-world editing dataset (encodeTime)12 ms13 ms8 ms165 ms
[B4] Apply real-world editing dataset (docSize)273,517 bytes226,981 bytes226,981 bytes129,116 bytes
[B4] Apply real-world editing dataset (parseTime)9 ms35 ms22 ms1,185 ms
[B4x100] Apply real-world editing dataset 100 times (time)175,938 ms295,197 msskippedskipped
[B4x100] Apply real-world editing dataset 100 times (encodeTime)1,194 ms573 msskippedskipped
[B4x100] Apply real-world editing dataset 100 times (docSize)27,335,558 bytes22,694,543 bytesskippedskipped
[B4x100] Apply real-world editing dataset 100 times (parseTime)771 ms1,185 msskippedskipped
[B3.5] 20√N clients concurrently insert text (time)199 ms48 ms49 ms1,869 ms
[B3.5] 20√N clients concurrently insert text (updateSize)160,590 bytes48,115 bytes48,135 bytes298,020 bytes
[B3.5] 20√N clients concurrently insert text (encodeTime)2 ms2 ms0 ms12 ms
[B3.5] 20√N clients concurrently insert text (docSize)24,969 bytes24,310 bytes24,325 bytes90,768 bytes
[B3.5] 20√N clients concurrently insert text (parseTime)54 ms44 ms38 ms67 ms
Replayable Event GraphNative Benchmarks