H-2697: Add harpc net

[indietyp]

🌟 What is the purpose of this PR?

This implements the harpc service layer in Rust, which uses libp2p to communicate a stream of Request and Response streams. Because this is such a critical part, extensive care has been taken to test every facet.

This building block is needed to create an axum-like interface in the future. The code has been written with performance in mind and is quite distributed. Each transaction on a connection will spawn two tasks, which handle receiving and transmitting data, back to a connection, which has two tasks to handle these. This might seem like many spawned tasks, but Tokio tasks are very lightweight, and one shouldn't be afraid to spawn thousands or millions, according to the Tokio docs. This clear separation between the different layers also enables a greater separation of concerns and easier testing.

Tasks in Tokio are very lightweight. Under the hood, they require only a single allocation and 64 bytes of memory. Applications should feel free to spawn thousands, if not millions of tasks.¹

Most of the code is tests, to ensure that any condition (I could think of) is tested to ensure everything is rock solid. This includes 138 different tests over the whole codebase.

The current problem is yamux, which seems prone to deadlock.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

• are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• affected the execution graph, and the turbo.json's have been updated to reflect this

Footnotes

1. https://tokio.rs/tokio/tutorial/spawning ↩

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 21.04%. Comparing base (cfd3295) to head (49e3a87).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4459      +/-   ##
==========================================
- Coverage   21.11%   21.04%   -0.07%     
==========================================
  Files         459      460       +1     
  Lines       15242    15290      +48     
  Branches     2273     2283      +10     
==========================================
  Hits         3218     3218              
- Misses      11983    12031      +48     
  Partials       41       41              

Flag	Coverage Δ
apps.hash-ai-worker-ts	1.81% <ø> (-0.03%)	⬇️
apps.hash-api	0.00% <ø> (ø)
blockprotocol.type-system	47.36% <ø> (ø)
deer	26.66% <ø> (ø)
error-stack	76.61% <ø> (ø)
local.hash-backend-utils	4.26% <ø> (-0.04%)	⬇️
local.hash-isomorphic-utils	0.00% <ø> (ø)
local.hash-subgraph	18.11% <ø> (ø)
sarif	89.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

Benchmark results

@rust/graph-benches – Integrations

scaling_read_entity_complete_one_depth

Function	Value	Mean	Flame graphs
entity_by_id	50 entities	$$1.55 \mathrm{s} \pm 4.24 \mathrm{ms}\left({\color{gray}0.830 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	10 entities	$$31.9 \mathrm{ms} \pm 197 \mathrm{μs}\left({\color{lightgreen}-36.969 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$21.4 \mathrm{ms} \pm 101 \mathrm{μs}\left({\color{gray}0.171 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	25 entities	$$73.1 \mathrm{ms} \pm 387 \mathrm{μs}\left({\color{gray}-2.760 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	5 entities	$$25.4 \mathrm{ms} \pm 155 \mathrm{μs}\left({\color{gray}-0.296 \mathrm{\%}}\right) $$	Flame Graph

representative_read_entity

Function	Value	Mean	Flame graphs
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1	$$17.1 \mathrm{ms} \pm 232 \mathrm{μs}\left({\color{gray}-2.056 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1	$$17.2 \mathrm{ms} \pm 220 \mathrm{μs}\left({\color{gray}2.13 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1	$$17.6 \mathrm{ms} \pm 238 \mathrm{μs}\left({\color{gray}1.21 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1	$$16.8 \mathrm{ms} \pm 187 \mathrm{μs}\left({\color{gray}0.933 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1	$$17.0 \mathrm{ms} \pm 237 \mathrm{μs}\left({\color{gray}-1.556 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1	$$16.4 \mathrm{ms} \pm 177 \mathrm{μs}\left({\color{gray}-3.569 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1	$$15.6 \mathrm{ms} \pm 186 \mathrm{μs}\left({\color{lightgreen}-7.816 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2	$$16.6 \mathrm{ms} \pm 198 \mathrm{μs}\left({\color{gray}3.57 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1	$$17.1 \mathrm{ms} \pm 236 \mathrm{μs}\left({\color{gray}2.19 \mathrm{\%}}\right) $$	Flame Graph

representative_read_multiple_entities

Function	Value	Mean	Flame graphs
link_by_source_by_property	depths: DT=255, PT=255, ET=255, E=255	$$2.00 \mathrm{s} \pm 9.96 \mathrm{ms}\left({\color{gray}-0.323 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=2, PT=2, ET=2, E=2	$$1.06 \mathrm{s} \pm 3.85 \mathrm{ms}\left({\color{gray}-1.061 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=2, ET=2, E=2	$$1.03 \mathrm{s} \pm 5.34 \mathrm{ms}\left({\color{gray}-1.870 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=0, E=2	$$97.3 \mathrm{ms} \pm 617 \mathrm{μs}\left({\color{gray}-0.430 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=2, E=2	$$413 \mathrm{ms} \pm 1.98 \mathrm{ms}\left({\color{gray}-1.633 \mathrm{\%}}\right) $$	Flame Graph
link_by_source_by_property	depths: DT=0, PT=0, ET=0, E=0	$$61.1 \mathrm{ms} \pm 592 \mathrm{μs}\left({\color{gray}0.600 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=255, PT=255, ET=255, E=255	$$2.82 \mathrm{s} \pm 8.50 \mathrm{ms}\left({\color{gray}-1.637 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=2, PT=2, ET=2, E=2	$$969 \mathrm{ms} \pm 4.71 \mathrm{ms}\left({\color{gray}-1.262 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=2, ET=2, E=2	$$967 \mathrm{ms} \pm 4.62 \mathrm{ms}\left({\color{gray}-3.028 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=0, E=2	$$41.4 \mathrm{ms} \pm 365 \mathrm{μs}\left({\color{gray}0.373 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=2, E=2	$$354 \mathrm{ms} \pm 1.93 \mathrm{ms}\left({\color{gray}-3.133 \mathrm{\%}}\right) $$	Flame Graph
entity_by_property	depths: DT=0, PT=0, ET=0, E=0	$$37.0 \mathrm{ms} \pm 338 \mathrm{μs}\left({\color{gray}1.12 \mathrm{\%}}\right) $$	Flame Graph

representative_read_entity_type

Function	Value	Mean	Flame graphs
get_entity_type_by_id	Account ID: d4e16033-c281-4cde-aa35-9085bf2e7579	$$1.41 \mathrm{ms} \pm 8.28 \mathrm{μs}\left({\color{gray}0.353 \mathrm{\%}}\right) $$	Flame Graph

scaling_read_entity_linkless

Function	Value	Mean	Flame graphs
entity_by_id	1000 entities	$$4.18 \mathrm{ms} \pm 20.6 \mathrm{μs}\left({\color{red}25.9 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	100 entities	$$2.57 \mathrm{ms} \pm 10.2 \mathrm{μs}\left({\color{gray}-1.134 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	10 entities	$$2.45 \mathrm{ms} \pm 11.4 \mathrm{μs}\left({\color{gray}0.517 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	10000 entities	$$13.3 \mathrm{ms} \pm 165 \mathrm{μs}\left({\color{gray}-1.383 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$2.46 \mathrm{ms} \pm 13.2 \mathrm{μs}\left({\color{gray}-0.185 \mathrm{\%}}\right) $$	Flame Graph

scaling_read_entity_complete_zero_depth

Function	Value	Mean	Flame graphs
entity_by_id	50 entities	$$4.49 \mathrm{ms} \pm 25.5 \mathrm{μs}\left({\color{gray}-3.085 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	10 entities	$$2.72 \mathrm{ms} \pm 12.0 \mathrm{μs}\left({\color{gray}-0.030 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	1 entities	$$2.48 \mathrm{ms} \pm 8.79 \mathrm{μs}\left({\color{gray}-0.671 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	25 entities	$$3.25 \mathrm{ms} \pm 77.8 \mathrm{μs}\left({\color{lightgreen}-13.772 \mathrm{\%}}\right) $$	Flame Graph
entity_by_id	5 entities	$$2.53 \mathrm{ms} \pm 13.4 \mathrm{μs}\left({\color{gray}0.373 \mathrm{\%}}\right) $$	Flame Graph