jwk updates #13253

zjma · 2024-05-11T02:14:28Z

Pull some shared JWK logic out into a crate and make it a dependency of pepper service and jwk-consensus.
Start fetching apple jwks in pepper service.

trunk-io · 2024-05-11T02:14:31Z

⏱️ 56m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-targeted-unit-tests	29m	🟩
rust-move-tests	13m	🟩
rust-lints	6m	🟩
run-tests-main-branch	4m	🟩
general-lints	2m	🟩
check-dynamic-deps	1m	🟩
semgrep/ci	27s	🟩
file_change_determinator	12s	🟩
file_change_determinator	10s	🟩
permission-check	5s	🟩
permission-check	2s	🟩
permission-check	2s	🟩
permission-check	2s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-targeted-unit-tests	29m	21m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

danielxiangzl · 2024-05-14T22:05:41Z

keyless/pepper/service/src/main.rs

-    jwk::start_jwk_refresh_loop(
-        "https://id.twitch.tv/oauth2",
-        "https://id.twitch.tv/oauth2/keys",
+        "https://appleid.apple.com",


So these are not on-chain?

crates/aptos-jwk-consensus/src/jwk_manager/mod.rs

alinush · 2024-05-22T20:44:41Z

crates/aptos-jwk-consensus/src/jwk_manager/mod.rs

-                    Duration::from_secs(10),
-                    local_observation_tx.clone(),
-                )
+            .flat_map(|provider| {


Why the flat_map? You are mapping provider to one value, not to an iterator over values no?

looks like i meant filter_map 🤦

alinush · 2024-05-22T20:47:14Z

crates/aptos-jwk-consensus/src/jwk_observer.rs

        let mut interval = tokio::time::interval(fetch_interval);
        interval.set_missed_tick_behavior(MissedTickBehavior::Delay);
        let mut close_rx = close_rx.into_stream();
+        let my_addr = if cfg!(feature = "smoke-test") {


What is this for? There are no comments on this function.

added comments

update Squashed commit of the following: commit b074196 Author: zhoujun.ma <zhoujun@aptoslabs.com> Date: Fri May 10 15:55:06 2024 -0700 debug commit f880993 Author: zhoujun.ma <zhoujun@aptoslabs.com> Date: Fri May 10 15:14:21 2024 -0700 debug lint update

github-actions · 2024-05-25T00:23:36Z

✅ Forge suite `compat` success on `3ffe0986b5fe4acb76544ae7ae85d73b91a6a411` ==> `2c7f78ee535822bb8e5982480918254c8b9d6246`

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246 (PR)
1. Check liveness of validators at old version: 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411
compatibility::simple-validator-upgrade::liveness-check : committed: 6789.159100231531 txn/s, latency: 4846.687951515151 ms, (p50: 4800 ms, p90: 8100 ms, p99: 8700 ms), latency samples: 247500
2. Upgrading first Validator to new version: 2c7f78ee535822bb8e5982480918254c8b9d6246
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 3132.1708503740633 txn/s, latency: 9940.93188059241 ms, (p50: 9700 ms, p90: 13800 ms, p99: 14300 ms), latency samples: 129640
3. Upgrading rest of first batch to new version: 2c7f78ee535822bb8e5982480918254c8b9d6246
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 3343.266600582348 txn/s, latency: 9302.100697370333 ms, (p50: 9400 ms, p90: 14100 ms, p99: 14400 ms), latency samples: 137660
4. upgrading second batch to new version: 2c7f78ee535822bb8e5982480918254c8b9d6246
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 4297.746338578865 txn/s, latency: 7449.21485743511 ms, (p50: 6600 ms, p90: 11700 ms, p99: 15200 ms), latency samples: 156420
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246 passed
Test Ok

github-actions · 2024-05-25T00:25:17Z

✅ Forge suite `realistic_env_max_load` success on `2c7f78ee535822bb8e5982480918254c8b9d6246`

two traffics test: inner traffic : committed: 7702.410182388868 txn/s, latency: 5091.7466755215255 ms, (p50: 4800 ms, p90: 6900 ms, p99: 10500 ms), latency samples: 3327740
two traffics test : committed: 99.98931619478398 txn/s, latency: 1938.959090909091 ms, (p50: 1900 ms, p90: 2200 ms, p99: 5700 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.232, avg: 0.209", "QsPosToProposal: max: 0.473, avg: 0.283", "ConsensusProposalToOrdered: max: 0.418, avg: 0.395", "ConsensusOrderedToCommit: max: 0.399, avg: 0.383", "ConsensusProposalToCommit: max: 0.787, avg: 0.778"]
Max round gap was 2 [limit 4] at version 318360. Max no progress secs was 4.740313 [limit 15] at version 1597226.
Test Ok

github-actions · 2024-05-25T00:27:20Z

✅ Forge suite `framework_upgrade` success on `3ffe0986b5fe4acb76544ae7ae85d73b91a6a411` ==> `2c7f78ee535822bb8e5982480918254c8b9d6246`

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246 (PR)
Upgrade the nodes to version: 2c7f78ee535822bb8e5982480918254c8b9d6246
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1172.3670111852432 txn/s, submitted: 1173.713527660571 txn/s, failed submission: 1.3465164753276146 txn/s, expired: 1.3465164753276146 txn/s, latency: 2605.791098774885 ms, (p50: 2100 ms, p90: 4500 ms, p99: 7800 ms), latency samples: 104480
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1201.7228287774806 txn/s, submitted: 1204.6973902348504 txn/s, failed submission: 2.974561457370001 txn/s, expired: 2.974561457370001 txn/s, latency: 2567.3517612338155 ms, (p50: 2100 ms, p90: 4500 ms, p99: 9000 ms), latency samples: 105040
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246 passed
Upgrade the remaining nodes to version: 2c7f78ee535822bb8e5982480918254c8b9d6246
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1153.2828117040337 txn/s, submitted: 1156.583563884068 txn/s, failed submission: 3.300752180034441 txn/s, expired: 3.300752180034441 txn/s, latency: 2686.839477199008 ms, (p50: 2100 ms, p90: 4500 ms, p99: 9900 ms), latency samples: 104820
Test Ok

zjma changed the title ~~jwks common lib~~ jwks update May 11, 2024

zjma changed the title ~~jwks update~~ jwk updates May 11, 2024

zjma marked this pull request as ready for review May 11, 2024 02:17

zjma requested review from heliuchuan, zekun000, danielxiangzl and alinush May 11, 2024 02:17

danielxiangzl reviewed May 14, 2024

View reviewed changes

danielxiangzl approved these changes May 15, 2024

View reviewed changes

alinush reviewed May 22, 2024

View reviewed changes

crates/aptos-jwk-consensus/src/jwk_manager/mod.rs Show resolved Hide resolved

alinush reviewed May 22, 2024

View reviewed changes

alinush approved these changes May 22, 2024

View reviewed changes

zjma enabled auto-merge (squash) May 24, 2024 17:53

This comment has been minimized.

Sign in to view

jwk utils

d1e7e49

update Squashed commit of the following: commit b074196 Author: zhoujun.ma <zhoujun@aptoslabs.com> Date: Fri May 10 15:55:06 2024 -0700 debug commit f880993 Author: zhoujun.ma <zhoujun@aptoslabs.com> Date: Fri May 10 15:14:21 2024 -0700 debug lint update

zjma force-pushed the zjma/apple-jwks-in-pepper-service branch from df5202c to d1e7e49 Compare May 24, 2024 22:55

This comment has been minimized.

Sign in to view

also fix flaky test

2c7f78e

This comment has been minimized.

Sign in to view

zjma merged commit f1c0744 into main May 25, 2024
50 of 51 checks passed

zjma deleted the zjma/apple-jwks-in-pepper-service branch May 25, 2024 00:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jwk updates #13253

jwk updates #13253

zjma commented May 11, 2024 •

edited

trunk-io bot commented May 11, 2024 •

edited

danielxiangzl May 14, 2024

zjma May 14, 2024

alinush May 22, 2024

zjma May 22, 2024

zjma May 24, 2024

alinush May 22, 2024

zjma May 24, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented May 25, 2024

github-actions bot commented May 25, 2024

github-actions bot commented May 25, 2024

jwk updates #13253

jwk updates #13253

Conversation

zjma commented May 11, 2024 • edited

trunk-io bot commented May 11, 2024 • edited

danielxiangzl May 14, 2024

Choose a reason for hiding this comment

zjma May 14, 2024

Choose a reason for hiding this comment

alinush May 22, 2024

Choose a reason for hiding this comment

zjma May 22, 2024

Choose a reason for hiding this comment

zjma May 24, 2024

Choose a reason for hiding this comment

alinush May 22, 2024

Choose a reason for hiding this comment

zjma May 24, 2024

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented May 25, 2024

✅ Forge suite compat success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246

github-actions bot commented May 25, 2024

✅ Forge suite realistic_env_max_load success on 2c7f78ee535822bb8e5982480918254c8b9d6246

github-actions bot commented May 25, 2024

✅ Forge suite framework_upgrade success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 2c7f78ee535822bb8e5982480918254c8b9d6246

zjma commented May 11, 2024 •

edited

trunk-io bot commented May 11, 2024 •

edited

✅ Forge suite `compat` success on `3ffe0986b5fe4acb76544ae7ae85d73b91a6a411` ==> `2c7f78ee535822bb8e5982480918254c8b9d6246`

✅ Forge suite `realistic_env_max_load` success on `2c7f78ee535822bb8e5982480918254c8b9d6246`

✅ Forge suite `framework_upgrade` success on `3ffe0986b5fe4acb76544ae7ae85d73b91a6a411` ==> `2c7f78ee535822bb8e5982480918254c8b9d6246`