eth_pubsub support #103

tgmichel · 2020-08-20T13:32:12Z

This PR add support for subscription RPC calls - eth_subscribe and eth_unsubscribe.

Included

EthPubSubApi implementation.
Service integration.
Support for Kind::NewHeads, Kind::Logs, Kind::NewPendingTransactions, Kind::Syncing subscriptions.

Example

While running the template locally, in PolkadotJs the subscription to Kind::NewHeads can be tested using the Javascript console provided in the UI with:

let ws = new WebSocket('ws://localhost:9944');

ws.onopen = function(){
  ws.send(JSON.stringify({"jsonrpc":"2.0","id":1,"method":"eth_subscribe","params":["newHeads"]}))
}    

ws.onmessage = function(a) {
    console.log(JSON.stringify(JSON.parse(a.data).params.result,null,2));
}

crystalin

Looks good so far

.gitignore

rpc/src/eth_pubsub.rs

tgmichel · 2020-08-26T14:29:05Z

Implemented Kind::Logs subscription type.

Each subscription message expects a single Log, while the storage update for receipt logs occurs once per block, so the storage notification is triggered with a single Vec<Log> Stream. This would not send a message to the subscriber per Log, but rather a single message with a Vec<Log>. We don't want that.

The proposed solution is flat_map the notification Stream creating a second Stream that will resolve a future per Log contained in the original Vec. That will emit i.e. 5 single Log messages to the subscriber if the latest block contained 5 Logs in total.

tgmichel · 2020-08-26T16:23:53Z

Same strategy for Kind::NewPendingTransactions.

tgmichel · 2020-08-28T10:02:52Z

#107 will include several changes in the project, makes sense to wait until that is merged and then adapt this PR - should be reasonably easy.

As a side note, both eth and eth_pubsub modules may end up sharing some stuff like native_block_number method and the filtering logic for Logs. We will need to decouple that.

# Conflicts: # frame/ethereum/src/lib.rs # rpc/src/lib.rs # template/node/src/rpc.rs

tgmichel · 2020-08-31T17:30:56Z

@crystalin @sorpaas I tried the NewHeads today and it works. Tomorrow I will try the logs and pending subscriptions, if that works I will send the PR.

Please do a quick review over this and let me know.

# Conflicts: # rpc/src/lib.rs # template/node/src/rpc.rs

tgmichel · 2020-09-08T08:15:14Z

Added Kind::Syncing implementation. Still not sure if this is the right way to do it, didn't find other way.

We want to keep track SyncState through the NetworkService::is_major_syncing method. Afaik there is no way of subscribe to a notification that streams whenever the NetworkWorker changes the syncing state.

What I did is just subscribe to a storage notification that streams per block, and then send to the subscriber only if the syncing state changed from the previous check.

crystalin · 2020-09-08T11:26:41Z

@tgmichel , did you look at adding the event into substrate directly ? Is that a long work

tgmichel · 2020-09-08T12:45:44Z

@tgmichel , did you look at adding the event into substrate directly ? Is that a long work

I don't want to mess with the NetworkWorker poll if possible, but we may end up going back to do more research on it if we find this approach does not work.

sorpaas · 2020-09-11T10:00:26Z

Please pull master.

# Conflicts: # frame/ethereum/src/lib.rs # rpc/Cargo.toml # rpc/src/lib.rs # template/runtime/src/lib.rs

tgmichel · 2020-09-21T08:45:08Z

@sorpaas Updated. Pushed some minor runtime changes for pallet-balances, required once paritytech/substrate#7048 goes through and we merge master to https://github.com/paritytech/substrate/tree/frontier.

tgmichel · 2020-09-21T16:45:25Z

Bumped jsonrpc-core and jsonrpc-pubsub to 15.0.0 to not overlap with https://github.com/paritytech/substrate/tree/frontier.

Added a minor runtime fix to add WeightInfo to pallet-grandpa.

# Conflicts: # frame/ethereum/src/lib.rs

sorpaas · 2020-09-28T08:46:26Z

Really sorry about this, but can you pull master again?

# Conflicts: # rpc/src/lib.rs

tgmichel · 2020-09-28T10:24:58Z

@ermalkaleci @crystalin @sorpaas

This test https://github.com/paritytech/frontier/blob/master/ts-tests/tests/test-block.ts#L120 was previously skipped and recently was changed to be included in this PR:

https://github.com/paritytech/frontier/pull/135/files

I'm getting random behaviour from it.

You can reproduce it locally by running the tests. Sometimes passes and sometimes fails. The error is this - returned by createAndFinalizeBlock:

Error: Unexpected result: {"jsonrpc":"2.0","error":{"code":20000,"message":"Execution: Runtime panicked: Timestamp must be updated once in the block"},"id":1}

I will revert it back to skip so I can move forward. Please let me know.

sorpaas · 2020-09-28T10:36:26Z

@tgmichel I think that's alright. The error was not related to this PR -- we probably just missed a timestamp inherent or something like that in the manual seal. Still good to have it fixed in a separate PR though.

* Pubsub service setup * Add LatestHeader StorageValue, change Module name * Implement Kind::NewHeads for eth_subscribe * Styling fix * Add .cargo-remote.toml to .gitignore * Update Cargo.lock * pallet_ethereum LatestHeader to LatestBlock * Refactor to accommodate new Kinds * Implement Kind::Logs * Move Vec<Log> build to its own function * Added missing fields from Log response * Implement Kind::NewPendingTransactions * Implement eth_unsubscribe * Implement filtering for Kind::Logs * Small fixes * Add stream_build macro * Adapt to frontier consensus * Indentation cleanup * Formatting and cleanup * Remove unused imports * Remove Pending.take and add on_initialize * Fix Kind:NewPendingTransactions * Add NetworkService to pubsub rpc handler * Implement Kind::Syncing * Fix checker * pallet_ethereum handle Log response from evm execution * Runtime update substrate#12d0b7e * Return Vec<Log> in handle_exec * Fix checker * Runtime update substrate#93b2c36 * Bump jsonrpc-ore and jsonrpc-pubsub version * Update Cargo.lock * Update pallet-ethereum mock * Add missing const * Move internal error handler to crate root * Update pubsub to one-to-many * Skip previous block hash as parent test

tgmichel added 6 commits August 17, 2020 14:29

Pubsub service setup

a28b052

Add LatestHeader StorageValue, change Module name

7ab9409

Implement Kind::NewHeads for eth_subscribe

daa9156

Styling fix

577e8ba

Add .cargo-remote.toml to .gitignore

e2311ed

Update Cargo.lock

9a97535

crystalin reviewed Aug 20, 2020

View reviewed changes

.gitignore Show resolved Hide resolved

rpc/src/eth_pubsub.rs Show resolved Hide resolved

tgmichel added 2 commits August 21, 2020 11:28

pallet_ethereum LatestHeader to LatestBlock

bd8a91e

Refactor to accommodate new Kinds

ee1031b

crystalin mentioned this pull request Aug 24, 2020

Is event support planned? #105

Closed

Implement Kind::Logs

2a977b2

tgmichel added 3 commits August 26, 2020 16:33

Move Vec<Log> build to its own function

be8b2c1

Added missing fields from Log response

6379280

Implement Kind::NewPendingTransactions

634ba64

tgmichel added 2 commits August 27, 2020 09:22

Implement eth_unsubscribe

749bf17

Implement filtering for Kind::Logs

9682a60

tgmichel added 5 commits August 28, 2020 12:36

Small fixes

18f1fd5

Add stream_build macro

da1794c

Merge branch 'master' into tgmichel-pubsub2

847b421

# Conflicts: # frame/ethereum/src/lib.rs # rpc/src/lib.rs # template/node/src/rpc.rs

Adapt to frontier consensus

f8a95d8

Indentation cleanup

78abbfe

tgmichel force-pushed the tgmichel-pubsub2 branch from 81e43d4 to 78abbfe Compare August 31, 2020 15:47

tgmichel added 4 commits September 1, 2020 10:08

Formatting and cleanup

6255336

Merge branch 'master' into tgmichel-pubsub2

a8260ff

# Conflicts: # rpc/src/lib.rs # template/node/src/rpc.rs

Remove unused imports

58288f4

Remove Pending.take and add on_initialize

b130b98

pallet_ethereum handle Log response from evm execution

7f5db95

tgmichel mentioned this pull request Sep 8, 2020

evm: Add executor logs to execute_evm response paritytech/substrate#7048

Merged

tgmichel added 3 commits September 21, 2020 10:04

Merge remote-tracking branch 'upstream/master' into tgmichel-pubsub2

bb5b243

# Conflicts: # frame/ethereum/src/lib.rs # rpc/Cargo.toml # rpc/src/lib.rs # template/runtime/src/lib.rs

Runtime update substrate#12d0b7e

dd74fda

Return Vec<Log> in handle_exec

0e63b14

tgmichel requested a review from sorpaas as a code owner September 21, 2020 08:41

Fix checker

cb36fc2

tgmichel added 3 commits September 21, 2020 18:13

Runtime update substrate#93b2c36

001b50a

Bump jsonrpc-ore and jsonrpc-pubsub version

ac80504

Update Cargo.lock

fb5605c

tgmichel added 3 commits September 21, 2020 19:12

Update pallet-ethereum mock

b1cb9a8

Add missing const

843f80c

Merge branch 'master' into tgmichel-pubsub2

e31bd64

# Conflicts: # frame/ethereum/src/lib.rs

tgmichel added 3 commits September 28, 2020 10:59

Merge remote-tracking branch 'upstream/master' into tgmichel-pubsub2

ee574aa

# Conflicts: # rpc/src/lib.rs

Move internal error handler to crate root

9cb94f6

Update pubsub to one-to-many

a5b3f72

Skip previous block hash as parent test

aa7f78f

sorpaas approved these changes Sep 28, 2020

View reviewed changes

sorpaas merged commit c288899 into polkadot-evm:master Sep 28, 2020

tgmichel deleted the tgmichel-pubsub2 branch April 1, 2022 09:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

eth_pubsub support #103

eth_pubsub support #103

tgmichel commented Aug 20, 2020 •

edited

Loading

Uh oh!

crystalin left a comment

Uh oh!

Uh oh!

Uh oh!

tgmichel commented Aug 26, 2020 •

edited

Loading

Uh oh!

tgmichel commented Aug 26, 2020

Uh oh!

tgmichel commented Aug 28, 2020

Uh oh!

tgmichel commented Aug 31, 2020

Uh oh!

tgmichel commented Sep 8, 2020

Uh oh!

crystalin commented Sep 8, 2020

Uh oh!

tgmichel commented Sep 8, 2020

Uh oh!

sorpaas commented Sep 11, 2020

Uh oh!

tgmichel commented Sep 21, 2020

Uh oh!

tgmichel commented Sep 21, 2020

Uh oh!

sorpaas commented Sep 28, 2020

Uh oh!

tgmichel commented Sep 28, 2020 •

edited

Loading

Uh oh!

sorpaas commented Sep 28, 2020

Uh oh!

eth_pubsub support #103

eth_pubsub support #103

Conversation

tgmichel commented Aug 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

crystalin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tgmichel commented Aug 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgmichel commented Aug 26, 2020

Uh oh!

tgmichel commented Aug 28, 2020

Uh oh!

tgmichel commented Aug 31, 2020

Uh oh!

tgmichel commented Sep 8, 2020

Uh oh!

crystalin commented Sep 8, 2020

Uh oh!

tgmichel commented Sep 8, 2020

Uh oh!

sorpaas commented Sep 11, 2020

Uh oh!

tgmichel commented Sep 21, 2020

Uh oh!

tgmichel commented Sep 21, 2020

Uh oh!

sorpaas commented Sep 28, 2020

Uh oh!

tgmichel commented Sep 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sorpaas commented Sep 28, 2020

Uh oh!

tgmichel commented Aug 20, 2020 •

edited

Loading

tgmichel commented Aug 26, 2020 •

edited

Loading

tgmichel commented Sep 28, 2020 •

edited

Loading