Skip to content

apply_entry_to_state_machine before replicating to the majority of the cluster #132

@simon-fu

Description

@simon-fu

Hi, I have wrote testing code basing on mem-store example to verify async-raft.
step1: create 3 nodes, that is node0, node1, node2
step2: initialize node0, changing leader to node0
step3: node0 add_non_voter with node1 and node2
step4: node0 change_membership with all nodes
step5: node0 client_write a request
step6: node0 print error log "wrong append state ", it means apply_entry_to_state_machine before replicating to the majority of the cluster

Is it an issue of async-raft? or anything wrong in my code? here is the code:

https://github.com/simon-fu/rust-hub/blob/60c84658f0ce08ed22c85d7c0ded9a2e3d670e3f/other/src/bin/verify_async_raft.rs

Activity

drmingdrmer

drmingdrmer commented on Sep 9, 2021

@drmingdrmer
Contributor

There are several issues related to quorum check.

I fixed bugs I found here:
https://github.com/drmingdrmer/async-raft

  • fix: handle-vote should compare last_log_id in dictionary order, not in vector order
  • fix: race condition of concurrent snapshot-install and apply.
  • fix: a non-voter not in joint config should not block replication
  • fix: when append-entries, deleting entries after prev-log-id causes committed entry to be lost
  • fix: too many(50) inconsistent log should not live lock append-entries
  • fix: RaftCore.entries_cache is inconsistent with storage. removed it.
  • fix: snapshot replication does not need to send a last 0 size chunk
  • fix: install snapshot req with offset GE 0 should not start a new session.
  • fix: leader should not commit when there is no replication to voters.
  • fix: after 2 log compaction, membership should be able to be extract from prev compaction log
  • fix: when finalize_snapshot_installation, memstore should not load membership from its old log that are going to be overridden by snapshot.
  • fix: leader should re-create and send snapshot when threshold/2 < last_log_index - snapshot < threshold
  • fix: client_read has using wrong quorum=majority-1
  • fix: doc-include can only be used in nightly build
  • fix: when handle_update_match_index(), non-voter should also be considered, because when member change a non-voter is also count as a quorum member
  • fix: when calc quorum, the non-voter should be count
  • fix: a NonVoter should stay as NonVoter instead of Follower after restart
  • fix: discarded log in replication_buffer should be finally sent.
  • fix: a conflict is expected even when appending empty enties
  • fix: last_applied should be updated only when logs actually applied.

Most suspicious ones are:

Note that my fork changed some of the APIs, in case you'd like to see if my fork fixed your issues.

simon-fu

simon-fu commented on Sep 10, 2021

@simon-fu
Author

@drmingdrmer your fork works! could you merge your fork back to this repo?

drmingdrmer

drmingdrmer commented on Sep 10, 2021

@drmingdrmer
Contributor

@drmingdrmer your fork works! could you merge your fork back to this repo?

This is an incompatible fork. Merging it back may affect a lot current users.

This fork is used in our project and will be maintained in a long term by datafuse developers: https://github.com/datafuse-extras/async-raft

simon-fu

simon-fu commented on Sep 10, 2021

@simon-fu
Author

@drmingdrmer your fork works! could you merge your fork back to this repo?

This is an incompatible fork. Merging it back may affect a lot current users.

This fork is used in our project and will be maintained in a long term by datafuse developers: https://github.com/datafuse-extras/async-raft

Oh, sounds good.
Another question, what is the correct usage in Cargo.toml if my project denpends on datafuse fork?
Something like async-raft = "0.6.2-alpha.15" or async-raft = "path=/path/to/fork" ?

drmingdrmer

drmingdrmer commented on Sep 10, 2021

@drmingdrmer
Contributor

async-raft = { git = "https://github.com/datafuse-extras/async-raft", tag = "v0.6.2-alpha.14" }

Actually in our project all of the internal dependencies are tracked with git url. :DDD

simon-fu

simon-fu commented on Sep 11, 2021

@simon-fu
Author

async-raft = { git = "https://github.com/datafuse-extras/async-raft", tag = "v0.6.2-alpha.14" }

Actually in our project all of the internal dependencies are tracked with git url. :DDD

Execllent! That is what I need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @drmingdrmer@simon-fu

        Issue actions

          apply_entry_to_state_machine before replicating to the majority of the cluster · Issue #132 · async-raft/async-raft