multi: Add cache layer #660
Conversation
|
Is this change going to be backward compatible for non-politeiawww clients of politeiad? |
|
@sndurkin the politeiad The way its written now, a cache must be used with politeiad. It could be updated to have some type of Edit: the cache now defaults to being disabled on politeiad. The |
lukebp
force-pushed
the
cache
branch
3 times, most recently
from
e4b4cb0
to
0006904
Compare
This commit adds a cache layer to politeiad and politeiawww. The politeiad git repositories serve as the source of truth for proposal data and a CockroachDB database acts as a cache for proposal data. politeiad has read and write access to the cache. politeiawww has only read access to the cache. The flow of data is as follows: 1. politeiawww receives a command from a user 2. politeiawww creates a politeiad request for the command and sends it 3. politeiad writes the command data to the git repository and updates the cache 4. politeiad returns a response to politeiawww 5. politeiawww reads the cache if needed 6. politeiawww returns a response to the user The cache is automatically rebuilt on cache version change. You can also force the cache to rebuilt by using the flag --buildcache on politeiad startup. The politeiad record and all record metadata has been removed from the politeiawww inventory. The only thing left in the inventory is decred plugin data. The decred plugin data will be added to the cache and removed from the inventory in a future PR. Since this commit touched so much of the code for the `/proposals` endpoints, I began cleaning up `backend.go` and moving the updated `/proposals` enpoint code to `proposals.go`. I had to delete all of the tests that relied on the proposal record in the inventory. I re-wrote most of these tests in `proposals_test.go`.
decredplugin/decredplugin.go
Outdated
| // InventoryReply returns the decred plugin inventory. | ||
| type InventoryReply struct { | ||
| Comments []Comment `json:"comments"` // Comments | ||
| CommentLikes []LikeComment `json:"commentlikes"` // Comment likes |
There was a problem hiding this comment.
The asymmetry is bothering my OCD!
politeiad/backend/gitbe/decred.go
Outdated
| decredPluginJournals = "journals" | ||
| decredPluginIdentity = "fullidentity" | ||
| decredPluginJournals = "journals" | ||
| decredPluginInventoryCmd = "inventorycmd" |
There was a problem hiding this comment.
Why is this postfixed cmd?
There was a problem hiding this comment.
It doesn't need to be. It has been shortened to "inventory".
politeiad/cache/cache.go
Outdated
| Settings []PluginSetting // Settings | ||
| } | ||
|
|
||
| // PluginDriver is the interface that all cache plugins must implement. |
There was a problem hiding this comment.
Explain what this does and why it exists.
| } | ||
| defer rows.Close() | ||
|
|
||
| records := make([]Record, 0, 1024) // PNOOMA |
There was a problem hiding this comment.
Glad you picked up on the PNOOMA development methodology!
* fix prop display * clear warning
Closes #546.
This PR adds a cache layer to politeiad and politeiawww. The politeiad git repositories serve as the source of truth for proposal data and a CockroachDB database acts as a cache for proposal data.
The cache is not required if you're running just politeiad. politeiad has the cache disable by default. If you're running the full politeia stack, politeiad and politeiawww, running the cache is required.
politeiad has read and write access to the cache. politeiawww has only read access to the cache. The flow of data is as follows:
The cache will build itself automatically on startup if it hasn't been built yet or it the version changes. You can also force a cache rebuild by using the
--buildcacheflag when starting politeiad.This PR does not attempt to optimize the cache performance at all. Bottlenecks will be identified and performance will be optimized in a future PR.
Other major changes this PR makes:
SetUnvettedStatus,SetVettedStatus, andUpdateRecord. These responses no longer return the full record. They only return the status of the update.votedetails,getcomment,commentlikes, andinventory.