cache: add cache package and AddressCache for dcrpg #1051

chappjc · 2019-02-27T16:23:59Z

Add a new cache package, starting with addresscache.go, which includes
AddressCache and AddressCacheItem.
Use AddressCache in ChainDB.
Remove DevFundBalance since it benefits from the general AddressCache.
Remove addressCounts since AddressCache includes balance.
To update an address' data, add RetrieveAllMainchainAddressTxns and
RetrieveAllMainchainAddressMergedTxns, which do not use offset or N.
Also cache transaction view type row counts via the new
(*ChainDB).CountTransactions function, which use cache or DB queries to
get the row counts for any of the supported txn view types.

Also:

Use vouts table primary key in SelectAddressUnspentWithTxn.
Avoid height query in AddressUTXO, using latest stored height instead.
fix: RetrieveAddressUTXOs forgot the block time
Rename AddrTxnType to AddrTxnViewType.
Remove unnecessary TimeDefs in scanAddressMergedRows and
scanAddressQueryRows.
Add Height() int64 to explorer/explorerDataSource interface to use the
no-query getter. This speeds up BlockHashPathOrIndexCtx middleware.

chappjc

Annotated for review.

chappjc · 2019-02-28T14:53:36Z

api/insight/apiroutes.go

-		return
-	}
-
+	height := c.BlockData.ChainDB.Height()


There is no reason for DB queries when the best block info is stored in ChainDB.

chappjc · 2019-02-28T14:54:24Z

db/cache/addresscache.go

+	}
+}
+
+// TryLock will attempt to obtain either an exclusive updating lock. Trylock


will revise this comment

chappjc · 2019-02-28T14:55:34Z

db/cache/addresscache.go

+func (d *AddressCacheItem) Balance() (*dbtypes.AddressBalance, *BlockID) {
+	d.RLock()
+	defer d.RUnlock()
+	if d.balance == nil {


Important to return a nil *BlockID in the event of a cache miss as checking this pointer is now the method for detecting a cache miss.

chappjc · 2019-02-28T14:56:26Z

db/cache/addresscache.go

+	DevAddress string
+}
+
+// NewAddressCache constructs a AddressCache.


need to doc the capacity arg

chappjc · 2019-02-28T14:57:40Z

db/cache/addresscache.go

+func (ac *AddressCache) Transactions(addr string, N, offset int64, txnType dbtypes.AddrTxnViewType) ([]*dbtypes.AddressRow, *BlockID, error) {
+	aci := ac.addressCacheItem(addr)
+	if aci == nil {
+		return nil, nil, nil /*fmt.Errorf("uninitialized address cache")*/


I can remove this comment. It's just a cache miss here, not an "uninitialized address cache".

chappjc · 2019-02-28T15:09:32Z

db/dcrpg/pgblockchain.go

@@ -2540,6 +2658,17 @@ func (pgb *ChainDB) StoreBlock(msgBlock *wire.MsgBlock, winningTickets []string,
 	numVouts = errStk.numVouts + errReg.numVouts
 	numAddresses = errStk.numAddresses + errReg.numAddresses

+	// Merge the affected addresses.


should explain this is for cache expiration

chappjc · 2019-02-28T15:09:50Z

db/dcrpg/pgblockchain.go

@@ -2578,9 +2707,9 @@ func (pgb *ChainDB) StoreBlock(msgBlock *wire.MsgBlock, winningTickets []string,
 		return
 	}

-	// If not in batch sync, lazy update the dev fund balance
+	// If not in batch sync, lazy update the dev fund balance.


should update this comment too.

chappjc · 2019-02-28T15:11:16Z

db/dcrpg/queries.go

@@ -1585,12 +1616,11 @@ func scanAddressQueryRows(rows *sql.Rows, queryType int) (ids []uint64, addressR
 	for rows.Next() {
 		var id uint64
 		var addr dbtypes.AddressRow
-		var txHash sql.NullString
-		var blockTime dbtypes.TimeDef
+		var matchingTxHash sql.NullString


txHash is never nul or empty. matchingTxHash is often empty, but I'm not even sure it is allowed to be null. This is just to be safe.

chappjc · 2019-02-28T15:12:07Z

db/dcrpg/queries.go

 			addr.AtomsCredit = addr.Value
 		case debitQuery:
+			// addr.IsFunding == false


Can remove these two commented lines. IsFunding is set directly by rows.Scan

chappjc · 2019-02-28T15:12:44Z

explorer/explorermiddleware.go

-						"", ExpStatusError)
-					return
-				}
+				maxHeight = exp.explorerSource.Height()


Another unnecessary DB query.

buck54321

I'm having trouble finding anything to complain about. Looks great.

db/dcrpg/pgblockchain.go

use vouts primary key in SelectAddressUnspentWithTxn avoid height query in AddressUTXO, using latest stored height instead fix: RetrieveAddressUTXOs forgot the block time rename AddrTxnType to AddrTxnViewType add addresscache.go with AddressCache and AddressCacheItem Use AddressCache in ChainDB. Remove DevFundBalance since it benefits from the general AddressCache. Remove addressCounts since AddressCache includes balance. Remove unnecessary TimeDefs in scanAddressMergedRows and scanAddressQueryRows. To update an address' data, add RetrieveAllMainchainAddressTxns and RetrieveAllMainchainAddressMergedTxns, which do not use offset or N. Add Height() int64 to explorer/explorerDataSource interface to use the no-query getter. This speeds up BlockHashPathOrIndexCtx middleware. make cache package in db/cache set capacity of cap, and respect it when adding items When enforcing capacity, do not purge the project fund cache item. remove redundant and conflicting functions

chappjc changed the title ~~cache: add cache package and AddressCache for dcrpg~~ [WIP] cache: add cache package and AddressCache for dcrpg Feb 27, 2019

chappjc force-pushed the insight-api-cache branch from 2a17334 to e9d77b0 Compare February 27, 2019 16:58

chappjc changed the title ~~[WIP] cache: add cache package and AddressCache for dcrpg~~ cache: add cache package and AddressCache for dcrpg Feb 27, 2019

chappjc added the needs testing label Feb 27, 2019

chappjc requested a review from buck54321 February 27, 2019 21:42

chappjc mentioned this pull request Feb 27, 2019

caching and query tuning for InsightAddressTransactions #1053

Merged

chappjc force-pushed the insight-api-cache branch 3 times, most recently from ed32871 to 1ca4477 Compare February 28, 2019 14:44

chappjc commented Feb 28, 2019

View reviewed changes

buck54321 approved these changes Feb 28, 2019

View reviewed changes

db/dcrpg/pgblockchain.go Show resolved Hide resolved

db/dcrpg/pgblockchain.go Outdated Show resolved Hide resolved

chappjc added 6 commits February 28, 2019 15:29

support the transaction view counts

f195c1d

only expire addresses touched by new block

a2bc324

docs, and rename AddressCache.DevAddress to ProjectAddress

340bca3

more docs

5f88619

tidy docs

ae0fa96

chappjc force-pushed the insight-api-cache branch from 9c5d147 to ae0fa96 Compare February 28, 2019 21:33

chappjc merged commit 4432f59 into decred:master Feb 28, 2019

chappjc deleted the insight-api-cache branch February 28, 2019 21:36

chappjc mentioned this pull request Mar 13, 2019

Caching, Sum address data #134

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

cache: add cache package and AddressCache for dcrpg #1051

cache: add cache package and AddressCache for dcrpg #1051

chappjc commented Feb 27, 2019 •

edited

Loading

Uh oh!

chappjc left a comment

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

chappjc Feb 28, 2019

Uh oh!

buck54321 left a comment

Uh oh!

Uh oh!

Uh oh!

cache: add cache package and AddressCache for dcrpg #1051

cache: add cache package and AddressCache for dcrpg #1051

Conversation

chappjc commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chappjc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

buck54321 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chappjc commented Feb 27, 2019 •

edited

Loading