Why do Bitcoin miners sometimes produce empty blocks despite there being unconfirmed transactions in the mempool?
In order to work on finding a valid block to become the new tip of the blockchain, miners need to set half a dozen fields for the header of the new block. The only two fields that are highly dependent on the data from the previous block are the previous block hash and the merkle root of all the transactions in the new block.
There is a period of time after which a miner has received the latest block's header from its network peers but has not yet checked all of the transactions in the block and updated its UTXO set. Because mining is essentially a global race to find the next block, every second of work counts - miners want to always be working on the next block. Thus during this period of time a miner typically avoids putting any transactions in the block (apart from the coinbase transaction), as they don't know which transactions were in the previous block and including any conflicting transactions could result in a double spend which would cause the network to reject the block and not pay the miner their reward. As such, during this period the merkle root of the next block template will be for a merkle tree that only contains a coinbase transaction.The Theory
If you read my previous research into block time variance then you may recall that we can calculate the expected amount of blocks generated X seconds after the previous block. On average it takes my node at statoshi.info 2 seconds to process a block, but sometimes it takes as long at 7 seconds.
Let's be conservative and assume that it can take up to 10 seconds for a mining pool to validate the latest block and update their mining template. How many blocks should we expect will be mined less than 10 seconds after the previous block?
1 - exp(-10/600) = 0.0165... about 1.7% of blocks, or about 2 blocks per day. So we should expect that in a situat...