|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.apache.hadoop.hbase.io.hfile.LruBlockCache
public class LruBlockCache
A block cache implementation that is memory-aware using HeapSize,
memory-bound using an LRU eviction algorithm, and concurrent: backed by a
ConcurrentHashMap and with a non-blocking eviction thread giving
constant-time cacheBlock(java.lang.String, org.apache.hadoop.hbase.io.hfile.Cacheable, boolean) and getBlock(java.lang.String, boolean) operations.
Contains three levels of block priority to allow for scan-resistance and in-memory families. A block is added with an inMemory flag if necessary, otherwise a block becomes a single access priority. Once a blocked is accessed again, it changes to multiple access. This is used to prevent scans from thrashing the cache, adding a least-frequently-used element to the eviction algorithm.
Each priority is given its own chunk of the total cache to ensure fairness during eviction. Each priority will retain close to its maximum size, however, if any priority is not using its entire chunk the others are able to grow beyond their chunk size.
Instantiated at a minimum with the total size and average block size. All sizes are in bytes. The block size is not especially important as this cache is fully dynamic in its sizing of blocks. It is only used for pre-allocating data structures and in initial heap estimation of the map.
The detailed constructor defines the sizes for the three priorities (they should total to the maximum size defined). It also sets the levels that trigger and control the eviction thread.
The acceptable size is the cache size level which triggers the eviction process to start. It evicts enough blocks to get the size below the minimum size specified.
Eviction happens in a separate thread and involves a single full-scan of the map. It determines how many bytes must be freed to reach the minimum size, and then while scanning determines the fewest least-recently-used blocks necessary from each of the three priorities (would be 3 times bytes to free). It then uses the priority chunk sizes to evict fairly according to the relative sizes and usage.
| Field Summary | |
|---|---|
static long |
CACHE_FIXED_OVERHEAD
|
| Constructor Summary | |
|---|---|
LruBlockCache(long maxSize,
long blockSize)
Default constructor. |
|
LruBlockCache(long maxSize,
long blockSize,
boolean evictionThread)
Constructor used for testing. |
|
LruBlockCache(long maxSize,
long blockSize,
boolean evictionThread,
int mapInitialSize,
float mapLoadFactor,
int mapConcurrencyLevel,
float minFactor,
float acceptableFactor,
float singleFactor,
float multiFactor,
float memoryFactor)
Configurable constructor. |
|
| Method Summary | |
|---|---|
void |
cacheBlock(String blockName,
Cacheable buf)
Cache the block with the specified name and buffer. |
void |
cacheBlock(String blockName,
Cacheable buf,
boolean inMemory)
Cache the block with the specified name and buffer. |
static long |
calculateOverhead(long maxSize,
long blockSize,
int concurrency)
|
protected long |
evictBlock(CachedBlock block)
|
boolean |
evictBlock(String blockName)
Evict block from cache. |
int |
evictBlocksByPrefix(String prefix)
Evicts all blocks whose name starts with the given prefix. |
Cacheable |
getBlock(String blockName,
boolean caching)
Get the buffer of the block with the specified name. |
List<BlockCacheColumnFamilySummary> |
getBlockCacheColumnFamilySummaries(org.apache.hadoop.conf.Configuration conf)
Performs a BlockCache summary and returns a List of BlockCacheColumnFamilySummary objects. |
long |
getBlockCount()
Returns the number of blocks currently cached in the block cache. |
long |
getCurrentSize()
Get the current size of this cache. |
long |
getEvictedCount()
Get the number of blocks that have been evicted during the lifetime of this cache. |
long |
getEvictionCount()
Get the number of eviction runs that have occurred |
long |
getFreeSize()
Get the current size of this cache. |
long |
getMaxSize()
Get the maximum size of this cache. |
CacheStats |
getStats()
Get counter statistics for this cache. |
long |
heapSize()
|
void |
logStats()
|
void |
setMaxSize(long maxSize)
|
void |
shutdown()
Shutdown the cache. |
long |
size()
Get the size of this cache (number of cached blocks) |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
|---|
public static final long CACHE_FIXED_OVERHEAD
| Constructor Detail |
|---|
public LruBlockCache(long maxSize,
long blockSize)
All other factors will be calculated based on defaults specified in this class.
maxSize - maximum size of cache, in bytesblockSize - approximate size of each block, in bytes
public LruBlockCache(long maxSize,
long blockSize,
boolean evictionThread)
public LruBlockCache(long maxSize,
long blockSize,
boolean evictionThread,
int mapInitialSize,
float mapLoadFactor,
int mapConcurrencyLevel,
float minFactor,
float acceptableFactor,
float singleFactor,
float multiFactor,
float memoryFactor)
maxSize - maximum size of this cache, in bytesblockSize - expected average size of blocks, in bytesevictionThread - whether to run evictions in a bg thread or notmapInitialSize - initial size of backing ConcurrentHashMapmapLoadFactor - initial load factor of backing ConcurrentHashMapmapConcurrencyLevel - initial concurrency factor for backing CHMminFactor - percentage of total size that eviction will evict untilacceptableFactor - percentage of total size that triggers evictionsingleFactor - percentage of total size for single-access blocksmultiFactor - percentage of total size for multiple-access blocksmemoryFactor - percentage of total size for in-memory blocks| Method Detail |
|---|
public void setMaxSize(long maxSize)
public void cacheBlock(String blockName,
Cacheable buf,
boolean inMemory)
It is assumed this will NEVER be called on an already cached block. If that is done, it is assumed that you are reinserting the same exact block due to a race condition and will update the buffer but not modify the size of the cache.
cacheBlock in interface BlockCacheblockName - block namebuf - block bufferinMemory - if block is in-memory
public void cacheBlock(String blockName,
Cacheable buf)
It is assumed this will NEVER be called on an already cached block. If that is done, it is assumed that you are reinserting the same exact block due to a race condition and will update the buffer but not modify the size of the cache.
cacheBlock in interface BlockCacheblockName - block namebuf - block buffer
public Cacheable getBlock(String blockName,
boolean caching)
getBlock in interface BlockCacheblockName - block namecaching - Whether this request has caching enabled (used for stats)
public boolean evictBlock(String blockName)
BlockCache
evictBlock in interface BlockCacheblockName - Block name to evict
public int evictBlocksByPrefix(String prefix)
This is used for evict-on-close to remove all blocks of a specific HFile. The prefix would be the HFile/StoreFile name (a UUID) followed by an underscore, because HFile v2 block names in cache are of the form "<storeFileUUID>_<blockOffset>".
evictBlocksByPrefix in interface BlockCacheprotected long evictBlock(CachedBlock block)
public long getMaxSize()
public long getCurrentSize()
getCurrentSize in interface BlockCachepublic long getFreeSize()
getFreeSize in interface BlockCachepublic long size()
size in interface BlockCachepublic long getBlockCount()
BlockCache
getBlockCount in interface BlockCachepublic long getEvictionCount()
public long getEvictedCount()
getEvictedCount in interface BlockCachepublic void logStats()
public CacheStats getStats()
Includes: total accesses, hits, misses, evicted blocks, and runs of the eviction processes.
getStats in interface BlockCachepublic long heapSize()
heapSize in interface HeapSize
public static long calculateOverhead(long maxSize,
long blockSize,
int concurrency)
public List<BlockCacheColumnFamilySummary> getBlockCacheColumnFamilySummaries(org.apache.hadoop.conf.Configuration conf)
throws IOException
BlockCache
getBlockCacheColumnFamilySummaries in interface BlockCacheconf - HBaseConfiguration
IOException - exceptionpublic void shutdown()
BlockCache
shutdown in interface BlockCache
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||