Cache for buckets in a part of a file. More...

#include <BucketCache.h>

Public Member Functions
	BucketCache (BucketFile file, Int64 startOffset, uInt bucketSize, uInt nrOfBuckets, uInt cacheSize, void ownerObject, BucketCacheToLocal readCallBack, BucketCacheFromLocal writeCallBack, BucketCacheAddBuffer addCallBack, BucketCacheDeleteBuffer deleteCallBack)
	Create the cache for (a part of) a file. More...

	~BucketCache ()

Bool	flush (uInt fromSlot=0)
	Flush the cache from the given slot on. More...

void	clear (uInt fromSlot=0, Bool doFlush=True)
	Clear the cache from the given slot on. More...

void	resize (uInt cacheSize)
	Resize the cache. More...

void	resync (uInt nrBucket, uInt nrOfFreeBucket, Int firstFreeBucket)
	Resynchronize the object (after another process updated the file). More...

uInt	nBucket () const
	Get the current nr of buckets in the file. More...

uInt	cacheSize () const
	Get the current cache size (in buckets). More...

void	setDirty ()
	Set the dirty bit for the current bucket. More...

char *	getBucket (uInt bucketNr)
	Make another bucket current. More...

void	extend (uInt nrBucket)
	Extend the file with the given number of buckets. More...

uInt	addBucket (char *data)
	Add a bucket to the file and make it the current one. More...

void	removeBucket ()
	Remove the current bucket; i.e. More...

void	get (char *buf, uInt length, Int64 offset)
	Get a part from the file outside the cached area. More...

void	put (const char *buf, uInt length, Int64 offset)
	Put a part from the file outside the cached area. More...

Int	firstFreeBucket () const
	Get the bucket number of the first free bucket. More...

uInt	nFreeBucket () const
	Get the number of free buckets. More...

void	initStatistics ()
	(Re)initialize the cache statistics. More...

void	showStatistics (ostream &os) const
	Show the statistics. More...

Private Member Functions
	BucketCache (const BucketCache &)
	Copy constructor is not possible. More...

BucketCache &	operator= (const BucketCache &)
	Assignment is not possible. More...

void	setLRU ()
	Set the LRU information for the current slot. More...

void	getSlot (uInt bucketNr)
	Get a cache slot for the bucket. More...

void	writeBucket (uInt slotNr)
	Write a bucket. More...

void	readBucket (uInt slotNr)
	Read a bucket. More...

void	initializeBuckets (uInt bucketNr)
	Initialize the bucket buffer. More...

void	checkOffset (uInt length, Int64 offset) const
	Check if the offset of a non-cached part is correct. More...

Private Attributes
BucketFile *	its_file
	The file used. More...

void *	its_Owner
	The owner object. More...

BucketCacheToLocal	its_ReadCallBack
	The read callback function. More...

BucketCacheFromLocal	its_WriteCallBack
	The write callback function. More...

BucketCacheAddBuffer	its_InitCallBack
	The add bucket callback function. More...

BucketCacheDeleteBuffer	its_DeleteCallBack
	The delete callback function. More...

Int64	its_StartOffset
	The starting offsets of the buckets in the file. More...

uInt	its_BucketSize
	The bucket size. More...

uInt	its_CurNrOfBuckets
	The current nr of buckets in the file. More...

uInt	its_NewNrOfBuckets
	The new nr of buckets in the file (after extension). More...

uInt	its_CacheSize
	The size of the cache (i.e. More...

uInt	its_CacheSizeUsed
	The nr of slots used in the cache. More...

PtrBlock< char * >	its_Cache
	The cache itself. More...

uInt	its_ActualSlot
	The cache slot actually used. More...

Block< Int >	its_SlotNr
	The slot numbers of the buckets in the cache (-1 = not in cache). More...

Block< uInt >	its_BucketNr
	The buckets in the cache. More...

Block< uInt >	its_Dirty
	Determine if a block is dirty (i.e. More...

Block< uInt >	its_LRU
	Determine when a block is used for the last time. More...

uInt	its_LRUCounter
	The Least Recently Used counter. More...

char *	its_Buffer
	The internal buffer. More...

uInt	its_NrOfFree
	The number of free buckets. More...

Int	its_FirstFree
	The first free bucket (-1 = no free buckets). More...

uInt	naccess_p
	The statistics. More...

uInt	nread_p

uInt	ninit_p

uInt	nwrite_p

Detailed Description

Cache for buckets in a part of a file.

Intended use:

Public interface

Review Status

Reviewed By:: UNKNOWN
Date Reviewed:: before2004/08/25

Prerequisite

BucketFile

Etymology

BucketCache implements a cache for buckets in (a part of) a file.

Synopsis

A cache may allow more efficient quasi-random IO. It can, for instance, be used when a limited number of blocks in a file have to be accessed again and again.

The class BucketCache provides such a cache. It can be used on a consecutive part of a file as long as that part is not simultaneously accessed in another way (including another BucketCache object).

BucketCache stores the data as given. It uses (see (group=BucketCache_CallBack))callback functions to allocate/delete buffers and to convert the data to/from local format.

When a new bucket is needed and all slots in the cache are used, BucketCache will remove the least recently used bucket from the cache. When the dirty flag is set, it will first be written.

BucketCache maintains a list of free buckets. Initially this list is empty. When a bucket is removed, it is added to the free list. AddBucket will take buckets from the free list before extending the file.

Since it is possible to handle only a part of a file by a BucketCache object, it is also possible to have multiple BucketCache objects on the same file (as long as they access disjoint parts of the file). Each BucketCache object can have its own bucket size. This can, for example, be used to have tiled arrays with different tile shapes in the same file.

Statistics are kept to know how efficient the cache is working. It is possible to initialize and show the statistics.

Motivation

A cache may reduce IO traffix considerably. Furthermore it is more efficient to keep a cache in local format. In that way conversion to/from local only have to be done when data gets read/written. It also allows for precalculations.

Example

 // Define the callback function for reading a bucket.
 char* bToLocal (void*, const char* data)
 {
 char* ptr = new char[32768];
 memcpy (ptr, data, 32768);
 return ptr;
 }
 // Define the callback function for writing a bucket.
 void bFromLocal (void*, char* data, const char* local)
 {
 memcpy (data, local, 32768);
 }
 // Define the callback function for initializing a new bucket.
 char* bAddBuffer (void*)
 {
 char* ptr = new char[32768];
 for (uInt i=0; i++; i<32768) {
 ptr[i] = 0;
 }
 return ptr;
 }
 // Define the callback function for deleting a bucket.
 void bDeleteBuffer (void*, char* buffer)
 {
 delete [] buffer;
 }
 
 void someFunc()
 {
 // Open the filebuf.
 BucketFile file(...);
 file.open();
 uInt i;
 // Create a cache for the part of the file starting at offset 512
 // consisting of 1000 buckets. The cache consists of 10 buckets.
 // Each bucket is 32768 bytes.
 BucketCache cache (&file, 512, 32768, 1000, 10, 0,
 bToLocal, bFromLocal, bAddBuffer, bDeleteBuffer);
 // Write all buckets into the file.
 for (i=0; i<100; i++) {
 char* buf = new char[32768];
 cache.addBucket (buf);
 }
 Flush the cache to write all buckets in it.
 cache.flush();
 // Read all buckets from the file.
 for (i=0; i<1000; i++) {
 char* buf = cache.getBucket(i);
..\.
 }
 cout << cache.nBucket() << endl;
 }

To Do

When ready, use HashMap for the internal maps.

Definition at line 217 of file BucketCache.h.

Constructor & Destructor Documentation

casacore::BucketCache::BucketCache	(	BucketFile *	file,
		Int64	startOffset,
		uInt	bucketSize,
		uInt	nrOfBuckets,
		uInt	cacheSize,
		void *	ownerObject,
		BucketCacheToLocal	readCallBack,
		BucketCacheFromLocal	writeCallBack,
		BucketCacheAddBuffer	addCallBack,
		BucketCacheDeleteBuffer	deleteCallBack
	)

Create the cache for (a part of) a file.

The file part used starts at startOffset. Its length is bucketSize*nrOfBuckets bytes. When the file is smaller, the remainder is indicated as an extension similarly to the behaviour of function extend.

casacore::BucketCache::~BucketCache ( )

casacore::BucketCache::BucketCache ( const BucketCache & )

private

Copy constructor is not possible.

Member Function Documentation

uInt casacore::BucketCache::addBucket ( char * data )

Add a bucket to the file and make it the current one.

When no more cache slots are available, the one least recently used is flushed.
When no free buckets are available, the file will be extended with one bucket. It returns the new bucket number. The buffer must have been allocated on the heap. It will get part of the cache; its contents are not copied. Thus the buffer should hereafter NOT be used for other purposes. It will be deleted later via the DeleteBuffer callback function. The data is copied into the bucket. A pointer to the data in local format is returned.

uInt casacore::BucketCache::cacheSize ( ) const

inline

Get the current cache size (in buckets).

Definition at line 403 of file BucketCache.h.

References its_CacheSize.

void casacore::BucketCache::checkOffset	(	uInt	length,
		Int64	offset
	)		const

private

Check if the offset of a non-cached part is correct.

void casacore::BucketCache::clear	(	uInt	fromSlot = `0`,
		Bool	doFlush = `True`
	)

Clear the cache from the given slot on.

By default the entire cache is cleared. It will remove the buckets in the cleared part. If wanted and needed, the buckets are flushed to the file before removing them. It can be used to enforce rereading buckets from the file.

void casacore::BucketCache::extend ( uInt nrBucket )

Extend the file with the given number of buckets.

The buckets get initialized when they are acquired (using getBucket) for the first time.

Int casacore::BucketCache::firstFreeBucket ( ) const

inline

Get the bucket number of the first free bucket.

-1 = no free buckets.

Definition at line 406 of file BucketCache.h.

References its_FirstFree.

Bool casacore::BucketCache::flush ( uInt fromSlot = 0 )

Flush the cache from the given slot on.

By default the entire cache is flushed. When the entire cache is flushed, possible remaining uninitialized buckets will be initialized first. A True status is returned when buckets had to be written.

void casacore::BucketCache::get	(	char *	buf,
		uInt	length,
		Int64	offset
	)

Get a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

char* casacore::BucketCache::getBucket ( uInt bucketNr )

Make another bucket current.

When no more cache slots are available, the one least recently used is flushed. The data in the bucket is converted using the ToLocal callback function. When the bucket does not exist yet in the file, it gets added and initialized using the AddBuffer callback function. A pointer to the data in converted format is returned.

void casacore::BucketCache::getSlot ( uInt bucketNr )

private

Get a cache slot for the bucket.

void casacore::BucketCache::initializeBuckets ( uInt bucketNr )

private

Initialize the bucket buffer.

The uninitialized buckets before this bucket are also initialized. It returns a pointer to the buffer.

void casacore::BucketCache::initStatistics ( )

(Re)initialize the cache statistics.

uInt casacore::BucketCache::nBucket ( ) const

Get the current nr of buckets in the file.

uInt casacore::BucketCache::nFreeBucket ( ) const

inline

Get the number of free buckets.

Definition at line 409 of file BucketCache.h.

References its_NrOfFree.

BucketCache& casacore::BucketCache::operator= ( const BucketCache & )

private

Assignment is not possible.

void casacore::BucketCache::put	(	const char *	buf,
		uInt	length,
		Int64	offset
	)

Put a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

void casacore::BucketCache::readBucket ( uInt slotNr )

private

Read a bucket.

void casacore::BucketCache::removeBucket ( )

Remove the current bucket; i.e.

add it to the beginning of the free bucket list.

void casacore::BucketCache::resize ( uInt cacheSize )

Resize the cache.

When the cache gets smaller, the latter buckets are cached out. It does not take "least recently used" into account.

void casacore::BucketCache::resync	(	uInt	nrBucket,
		uInt	nrOfFreeBucket,
		Int	firstFreeBucket
	)

Resynchronize the object (after another process updated the file).

It clears the cache (so all data will be reread) and sets the new sizes.

void casacore::BucketCache::setDirty ( )

Set the dirty bit for the current bucket.

void casacore::BucketCache::setLRU ( )

private

Set the LRU information for the current slot.

void casacore::BucketCache::showStatistics ( ostream & os ) const

Show the statistics.

void casacore::BucketCache::writeBucket ( uInt slotNr )

private

Write a bucket.

Member Data Documentation

uInt casacore::BucketCache::its_ActualSlot

private

The cache slot actually used.

Definition at line 350 of file BucketCache.h.

Block<uInt> casacore::BucketCache::its_BucketNr

private

The buckets in the cache.

Definition at line 354 of file BucketCache.h.

uInt casacore::BucketCache::its_BucketSize

private

The bucket size.

Definition at line 338 of file BucketCache.h.

char* casacore::BucketCache::its_Buffer

private

The internal buffer.

Definition at line 362 of file BucketCache.h.

PtrBlock<char*> casacore::BucketCache::its_Cache

private

The cache itself.

Definition at line 348 of file BucketCache.h.

uInt casacore::BucketCache::its_CacheSize

private

The size of the cache (i.e.

#buckets fitting in it).

Definition at line 344 of file BucketCache.h.

Referenced by cacheSize().

uInt casacore::BucketCache::its_CacheSizeUsed

private

The nr of slots used in the cache.

Definition at line 346 of file BucketCache.h.

uInt casacore::BucketCache::its_CurNrOfBuckets

private

The current nr of buckets in the file.

Definition at line 340 of file BucketCache.h.

BucketCacheDeleteBuffer casacore::BucketCache::its_DeleteCallBack

private

The delete callback function.

Definition at line 334 of file BucketCache.h.

Block<uInt> casacore::BucketCache::its_Dirty

private

Determine if a block is dirty (i.e.

changed) (1=dirty).

Definition at line 356 of file BucketCache.h.

BucketFile* casacore::BucketCache::its_file

private

The file used.

Definition at line 324 of file BucketCache.h.

Int casacore::BucketCache::its_FirstFree

private

The first free bucket (-1 = no free buckets).

Definition at line 366 of file BucketCache.h.

Referenced by firstFreeBucket().

BucketCacheAddBuffer casacore::BucketCache::its_InitCallBack

private

The add bucket callback function.

Definition at line 332 of file BucketCache.h.

Block<uInt> casacore::BucketCache::its_LRU

private

Determine when a block is used for the last time.

Definition at line 358 of file BucketCache.h.

uInt casacore::BucketCache::its_LRUCounter

private

The Least Recently Used counter.

Definition at line 360 of file BucketCache.h.

uInt casacore::BucketCache::its_NewNrOfBuckets

private

The new nr of buckets in the file (after extension).

Definition at line 342 of file BucketCache.h.

uInt casacore::BucketCache::its_NrOfFree

private

The number of free buckets.

Definition at line 364 of file BucketCache.h.

Referenced by nFreeBucket().

void* casacore::BucketCache::its_Owner

private

The owner object.

Definition at line 326 of file BucketCache.h.

BucketCacheToLocal casacore::BucketCache::its_ReadCallBack

private

The read callback function.

Definition at line 328 of file BucketCache.h.

Block<Int> casacore::BucketCache::its_SlotNr

private

The slot numbers of the buckets in the cache (-1 = not in cache).

Definition at line 352 of file BucketCache.h.

Int64 casacore::BucketCache::its_StartOffset

private

The starting offsets of the buckets in the file.

Definition at line 336 of file BucketCache.h.

BucketCacheFromLocal casacore::BucketCache::its_WriteCallBack

private

The write callback function.

Definition at line 330 of file BucketCache.h.

uInt casacore::BucketCache::naccess_p

private

The statistics.

Definition at line 368 of file BucketCache.h.

uInt casacore::BucketCache::ninit_p

private

Definition at line 370 of file BucketCache.h.

uInt casacore::BucketCache::nread_p

private

Definition at line 369 of file BucketCache.h.

uInt casacore::BucketCache::nwrite_p

private

Definition at line 371 of file BucketCache.h.

The documentation for this class was generated from the following file:

casa/casacore/casa/IO/BucketCache.h

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Intended use:

Review Status

Prerequisite

Etymology

Synopsis

Motivation

Example

To Do

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation