Cache for buckets in a part of a file. More...

#include <BucketCache.h>

Public Member Functions
	BucketCache (BucketFile file, Int64 startOffset, uInt bucketSize, uInt nrOfBuckets, uInt cacheSize, void ownerObject, BucketCacheToLocal readCallBack, BucketCacheFromLocal writeCallBack, BucketCacheAddBuffer addCallBack, BucketCacheDeleteBuffer deleteCallBack)
	Create the cache for (a part of) a file.
	~BucketCache ()
Bool	flush (uInt fromSlot=0)
	Flush the cache from the given slot on.
void	clear (uInt fromSlot=0, Bool doFlush=True)
	Clear the cache from the given slot on.
void	resize (uInt cacheSize)
	Resize the cache.
void	resync (uInt nrBucket, uInt nrOfFreeBucket, Int firstFreeBucket)
	Resynchronize the object (after another process updated the file).
uInt	nBucket () const
	Get the current nr of buckets in the file.
uInt	cacheSize () const
	Get the current cache size (in buckets).
void	setDirty ()
	Set the dirty bit for the current bucket.
char *	getBucket (uInt bucketNr)
	Make another bucket current.
void	extend (uInt nrBucket)
	Extend the file with the given number of buckets.
uInt	addBucket (char *data)
	Add a bucket to the file and make it the current one.
void	removeBucket ()
	Remove the current bucket; i.e.
void	get (char *buf, uInt length, Int64 offset)
	Get a part from the file outside the cached area.
void	put (const char *buf, uInt length, Int64 offset)
	Put a part from the file outside the cached area.
Int	firstFreeBucket () const
	Get the bucket number of the first free bucket.
uInt	nFreeBucket () const
	Get the number of free buckets.
void	initStatistics ()
	(Re)initialize the cache statistics.
void	showStatistics (ostream &os) const
	Show the statistics.
Private Member Functions
	BucketCache (const BucketCache &)
	Copy constructor is not possible.
BucketCache &	operator= (const BucketCache &)
	Assignment is not possible.
void	setLRU ()
	Set the LRU information for the current slot.
void	getSlot (uInt bucketNr)
	Get a cache slot for the bucket.
void	writeBucket (uInt slotNr)
	Write a bucket.
void	readBucket (uInt slotNr)
	Read a bucket.
void	initializeBuckets (uInt bucketNr)
	Initialize the bucket buffer.
void	checkOffset (uInt length, Int64 offset) const
	Check if the offset of a non-cached part is correct.
Private Attributes
BucketFile *	its_file
	The file used.
void *	its_Owner
	The owner object.
BucketCacheToLocal	its_ReadCallBack
	The read callback function.
BucketCacheFromLocal	its_WriteCallBack
	The write callback function.
BucketCacheAddBuffer	its_InitCallBack
	The add bucket callback function.
BucketCacheDeleteBuffer	its_DeleteCallBack
	The delete callback function.
Int64	its_StartOffset
	The starting offsets of the buckets in the file.
uInt	its_BucketSize
	The bucket size.
uInt	its_CurNrOfBuckets
	The current nr of buckets in the file.
uInt	its_NewNrOfBuckets
	The new nr of buckets in the file (after extension).
uInt	its_CacheSize
	The size of the cache (i.e.
uInt	its_CacheSizeUsed
	The nr of slots used in the cache.
PtrBlock< char * >	its_Cache
	The cache itself.
uInt	its_ActualSlot
	The cache slot actually used.
Block< Int >	its_SlotNr
	The slot numbers of the buckets in the cache (-1 = not in cache).
Block< uInt >	its_BucketNr
	The buckets in the cache.
Block< uInt >	its_Dirty
	Determine if a block is dirty (i.e.
Block< uInt >	its_LRU
	Determine when a block is used for the last time.
uInt	its_LRUCounter
	The Least Recently Used counter.
char *	its_Buffer
	The internal buffer.
uInt	its_NrOfFree
	The number of free buckets.
Int	its_FirstFree
	The first free bucket (-1 = no free buckets).
uInt	naccess_p
	The statistics.
uInt	nread_p
uInt	ninit_p
uInt	nwrite_p

Detailed Description

Cache for buckets in a part of a file.

Intended use:

Public interface

Review Status

Reviewed By:: UNKNOWN
Date Reviewed:: before2004/08/25

Prerequisite

BucketFile

Etymology

BucketCache implements a cache for buckets in (a part of) a file.

Synopsis

A cache may allow more efficient quasi-random IO. It can, for instance, be used when a limited number of blocks in a file have to be accessed again and again.

The class BucketCache provides such a cache. It can be used on a consecutive part of a file as long as that part is not simultaneously accessed in another way (including another BucketCache object).

BucketCache stores the data as given. It uses (see (group=BucketCache_CallBack))callback functions to allocate/delete buffers and to convert the data to/from local format.

When a new bucket is needed and all slots in the cache are used, BucketCache will remove the least recently used bucket from the cache. When the dirty flag is set, it will first be written.

BucketCache maintains a list of free buckets. Initially this list is empty. When a bucket is removed, it is added to the free list. AddBucket will take buckets from the free list before extending the file.

Since it is possible to handle only a part of a file by a BucketCache object, it is also possible to have multiple BucketCache objects on the same file (as long as they access disjoint parts of the file). Each BucketCache object can have its own bucket size. This can, for example, be used to have tiled arrays with different tile shapes in the same file.

Statistics are kept to know how efficient the cache is working. It is possible to initialize and show the statistics.

Motivation

A cache may reduce IO traffix considerably. Furthermore it is more efficient to keep a cache in local format. In that way conversion to/from local only have to be done when data gets read/written. It also allows for precalculations.

Example

     // Define the callback function for reading a bucket.
     char* bToLocal (void*, const char* data)
     {
       char* ptr = new char[32768];
       memcpy (ptr, data, 32768);
       return ptr;
     }
     // Define the callback function for writing a bucket.
     void bFromLocal (void*, char* data, const char* local)
     {
       memcpy (data, local, 32768);
     }
     // Define the callback function for initializing a new bucket.
     char* bAddBuffer (void*)
     {
       char* ptr = new char[32768];
       for (uInt i=0; i++; i<32768) {
        ptr[i] = 0;
       }
       return ptr;
     }
     // Define the callback function for deleting a bucket.
     void bDeleteBuffer (void*, char* buffer)
     {
       delete [] buffer;
     }
   
     void someFunc()
     {
       // Open the filebuf.
       BucketFile file(...);
       file.open();
       uInt i;
       // Create a cache for the part of the file starting at offset 512
       // consisting of 1000 buckets. The cache consists of 10 buckets.
       // Each bucket is 32768 bytes.
       BucketCache cache (&file, 512, 32768, 1000, 10, 0,
                          bToLocal, bFromLocal, bAddBuffer, bDeleteBuffer);
       // Write all buckets into the file.
       for (i=0; i<100; i++) {
         char* buf = new char[32768];
         cache.addBucket (buf);
       }
       Flush the cache to write all buckets in it.
       cache.flush();
       // Read all buckets from the file.
       for (i=0; i<1000; i++) {
         char* buf = cache.getBucket(i);
         ..\.
       }
       cout << cache.nBucket() << endl;
     }

To Do

When ready, use HashMap for the internal maps.

Definition at line 217 of file BucketCache.h.

Constructor & Destructor Documentation

casa::BucketCache::BucketCache	(	BucketFile *	file,
		Int64	startOffset,
		uInt	bucketSize,
		uInt	nrOfBuckets,
		uInt	cacheSize,
		void *	ownerObject,
		BucketCacheToLocal	readCallBack,
		BucketCacheFromLocal	writeCallBack,
		BucketCacheAddBuffer	addCallBack,
		BucketCacheDeleteBuffer	deleteCallBack
	)

Create the cache for (a part of) a file.

The file part used starts at startOffset. Its length is bucketSize*nrOfBuckets bytes. When the file is smaller, the remainder is indicated as an extension similarly to the behaviour of function extend.

casa::BucketCache::~BucketCache ( )

casa::BucketCache::BucketCache ( const BucketCache & ) [private]

Copy constructor is not possible.

Member Function Documentation

uInt casa::BucketCache::addBucket ( char * data )

Add a bucket to the file and make it the current one.

When no more cache slots are available, the one least recently used is flushed.
When no free buckets are available, the file will be extended with one bucket. It returns the new bucket number. The buffer must have been allocated on the heap. It will get part of the cache; its contents are not copied. Thus the buffer should hereafter NOT be used for other purposes. It will be deleted later via the DeleteBuffer callback function. The data is copied into the bucket. A pointer to the data in local format is returned.

uInt casa::BucketCache::cacheSize ( ) const [inline]

Get the current cache size (in buckets).

Definition at line 403 of file BucketCache.h.

References its_CacheSize.

void casa::BucketCache::checkOffset	(	uInt	length,
		Int64	offset
	)		const `[private]`

Check if the offset of a non-cached part is correct.

void casa::BucketCache::clear	(	uInt	fromSlot = `0`,
		Bool	doFlush = `True`
	)

Clear the cache from the given slot on.

By default the entire cache is cleared. It will remove the buckets in the cleared part. If wanted and needed, the buckets are flushed to the file before removing them. It can be used to enforce rereading buckets from the file.

void casa::BucketCache::extend ( uInt nrBucket )

Extend the file with the given number of buckets.

The buckets get initialized when they are acquired (using getBucket) for the first time.

Int casa::BucketCache::firstFreeBucket ( ) const [inline]

Get the bucket number of the first free bucket.

-1 = no free buckets.

Definition at line 406 of file BucketCache.h.

References its_FirstFree.

Bool casa::BucketCache::flush ( uInt fromSlot = 0 )

Flush the cache from the given slot on.

By default the entire cache is flushed. When the entire cache is flushed, possible remaining uninitialized buckets will be initialized first. A True status is returned when buckets had to be written.

void casa::BucketCache::get	(	char *	buf,
		uInt	length,
		Int64	offset
	)

Get a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

char* casa::BucketCache::getBucket ( uInt bucketNr )

Make another bucket current.

When no more cache slots are available, the one least recently used is flushed. The data in the bucket is converted using the ToLocal callback function. When the bucket does not exist yet in the file, it gets added and initialized using the AddBuffer callback function. A pointer to the data in converted format is returned.

void casa::BucketCache::getSlot ( uInt bucketNr ) [private]

Get a cache slot for the bucket.

void casa::BucketCache::initializeBuckets ( uInt bucketNr ) [private]

Initialize the bucket buffer.

The uninitialized buckets before this bucket are also initialized. It returns a pointer to the buffer.

void casa::BucketCache::initStatistics ( )

(Re)initialize the cache statistics.

uInt casa::BucketCache::nBucket ( ) const

Get the current nr of buckets in the file.

uInt casa::BucketCache::nFreeBucket ( ) const [inline]

Get the number of free buckets.

Definition at line 409 of file BucketCache.h.

References its_NrOfFree.

BucketCache& casa::BucketCache::operator= ( const BucketCache & ) [private]

Assignment is not possible.

void casa::BucketCache::put	(	const char *	buf,
		uInt	length,
		Int64	offset
	)

Put a part from the file outside the cached area.

It is checked if that part is indeed outside the cached file area.

void casa::BucketCache::readBucket ( uInt slotNr ) [private]

Read a bucket.

void casa::BucketCache::removeBucket ( )

Remove the current bucket; i.e.

add it to the beginning of the free bucket list.

void casa::BucketCache::resize ( uInt cacheSize )

Resize the cache.

When the cache gets smaller, the latter buckets are cached out. It does not take "least recently used" into account.

void casa::BucketCache::resync	(	uInt	nrBucket,
		uInt	nrOfFreeBucket,
		Int	firstFreeBucket
	)

Resynchronize the object (after another process updated the file).

It clears the cache (so all data will be reread) and sets the new sizes.

void casa::BucketCache::setDirty ( )

Set the dirty bit for the current bucket.

void casa::BucketCache::setLRU ( ) [private]

Set the LRU information for the current slot.

void casa::BucketCache::showStatistics ( ostream & os ) const

Show the statistics.

void casa::BucketCache::writeBucket ( uInt slotNr ) [private]

Write a bucket.

Member Data Documentation

uInt casa::BucketCache::its_ActualSlot [private]

The cache slot actually used.

Definition at line 350 of file BucketCache.h.

Block<uInt> casa::BucketCache::its_BucketNr [private]

The buckets in the cache.

Definition at line 354 of file BucketCache.h.

uInt casa::BucketCache::its_BucketSize [private]

The bucket size.

Definition at line 338 of file BucketCache.h.

char* casa::BucketCache::its_Buffer [private]

The internal buffer.

Definition at line 362 of file BucketCache.h.

PtrBlock<char*> casa::BucketCache::its_Cache [private]

The cache itself.

Definition at line 348 of file BucketCache.h.

uInt casa::BucketCache::its_CacheSize [private]

The size of the cache (i.e.

#buckets fitting in it).

Definition at line 344 of file BucketCache.h.

Referenced by cacheSize().

uInt casa::BucketCache::its_CacheSizeUsed [private]

The nr of slots used in the cache.

Definition at line 346 of file BucketCache.h.

uInt casa::BucketCache::its_CurNrOfBuckets [private]

The current nr of buckets in the file.

Definition at line 340 of file BucketCache.h.

BucketCacheDeleteBuffer casa::BucketCache::its_DeleteCallBack [private]

The delete callback function.

Definition at line 334 of file BucketCache.h.

Block<uInt> casa::BucketCache::its_Dirty [private]

Determine if a block is dirty (i.e.

changed) (1=dirty).

Definition at line 356 of file BucketCache.h.

BucketFile* casa::BucketCache::its_file [private]

The file used.

Definition at line 324 of file BucketCache.h.

Int casa::BucketCache::its_FirstFree [private]

The first free bucket (-1 = no free buckets).

Definition at line 366 of file BucketCache.h.

Referenced by firstFreeBucket().

BucketCacheAddBuffer casa::BucketCache::its_InitCallBack [private]

The add bucket callback function.

Definition at line 332 of file BucketCache.h.

Block<uInt> casa::BucketCache::its_LRU [private]

Determine when a block is used for the last time.

Definition at line 358 of file BucketCache.h.

uInt casa::BucketCache::its_LRUCounter [private]

The Least Recently Used counter.

Definition at line 360 of file BucketCache.h.

uInt casa::BucketCache::its_NewNrOfBuckets [private]

The new nr of buckets in the file (after extension).

Definition at line 342 of file BucketCache.h.

uInt casa::BucketCache::its_NrOfFree [private]

The number of free buckets.

Definition at line 364 of file BucketCache.h.

Referenced by nFreeBucket().

void* casa::BucketCache::its_Owner [private]

The owner object.

Definition at line 326 of file BucketCache.h.

BucketCacheToLocal casa::BucketCache::its_ReadCallBack [private]

The read callback function.

Definition at line 328 of file BucketCache.h.

Block<Int> casa::BucketCache::its_SlotNr [private]

The slot numbers of the buckets in the cache (-1 = not in cache).

Definition at line 352 of file BucketCache.h.

Int64 casa::BucketCache::its_StartOffset [private]

The starting offsets of the buckets in the file.

Definition at line 336 of file BucketCache.h.

BucketCacheFromLocal casa::BucketCache::its_WriteCallBack [private]

The write callback function.

Definition at line 330 of file BucketCache.h.

uInt casa::BucketCache::naccess_p [private]

The statistics.

Definition at line 368 of file BucketCache.h.

uInt casa::BucketCache::ninit_p [private]

Definition at line 370 of file BucketCache.h.

uInt casa::BucketCache::nread_p [private]

Definition at line 369 of file BucketCache.h.

uInt casa::BucketCache::nwrite_p [private]

Definition at line 371 of file BucketCache.h.

The documentation for this class was generated from the following file:

casacore/casa/IO/BucketCache.h

Public Member Functions

Private Member Functions

Private Attributes