TSMCube.h

Classes

TSMCube -- Tiled hypercube in a table (full description)

class TSMCube

Types

enum AccessType

NoAccess
CellAccess
SliceAccess
ColumnAccess
ColumnSliceAccess

Interface

Public Members
TSMCube (TiledStMan* stman, TSMFile* file)
TSMCube (TiledStMan* stman, TSMFile* file, const IPosition& cubeShape, const IPosition& tileShape, const Record& values)
TSMCube (TiledStMan* stman, AipsIO& ios)
TSMCube (TiledStMan* stman, TSMFile* file, const IPosition& cubeShape, const IPosition& tileShape, Int64 fileOffset)
~TSMCube()
void flushCache()
void clearCache (Bool doFlush = True)
void emptyCache()
void showCacheStatistics (ostream& os) const
void putObject (AipsIO& ios)
Int getObject (AipsIO& ios)
void resync (AipsIO& ios)
Bool isExtensible() const
uInt bucketSize() const
uInt localTileLength() const
void setShape (const IPosition& cubeShape, const IPosition& tileShape)
const IPosition& cubeShape() const
const IPosition& tileShape() const
IPosition cellShape() const
uInt coordinateSize (const String& coordinateName) const
const Record& valueRecord() const
Record& rwValueRecord()
Bool matches (const PtrBlock<TSMColumn*>& idColSet, const Record& idValues)
void extend (uInt nr, const Record& coordValues, const TSMColumn* lastCoordColumn)
void extendCoordinates (const Record& coordValues, const String& coordName, uInt length)
void accessSection (const IPosition& start, const IPosition& end, char* section, uInt colnr, uInt localPixelSize, Bool writeFlag)
void accessStrided (const IPosition& start, const IPosition& end, const IPosition& stride, char* section, uInt colnr, uInt localPixelSize, Bool writeFlag)
uInt cacheSize() const
uInt calcCacheSize (const IPosition& sliceShape, const IPosition& windowStart, const IPosition& windowLength, const IPosition& axisPath) const
void setCacheSize (const IPosition& sliceShape, const IPosition& windowStart, const IPosition& windowLength, const IPosition& axisPath, Bool forceSmaller, Bool userSet)
void setCacheSize (uInt cacheSize, Bool forceSmaller, Bool userSet)
uInt validateCacheSize (uInt cacheSize) const
Bool userSetCache() const
AccessType getLastColAccess() const
const IPosition& getLastColSlice() const
void setLastColAccess (AccessType type)
void setLastColSlice (const IPosition& slice)
Private Members
TSMCube (const TSMCube&)
TSMCube& operator= (const TSMCube&)
void setup()
void setupNrTiles()
IPosition adjustTileShape (const IPosition& cubeShape, const IPosition& tileShape) const
BucketCache* getCache()
void makeCache()
void accessLine (char* section, uInt pixelOffset, uInt localPixelSize, Bool writeFlag, BucketCache* cachePtr, const IPosition& startTile, uInt endTile, const IPosition& startPixelInFirstTile, uInt endPixelInLastTile, uInt lineIndex)
static char* readCallBack (void* owner, const char* external)
static void writeCallBack (void* owner, char* external, const char* local)
static char* initCallBack (void* owner)
static void deleteCallBack (void* owner, char* buffer)
char* readTile (const char* external)
void writeTile (char* external, const char* local)
void resizeTileSections()

Description

Prerequisite

Etymology

TSMCube represents a hypercube in the Tiled Storage Manager.

Synopsis

TSMCube defines a tiled hypercube. The data is stored in a TSMFile object and accessed using a BucketCache object. The hypercube can be extensible in its last dimension to support tables with a size which is not known in advance.
Normally hypercubes share the same TSMFile object, but extensible hypercubes have their own TSMFile object (to be extensible). When the hypercolumn has multiple data columns, their cells share the same tiles. Per tile data column A appears first, thereafter B, etc..
The data in the cache is held in external format and is converted when accessed. The alternative would be to hold it in the cache in local format and convert it when read/written from the file. It was felt that the latter approach would generate more needless conversions.

The possible id and coordinate values are stored in a Record object. They are written in the main hypercube AipsIO file.

TSMCube uses the maximum cache size set for a Tiled Storage manager. The description of class ROTiledStManAccessor contains a discussion about the effect of setting the maximum cache size.

Motivation

TSMCube encapsulates all operations on a hypercube.

Member Description

enum AccessType

Define the possible access types for TSMDataColumn.

TSMCube (TiledStMan* stman, TSMFile* file)

Construct an object with a still undefined hypercube. It can be filled later with setShape. It is used by TiledCellStMan which may know its hypercube shape only when ArrayColumn::setShape is done.

TSMCube (TiledStMan* stman, TSMFile* file, const IPosition& cubeShape, const IPosition& tileShape, const Record& values)

Construct the hypercube using the given file with the given shape. The record contains the id and possible coordinate values.

TSMCube (TiledStMan* stman, AipsIO& ios)

Reconstruct the hypercube by reading its data from the AipsIO stream. It will link itself to the correct TSMFile. The TSMFile objects must have been reconstructed in advance.

TSMCube (TiledStMan* stman, TSMFile* file, const IPosition& cubeShape, const IPosition& tileShape, Int64 fileOffset)

Construct a TSMCube object for an existing array in a file. It is meant to be used by class=TiledFileAccess>TiledFileAccess.

~TSMCube()

void flushCache()

Flush the data in the cache.

void clearCache (Bool doFlush = True)

Clear the cache, so data will be reread. If wanted, the data is flushed before the cache is cleared.

void emptyCache()

Empty the cache. It will flush the cache as needed and remove all buckets from it resulting in a possibly large drop in memory used. It'll also clear the userSetCache_p flag.

void showCacheStatistics (ostream& os) const

Show the cache statistics.

void putObject (AipsIO& ios)

Put the data of the object into the AipsIO stream.

Int getObject (AipsIO& ios)

Get the data of the object from the AipsIO stream. It returns the data manager sequence number, which is -1 if no file is attached to the cube (for cells without a value).

void resync (AipsIO& ios)

Resync the object with the data file. It reads the object, and adjusts the cache.

Bool isExtensible() const

Is the hypercube extensible?

uInt bucketSize() const

Get the bucket size (which is the length of a tile in external format).

uInt localTileLength() const

Get the lenghth of a tile in local format.

void setShape (const IPosition& cubeShape, const IPosition& tileShape)

Set the hypercube shape. This is only possible when the shape was not defined yet.

const IPosition& cubeShape() const

Get the shape of the hypercube.

const IPosition& tileShape() const

Get the shape of the tiles.

IPosition cellShape() const

Get the shape of the data cells in the cube.

uInt coordinateSize (const String& coordinateName) const

Get the size of a coordinate (i.e. the number of values in it). If not defined, it returns zero.

const Record& valueRecord() const
Record& rwValueRecord()

Get the record containing the id and coordinate values. It is used by TSMIdColumn and TSMCoordColumn.

Bool matches (const PtrBlock<TSMColumn*>& idColSet, const Record& idValues)

Test if the id values match.

void extend (uInt nr, const Record& coordValues, const TSMColumn* lastCoordColumn)

Extend the last dimension of the cube with the given number. The record can contain the coordinates of the elements added.

void extendCoordinates (const Record& coordValues, const String& coordName, uInt length)

Extend the coordinates vector for the given coordinate to the given length with the given coordValues. It will be initialized to zero if no coordValues are given. If the coordinate vector does not exist yet, it will be created.

void accessSection (const IPosition& start, const IPosition& end, char* section, uInt colnr, uInt localPixelSize, Bool writeFlag)

Read or write a section in the cube. It is assumed that the section buffer is long enough.

void accessStrided (const IPosition& start, const IPosition& end, const IPosition& stride, char* section, uInt colnr, uInt localPixelSize, Bool writeFlag)

Read or write a section in a strided way. It is assumed that the section buffer is long enough.

uInt cacheSize() const

Get the current cache size (in buckets).

uInt calcCacheSize (const IPosition& sliceShape, const IPosition& windowStart, const IPosition& windowLength, const IPosition& axisPath) const

Calculate the cache size (in buckets) for the given slice and access path.

void setCacheSize (const IPosition& sliceShape, const IPosition& windowStart, const IPosition& windowLength, const IPosition& axisPath, Bool forceSmaller, Bool userSet)

Set the cache size for the given slice and access path.

void setCacheSize (uInt cacheSize, Bool forceSmaller, Bool userSet)

Resize the cache object. When forceSmaller is False, the cache will only be resized when it grows. When the given size exceeds the maximum size with more than 10%, the maximum size will be used. The cacheSize has to be given in buckets.
The flag userSet inidicates if the cache size is set by the user (by an Accessor object) or automatically (by TSMDataColumn).

uInt validateCacheSize (uInt cacheSize) const

Validate the cache size (in buckets). This means it will return the given cache size if smaller than the maximum cache size. Otherwise the maximum is returned.

Bool userSetCache() const

Determine if the user set the cache size (using setCacheSize).

AccessType getLastColAccess() const
const IPosition& getLastColSlice() const
void setLastColAccess (AccessType type)
void setLastColSlice (const IPosition& slice)

Functions for TSMDataColumn to keep track of the last type of access to a hypercube. It uses it to determine if the cache has to be reset.

TSMCube (const TSMCube&)

Forbid copy constructor.

TSMCube& operator= (const TSMCube&)

Forbid assignment.

void setup()
void setupNrTiles()

Initialize the various variables.

IPosition adjustTileShape (const IPosition& cubeShape, const IPosition& tileShape) const

Adjust the tile shape to the hypercube shape. A size of 0 gets set to 1. A tile size > cube size gets set to the cube size.

BucketCache* getCache()

Get the cache object. This will construct the cache object if not present yet.

void makeCache()

Construct the cache object (if not constructed yet).

void accessLine (char* section, uInt pixelOffset, uInt localPixelSize, Bool writeFlag, BucketCache* cachePtr, const IPosition& startTile, uInt endTile, const IPosition& startPixelInFirstTile, uInt endPixelInLastTile, uInt lineIndex)

Access a line in a more optimized way.

static char* readCallBack (void* owner, const char* external)
static void writeCallBack (void* owner, char* external, const char* local)
static char* initCallBack (void* owner)
static void deleteCallBack (void* owner, char* buffer)

Define the callback functions for the BucketCache.

char* readTile (const char* external)
void writeTile (char* external, const char* local)

Define the functions doing the actual read and write of the data in the tile and converting it to/from local format.

void resizeTileSections()

Resize the IPosition member variables used in accessSection() if nrdim_p changes value.