CWB
Macros | Functions
storage.c File Reference
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include "globals.h"
#include "endian.h"
#include "macros.h"
#include "storage.h"

Macros

#define MMAPFLAGS   MAP_PRIVATE
 Flags used in calls to mmap. More...
 
#define MMAP_EMPTY_LEN   8 /* can't mmap() 0 bytes, so mmap() MMAP_EMPTY_LEN bytes beyond end-of-file for empty files */
 

Functions

void NwriteInt (int val, FILE *fd)
 Writes a 32-bit integer to file, converting to network byte order. More...
 
void NreadInt (int *val, FILE *fd)
 Reads a 32-bit integer from file, converting from network byte order. More...
 
void NwriteInts (int *vals, int nr_vals, FILE *fd)
 Writes an array of 32-bit integers to file, converting to network byte order. More...
 
void NreadInts (int *vals, int nr_vals, FILE *fd)
 Reads an array of 32-bit integers from file, converting from network byte order. More...
 
void init_mblob (MemBlob *blob)
 Clears all fields in a MemBlob, regardless of their usage, and puts the blob back to its virginal state. More...
 
int alloc_mblob (MemBlob *blob, int nr_items, int item_size, int clear_blob)
 Allocates memory for a blob of the requested size, using malloc(2). More...
 
void mfree (MemBlob *blob)
 Frees the memory used by a MemBlob. More...
 
void * mmapfile (char *filename, size_t *len_ptr, char *mode)
 Maps a file into memory in either read or write mode. More...
 
void * mallocfile (char *filename, size_t *len_ptr, char *mode)
 Maps a file into memory. More...
 
int read_file_into_blob (char *filename, int allocation_method, int item_size, MemBlob *blob)
 Reads the contents of a file into memory represented by blob. More...
 
int write_file_from_blob (char *filename, MemBlob *blob, int convert_to_nbo)
 Writes the data stored in a blob to file. More...
 

Macro Definition Documentation

#define MMAP_EMPTY_LEN   8 /* can't mmap() 0 bytes, so mmap() MMAP_EMPTY_LEN bytes beyond end-of-file for empty files */

Referenced by mfree(), and mmapfile().

#define MMAPFLAGS   MAP_PRIVATE

Flags used in calls to mmap.

TODO: actually, only used when calling mmap() for read: should be renamed to indicate this.

See also
mmapfile

Referenced by mmapfile().

Function Documentation

int alloc_mblob ( MemBlob blob,
int  nr_items,
int  item_size,
int  clear_blob 
)

Allocates memory for a blob of the requested size, using malloc(2).

A block of memory holding nr_items of size item_size is created in the specified MemBlob.

Parameters
blobThe MemBlob in which to place the memory.
nr_itemsThe number of items the MemBlob is to hold as data.
item_sizeThe size of one item.
clear_blobboolean: if true, all bytes in the data space will be initialised to 0
Returns
boolean: true 1 if OK, false on error

References TMblob::allocation_method, TMblob::changed, cl_calloc(), cl_malloc(), TMblob::data, TMblob::fname, TMblob::fsize, TMblob::item_size, MALLOCED, TMblob::nr_items, TMblob::offset, TMblob::size, SIZE_BIT, and TMblob::writeable.

void init_mblob ( MemBlob blob)

Clears all fields in a MemBlob, regardless of their usage, and puts the blob back to its virginal state.

Note that it doesn't free blob->data - just sets it to NULL.

References TMblob::allocation_method, TMblob::changed, TMblob::data, TMblob::fname, TMblob::fsize, TMblob::item_size, TMblob::nr_items, TMblob::offset, TMblob::size, UNALLOCATED, and TMblob::writeable.

Referenced by declare_component(), and mfree().

void* mallocfile ( char *  filename,
size_t *  len_ptr,
char *  mode 
)

Maps a file into memory.

This function does virtually the same as mmapfile (same parameters, same return value), but the memory is taken with malloc(3), not with mmap(2).

See also
mmapfile

References cl_free, cl_malloc(), and O_BINARY.

Referenced by read_file_into_blob().

void mfree ( MemBlob blob)

Frees the memory used by a MemBlob.

This works regardless of the method used to allocate the blob.

References TMblob::allocation_method, cl_free, TMblob::data, TMblob::fname, init_mblob(), MALLOCED, MMAP_EMPTY_LEN, MMAPPED, munmap(), TMblob::size, and UNALLOCATED.

Referenced by comp_drop_component().

void* mmapfile ( char *  filename,
size_t *  len_ptr,
char *  mode 
)

Maps a file into memory in either read or write mode.

Parameters
filenameName of the file to map.
len_ptrThe number of bytes the returned pointer points to.
modeCan be either "r", "w", "rb" or "wb". (a "+" can be used too, but is ignored.) If mode is "r", the amount of memory allocated (size of file) is placed in len_ptr. If mode is "w", len_ptr is taken as an input parameter (*len_ptr bytes are allocated).
Returns
The contents of file in filename as a pointer to a memory area.

References MAP_FAILED, MAP_SHARED, mmap(), MMAP_EMPTY_LEN, MMAPFLAGS, O_BINARY, PROT_READ, and PROT_WRITE.

Referenced by read_file_into_blob().

void NreadInt ( int *  val,
FILE *  fd 
)

Reads a 32-bit integer from file, converting from network byte order.

This function does all the error checking for you, and will abort the program if the int cannot be read.

Parameters
valLocation to put the resulting int.
fdFile handle to read from

References word.

Referenced by decode_check_huff(), and ReadHCD().

void NreadInts ( int *  vals,
int  nr_vals,
FILE *  fd 
)

Reads an array of 32-bit integers from file, converting from network byte order.

This function does all the error checking for you, and will abort the program if the requested number of ints cannot be read.

Parameters
valsPointer to location to put the resulting array of ints. (This memory must have been allocated by the caller.)
nr_valsNumber of integers to read.
fdFile handle to read from

References word.

Referenced by ReadHCD().

void NwriteInt ( int  val,
FILE *  fd 
)

Writes a 32-bit integer to file, converting to network byte order.

Other than the byte order conversion, amd the fact that it exits the program upon error (so the user of this function can assume success), this is the same as

fwrite(&val, sizeof(int), 1, fd) .

Parameters
valThe integer to write.
fdFile handle to write to.

References word.

Referenced by compress_reversed_index(), compute_code_lengths(), creat_rev_corpus(), encode_add_wattr_line(), main(), range_close(), sencode_write_region(), and WriteHCD().

void NwriteInts ( int *  vals,
int  nr_vals,
FILE *  fd 
)

Writes an array of 32-bit integers to file, converting to network byte order.

Other than the byte order conversion, this is the same as fwrite(vals, sizeof(int), nr_vals, fd) .

Parameters
valsPointer to the location of the block of integers to write.
nr_valsNumber of integers to write.
fdFile handle to write to.

References word.

Referenced by creat_rev_corpus(), write_file_from_blob(), and WriteHCD().

int read_file_into_blob ( char *  filename,
int  allocation_method,
int  item_size,
MemBlob blob 
)

Reads the contents of a file into memory represented by blob.

You can choose the memory allocation method - MMAPPED is faster, but writeable areas of memory should be taken with care. MALLOCED is slower (and far more space consuming), but writing data into malloced memory is no problem.

In Windows, the read is always binary-mode.

Parameters
filenameThe file to read in.
allocation_methodMMAPPED or MALLOCED (see function description)
item_sizeThis is used for MemBlob access methods, it is simply copied into the MemBlob data structure.
blobThe MemBlob to read the file into. It must not be in use – the fields are overwritten.
Returns
0 on failure, 1 if everything went fine.

References TMblob::allocation_method, TMblob::changed, TMblob::data, TMblob::item_size, MALLOCED, mallocfile(), mmapfile(), MMAPPED, TMblob::nr_items, TMblob::size, SIZE_BIT, UNALLOCATED, and TMblob::writeable.

Referenced by creat_freqs(), creat_sort_lexicon(), and load_component().

int write_file_from_blob ( char *  filename,
MemBlob blob,
int  convert_to_nbo 
)

Writes the data stored in a blob to file.

Note that here we are writing to a new file: we are not writing back to the file the data came from, if MMAPPED.

In Windows, the write is always binary-mode.

Parameters
filenameThe file to write to.
blobThe MemBlob to write to file.
convert_to_nboboolean: if true, data is converted to network byte order before it's written.
Returns
0 on failure, 1 if everything went fine.

References TMblob::allocation_method, TMblob::changed, TMblob::data, MALLOCED, MMAPPED, NwriteInts(), TMblob::size, and UNALLOCATED.

Referenced by creat_freqs(), creat_rev_corpus_idx(), and creat_sort_lexicon().