CWB
|
Data Structures | |
struct | TMblob |
The MemBlob object. More... | |
Macros | |
#define | MAP_FAILED ((void *) -1) |
#define | UNALLOCATED 0 |
Flag: no memory has been allocated. More... | |
#define | MMAPPED 1 |
Flag: indicates use of mmap() to allocate memory in a MemBlob. More... | |
#define | MALLOCED 2 |
Flag: indicates use of malloc() to allocate memory. More... | |
#define | MEMBLOB_UNALLOCATED 0 |
Flag: indicates no memory has been allocated. More... | |
#define | MEMBLOB_MMAPPED 1 |
Flag: indicates use of mmap() to allocate memory in a MemBlob. More... | |
#define | MEMBLOB_MALLOCED 2 |
Flag: indicates use of malloc() to allocate memory in a MemBlob. More... | |
#define | SIZE_BIT 0 |
#define | SIZE_BYTE sizeof(char) |
#define | SIZE_SHINT sizeof(short) |
#define | SIZE_INT sizeof(int) |
#define | SIZE_LONG sizeof(long) |
#define | memblob_read_from_file read_file_into_blob |
#define | memblob_write_to_file write_file_from_blob |
#define | memblob_free mfree |
#define | memblob_clear init_mblob |
#define | memblob_allocate alloc_mblob |
Typedefs | |
typedef struct TMblob | MemBlob |
The MemBlob object. More... | |
Functions | |
void | NwriteInt (int val, FILE *fd) |
Writes a 32-bit integer to file, converting to network byte order. More... | |
void | NreadInt (int *val, FILE *fd) |
Reads a 32-bit integer from file, converting from network byte order. More... | |
void | NwriteInts (int *vals, int nr_vals, FILE *fd) |
Writes an array of 32-bit integers to file, converting to network byte order. More... | |
void | NreadInts (int *vals, int nr_vals, FILE *fd) |
Reads an array of 32-bit integers from file, converting from network byte order. More... | |
void | mfree (MemBlob *blob) |
Frees the memory used by a MemBlob. More... | |
void | init_mblob (MemBlob *blob) |
Clears all fields in a MemBlob, regardless of their usage, and puts the blob back to its virginal state. More... | |
int | alloc_mblob (MemBlob *blob, int nr_items, int item_size, int clear_blob) |
Allocates memory for a blob of the requested size, using malloc(2). More... | |
int | read_file_into_blob (char *filename, int allocation_method, int item_size, MemBlob *blob) |
Reads the contents of a file into memory represented by blob. More... | |
int | write_file_from_blob (char *filename, MemBlob *blob, int convert_to_nbo) |
Writes the data stored in a blob to file. More... | |
void * | mmapfile (char *filename, size_t *len_ptr, char *mode) |
Maps a file into memory in either read or write mode. More... | |
void * | mallocfile (char *filename, size_t *len_ptr, char *mode) |
Maps a file into memory. More... | |
#define MALLOCED 2 |
Flag: indicates use of malloc() to allocate memory.
Referenced by alloc_mblob(), creat_freqs(), creat_rev_corpus_idx(), creat_sort_lexicon(), mfree(), read_file_into_blob(), and write_file_from_blob().
#define MAP_FAILED ((void *) -1) |
Referenced by mmap(), and mmapfile().
#define memblob_allocate alloc_mblob |
#define memblob_clear init_mblob |
#define memblob_free mfree |
#define MEMBLOB_MALLOCED 2 |
Flag: indicates use of malloc() to allocate memory in a MemBlob.
#define MEMBLOB_MMAPPED 1 |
Flag: indicates use of mmap() to allocate memory in a MemBlob.
#define memblob_read_from_file read_file_into_blob |
#define MEMBLOB_UNALLOCATED 0 |
Flag: indicates no memory has been allocated.
#define memblob_write_to_file write_file_from_blob |
#define MMAPPED 1 |
Flag: indicates use of mmap() to allocate memory in a MemBlob.
Referenced by load_component(), mfree(), read_file_into_blob(), and write_file_from_blob().
#define SIZE_BIT 0 |
Referenced by alloc_mblob(), and read_file_into_blob().
#define SIZE_BYTE sizeof(char) |
#define SIZE_INT sizeof(int) |
Referenced by creat_rev_corpus_idx().
#define SIZE_LONG sizeof(long) |
#define SIZE_SHINT sizeof(short) |
#define UNALLOCATED 0 |
Flag: no memory has been allocated.
Referenced by init_mblob(), mfree(), read_file_into_blob(), and write_file_from_blob().
The MemBlob object.
This object, unsurprisingly, represents a blob of memory.
int alloc_mblob | ( | MemBlob * | blob, |
int | nr_items, | ||
int | item_size, | ||
int | clear_blob | ||
) |
Allocates memory for a blob of the requested size, using malloc(2).
A block of memory holding nr_items of size item_size is created in the specified MemBlob.
blob | The MemBlob in which to place the memory. |
nr_items | The number of items the MemBlob is to hold as data. |
item_size | The size of one item. |
clear_blob | boolean: if true, all bytes in the data space will be initialised to 0 |
References TMblob::allocation_method, TMblob::changed, cl_calloc(), cl_malloc(), TMblob::data, TMblob::fname, TMblob::fsize, TMblob::item_size, MALLOCED, TMblob::nr_items, TMblob::offset, TMblob::size, SIZE_BIT, and TMblob::writeable.
void init_mblob | ( | MemBlob * | blob | ) |
Clears all fields in a MemBlob, regardless of their usage, and puts the blob back to its virginal state.
Note that it doesn't free blob->data - just sets it to NULL.
References TMblob::allocation_method, TMblob::changed, TMblob::data, TMblob::fname, TMblob::fsize, TMblob::item_size, TMblob::nr_items, TMblob::offset, TMblob::size, UNALLOCATED, and TMblob::writeable.
Referenced by declare_component(), and mfree().
void* mallocfile | ( | char * | filename, |
size_t * | len_ptr, | ||
char * | mode | ||
) |
Maps a file into memory.
This function does virtually the same as mmapfile (same parameters, same return value), but the memory is taken with malloc(3), not with mmap(2).
References cl_free, cl_malloc(), and O_BINARY.
Referenced by read_file_into_blob().
void mfree | ( | MemBlob * | blob | ) |
Frees the memory used by a MemBlob.
This works regardless of the method used to allocate the blob.
References TMblob::allocation_method, cl_free, TMblob::data, TMblob::fname, init_mblob(), MALLOCED, MMAP_EMPTY_LEN, MMAPPED, munmap(), TMblob::size, and UNALLOCATED.
Referenced by comp_drop_component().
void* mmapfile | ( | char * | filename, |
size_t * | len_ptr, | ||
char * | mode | ||
) |
Maps a file into memory in either read or write mode.
filename | Name of the file to map. |
len_ptr | The number of bytes the returned pointer points to. |
mode | Can be either "r", "w", "rb" or "wb". (a "+" can be used too, but is ignored.) If mode is "r", the amount of memory allocated (size of file) is placed in len_ptr. If mode is "w", len_ptr is taken as an input parameter (*len_ptr bytes are allocated). |
References MAP_FAILED, MAP_SHARED, mmap(), MMAP_EMPTY_LEN, MMAPFLAGS, O_BINARY, PROT_READ, and PROT_WRITE.
Referenced by read_file_into_blob().
void NreadInt | ( | int * | val, |
FILE * | fd | ||
) |
Reads a 32-bit integer from file, converting from network byte order.
This function does all the error checking for you, and will abort the program if the int cannot be read.
val | Location to put the resulting int. |
fd | File handle to read from |
References word.
Referenced by decode_check_huff(), and ReadHCD().
void NreadInts | ( | int * | vals, |
int | nr_vals, | ||
FILE * | fd | ||
) |
Reads an array of 32-bit integers from file, converting from network byte order.
This function does all the error checking for you, and will abort the program if the requested number of ints cannot be read.
vals | Pointer to location to put the resulting array of ints. (This memory must have been allocated by the caller.) |
nr_vals | Number of integers to read. |
fd | File handle to read from |
References word.
Referenced by ReadHCD().
void NwriteInt | ( | int | val, |
FILE * | fd | ||
) |
Writes a 32-bit integer to file, converting to network byte order.
Other than the byte order conversion, amd the fact that it exits the program upon error (so the user of this function can assume success), this is the same as
fwrite(&val, sizeof(int), 1, fd) .
val | The integer to write. |
fd | File handle to write to. |
References word.
Referenced by compress_reversed_index(), compute_code_lengths(), creat_rev_corpus(), encode_add_wattr_line(), main(), range_close(), sencode_write_region(), and WriteHCD().
void NwriteInts | ( | int * | vals, |
int | nr_vals, | ||
FILE * | fd | ||
) |
Writes an array of 32-bit integers to file, converting to network byte order.
Other than the byte order conversion, this is the same as fwrite(vals, sizeof(int), nr_vals, fd) .
vals | Pointer to the location of the block of integers to write. |
nr_vals | Number of integers to write. |
fd | File handle to write to. |
References word.
Referenced by creat_rev_corpus(), write_file_from_blob(), and WriteHCD().
int read_file_into_blob | ( | char * | filename, |
int | allocation_method, | ||
int | item_size, | ||
MemBlob * | blob | ||
) |
Reads the contents of a file into memory represented by blob.
You can choose the memory allocation method - MMAPPED is faster, but writeable areas of memory should be taken with care. MALLOCED is slower (and far more space consuming), but writing data into malloced memory is no problem.
In Windows, the read is always binary-mode.
filename | The file to read in. |
allocation_method | MMAPPED or MALLOCED (see function description) |
item_size | This is used for MemBlob access methods, it is simply copied into the MemBlob data structure. |
blob | The MemBlob to read the file into. It must not be in use – the fields are overwritten. |
References TMblob::allocation_method, TMblob::changed, TMblob::data, TMblob::item_size, MALLOCED, mallocfile(), mmapfile(), MMAPPED, TMblob::nr_items, TMblob::size, SIZE_BIT, UNALLOCATED, and TMblob::writeable.
Referenced by creat_freqs(), creat_sort_lexicon(), and load_component().
int write_file_from_blob | ( | char * | filename, |
MemBlob * | blob, | ||
int | convert_to_nbo | ||
) |
Writes the data stored in a blob to file.
Note that here we are writing to a new file: we are not writing back to the file the data came from, if MMAPPED.
In Windows, the write is always binary-mode.
filename | The file to write to. |
blob | The MemBlob to write to file. |
convert_to_nbo | boolean: if true, data is converted to network byte order before it's written. |
References TMblob::allocation_method, TMblob::changed, TMblob::data, MALLOCED, MMAPPED, NwriteInts(), TMblob::size, and UNALLOCATED.
Referenced by creat_freqs(), creat_rev_corpus_idx(), and creat_sort_lexicon().