{ "access": { "embargo": { "active": false, "reason": null }, "files": "public", "record": "public", "status": "open" }, "created": "2018-02-07T20:18:34.675171+00:00", "custom_fields": {}, "deletion_status": { "is_deleted": false, "status": "P" }, "files": { "count": 1, "enabled": true, "entries": { "syhlalala/PAM-AE-v1.0.zip": { "checksum": "md5:0492811def5dac2f56beb2d527b435ea", "ext": "zip", "id": "29c15ebe-7b02-402c-baeb-411821aca16c", "key": "syhlalala/PAM-AE-v1.0.zip", "metadata": null, "mimetype": "application/zip", "size": 35603244 } }, "order": [], "total_bytes": 35603244 }, "id": "1168703", "is_draft": false, "is_published": true, "links": { "access": "https://zenodo.org/api/records/1168703/access", "access_links": "https://zenodo.org/api/records/1168703/access/links", "access_request": "https://zenodo.org/api/records/1168703/access/request", "access_users": "https://zenodo.org/api/records/1168703/access/users", "archive": "https://zenodo.org/api/records/1168703/files-archive", "archive_media": "https://zenodo.org/api/records/1168703/media-files-archive", "communities": "https://zenodo.org/api/records/1168703/communities", "communities-suggestions": "https://zenodo.org/api/records/1168703/communities-suggestions", "doi": "https://doi.org/10.5281/zenodo.1168703", "draft": "https://zenodo.org/api/records/1168703/draft", "files": "https://zenodo.org/api/records/1168703/files", "latest": "https://zenodo.org/api/records/1168703/versions/latest", "latest_html": "https://zenodo.org/records/1168703/latest", "media_files": "https://zenodo.org/api/records/1168703/media-files", "parent": "https://zenodo.org/api/records/1168702", "parent_doi": "https://zenodo.org/doi/10.5281/zenodo.1168702", "parent_html": "https://zenodo.org/records/1168702", "requests": "https://zenodo.org/api/records/1168703/requests", "reserve_doi": "https://zenodo.org/api/records/1168703/draft/pids/doi", "self": "https://zenodo.org/api/records/1168703", "self_doi": "https://zenodo.org/doi/10.5281/zenodo.1168703", "self_html": "https://zenodo.org/records/1168703", "self_iiif_manifest": "https://zenodo.org/api/iiif/record:1168703/manifest", "self_iiif_sequence": "https://zenodo.org/api/iiif/record:1168703/sequence/default", "versions": "https://zenodo.org/api/records/1168703/versions" }, "media_files": { "count": 0, "enabled": false, "entries": {}, "order": [], "total_bytes": 0 }, "metadata": { "creators": [ { "person_or_org": { "family_name": "syhlalala", "name": "syhlalala", "type": "personal" } } ], "description": "PAM\n
PAM (Parallel Augmented Maps) is a parallel C++ library implementing the interface for augmented maps [1]. It is designed for maintaining an ordered map data structure while efficiently answering range-based and related queries. In the experiments we use the interface in four examples: augmented-sums, interval-queries, 2d range-queries, and an inverted index. The released code includes the examples and scripts for running the specific experiments reported in the paper. It is also designed so it is easy to try in many other scenarios (different sizes, different numbers of cores, and other operations described in the paper, but not reported in the experiments).
\nMore details and examples can be found in our paper [1].
\nUsage:\nTo define an augmented map using PAM, user need to specify the parameters including type names and (static) functions in the entry structure ``entry''.
\nThen an augmented map is defined with C++ template as
\naugmented_map<entry>.\n
\nNote that a plain ordered map is defined as an augmented map with no augmentation (i.e., it only has K, <_K and V in its entry) and a plain ordered set is similarly defined as an augmented map with no augmentation and no value type.
\nHere is an example of defining an augmented map \"m\" that has integer keys and values and is augmented with value sums (similar as the augmented sum example in our paper [1]):
\nstruct entry {\n using key_t = int;\n using val_t = int;\n using aug_t = int;\n static bool comp(key_t a, key_t b) { \n return a < b;}\n static aug_t get_empty() { return 0;}\n static aug_t from_entry(key_t k, val_t v) { \n return v;}\n static aug_t combine(aug_t a, aug_t b) { \n return a+b;}};\naugmented_map<entry> m;\n
\nAnother quick example can be found in [1], which shows how to implement an interval tree using the PAM interface.
\nHardware dependencies\nAny modern (2010+) x86-based multicore machines. Relies on 128-bit CMPXCHG (requires -mcx16 compiler flag) but does not need hardware transactional memory (TSX). Most examples given in our scripts require 64GB memory, but range_query requires 256GB memory and aug_sum on the large input requires 1TB memory. All the examples can take smaller input size by setting command line arguments.
\nSoftware dependencies\nPAM requires g++ 5.4.0 or later versions supporting the Cilk Plus extensions. The scripts that we provide in the repository use \"numactl\" for better performance. All tests can also run directly without \"numactl\".
\nWe use python to write a script to organize all results and compute speedup. It is not required to run tests.
\nDatasets\nWe use the publicly available Wikipedia database (dumped on Oct. 1, 2016) for the inverted index experiment. We release a sample (1% of total size) in the github repository (35MB compressed). The full data (3.5TB compressed) is available on request. All other applications use randomly generated data.
\nExperiment Workflow:\nAt the top level there is a makefile (make) and a script for compiling and running all timings (./run_all). The source code\nof the library is provided in the directory c++/, and the other directories\neach corresponds to some examples of applications. There are\nfour example applications provided in our repository:
\nIn each of the directories there is a separated makefile and a script to run the timings for the corresponding application.
\nAll tests include parallel and sequential running times. The sequential versions are the algorithms running directly on one thread, and the parallel versions use all threads on the machine using \"numactl -i all\".
\nTo run all tests, just type the following at the top level:
\nmake\n./run_all.sh\n
\nBy default the script will not include the tests on very large input sizes (10 billion), which costs long. Users can use
\nmake\n./run_all.sh -l\n
\nto include large tests.
\nTo run separated tests on each application, users can also go to each sub-directory to run the scripts.
\nWe recommend to use numactl -i all on all parallel tests.
\nAugmented Sum (/aug_sum/)\nUsing
\nmake\n
\nwill give two executable files aug_sum (augmented version) and aug_sumNA (non-augmented version). This is done by setting flag -DNO_AUG in compiling time.
\nUsing
\n./run_aug_sum.sh -l\n
\nwill run all experiments as shown in Table 3 in [1] on augmented sum. If you do not want to run on large input, remove \"-l\".
\nUsing
\n./runall [-r rounds] [-p threads]\n
\nwill run all functions as shown in Table 3 in [1] with 'rounds' rounds and 'threads' threads. By default rounds=3 and threads=nproc --all
(maximum number of threads on the machine).
Both scripts will output to both stdout and a file res.txt. The script run_aug_sum.sh will then call a python code to give all results (timings and speedups) in a file data.txt.
\nIf user wants to directly run our executable file (aug_sum or aug_sumNA), the arguments are listed as follows:
\n./aug_sum [-n size1] [-m size2] [-r rounds] [-p] <testid>\n
\nInterval Tree (/interval/)\nUsing
\nmake\n
\nwill give the executable file (interval).
\nUsing
\nrun_interval\n
\nwill give the same experiment of interval trees as shown in Table 5 in [1].
\nTo directly run the executable file (interval), one can try:
\n./interval n q r\n
\nwhere n stands for the number of intervals, q is the number of querys, r is the number of rounds. By default n=100000000, q=n, r=5.
\nRange Tree (/range_query/)\nUsing
\nmake\n
\nwill give the executable file (rt_test).
\nUsing
\n./run_range\n
\nwill give the same experiment of range trees as shown in Table 5 in [1].
\nTo directly run the executable file (rt_test), one can try:
\n./rt_test [-n size] [-l rmin] [-h rmax] [-r rounds] [-q queries] [-w window] [-t query_type]\n
\nwhere 'size' stands for the number of points, 'rmin' and 'rmax' are the upper and lower bound of coordinates, 'rounds' is the number of rounds, 'queries' is the number of queries, 'window' is the query window size (for one dimension), 'query_type' is 0 for query-all, and 1 for query-ssum. By default n=100000000, l=0, h=1000000000, r=3, q=1000, w=1000000, t=0.
\nInverted Index\nUsing
\nmake\n
\nwill give the executable file (index).
\nUsing
\n./run_index\n
\nwill give the same type of experiment of inverted index as shown in Table 6 in [1], but on a smaller input size.
\nTo directly run the executable file (index), one can try:
\n./index [-o] [-v] [-n max_chars] [-q num_queries] [-f file]\n
\nwhere '-o' indicates an output file of query results, '-v' means to output verbose information, '-n' means the length to read from a file, '-q' is the number of queries, and '-f' is the input file. By default n=1000000000000 (just read the whole file), q=10000, f='wiki_small.txt'.
\nReference\n[1] Yihan Sun, Daniel Ferizovic, and Guy E. Blelloch. PAM: Parallel Augmented Maps. PPoPP 2018.
", "publication_date": "2018-02-07", "publisher": "Zenodo", "related_identifiers": [ { "identifier": "https://github.com/syhlalala/PAM-AE/tree/v1.0", "relation_type": { "id": "issupplementto", "title": { "de": "Erg\u00e4nzt", "en": "Is supplement to" } }, "scheme": "url" } ], "resource_type": { "id": "software", "title": { "de": "Software", "en": "Software" } }, "rights": [ { "description": { "en": "" }, "id": "other-open", "title": { "en": "Other (Open)" } } ], "title": "syhlalala/PAM-AE: PAM", "version": "v1.0" }, "parent": { "access": { "owned_by": { "user": 41869 } }, "communities": {}, "id": "1168702", "pids": { "doi": { "client": "datacite", "identifier": "10.5281/zenodo.1168702", "provider": "datacite" } } }, "pids": { "doi": { "client": "datacite", "identifier": "10.5281/zenodo.1168703", "provider": "datacite" }, "oai": { "identifier": "oai:zenodo.org:1168703", "provider": "oai" } }, "revision_id": 6, "stats": { "all_versions": { "data_volume": 391635684.0, "downloads": 11, "unique_downloads": 11, "unique_views": 403, "views": 423 }, "this_version": { "data_volume": 391635684.0, "downloads": 11, "unique_downloads": 11, "unique_views": 402, "views": 422 } }, "status": "published", "updated": "2020-01-25T07:25:02.610997+00:00", "versions": { "index": 1, "is_latest": true } }