PyHelpers: an open-source toolkit for facilitating Python users' data manipulation tasks

PyHelpers is a lightweight open-source Python package designed as a versatile toolkit for simplifying data (pre)processing tasks. It offers a comprehensive range of practical utilities to facilitate common data manipulation operations. These utilities encompass the ability to read and write file-like objects, efficiently handle various data types such as geographical and textual data, and establish streamlined communication with relational databases like PostgreSQL and Microsoft SQL Server.

 

This release:

1.5.2 (13 September 2023)

      Notable changes since 1.5.1:

• Turned the module dbms into a sub-package.

      For more information and detailed specifications, check out PyHelpers 1.5.2 documentation.

 

Release history:

1.5.1 (17 June 2023)

      Notable changes since 1.5.0:

• Renamed arguments for the functions of saving/loading data within the module store.
• Modified the following functions:
  • save_spreadsheet() in the module store;
  • find_similar_str() in the module text.

      For more information and detailed specifications, check out PyHelpers 1.5.1 documentation.

1.5.0 (8 June 2023)

      Notable changes since 1.4.6:

• Improved the structure of the package and a few functions and class methods.
• Added the following new functions/classes:
  • path2linux(), validate_filename(), get_file_pathnames(), and check_files_exist() to the module dir;
  • GitHubFileDownloader() and func_running_time() to the module ops.

      For more information and detailed specifications, check out PyHelpers 1.5.0 documentation.

1.4.6 (26 February 2023)

      Notable changes since 1.4.5:

• Improved the module dbms (for migrating to SQLAlchemy 2.0).
• Improved the following functions (with bug fixes):
  • cmap_discretisation() in the module ops;
  • find_closest_points() in the module geom;
  • save_web_page_as_pdf() in the module store.

      For more information and detailed specifications, check out PyHelpers 1.4.6 documentation.

1.4.5 (9 January 2023)

      Notable changes since 1.4.4:

• Improved the classes PostgreSQL and MSSQL in the module dbms (for migrating to SQLAlchemy 2.0).

      For more information and detailed specifications, check out PyHelpers 1.4.5 documentation.

1.4.4 (3 January 2023)

      Notable changes since 1.4.3:

• Improved a few functions in the module geom (as per warnings by Shapely 2.0), including drop_axis(), project_point_to_line(), get_coordinates_as_array(), find_closest_point() and get_rectangle_centroid().

      For more information and detailed specifications, check out PyHelpers 1.4.4 documentation.

1.4.3 (11 November 2022)

      Notable changes since 1.4.2:

• Improved the following functions (with bug fixes):
  • is_dir() in the module dirs;
  • find_closest_point(), find_closest_points(), project_point_to_line(), transform_geom_point_type() and find_shortest_path() in the module geom;
  • load_user_agent_strings(), cmap_discretisation() and colour_bar_index() in the module ops;
  • pd_preferences() in the module settings;
  • save_spreadsheets() and xlsx_to_csv() in the module store;
  • find_similar_str() and remove_punctuation() in the module text.
• Added the following new functions:
  • get_coordinates_as_array() to the module geom;
  • get_relative_path() to the module ops.

      For more information and detailed specifications, check out PyHelpers 1.4.3 documentation.

1.4.2 (15 August 2022)

      Notable changes since 1.4.1:

• Renamed the parameter method to engine for the following functions:
  • find_similar_str() in the module text;
  • save_json(), load_json(), markdown_to_rst() and xlsx_to_csv() in the module store.
• Improved the functions save_data(), load_csv(), load_data() in the module store.

      For more information and detailed specifications, check out PyHelpers 1.4.2 documentation.

1.4.1 (17 July 2022)

      Notable changes since 1.4.0:

• Improved the following functions (with bug fixes):
  • update_dict() in the module ops;
  • gdal_configurations() and mpl_preferences() in the module settings.

      For more information and detailed specifications, check out PyHelpers 1.4.1 documentation.

1.4.0 (13 June 2022)

      Notable changes since 1.3.4:

• Renamed the module dir to dirs (to avoid a possible redefinition of, or any potential conflict with, the Python built-in function dir()).

      For more information and detailed specifications, check out PyHelpers 1.4.0 documentation.

1.3.4 (13 June 2022)

      Notable changes since 1.3.3:

• Enabled direct access to all modules through importing pyhelpers, and all functions/classes in each module through importing pyhelpers..
• Improved the following function/class (with bug fixes):
  • find_similar_str() in the module text;
  • PostgreSQL in the module dbms.

      For more information and detailed specifications, check out PyHelpers 1.3.4 documentation.

1.3.3 (12 May 2022)

      Notable changes since 1.3.2:

• Improved the following class/functions with bug fixes:
  • PostgreSQL in the module dbms;
  • save_data() and save_svg_as_emf() in the module store;
  • delete_dir() in the module dir;
  • euclidean_distance_between_texts() and cosine_similarity_between_texts() in the module text.
• Added a new class MSSQL to the module dbms.

      For more information and detailed specifications, check out PyHelpers 1.3.3 documentation.

1.3.2 (14 March 2022)

      Notable changes since 1.3.1:

• Added the following new functions:
  • xlsx_to_csv() to the module store;
  • np_shift() to the module ops;
  • project_point_to_line() and find_shortest_path() to the module geom.

      For more information and detailed specifications, check out PyHelpers 1.3.2 documentation.

1.3.1 (10 February 2022)

      Notable changes since 1.3.0:

• Changed LICENSE to GPLv3+.
• Renamed the following functions:
  • save() to save_data() in the module store;
  • get_user_agent_strings() to load_user_agent_strings() in the module ops.
• Improved the functions:
  • remove_punctuation(), find_similar_str() and convert_md_to_rst() in the module text;
  • merge_dicts() in the module ops.
• Added a few constants cached as the package/module is imported.
• Added the following new functions/methods:
  • load_csv() and load_data() to the module store;
  • numeral_english_to_arabic() to the module text;
  • PostgreSQL.list_schema_names() to the module dbms;
  • find_executable() and compare_dicts() to the module ops.

      For more information and detailed specifications, check out PyHelpers 1.3.1 documentation.

1.3.0 (6 January 2022)

      Notable changes since 1.2.18:

• Renamed the module sql to dbms.
• Renamed the following functions:
  • in the module dir:
    • validate_input_data_dir() to validate_dir();
  • in the module ops:
    • update_nested_dict() to update_dict();
    • get_all_values_from_nested_dict() to get_dict_values();
    • remove_multiple_keys_from_dict() to remove_dict_keys();
    • instantiate_requests_session() to init_requests_session().
• Improved the following functions/class:
  • PostgreSQL in the module dbms (previously sql);
  • wgs84_to_osgb36() and osgb36_to_wgs84() in the module geom.
  • parse_size() (newly added), init_requests_session() and download_file_from_url() in the module ops;
  • load_json() in the module store;
  • find_similar_str() in the module text;
• Added the following functions to the module ops: parse_size(), get_number_of_chunks(), loop_in_pairs(), is_url() and update_dict_keys().
• Reduced the number of essential dependencies.

      For more information and detailed specifications, check out PyHelpers 1.3.0 documentation.

1.2.18 (20 October 2021)

      Notable changes since 1.2.17:

• Improved the module ops by:
  • fixing the issue in fake_requests_headers() that may occasionally raise IndexError;
  • adding a new function get_fake_user_agent().

      For more information and detailed specifications, check out PyHelpers 1.2.18 documentation.

1.2.17 (1 October 2021)

      Notable changes since 1.2.16:

• Improved the function fake_requests_headers() to avoid raising IndexError.

      For more information and detailed specifications, check out PyHelpers 1.2.17 documentation.

1.2.16 (20 September 2021)

      Notable changes since 1.2.15:

• Renamed the function is_dirname() to is_dir() in the module dir.
• Improved the following functions (with bug fixes):
  • cd() in the module dir;
  • find_closest_points() in the module geom;
  • confirmed(), swap_cols(), swap_rows(), fake_requests_headers() and download_file_from_url() in the module ops;
  • all the functions in the module settings;
  • save_multiple_spreadsheets(), save_json() and load_json() in the module store;
  • find_similar_str() in the module text.
• Added the following new functions:
  • go_from_altered_cwd() to the module dir;
  • drop_axis() to the module geom;
  • instantiate_requests_session() to the module ops;
  • save_joblib() and load_joblib() to the module store.

      For more information and detailed specifications, check out PyHelpers 1.2.16 documentation.

1.2.15 (19 April 2021)

      Notable changes since 1.2.14:

• Improved the following functions/methods with bug fixes:
  • find_similar_str() and convert_md_to_rst() in the module text;
  • download_file_from_url() in the module ops;
  • save_svg_as_emf() in the module store;
  • PostgreSQL.import_data() and PostgreSQL.drop_schema() in the module sql.
• Added a new function is_downloadable() to the module ops.

      For more information and detailed specifications, check out PyHelpers 1.2.15 documentation.

1.2.14 (27 March 2021)

      Notable changes since 1.2.13:

• Rectified an error in the specification of dependencies for package installation.

      For more information and detailed specifications, check out PyHelpers 1.2.14 documentation.

1.2.13 (26 March 2021)

      Notable changes since 1.2.12:

• Improved the module sql with a bug fix for errors raised when importing SQLAlchemy 1.4.0+.

      For more information and detailed specifications, check out PyHelpers 1.2.13 documentation.

1.2.12 (22 March 2021)

      Notable changes since 1.2.11:

• Resolved an issue that failed importing modules (after updating or reinstalling the package).
• Added a new function gps_to_utc() to the module ops.

      For more information and detailed specifications, check out PyHelpers 1.2.12 documentation.

1.2.11 (3 March 2021)

      Notable changes since 1.2.10:

• Fixed a bug in the method PostgreSQL.import_data() in the module sql.

      For more information and detailed specifications, check out PyHelpers 1.2.11 documentation.

1.2.10 (1 February 2021)

      Notable changes since 1.2.9:

• Improved the following functions/method with bug fixes to:
  • np_preferences() in the module settings;
  • save_multiple_spreadsheets() in the module store;
  • PostgreSQL.read_sql_query() in the module sql.
• Added new functions/methods:
  • eval_dtype(), swap_cols() and swap_rows() to the module ops;
  • get_rectangle_centroid() to the module geom;
  • .list_table_names() and .alter_table_schema() to the class PostgreSQL in the module sql.

      For more information and detailed specifications, check out PyHelpers 1.2.10 documentation.

1.2.9 (5 January 2021)

      Note that the release 1.2.8 had been permanently deleted.

      Notable changes since 1.2.7:

• Made modifications with bug fixes to the following functions/class:
  • validate_input_data_dir() in the module dir;
  • confirmed() and fake_requests_headers() in the module ops;
  • PostgreSQL in the module sql.
• Added new functions:
  • get_obj_attr() to the module ops;
  • get_db_address() to the module sql.

1.2.7 (17 November 2020)

      Notable changes since 1.2.6:

• Fixed a minor bug in the method PostgreSQL.import_data() in the module sql
• Added new functions:
  • merge_dicts(), is_network_connected() and is_url_connectable() to the module ops;
  • get_acronym() and extract_words1upper() to the module text.

1.2.6 (6 November 2020)

      Notable changes since 1.2.5:

• Renamed two functions in the module geom:
  • find_closest_point_from() to find_closest_point();
  • find_closest_points_between() to find_closest_points().
• Reduced the number of essential dependencies for installing the package (see also the installation note).
• Enabled direct access to all functions/classes from importing the package without having to specifying the modules they reside in.
• Fixed a minor bug in the function fake_requests_headers() in the module ops.

1.2.5 (12 October 2020)

      Notable changes since 1.2.4:

• Renamed the function rm_dir() to delete_dir() in the module dir.
• Moved the function save_web_page_as_pdf() from the module text to store.
• Removed two functions from the module ops: get_variable_name() and get_variable_names().
• Improved the following function/class:
  • save() in the module store;
  • PostgreSQL in the module sql.

1.2.4 (7 September 2020)

      Notable changes since 1.2.3:

• Improved the function fake_requests_headers() in the module ops.

1.2.3 (7 September 2020)

      Notable changes since 1.2.2:

• Fixed a few bugs in the following functions/class:
  • download_file_from_url() in the module ops;
  • get_specific_filepath_info() in the module store;
  • PostgreSQL in the module sql.
• Added a new function fake_requests_headers() to the module ops.

1.2.2 (18 July 2020)

      Notable changes since 1.2.1:

• Improved the following functions:
  • get_specific_filepath_info() in the module store;
  • split_list() and get_all_values_from_nested_dict() in the module ops.

1.2.1 (1 July 2020)

      Note that the release 1.2.0 had been permanently deleted.

      Notable changes since 1.1.2:

• Integrated the module download into ops and made the following changes:
  • renamed the function download() to download_file_from_url();
  • added a new function convert_md_to_rst();
  • improved the following functions: split_list_by_size(), split_list(), get_extreme_outlier_bounds(),
    interquartile_range(), find_closest_date() and detect_nan_for_str_column().
• Moved the function save_web_page_as_pdf() from the module store to text.
• Improved the following functions/class:
  • get_specific_filepath_info(), save_fig(), save_svg_as_emf(), seven_zip() and load_csr_matrix() in the module store;
  • PostgreSQL in the module sql.

1.1.2 (30 May 2020)

      Note that the release 1.1.1 had been permanently deleted.

      Notable changes since 1.1.0:

• Renamed the following function/method:
  • save_excel() to save_spreadsheet() in the module store;
  • .disconnect() to .disconnect_database() of the class PostgreSQL in the module sql.
• Removed the function csr_matrix_to_dict() from the module text.
• Removed redundant parameters from functions in the module store.
• Improved the following functions by adding optional parameters:
  • cd(), cdd(), cd_data and rm_dir() in the module dir;
  • download() in the module download;
  • save_svg_as_emf() and save_web_page_as_pdf() in the module store.
• Added new functions:
  • to the module geom: sketch_square();
  • to the module ops: create_rotation_matrix() and dict_to_dataframe();
  • to the module store: save_multiple_spreadsheets(), load_multiple_spreadsheets(),
    unzip(), seven_zip() and load_csr_matrix();
  • to the module text: remove_punctuation(), count_words(), calculate_idf(), calculate_tf_idf(),
    euclidean_distance_between_texts() and cosine_similarity_between_texts().

1.1.0 (22 April 2020)

      Notable changes since 1.0.26:

• Made a few modifications to the class sql.PostgreSQL to speed up importing data into, and retrieving data from, a PostgreSQL database.
• Fixed a minor bug in the function text.find_matched_str().
• Improved the two modules store and text.

1.0.26 (12 March 2020)

      Notable changes since 1.0.25:

• Fixed a few bugs in the module store.

1.0.25 (6 March 2020)

      Notable changes since 1.0.24:

• Improved the functions, osgb36_to_wgs84() and wgs84_to_osgb36(), in the module geom.

1.0.24 (28 January 2020)

      Note that the release 1.0.23 had been permanently deleted.

      Notable changes since 1.0.22:

• Fixed a bug in the module sql.

1.0.22 (27 January 2020)

      Notable changes since 1.0.21:

• Renamed the function divide_list_into_chunks() to split_list_by_size() in the module ops.
• Added two new functions to the module ops: split_list() and split_iterable().
• Improved the following function/class:
  • gdal_configurations() in the module settings;
  • PostgreSQL in the module sql.

1.0.21 (20 January 2020)

      Notable changes since 1.0.20:

• Added a new module sql.
• Added a new function detect_nan_for_str_column() to the module ops.

1.0.20 (7 January 2020)

      Notable changes since 1.0.19:

• Fixed a few minor bugs in the module store.

1.0.19 (28 November 2019)

      Note that the release 1.0.18 had been permanently deleted.

      Notable changes since 1.0.17:

• Renamed the module misc to ops; moved the function csr_matrix_to_dict() to the module text.
• Fixed minor bugs:
  • save_feather() and load_feather() in the module store;
  • download() in the module download;
  • mpl_preferences() in the module settings.
• Added new functions:
  • show_square(), locate_square_vertices() and locate_square_vertices_calc() to the module geom;
  • is_dirname() to the module dir.
• Improved the module store with modifications to the functions: save_pickle(), load_pickle(), save_json(), load_json() and save().

1.0.17 (10 September 2019)

      Notable changes since 1.0.16:

• Improved the function rm_dir() in the module dir.
• Tidied up the code of the three modules: misc, geom and store.

1.0.16 (3 September 2019)

      This is a release of a brand-new version.

      Note that the initial releases (of early versions up to 1.0.15) had been permanently deleted.