Sage-Bionetworks/schematic: v23.9.2

What's Changed

• implement GE list rule by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/655
• Develop initial rule pairs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/802
• Cross Manifest Validation: Value Level Comparisons and Reworked Warning Messages by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/751
• Develop schema viz tool tests by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/659
• make sorting case insensitive by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/830
• Develop fix submit keyerror by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/831
• updating the release documentation by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/817
• Develop an endpoint to return data types of columns in a manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/815
• Develop rule combination by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/766
• BugFix: Import dataType from CLI as list not string by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/836
• BugFix: Manifest Import - Mask null cells as empty strings instead of NaNs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/837
• BugFix: list like behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/840
• Check type of 'url' entry when raising errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/844
• Fix example data model list validation rule and update "list" in ManifestGenerator to handle both list strict and list like by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/847
• add warning to validate_manifest endpoint by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/859
• Develop NA string behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/842
• Develop blank annotation behavior by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/862
• Develop Missing Entity Handling by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/871
• rename manifest based on manifest_basename when submit to synapse by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/869
• Output errors generated from jsonSchema (when submitting manifest) to the console by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/873
• Develop manifest submission error handle by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/879
• do not create dir if located in cwd by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/881
• Update annotations unit test by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/880
• Develop optional rule args by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/843
• Containerizing schematic (and update dependencies as needed) by @BrunoGrandePhD in https://github.com/Sage-Bionetworks/schematic/pull/882
• add endpoint find_class_specific_properties by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/890
• add route get_subgraph_by_edge_type by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/891
• create endpoint to return schema as a pickle file by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/892
• poetry documentation by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/895
• Add support for Schematic API to Dockerfile by @BrunoGrandePhD in https://github.com/Sage-Bionetworks/schematic/pull/887
• Create project table manifests, move entities to new project by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/829
• add concurrency to github action by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/896
• HOTFIX: Mixed type column manifest table upload by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/902
• update asset view table endpoint to allow returning a json by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/912
• added get_node_range endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/904
• added get_property_label endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/905
• added node_dependencies endpoint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/906
• Allow schema.org schema to be loaded and used by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/903
• add example workbook to demo cli functions by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/916
• change manifest name for censored manifest when upload by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/931
• Add Restriction Flag to Make Synapse Table Method by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/935
• Develop Table Uploads: Replace by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/917
• allow users to upload csv or json file when submitting manifest using API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/908
• allow users to upload csv or json file when validating manifest using API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/910
• Develop table schema fix replacements by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/947
• Fixing parsing of table schema parameters on table replace by @milen-sage in https://github.com/Sage-Bionetworks/schematic/pull/945
• Develop schema viz tool cors by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/942
• Move Schema Visualization to Schematic by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/645
• add default max size for columns missing this key by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/952
• populate manifest as an excel spreadsheet without sending data to Google api by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/940
• Skip Trashed Entities and Entities that "do not exist" by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/985
• Reword conditional requirements for the attribute table by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/987
• add tests related to manifest get by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/979
• fix api endpoint for docker container by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/974
• Remove unused libraries by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/991
• Remove Manifest Table Submission Dependency on Project Scope when not Validating by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/989
• HOTFIX: Extend table replacement sleep duration to ensure synapse deletion operation completes by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/999
• Pass config.yml as an environment variable to docker container by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/975
• update error messages in synapseStore when manifest_basename key is missing in config.yml by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/997
• Added instructions for installing and using schematic and the schematic API docker containers to README by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/953
• Develop pdoc by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/957
• correct typo: master_basename to manifest_basename in error message by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1005
• Fixed typo in conditional requirement for schema viz attribute table by adding quotes by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1003
• Added tests for all API endpoints by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/995
• Fixed an issue where the DCA would disconnect on manifest submission by casting UUIDs to type string by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1006
• Reminded users to run synapse login --rememberMe after schematic init step in ReadMe by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1004
• Added the option to populate manifest as an excel spreadsheet to avoid sending metadata to Google APIs by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/994
• For schematic APIs, return excel file instead of excel file path by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/988
• Emphasized that users should download the config.yml from develop branch in ReadMe by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1007
• Removed validity requirement for unrequired attribute entries, allow users to specify validation rule conformity required for entries, fix DCA disconnect caused by JSON Schema validation errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1000
• Strip hyphens from Synapse annotation keys by @milen-sage in https://github.com/Sage-Bionetworks/schematic/pull/1020
• Fixed a bug where in some instances manifests would be uploaded with different display and downloadAs names by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1017
• Added an API endpoint to check if a given node is required or not by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1024
• added more instructions of running CLI in jupyter notebook by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1031
• Added a slack bot to notify successful new releases by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1025
• Turned service_account_creds.json as an environment variable and default way of authentication by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1015
• Changed base docker image and removed package for security reasons by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1037
• Add support for treating date entries as type datetime by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1041
• Automatically updated CLI documentation on Readthedocs by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1047
• Deprecated token pickle and credentials.json by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1040
• Introduce tests to cover current table operations: creation and replacement by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1046
• Skip api tests when rule combination tests are run by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1068
• Added workflow to deploy schematic docker container in Github container registry by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1062
• Remove schematic support for Python v3.7 and v3.8 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1090
• Refactor table operations structure in asset store by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1069
• Added input_token as a parameter for /manifest/get endpoint to fix credential issues when getting an existing manifest on AWS by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1080
• Fixed getProjectManifests function in synapse storage by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1084
• Develop api node display names by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1094
• Create API endpoint for get_node_validation_rules by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1095
• Update schematic dependencies by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1092
• Raise errors for wrong schema errors by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1073
• Set default of "table_manipulation" as "replace" in API endpoint when users enter None and updated tests by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1115
• Update synapseClient dependency and api for manifest table uploads by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1101
• set pyopenssl = "^23.0.0" by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1125
• added date GE rule by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1103
• Implement table upsert feature by using schematic-db by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1081
• Add use_schema_label parameter to manifest submission endpoint, separate manifest submission and table upsert tests by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1129
• Delete GE checkpoint after completion of GE validation by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1136
• Remove try: catch: block from manifest submission command function by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1130
• Save all properties that are Included in the domain of a Class by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1134
• Display exceptions raised during validation with Great expectations, allow exclusion of upper bound OR lower bound for inRange rule by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1131
• Update Documentation - python/package versions and POCs by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1139
• Increase buffer size to a higher limit to deal with long token by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1144
• lock schematic-db to version 0.0.6 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1145
• use try: finally: to delete checkpoint even if running the checkpoint fails or errors out by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1155
• Allowed CORS on given routes instead of all routes by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1168
• Added restrict rules param to manifest/validate by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1178
• Bug Fix: remedy negation of table manipulation specification by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1186
• Added an endpoint to check entity type on Synapse and an endpoint to check if an entity is in the asset view by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1078
• add restrict rules control to manifest validate by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1189
• Added a parameter to control if GE gets used when using manifest/validate endpoint by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1177
• Propagate logger level entered in from command line to other schematic submodules by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1180
• Add timing of validation operations to DEBUG log by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1181
• Standardize validation error format and type by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1183
• Added function to calculate and clear cache by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1190
• Develop add file only manifest submission option by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1175
• Create example data model for single rule benchmarking by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1193
• Develop api tests for benchmarking single rule validation performance by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1184
• Fixed column headers problem when generating an Excel spreadsheet for getting an existing manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1164
• Do not index attribute visualization dataframe when converting to csv for component viz endpoint by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1196
• Remove restrictions on rule number and allowed combinations by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1203
• Start checking mypy and black by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1204
• Add IsNA rule to validation suite | FDS-81 FDS-232 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1200
• Add API endpoint to visualize attributes for a specific component by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1195
• Added functionality to download a manifest by using the manifest id by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1192
• Introduce workflow for API tests by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1208
• Remove requirement for .synapseConfig file to use upsert feature by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1207
• Parallelize operations in load_df by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1100
• Id Column - fix bug preventing table updates by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1214
• Id Column - fix bug where there would be two columns with uuid values by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1215
• Merge AWS deployment branch to develop by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1142
• Id Column - fix bug preventing upserts by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1216
• Downloaded manifests to temporary folders on AWS by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1210
• FDS-293 Fix column mismatch for Excel File Based Manifest Generation by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1213
• Feature fds 361 pylint by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1212
• Changed workflow to run poetry version 1.2.0 by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1222
• remove breakpoint from command by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1224
• Restrict typing-extensions package to versions before v4.6.0 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1228
• Make aws group dependencies optional and restrict typing_extensions package version by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1227
• Change Uuid column name to Id by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1211
• Avoided publishing minor releases to Docker Hub by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1231
• Remove lock on schematic_db version - FDS-287 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1232
• feat: added nginx to encrypt the communication between ALB and API by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1202
• Installed schematic in docker file by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1230
• fix: Fixed unit test in test_manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1233
• Fix formatting issues when pulling data from Synapse to Excel Manifest by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1234
• fix: fixed tests in test_api.py by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1225
• Added test manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1238
• [Bug fix]: Tried install pdoc again in github action workflow pdoc.yml by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1239
• Replaced synapse API calls with synapse Python client call by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1235
• Feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1241
• Added hide-blanks parameter when submitting manifest through API by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1242
• Revert "Feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment" by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1243
• feat: allowed production tag, staging tag, and manual run on existing tags to trigger building docker images for AWS deployment by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1244
• [Bug fix]: Fix regex in docker build workflow by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1246
• Revert "[Bug fix]: Fix regex in docker build workflow" by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1248
• [Bug fix]: Fix regex in docker build workflow by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1249
• Feature fds 273 coniguration by @andrewelamb in https://github.com/Sage-Bionetworks/schematic/pull/1219
• Expose strict_validation option for manifest/generate endpoint by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1253
• Optimize performance in table upsert backwards compatibility scenarios by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1229
• Fix df_utils/load_df so it more accurately captures integers to prevent Regex Errors. by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1240
• Use FAIR Data service desk for issues by @afwillia in https://github.com/Sage-Bionetworks/schematic/pull/1257
• Update Black to 2023 version: 23.7.0 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1262
• Update mypy to latest version by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1263
• Updated cli help text for parameter json_schema by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1264
• Modified Nginx parameters to fix submission issue by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1255
• modify tag by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1251
• BugFix: Fixed an issue where files on synapse were not being annotated correctly by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1254
• [bug fix] Fixed errors when calling get_empty_manifest by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1260
• [bug fix] Updated parameters being used when generating existing manifests in test_api.py by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1261
• Update schematic_db minimum version to v0.0.29 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1266
• Fix Google Sheet column info mismatch FDS-675 by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1265
• Only add validation rules to google sheets by @mialy-defelice in https://github.com/Sage-Bionetworks/schematic/pull/1252
• Make all entity annotations comply with synapse FDS-481 FDS-726 by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1267
• [bug fix] Updated docker file when building docker images for schematic AWS deployment by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1269
• update channel id to fair-data-team channel by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1272
• [bug fix] Updated manifest download endpoint to avoid querying file view by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1270
• [bug fix] Fixed typo in manifest generator by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1275
• Add API endpoint that returns the current version of schematic by @GiaJordan in https://github.com/Sage-Bionetworks/schematic/pull/1277
• [bug fix] Updated .synapseCache, functions to calculate cache, and cleared manifests before each download by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1268
• [bug fix] Renamed function related to setting background color of required columns by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1276
• [bug fix] Added includesType parameter when using walk function by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1281
• [bug fix] Do not pull annotations when use_annotations set to False and there is no existing manifests by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1278
• Add unit test for add_root_to_component function by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1282
• [bug fix] Skipped submission when running API tests remotely by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1280
• Refactored logic when pulling annotations by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1279
• [bug fix] Fix issues when generating an existing manifest as an excel sheet by @linglp in https://github.com/Sage-Bionetworks/schematic/pull/1284

• @afwillia made their first contribution in https://github.com/Sage-Bionetworks/schematic/pull/1257

Full Changelog: https://github.com/Sage-Bionetworks/schematic/compare/v22.07.1...v23.9.2