This directory is the unified automation entry point for the four measurement modules, allowing the full workflow to be executed with a single command.

• Updatechk
• Uniqueness
• Vulnerability
• Maliciousness

The workflow is currently orchestrated by run_pipeline.py, and each module writes its results to its own res directory.

• run_pipeline.py: main entry point responsible for argument parsing, module scheduling, and manifest generation
• pipeline_config.py: default input paths and output directory layout
• Updatechk/res
• Uniqueness/link_dup/res
• Vulnerability/res
• Maliciousness/res
• manifests: pipeline manifest files generated for each run

manifests/ stores a traceable snapshot of each pipeline run. File names use the following format:

• manifests/<run_id>_pipeline_manifest.json

Each manifest includes:

• input arguments for the run (dataset_root, metadata_yaml, workspace_root)
• a snapshot of the resolved paths for the four modules and output directories
• the status of each module (completed / skipped)
• key output paths for each module
• execution script lists and runtime information for some modules

Use cases:

• reproducing experiments
• investigating why a module produced no output
• comparing differences between runs

From the Automated-process-ase directory:

python run_pipeline.py --run-id ase_full_20260326 --clean

• --dataset-root: root directory of the repositories to be analyzed
• --metadata-yaml: marketplace metadata file
• --workspace-root: root output directory
• --run-id: run ID for the current execution, used to name the manifest
• --max-projects: process only the first N projects for quick testing
• --clean: clear each module's res directory before running
• --verbose: print full logs from child scripts
• --skip-updatechk
• --skip-uniqueness
• --skip-vulnerability
• --skip-maliciousness
• --uniqueness-run-featured: run the uniqueness featured collection script first

Default order:

1. Updatechk
2. Uniqueness
3. Vulnerability
4. Maliciousness

Any module can be skipped with the corresponding --skip-* flag.

When the pipeline finishes, it prints:

• pipeline_manifest
• the root res directory for each module

For example:

• Updatechk/res/api_check_res.csv
• Uniqueness/link_dup/res/summary_duplicate_links.csv
• Vulnerability/res/validated_res.csv
• Maliciousness/res/tool_poisoning_res.csv

• The default logging mode hides child-script stdout to reduce terminal noise, while keeping stderr output such as progress bars.