NTIS–WoS Matched Papers

This document accompanies the Zenodo release that links National Science & Technology Information Service (NTIS)-funded publications to Web of Science (WoS) metadata. It summarises the source data, cleansing pipeline, matching logic, key variables, and considerations for public sharing.

Dataset Overview

• Purpose: expose a reproducible mapping between NTIS paper records and WoS identifiers so downstream users can validate or extend the linkage.
• Public payload: the Zenodo table contains only six columns — NTIS project ID, DOI_NTIS, DOI_NTIS_cleaned, validity_flag, DOI_WoS, uid_WoS. Everything else (titles, bibliometrics, provenance flags) remains internal.
• Matching keys: cleansing is still performed on the NTIS DOI; when a DOI cannot be recovered, an internal title_vol comparison is used to source the WoS DOI/uid before the public table is trimmed down.

Source Material

Only the fields required for linking (year, title, DOI, SCI flag, NTIS project ID) are pulled into the working Polars DataFrame.

Processing Summary

1. DOI cleansing (Polars)

   Simple replacements (looped sequentially):

   • '99999999999'→''

   • ' '→''

   • '\\\\n'→''

   • '?'→''
   • '링크'→''

   • 'ㄴ'→''

   • ','→'.'

   • ']'→'',
   • '%28'→'('

   • '%29'→')'

   
   Regex replacements (Rust-style backrefs):

   • .*?(10\\\\.\\\\d{4,9}/[-._;()/:a-z0-9]+) → $1
   • (<https://dx>\\\\.doi\\\\.org/|http://dx\\\\.doi\\\\.org/|dx\\\\.doi\\\\.org/) → ``
   • ^0\\\\. → 10.
   • (10\\\\.\\\\d{4,9})\\\\. → $1/
   • (10\\\\.\\\\d{4,9})([a-z0-9]+) → $1/$2
   • %2f → /
   • ^(\\\\d{2})(\\\\d{4}/) → $1.$2 
   • (^/|^:) → ``

   Validation flag:

   • doi_pattern = r'(?i)^10.\\\\d{4,9}/[-._;()/:a-z0-9]+$'

2. WoS title normalisation

   HTML-like markup is stripped so titles align with NTIS records:

   • pattern = re.compile(r"<[^>]*>")

3. Matching workflow

   1. Primary DOI join — DOI_NTIS_cleaned (NTIS) ↔ DOI_WoS (WoS).
   2. Fallback exact-title join — for rows still missing DOI, match TITLE+VOL_NTIS ↔ TITLE+VOL_WoS.

Table Description

All other columns (titles, title_vol, bibliometrics, internal match strategy) are retained only in the private diagnostic parquet.

Validation & Coverage Checks

• cnt_valid_doi = df_pap_sub.group_by('is_valid_doi').len() — proportion of records with syntactically valid DOIs.
• df_merged.drop_nulls(subset='uid').height / df_merged.height — overall linkage rate after applying DOI-first then title_vol.
• df_wos.filter(pl.col('wos_title_clean').is_in(...)) — WoS slice that now maps to NTIS, supporting manual audits.

Limitations & Caveats

• Hangul keyboard recovery rules from earlier experiments are disabled; enable them if new NTIS dumps show IME artefacts.
• Title fallback relies on exact equality. A string-grouper prototype exists later in the notebook but is not part of the published pipeline.
• When multiple DOIs appear in a single cell, only the first DOI is retained (design choice).

Sharing Considerations

• The published six-column table carries only project/publication identifiers and DOI variants; no personal data or institutional review details are exposed.
• Keep the richer diagnostic parquet internal (it includes titles, bibliometrics, and intermediate flags).
• Consider adding a short note in the Zenodo description that non-essential NTIS/WoS metadata has been withheld intentionally.

This documentation should be packaged with the Zenodo upload so users understand the provenance of the matched dataset and the meaning of the key fields.

Acknowledgement

This dataset is a research output produced under KISTI’s 2025 Basic Project (Project No.: K25L4M2C3).