2020-07-24 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 198338 CORD UIDs (new: 211, removed: 5) Full text: PDF - 91043 json (new: 2501, removed: 129) PMC - 64491 json (new: 302, removed: 2) --------------------------- 2020-07-23 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 198129 CORD UIDs (new: 718, removed: 9) Full text: PDF - 88670 json (new: 1404, removed: 28) PMC - 64191 json (new: 1) --------------------------- 2020-07-22 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 197412 CORD UIDs (new: 783, removed: 4) Full text: PDF - 87293 json (new: 835, removed: 397) PMC - 64190 json (new: 567) --------------------------- 2020-07-21 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 196630 CORD UIDs (new: 904, removed: 3) Full text: PDF - 86853 json (new: 854, removed: 1636) PMC - 63623 json (new: 229) --------------------------- 2020-07-20 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 195711 CORD UIDs (new: 3175, removed: 17) Full text: PDF - 87634 json (new: 3296, removed: 89) PMC - 63394 json (new: 658) --------------------------- 2020-07-16 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 192509 CORD UIDs (new: 1176, removed: 17) Full text: PDF - 84426 json (new: 804, removed: 1829) PMC - 62736 json (new: 770) --------------------------- 2020-07-15 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 191355 CORD UIDs (new: 717, removed: 6) Full text: PDF - 85451 json (new: 1428, removed: 1491) PMC - 61966 json (new: 341, removed: 1) --------------------------- 2020-07-14 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 190647 CORD UIDs (new: 1256, removed: 7) Full text: PDF - 85513 json (new: 2190, removed: 320) PMC - 61626 json (new: 541) --------------------------- 2020-07-13 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 189384 CORD UIDs (new: 1457, removed: 17) Full text: PDF - 83641 json (new: 831, removed: 75) PMC - 61085 json (new: 363) --------------------------- 2020-07-12 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 187938 CORD UIDs (new: 647, removed: 1) Full text: PDF - 82884 json (new: 317, removed: 3604) PMC - 60722 json (new: 2) --------------------------- 2020-07-11 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 187291 CORD UIDs (new: 359, removed: 4) Full text: PDF - 86171 json (new: 2986, removed: 313) PMC - 60720 json (new: 7, removed: 2) --------------------------- 2020-07-10 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 186929 CORD UIDs (new: 850, removed: 6) Full text: PDF - 83496 json (new: 366, removed: 2446) PMC - 60715 json (new: 468, removed: 2) --------------------------- 2020-07-09 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 186084 CORD UIDs (new: 1153, removed: 4) Full text: PDF - 85576 json (new: 3244, removed: 143) PMC - 60249 json (new: 265, removed: 2) --------------------------- 2020-07-08 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 184929 CORD UIDs (new: 1758, removed: 2) Full text: PDF - 82474 json (new: 1749, removed: 109) PMC - 59986 json (new: 272, removed: 2) --------------------------- 2020-07-07 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 183141 CORD UIDs (new: 869, removed: 54) Full text: PDF - 80833 json (new: 481, removed: 3085) PMC - 59716 json (new: 767) --------------------------- 2020-07-06 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 182328 CORD UIDs (new: 546, removed: 3) Full text: PDF - 83436 json (new: 2988, removed: 304) PMC - 58949 json (new: 1, removed: 2) --------------------------- 2020-07-05 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 181778 CORD UIDs (new: 319, removed: 2) Full text: PDF - 80750 json (new: 838, removed: 526) PMC - 58950 json (new: 0) --------------------------- 2020-07-04 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 181456 CORD UIDs (new: 1294, removed: 33) Full text: PDF - 80438 json (new: 729, removed: 789) PMC - 58950 json (new: 232, removed: 1) --------------------------- 2020-07-03 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 180153 CORD UIDs (new: 9671, removed: 49) Full text: PDF - 80497 json (new: 2295, removed: 547) PMC - 58719 json (new: 770, removed: 5) --------------------------- 2020-07-01 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 169821 CORD UIDs (new: 1157, removed: 13) Full text: PDF - 78747 json (new: 715, removed: 1424) PMC - 57954 json (new: 687) --------------------------- 2020-06-30 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 168679 CORD UIDs (new: 874, removed: 12) Full text: PDF - 79455 json (new: 1569, removed: 140) PMC - 57267 json (new: 2, removed: 1) --------------------------- 2020-06-29 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 167822 CORD UIDs (new: 335, removed: 3) Full text: PDF - 78025 json (new: 241, removed: 1355) PMC - 57266 json (new: 229) --------------------------- 2020-06-28 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 167470 CORD UIDs (new: 3603, removed: 7) Full text: PDF - 79138 json (new: 1638, removed: 208) PMC - 57037 json (new: 0) --------------------------- 2020-06-27 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 163835 CORD UIDs (new: 292, removed: 1) Full text: PDF - 77707 json (new: 242, removed: 3168) PMC - 57037 json (new: 15) --------------------------- 2020-06-26 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 163544 CORD UIDs (new: 2011, removed: 29) Full text: PDF - 80632 json (new: 3122, removed: 55) PMC - 57022 json (new: 556, removed: 3) --------------------------- 2020-06-25 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 161564 CORD UIDs (new: 1486, removed: 7) Full text: PDF - 77564 json (new: 1957, removed: 1067) PMC - 56469 json (new: 308, removed: 1) --------------------------- 2020-06-24 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 160082 CORD UIDs (new: 838, removed: 1110) Full text: PDF - 76674 json (new: 607, removed: 2595) PMC - 56162 json (new: 498, removed: 1) --------------------------- 2020-06-23 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 160356 CORD UIDs (new: 1413, removed: 7) Full text: PDF - 78660 json (new: 3670, removed: 319) PMC - 55665 json (new: 965, removed: 1) --------------------------- 2020-06-22 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 158947 CORD UIDs (new: 103, removed: 3) Full text: PDF - 75308 json (new: 113, removed: 2171) PMC - 54701 json (new: 1) --------------------------- 2020-06-21 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 158847 CORD UIDs (new: 128, removed: 2) Full text: PDF - 77366 json (new: 1030, removed: 594) PMC - 54700 json (new: 0, removed: 1) --------------------------- 2020-06-20 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 158721 CORD UIDs (new: 453, removed: 8) Full text: PDF - 76928 json (new: 978, removed: 659) PMC - 54701 json (new: 571, removed: 7) --------------------------- 2020-06-19 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 158274 CORD UIDs (new: 573, removed: 10) Full text: PDF - 76608 json (new: 834, removed: 18) PMC - 54137 json (new: 1, removed: 1) --------------------------- 2020-06-18 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 157712 CORD UIDs (new: 323, removed: 6) Full text: PDF - 75791 json (new: 1718, removed: 883) PMC - 54137 json (new: 404, removed: 1) --------------------------- 2020-06-17 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 157394 CORD UIDs (new: 11122, removed: 10) Full text: PDF - 74955 json (new: 2648, removed: 305) PMC - 53734 json (new: 286) --------------------------- 2020-06-16 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 146525 CORD UIDs (new: 3596, removed: 19) Full text: PDF - 72612 json (new: 690, removed: 763) PMC - 53448 json (new: 358) --------------------------- 2020-06-15 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 142905 CORD UIDs (new: 471, removed: 3) Full text: PDF - 72683 json (new: 961, removed: 263) PMC - 53090 json (new: 470) --------------------------- 2020-06-14 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 142436 CORD UIDs (new: 859, removed: 1) Full text: PDF - 71984 json (new: 500, removed: 1218) PMC - 52620 json (new: 1) --------------------------- 2020-06-13 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 141575 CORD UIDs (new: 791, removed: 7) Full text: PDF - 72702 json (new: 1669, removed: 996) PMC - 52619 json (new: 6, removed: 1) --------------------------- 2020-06-12 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 140780 CORD UIDs (new: 280, removed: 1) Full text: PDF - 72027 json (new: 258, removed: 1943) PMC - 52614 json (new: 270, removed: 1) --------------------------- 2020-06-11 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 140499 CORD UIDs (new: 706, removed: 7) Full text: PDF - 73711 json (new: 3706, removed: 425) PMC - 52345 json (new: 452, removed: 4) --------------------------- 2020-06-10 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 139792 CORD UIDs (new: 1006, removed: 8) Full text: PDF - 70430 json (new: 1599, removed: 823) PMC - 51897 json (new: 1079) --------------------------- 2020-06-09 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 138794 CORD UIDs (new: 1388, removed: 9) Full text: PDF - 69652 json (new: 2817, removed: 324) PMC - 50818 json (new: 17) --------------------------- 2020-06-08 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 137411 CORD UIDs (new: 825, removed: 2) Full text: PDF - 67157 json (new: 120, removed: 1050) PMC - 50801 json (new: 794) --------------------------- 2020-06-07 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 136583 CORD UIDs (new: 434, removed: 0) Full text: PDF - 68087 json (new: 664, removed: 730) PMC - 50007 json (new: 5) --------------------------- 2020-06-06 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 136136 CORD UIDs (new: 60, removed: 7) Full text: PDF - 68152 json (new: 681, removed: 1658) PMC - 50002 json (new: 191, removed: 5) --------------------------- 2020-06-05 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 136082 CORD UIDs (new: 281, removed: 2) Full text: PDF - 69128 json (new: 753, removed: 537) PMC - 49816 json (new: 0) --------------------------- 2020-06-04 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 135803 CORD UIDs (new: 647, removed: 8) Full text: PDF - 68910 json (new: 3878, removed: 556) PMC - 49816 json (new: 1130) --------------------------- 2020-06-03 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 135149 CORD UIDs (new: 1364, removed: 6895) Full text: PDF - 65587 json (new: 560, removed: 221) PMC - 48686 json (new: 0, removed: 1) --------------------------- 2020-06-02 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 141109 CORD UIDs (new: 559, removed: 5) Full text: PDF - 65339 json (new: 473, removed: 1014) PMC - 48687 json (new: 536) --------------------------- 2020-06-01 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 140532 CORD UIDs (new: 601, removed: 36) Full text: PDF - 65893 json (new: 975, removed: 311) PMC - 48151 json (new: 488) --------------------------- 2020-05-31 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 139952 CORD UIDs (new: 939, removed: 16) Full text: PDF - 65222 json (new: 473, removed: 2716) PMC - 47663 json (new: 1, removed: 62) --------------------------- 2020-05-30 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 139061 CORD UIDs (new: 265, removed: 4) Full text: PDF - 67509 json (new: 2876, removed: 364) PMC - 47724 json (new: 563) --------------------------- 2020-05-29 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 138807 CORD UIDs (new: 1447, removed: 11) Full text: PDF - 64938 json (new: 822, removed: 1056) PMC - 47161 json (new: 0) --------------------------- 2020-05-28 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 137326 CORD UIDs (new: 1247, removed: 24) Full text: PDF - 65223 json (new: 2794, removed: 264) PMC - 47161 json (new: 278) --------------------------- 2020-05-27 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 136067 CORD UIDs (new: 1811, removed: 36) Full text: PDF - 62646 json (new: 2657, removed: 601) PMC - 46883 json (new: 3141, removed: 11) --------------------------- 2020-05-26 ---CHANGES--- No major changes. ---SUMMARY--- total metadata rows: 134206 CORD UIDs (new: 5317, removed: 130) Full text: PDF - 60562 json (new: 1710, removed: 781) PMC - 43753 json (new: 0) --------------------------- 2020-05-19 ---CHANGES--- Added Medline paper source Changed metadata column `url` to a ;-joined list Added Semantic Scholar paper identifier where available to metadata column `s2_id` The s2_id can be resolved to the Semantic Scholar paper page using the API at https://api.semanticscholar.org/ ---SUMMARY--- total metadata rows: 128492 CORD UIDs (new: 71901, removed: 7266) Full text: PDF - 59566 json (new: 15396, removed: 7704) PMC - 43753 json (new: 10259, removed: 9) --------------------------- 2020-05-12 ---CHANGES--- NOTE: THERE ARE SIGNIFICANT BREAKING CHANGES THIS WEEK. PLEASE READ CAREFULLY. * moving to one single full text release file (document_parses.tar.gz) that combines the json parses from all *.tar.gz files from previous releases * consolidated metadata columns `has_pdf_parse` and `full_text_file` into column `pdf_json_files`, which now stores the relative paths to the corresponding *.json files. note: this column is a ';' separated list of paths (some rows have multiple associated pdf parses). e.g. document_parses/pdf_json/e3d0d482ebd9a8ba81c254cc433f314142e72174.json * consolidated metadata columns `has_pmc_xml_parse`, `full_text_file` and pmcid lookup into column `pmc_json_files`, which now stores the relative paths to the corresponding *.xml.json files. e.g. document_parses/pmc_json/PMC125375.xml.json * renamed metadata columns: "Microsoft Academic Paper ID" -> "mag_id" "WHO #Covidence" -> "who_covidence_id" * added HTML table parses for many files; these are in the PDF json parses where available, under key "ref_entries", type "table", key "html". this rollout is incremental, and more tables will be added as they become available. SEE FAQ FOR DETAILED INFORMATION: https://discourse.cord-19.semanticscholar.org/t/faqs-about-cord-19-dataset/94 ---SUMMARY--- total metadata rows: 63571 CORD UIDs (new: 3691, removed: 15) Full text: PDF - 51882 json (new: 4950, removed: 1492) PMC - 33503 json (new: 13989, removed: 3337) --------------------------- 2020-05-01 ---CHANGES--- * added arXiv as a source of papers * added column arxiv_id in metadata.csv file to store arxiv identifiers ---SUMMARY--- total metadata rows: 59887 59851 CORD UIDs (new: 2522, removed: 4) arxiv: PDF - 787 full text (new: 787) biorxiv_medrxiv: PDF - 2670 full text (new: 543, removed: 151) comm_use_subset: PDF - 9918 full text (new: 183, removed: 34) PMC - 9540 full text (new: 171, removed: 21) custom_license: PDF - 32450 full text (new: 1295, removed: 221) PMC - 11000 full text (new: 430, removed: 45) noncomm_use_subset: PDF - 2584 full text (new: 79, removed: 13) PMC - 2311 full text (new: 61, removed: 8) --------------------------- 2020-04-24 ---CHANGES--- * no changes ---SUMMARY--- total metadata rows: 57366 57332 CORD UIDs (new: 4988, removed: 22) biorxiv_medrxiv: PDF - 2278 full text (new: 492, removed: 148) comm_use_subset: PDF - 9769 full text (new: 253, removed: 41) PMC - 9390 full text (new: 233, removed: 32) custom_license: PDF - 31376 full text (new: 4355, removed: 199) PMC - 10615 full text (new: 2630, removed: 33) noncomm_use_subset: PDF - 2518 full text (new: 68, removed: 16) PMC - 2258 full text (new: 56, removed: 10) --------------------------- 2020-04-17 ---CHANGES--- * no changes ---SUMMARY--- total metadata rows: 52398 52365 CORD UIDs (new: 1320, removed: 1) biorxiv_medrxiv: PDF - 1934 full text (new: 426, removed: 117) comm_use_subset: PDF - 9557 full text (new: 104, removed: 71) PMC - 9189 full text (new: 97, removed: 56) custom_license: PDF - 27220 full text (new: 882, removed: 167) PMC - 8018 full text (new: 224, removed: 8) noncomm_use_subset: PDF - 2466 full text (new: 23, removed: 47) PMC - 2212 full text (new: 22, removed: 27) --------------------------- 2020-04-10 ---CHANGES--- * minor change to metadata deduplication logic ---SUMMARY--- total metadata rows: 51078 51045 CORD UIDs (new: 3786, removed: 38) biorxiv_medrxiv: PDF - 1625 full text (new: 353, removed: 70) comm_use_subset: PDF - 9524 full text (new: 174, removed: 15) PMC - 9148 full text (new: 174, removed: 21) custom_license: PDF - 26505 full text (new: 3647, removed: 294) PMC - 7802 full text (new: 3081, removed: 52) noncomm_use_subset: PDF - 2490 full text (new: 130, removed: 17) PMC - 2217 full text (new: 128, removed: 4) --------------------------- 2020-04-03 ---CHANGES--- * added PMC XML parses; available under metadata file column "has_pmc_xml_parse" * renamed "has_full_text" metadata column to "has_pdf_parse" * restructured tar.gz outputs, example new directory structure: |-- comm_use_subset |-- pmc_json (contains all JSON derived from PMC XML parses; filenames are PMC_ID.xml.json) |-- pdf_json (contains all JSON derived from PDF parses) NOTE: some papers have both PMC XML and PDF parses ---SUMMARY--- total metadata rows: 47298 47296 CORD UIDs (new: 1524, removed: 2) biorxiv_medrxiv: PDF - 1342 full text (new: 348, removed: 59) comm_use_subset: PDF - 9365 full text (new: 70, removed: 20) PMC - 8995 full text (new: 8995, removed: 0) custom_license: PDF - 23152 full text (new: 2541, removed: 46) PMC - 4773 full text (new: 4773, removed: 0) noncomm_use_subset: PDF - 2377 full text (new: 54, removed: 27) PMC - 2093 full text (new: 2093, removed: 0) --------------------------- 2020-03-27 ---CHANGES--- * added column "cord_uid" to metadata file (this persistent identifier is a 8-char alphanumeric string unique to each entry) * added column "url" to metadata file (URL to paper landing page where available, usually DOI/PMC/Pubmed URL) * normalized publication date, DOIs * adjusted deduplication logic slightly (affects 3 papers) * removed some entries that correspond to indices or table of contents ---SUMMARY--- total metadata rows: 45774 biorxiv_medrxiv: 1053 full text (new: 194, removed: 26) comm_use_subset: 9315 full text (new: 210, removed: 13) custom_license: 20657 full text (new: 4218, removed: 520) noncomm_use_subset: 2350 full text (new: 6, removed: 9) --------------------------- 2020-03-20 ---CHANGES--- * normalized doi, authors, title, abstract strings * merged redundant rows in metadata file on doi, pmcid, and pubmed_id * metadata sha column can now include multiple files (some PMC files have multiple associated PDFs) * added column in metadata file "full_text_file" to signal the tar.gz file in which the full text json resides ---SUMMARY--- total metadata rows: 44220 biorxiv_medrxiv: 885 full text (new: 110) comm_use_subset: 9118 full text (new: 128) custom_license: 16959 full text (new: 15533) noncomm_use_subset: 2353 full text (new: 385)