# LitScan **LitScan** is a text mining pipeline that connects RNA sequences with the latest open access scientific literature. LitScan uses a collection of identifiers (Ids), gene names, and synonyms provided to RNAcentral by the [Expert Databases](expert-databases.md) to scan the papers available in [Europe PMC](https://europepmc.org), and is updated with every RNAcentral release. The papers are listed in the **Publications** tab of each sequence page, where they can be filtered by identifier, article type, the section of the paper where the Id is found, mentioned organism, journal, and year. For example, lncRNA `THRIL` is also known as `Linc1992`. Using LitScan, [the corresponding RNAcentral entry](https://rnacentral.org/rna/URS000075D66B/9606?tab=pub) includes papers about `THRIL`, `Linc1992`, and even `NR_110375` which is another Id for the same gene: [![LitScan results for THRIL](_static/img/litscan-thril.png)](https://rnacentral.org/rna/URS000075D66B/9606?tab=pub) [Browse all sequences with publications →](https://rnacentral.org/search?q=has_lit_scan:%22True%22) ## How LitScan works With every release, LitScan: 1. **Collects the Ids** (identifiers, gene names and synonyms) of RNAcentral sequences from the Expert Databases. Ids that are numbers, common English words such as `hairpin`, `nail`, `digit` or `eric`, or too short to be specific are left out to prevent false matches. 2. **Searches Europe PMC** for open access articles that mention each Id. New Ids are searched across all of Europe PMC, and Ids that were searched before are checked for articles published since the last search. The query is: ```text "" AND ("rna" OR "mrna" OR "ncrna" OR "lncrna" OR "rrna" OR "sncrna") AND IN_EPMC:Y AND OPEN_ACCESS:Y AND NOT SRC:PPR ``` where the RNA terms filter out possible false positives, `IN_EPMC:Y` and `OPEN_ACCESS:Y` require the full text to be openly available in Europe PMC, and `NOT SRC:PPR` excludes preprints, which are not peer-reviewed. 3. **Confirms the exact Id** in the full text of each article. Europe PMC's search treats spaces and special characters as separators, so it can return articles that only contain part of an Id. LitScan uses [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) to find the exact Id in the title, abstract, or body of the article, and extracts the sentences that mention it. Articles where the exact Id is not found are not shown. 4. **Annotates the articles.** Retracted articles are removed, articles that Expert Databases have curated for an RNA are marked, and the organisms mentioned in each article are identified using the [ORGANISMS project](https://organisms.jensenlab.org) ([paper](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0065390)), so they may not be entirely accurate. ### Example A [search on EuropePMC](https://europepmc.org/search?query=%22dme-bantam%22%20AND%20%28%22rna%22%20OR%20%22mrna%22%20OR%20%22ncrna%22%20OR%20%22lncrna%22%20OR%20%22rrna%22%20OR%20%22sncrna%22%29%20AND%20IN_EPMC%3AY%20AND%20OPEN_ACCESS%3AY%20AND%20NOT%20SRC%3APPR) for the microRNA precursor [dme-bantam](https://rnacentral.org/rna/URS00002F21DA/7227) ID yielded 13 results as of October 2023. However, the second step finds the exact string `dme-bantam` **only in 4 articles**, while the other 9 mention [dme-bantam-3p](https://rnacentral.org/rna/URS00004E9E38/7227) and/or [dme-bantam-5p](https://rnacentral.org/rna/URS000055786A/7227) and appear on the corresponding mature microRNA pages. [![LitScan results for dme-bantam](_static/img/litscan-dme-bantam.png)](https://rnacentral.org/rna/URS00002F21DA/7227?tab=pub) ## Use LitScan in your website The LitScan widget is implemented as an **embeddable component** that can be used by any [Expert Database](expert-databases.md) or any other website. LitScan has already been deployed on the Rfam website (for example, see the [SAM riboswitch](https://rfam.org/family/RF00162#tabview=tab10) page). Find out more about [how to integrate this widget into your website](https://github.com/RNAcentral/rnacentral-litscan). ## Finding the Ids used by LitScan The list of Ids searched for each Expert Database can be accessed programmatically, for example with the following Python script. EBI Search returns at most 100,000 results for a query, so for the largest databases (such as Ensembl, RefSeq and miRBase) the script retrieves only the first 100,000 entries. ```python """ Usage: python get_ids.py [database] Example: python get_ids.py rfam """ import itertools import json import math import requests # pip install requests import sys # EBI Search returns at most the first 100,000 results of a query MAX_RESULTS = 100000 def get_query(database, start): """ Function to build the query :param database: name of the Expert DB :param start: index of the first entry to fetch :return: query """ query = "?query=entry_type:metadata%20AND%20database:" \ + database \ + "&fields=job_id&size=100&start=" \ + str(start) \ + "&format=json" return query def get_results(database): """ Function to get the list of ids from a given database :param database: name of the Expert DB :return: results """ start = 0 endpoint = "https://www.ebi.ac.uk/ebisearch/ws/rest/rnacentral-litscan" query = get_query(database, start) db_request = requests.get(endpoint + query) hit_count = json.loads(db_request.text)["hitCount"] results = [] if hit_count > 0: # get ids get_result = json.loads(db_request.text) entries = [get_result["entries"]] results.append([item["fields"]["job_id"][0] for sublist in entries for item in sublist]) if hit_count > MAX_RESULTS: print("{} has {} entries; only the first {} can be retrieved".format( database, hit_count, MAX_RESULTS)) # fetch other ids if needed iter_number = int(math.ceil(min(hit_count, MAX_RESULTS) / 100.0)) for num in range(iter_number - 1): start += 100 query = get_query(database, start) new_request = requests.get(endpoint + query) new_request_result = json.loads(new_request.text) new_entries = [new_request_result["entries"]] results.append([item["fields"]["job_id"][0] for sublist in new_entries for item in sublist]) return results def main(): database = None if len(sys.argv) == 1: print("You must specify the database") exit() elif len(sys.argv) == 2: database = sys.argv[1] else: print("Usage: python get_ids.py rfam") exit() if database: # get list of ids results = get_results(database) if results: results = list(itertools.chain.from_iterable(results)) results = sorted(set(results)) # save results with open(database + '_ids.txt', 'w') as f: for item in results: f.write(item + '\n') else: print("No id found for database {}".format(database)) if __name__ == "__main__": main() ```