LitScan
LitScan is a text mining pipeline that connects RNA sequences with the latest open access scientific literature. LitScan uses a collection of identifiers (Ids), gene names, and synonyms provided to RNAcentral by the Expert Databases to scan the papers available in Europe PMC, and is updated with every RNAcentral release.
The papers are listed in the Publications tab of each sequence page, where they can be filtered by identifier, article type, the section of the paper where the Id is found, mentioned organism, journal, and year.
For example, lncRNA THRIL is also known as Linc1992. Using LitScan,
the corresponding RNAcentral entry
includes papers about THRIL, Linc1992, and even NR_110375 which is another Id for the same
gene:
Browse all sequences with publications →
How LitScan works
With every release, LitScan:
Collects the Ids (identifiers, gene names and synonyms) of RNAcentral sequences from the Expert Databases. Ids that are numbers, common English words such as
hairpin,nail,digitoreric, or too short to be specific are left out to prevent false matches.Searches Europe PMC for open access articles that mention each Id. New Ids are searched across all of Europe PMC, and Ids that were searched before are checked for articles published since the last search. The query is:
"<id>" AND ("rna" OR "mrna" OR "ncrna" OR "lncrna" OR "rrna" OR "sncrna") AND IN_EPMC:Y AND OPEN_ACCESS:Y AND NOT SRC:PPRwhere the RNA terms filter out possible false positives,
IN_EPMC:YandOPEN_ACCESS:Yrequire the full text to be openly available in Europe PMC, andNOT SRC:PPRexcludes preprints, which are not peer-reviewed.Confirms the exact Id in the full text of each article. Europe PMC’s search treats spaces and special characters as separators, so it can return articles that only contain part of an Id. LitScan uses regular expressions to find the exact Id in the title, abstract, or body of the article, and extracts the sentences that mention it. Articles where the exact Id is not found are not shown.
Annotates the articles. Retracted articles are removed, articles that Expert Databases have curated for an RNA are marked, and the organisms mentioned in each article are identified using the ORGANISMS project (paper), so they may not be entirely accurate.
Example
A search on EuropePMC
for the microRNA precursor dme-bantam ID yielded
13 results as of October 2023. However, the second step finds the exact string dme-bantam only
in 4 articles, while the other 9 mention
dme-bantam-3p and/or
dme-bantam-5p and appear on the corresponding
mature microRNA pages.
Use LitScan in your website
The LitScan widget is implemented as an embeddable component that can be used by any Expert Database or any other website. LitScan has already been deployed on the Rfam website (for example, see the SAM riboswitch page).
Find out more about how to integrate this widget into your website.
Finding the Ids used by LitScan
The list of Ids searched for each Expert Database can be accessed programmatically, for example with the following Python script. EBI Search returns at most 100,000 results for a query, so for the largest databases (such as Ensembl, RefSeq and miRBase) the script retrieves only the first 100,000 entries.
"""
Usage: python get_ids.py [database]
Example: python get_ids.py rfam
"""
import itertools
import json
import math
import requests # pip install requests
import sys
# EBI Search returns at most the first 100,000 results of a query
MAX_RESULTS = 100000
def get_query(database, start):
"""
Function to build the query
:param database: name of the Expert DB
:param start: index of the first entry to fetch
:return: query
"""
query = "?query=entry_type:metadata%20AND%20database:" \
+ database \
+ "&fields=job_id&size=100&start=" \
+ str(start) \
+ "&format=json"
return query
def get_results(database):
"""
Function to get the list of ids from a given database
:param database: name of the Expert DB
:return: results
"""
start = 0
endpoint = "https://www.ebi.ac.uk/ebisearch/ws/rest/rnacentral-litscan"
query = get_query(database, start)
db_request = requests.get(endpoint + query)
hit_count = json.loads(db_request.text)["hitCount"]
results = []
if hit_count > 0:
# get ids
get_result = json.loads(db_request.text)
entries = [get_result["entries"]]
results.append([item["fields"]["job_id"][0] for sublist in entries for item in sublist])
if hit_count > MAX_RESULTS:
print("{} has {} entries; only the first {} can be retrieved".format(
database, hit_count, MAX_RESULTS))
# fetch other ids if needed
iter_number = int(math.ceil(min(hit_count, MAX_RESULTS) / 100.0))
for num in range(iter_number - 1):
start += 100
query = get_query(database, start)
new_request = requests.get(endpoint + query)
new_request_result = json.loads(new_request.text)
new_entries = [new_request_result["entries"]]
results.append([item["fields"]["job_id"][0] for sublist in new_entries for item in sublist])
return results
def main():
database = None
if len(sys.argv) == 1:
print("You must specify the database")
exit()
elif len(sys.argv) == 2:
database = sys.argv[1]
else:
print("Usage: python get_ids.py rfam")
exit()
if database:
# get list of ids
results = get_results(database)
if results:
results = list(itertools.chain.from_iterable(results))
results = sorted(set(results))
# save results
with open(database + '_ids.txt', 'w') as f:
for item in results:
f.write(item + '\n')
else:
print("No id found for database {}".format(database))
if __name__ == "__main__":
main()

