LitScan

LitScan is a text mining pipeline that connects RNA sequences with the latest open access scientific literature. LitScan uses a collection of identifiers (Ids), gene names, and synonyms provided to RNAcentral by the Expert Databases to scan the papers available in Europe PMC, and is updated with every RNAcentral release.

The papers are listed in the Publications tab of each sequence page, where they can be filtered by identifier, article type, the section of the paper where the Id is found, mentioned organism, journal, and year.

For example, lncRNA THRIL is also known as Linc1992. Using LitScan, the corresponding RNAcentral entry includes papers about THRIL, Linc1992, and even NR_110375 which is another Id for the same gene:

LitScan results for THRIL

Browse all sequences with publications →

How LitScan works

With every release, LitScan:

  1. Collects the Ids (identifiers, gene names and synonyms) of RNAcentral sequences from the Expert Databases. Ids that are numbers, common English words such as hairpin, nail, digit or eric, or too short to be specific are left out to prevent false matches.

  2. Searches Europe PMC for open access articles that mention each Id. New Ids are searched across all of Europe PMC, and Ids that were searched before are checked for articles published since the last search. The query is:

    "<id>" AND ("rna" OR "mrna" OR "ncrna" OR "lncrna" OR "rrna" OR "sncrna") AND IN_EPMC:Y AND OPEN_ACCESS:Y AND NOT SRC:PPR
    

    where the RNA terms filter out possible false positives, IN_EPMC:Y and OPEN_ACCESS:Y require the full text to be openly available in Europe PMC, and NOT SRC:PPR excludes preprints, which are not peer-reviewed.

  3. Confirms the exact Id in the full text of each article. Europe PMC’s search treats spaces and special characters as separators, so it can return articles that only contain part of an Id. LitScan uses regular expressions to find the exact Id in the title, abstract, or body of the article, and extracts the sentences that mention it. Articles where the exact Id is not found are not shown.

  4. Annotates the articles. Retracted articles are removed, articles that Expert Databases have curated for an RNA are marked, and the organisms mentioned in each article are identified using the ORGANISMS project (paper), so they may not be entirely accurate.

Example

A search on EuropePMC for the microRNA precursor dme-bantam ID yielded 13 results as of October 2023. However, the second step finds the exact string dme-bantam only in 4 articles, while the other 9 mention dme-bantam-3p and/or dme-bantam-5p and appear on the corresponding mature microRNA pages.

LitScan results for dme-bantam

Use LitScan in your website

The LitScan widget is implemented as an embeddable component that can be used by any Expert Database or any other website. LitScan has already been deployed on the Rfam website (for example, see the SAM riboswitch page).

Find out more about how to integrate this widget into your website.

Finding the Ids used by LitScan

The list of Ids searched for each Expert Database can be accessed programmatically, for example with the following Python script. EBI Search returns at most 100,000 results for a query, so for the largest databases (such as Ensembl, RefSeq and miRBase) the script retrieves only the first 100,000 entries.

"""
Usage: python get_ids.py [database]
Example: python get_ids.py rfam
"""

import itertools
import json
import math
import requests  # pip install requests
import sys

# EBI Search returns at most the first 100,000 results of a query
MAX_RESULTS = 100000


def get_query(database, start):
    """
    Function to build the query
    :param database: name of the Expert DB
    :param start: index of the first entry to fetch
    :return: query
    """
    query = "?query=entry_type:metadata%20AND%20database:" \
            + database \
            + "&fields=job_id&size=100&start=" \
            + str(start) \
            + "&format=json"
    return query


def get_results(database):
    """
    Function to get the list of ids from a given database
    :param database: name of the Expert DB
    :return: results
    """
    start = 0
    endpoint = "https://www.ebi.ac.uk/ebisearch/ws/rest/rnacentral-litscan"
    query = get_query(database, start)
    db_request = requests.get(endpoint + query)
    hit_count = json.loads(db_request.text)["hitCount"]
    results = []

    if hit_count > 0:
        # get ids
        get_result = json.loads(db_request.text)
        entries = [get_result["entries"]]
        results.append([item["fields"]["job_id"][0] for sublist in entries for item in sublist])

        if hit_count > MAX_RESULTS:
            print("{} has {} entries; only the first {} can be retrieved".format(
                database, hit_count, MAX_RESULTS))

        # fetch other ids if needed
        iter_number = int(math.ceil(min(hit_count, MAX_RESULTS) / 100.0))
        for num in range(iter_number - 1):
            start += 100
            query = get_query(database, start)
            new_request = requests.get(endpoint + query)
            new_request_result = json.loads(new_request.text)
            new_entries = [new_request_result["entries"]]
            results.append([item["fields"]["job_id"][0] for sublist in new_entries for item in sublist])

        return results


def main():
    database = None

    if len(sys.argv) == 1:
        print("You must specify the database")
        exit()
    elif len(sys.argv) == 2:
        database = sys.argv[1]
    else:
        print("Usage: python get_ids.py rfam")
        exit()

    if database:
        # get list of ids
        results = get_results(database)

        if results:
            results = list(itertools.chain.from_iterable(results))
            results = sorted(set(results))

            # save results
            with open(database + '_ids.txt', 'w') as f:
                for item in results:
                    f.write(item + '\n')
        else:
            print("No id found for database {}".format(database))


if __name__ == "__main__":
    main()