Sequence search API

The RNAcentral sequence similarity search API allows you to perform most of the same actions through your own tools as you can through the web interface. Any tool capable of making HTTP requests can communicate with this API, for example curl.

Base URL: https://sequence-search.rnacentral.org/

The web interface accepts up to 50 sequences per submission, so use the API for larger batches.

How it works

The API is asynchronous:

  1. Submit a job with POST /api/submit-job, passing the sequence (10 to 7,000 nucleotides) and an optional list of expert databases to search, such as ["mirbase"]. Leave the list empty to search all of RNAcentral. The response contains a job_id.

  2. Poll GET /api/job-status/<job_id> until the status is finished (or error or not_found). The response also includes a progress percentage.

  3. Retrieve results from GET /api/job-results/<job_id>, which is paginated with page and page_size parameters (100 per page by default) and reports the total hit_count. Each entry in entries includes the rnacentral_id, description, e_value, identity, query_coverage and alignment. Unlike the web interface, which shows the top 1,000 hits, the API returns them all.

API throttling

Job submissions (/api/submit-job) are rate limited to 10 requests per minute per IP address. There is no rate limit on job status or results endpoints.

When the service is busy, a submission can also be refused with HTTP status 429 and a message asking you to wait. Wait a minute and submit again; the example script below does this automatically.

Example script

An example of a Python script to perform searches for each record of a FASTA file is provided below. It requires Python 3.5 or later and the biopython and requests packages (pip install biopython requests).

You can use this script to search for sequences in the RNAcentral database or in one of the Expert Databases. The results will be organized in a new directory. A file in JSON format containing the metadata and the search results will be created. This file will be named with the description of the sequence. Here is an example:

results
|-- URS00001F7A66_9031 Gallus gallus (chicken) gga-let-7g-5p.json
|-- ...
"""
Usage: python search.py [file] [database]

Examples:
python search.py file.fasta               # search in all databases
python search.py file.fasta mirbase       # search in miRBase only
"""
import json
import requests
import sys
import time
from Bio import SeqIO
from pathlib import Path

SERVER = "https://sequence-search.rnacentral.org/"


def get_sequence_search_result(description, sequence, job_id):
    """Poll job status then retrieve all paginated sequence search results."""
    job_status = "pending"
    while True:
        try:
            data = requests.get(SERVER + "api/job-status/" + job_id, timeout=30).json()
            job_status = data["status"]
            progress = data.get("progress", 0)
        except requests.exceptions.RequestException as e:
            print("Error checking job status, retrying: {}".format(e))
            time.sleep(10)
            continue

        if job_status == "finished":
            break
        elif job_status in ("error", "not_found"):
            print("Search failed for record: {}".format(description))
            return {
                "job_id": job_id,
                "status": job_status,
                "description": description,
                "sequence": str(sequence),
            }
        else:
            print("  {:.0f}% complete...".format(progress))
            time.sleep(10)

    # Fetch results page by page
    results = []
    page = 1
    page_size = 100
    hit_count = None
    while True:
        try:
            url = "{}api/job-results/{}?page={}&page_size={}".format(
                SERVER, job_id, page, page_size
            )
            data = requests.get(url, timeout=30).json()
        except requests.exceptions.RequestException as e:
            print("Error fetching results page {} for {}: {}".format(page, description, e))
            break

        if hit_count is None:
            hit_count = data.get("hit_count", 0)
        entries = data.get("entries", [])
        results.extend(entries)
        if not entries or len(results) >= hit_count:
            break
        page += 1

    return {
        "job_id": job_id,
        "hits": hit_count or 0,
        "status": job_status,
        "description": description,
        "sequence": str(sequence),
        "results": results,
    }


def main():
    if len(sys.argv) == 1:
        print("You must specify the FASTA file")
        exit()
    elif len(sys.argv) == 2:
        filename = sys.argv[1]
        database = []
    elif len(sys.argv) == 3:
        filename = sys.argv[1]
        database = [sys.argv[2]]
    else:
        print("Usage: python search.py file.fasta [database]")
        exit()

    Path("results").mkdir(parents=True, exist_ok=True)

    with open(filename, mode="r") as handle:
        for record in SeqIO.parse(handle, "fasta"):
            description = record.description
            sequence = record.seq
            data = {"databases": database, "sequence": str(sequence)}
            post_job = requests.post(SERVER + "api/submit-job", json=data)
            while post_job.status_code == 429:
                print("Rate limited, waiting a minute before resubmitting...")
                time.sleep(60)
                post_job = requests.post(SERVER + "api/submit-job", json=data)

            job_id = None
            if post_job.status_code == 200:
                job_id = post_job.json()["job_id"]
            else:
                print("Failed to submit job for record:\n{}\n{}".format(description, sequence))

            if job_id:
                result = get_sequence_search_result(description, sequence, job_id)
                if result:
                    hits = result.get("hits", 0)
                    if hits == 0:
                        print("No results found for {}".format(description))
                    else:
                        print("Saving {} hit(s) for {}".format(hits, description))
                        with open("results/" + description + ".json", "w") as f:
                            json.dump(result, f)


if __name__ == "__main__":
    main()

Results retention

Results are kept for up to 7 days, after which all data and metadata are deleted. They can be removed sooner when the service is under heavy load, so save the results once a job finishes.