Stock Trading Data Retrieval and Storage v. 1.

02

1. This project will be developed by the Developer in Python, in keeping with its best
practices.
2. The IP of this project, except the parts covered by Odoo’s licence, or the licence of
any other software components used to build this project, will belong to us.
3. This module will need a postgreSQL extension Citus Data or timescaleDB, which
allows postgreSQL to store Time Series Data.
4. This module will use a separate schema in the postgreSQL database.
5. Each source will have a identifier as without it duplication will be there.
6. It will work as follows:
6.1. Terraform will be used to instantiate a VPS with a private IP, inacceible
from regular Internet in a VPC on Amazon, GCP or Digital Ocean, assesible only
through a bastion host using openvpn.
6.2. postgreSQl will be installed using Terraform script only (no ssh
configuration) in a Docker Container, on the VPS in private IP space in the VPC.
6.3. Apache Kafka will be installed as a docker container on second VPS in
the VPC.
6.4. PostgreSQL & Kafka will be connected using dibizium installed in a
docker container, in any of two VPS.
6.5. Kafka will accept the streaming data you send from scraping.
6.6. Terraform will instantiate a third VPS in Private IP space in the VPC. This
is where your scripts will run via NAT through the bastion host.
6.7. You will scrape required information from various sources using proxies to
avoid throttling as we will be scraping data on multiple tickers at the same time.
The scraped information will be fed to postgresql via Kafka.
7. Scraping Options:
7.1. It will use a combination of scrapy, scrapyd and rarely, in case of very
large scrapes scrapinghub.com OR
7.1.1. It will use various pre-existing foss APIs posted on Github to fetch
Data from SEBI & various Indian exchanges like NSE, BSE, MCX, NCEDEX
etc. and Yahoo Finance & Google Finance etc.
7.2. Commercial Proxies will be identified & advised to us for subscription and
used for avoiding blocking & throttling issues.
8. Job is to use pre-existing APIs to fetch Data – EOD quotes, Intraday prices,
Snapshot Data about trading that these exchanges display on their sites,
Fundamental Stock Data & Historical price data and store everything in Odoo
database.
9. You must not think on writing API to do the job from scratch. You must use existing
API.
10. You must check out existing API’s on Github & other similar sources and tell us you
can do the job. What parts you can do with existing API & what API will you use
and what part has no API so will need coding from scratch.
11. Based on discussions, we will amend this document & enlarge the scope of this
project to include creation of fresh API, IF NEEDED.
12. This document is yet to be updated with exact data be scraped, as the above
description has been found to be inadequate after discussion with bidders. We will
be updating this document throughout the next 15 days, in parts.