Write your own blockchain scraper

Add your own blockchain scraper
In order to add your own scraper for a blockchain supply source, you must adhere to our format. We use Go modules for our scrapers, so that each data provider is living as an independent module.
Directory structure
The working directory for blockchain scrapers is diadata/internal/pkg/blockchain-scrapers/blockchains. Each scrapper should comply with the following folder structure currencyName/scrappers/currencySymbol/currencySymbol.go for example for bitcoin will look like bitcoin/scrapers/btc/btc.go. You must provide a Dockerfile for your blockchain client and for the Go wrapper that connects to our database. An example dockerfile is provided for the Go bindings with a bitcoind client scraping the Bitcoin blockchain. The file should be named build/Dockerfile-symbol for this example build/Dockerfile-btc
1
FROM golang:1.14 as build
2
​
3
WORKDIR $GOPATH
4
​
5
COPY . .
6
​
7
WORKDIR $GOPATH/src/github.com/diadata-org/diadata/internal/pkg/blockchain-scrapers/blockchains/bitcoin/scrapers/btc
8
​
9
RUN go install
10
​
11
FROM gcr.io/distroless/base
12
​
13
COPY --from=build /go/bin/btc /bin/btc
14
​
15
CMD ["btc"]
Copied!
After cloning a container capable of executing Go, the files from the working directory are copied into the container. Next, the go program located in diadata/internal/pkg/blockchain-scrapers/blockchains/bitcoin/scrapers/btc is built and installed and the finished binary is placed into a mininal distroless container. From there it is executed using the statement in the last line.
Blockchain Client
The bitcoind itself is initialized directly from the file docker-compose.bitcoin.yml. Ideally, all blockchain clients are run from there directly
1
version: '3.2'
2
​
3
services:
4
  bitcoin:
5
    image:
6
      kylemanna/bitcoind
7
    user: "1000"
8
    volumes:
9
      - /home/srv/bitcoin:/bitcoin
10
    command: btc_oneshot -printtoconsole -prune=550 -rpcallowip=::/0 -disablewallet -rpcpassword=mysecretrpcdiapassword -rpcuser=mysecretrpcdiauser
11
    logging:
12
      options:
13
        max-size: "50m"
14
    networks:
15
      - scrapers-network
16
    deploy:
17
      mode: global
18
      restart_policy:
19
        delay: 2s
20
        window: 20s
Copied!
Make sure to enable pruning so that only the latest mined blocks are stored in the container. Add persistency in /home/srv for the daemon data, so it can restart without needing to resynch the blockchain.
Go Wrapper
For your Go code, also add an entry into the docker-compose file so that your Dockerfile is automatically called. An example entry should look something like this
1
  btc:
2
    build:
3
      context: ../../../..
4
    build:
5
      context: $GOPATH
6
      dockerfile: $GOPATH/src/github.com/diadata-org/diadata/build/Dockerfile-btc
7
    image: ${DOCKER_HUB_LOGIN}/${STACKNAME}_btc:latest
8
    networks:
9
      - scrapers-network
10
    logging:
11
      options:
12
        max-size: "50m"
13
    secrets:
14
      - api_diadata
Copied!
Be sure to add your container to the scrapers-network virtual network. Your credentials for the DIA API should be handled by the secrets directive in the section of your blockchain client. To initialize the API from Go you can create
1
var api *dia.Client
Copied!
and then call
1
var c dia.ConfigApi
2
configFile := "/run/secrets/api_diadata"
3
err := gonfig.GetConf(configFile, &c)
4
if err != nil {
5
  configFile = "../config/secrets/api_diadata.json"
6
  err = gonfig.GetConf(configFile, &c)
7
}
8
if err != nil {
9
  log.Println(err)
10
} else {
11
  log.Println("Loaded secret in", configFile)
12
}
13
api = dia.NewClient(&c)
Copied!
For sending supply data to DIA, you can use SendSupply in the ApiClient.go file.
This is the basic structure of these files. Run your Docker instance by calling
1
docker run <your instance>
Copied!
from the main directory. With docker-compose up you can test the entire system with your cryptocurrency VM and your Go wrapper in a virtual network.

Write your own rate scraper

Write your own exchange scraper

Last modified 7mo ago

Export as PDF

Copy link

Contents

Add your own blockchain scraper

Directory structure

Blockchain Client

Go Wrapper