Write your own rate scraper

These instructions concern writing scrapers for single units characterised by a (floating point) number. For scrapers describing the relation between pairs of units, i.e. exchange rates see the instructions in exchangescrapers.md.
Instructions for the addition of a rate scraper
In order to add your own scraper for a new data source, you must adhere to our format. Create the package file UpdateMYRATE.go in the package /internal/pkg/ratescrapers. The central method is UpdateMYRATE(). This method acts on a RateScraper struct which is defined in RateScraper.go in the ratescrapers package. For instance, for the the Euro Short-Term Rate (ESTER) issued by the ECB, UpdateESTER.go would look like
Practical advice
Let's assume you want to scrape a data source that provides floating point information. Create a new file in exchange-scrapers/ and call it MySourceScraper.go. The main difference between scrapers is the Update() method, where the actual scraping is done.
1
func (s *RateScraper) UpdateESTER() error {
2
  // scraper code here
3
}
Copied!
Here, the type RateScraper from the ratescrapers package is something like this:
1
type RateScraper struct {
2
    // signaling channels
3
    shutdown chan nothing
4
    shutdownDone chan nothing
5
    // error handling;
6
    errorLock        sync.RWMutex
7
    error            error
8
    closed           bool
9
    ticker           *time.Ticker
10
    datastore        models.Datastore
11
    chanInterestRate chan *models.InterestRate
12
}
Copied!
The scraped data has to be written into a struct of type InterestRate from pkg/model/types.go
1
type InterestRate struct {
2
    Symbol             string
3
    Value              float64
4
    PublicationTime    time.Time
5
    EffectiveDate      time.Time  
6
    Source             string
7
}
Copied!
and sent to the channel chanInterestRate of s. In order to write a new scraper, it is not imperative to understand the architecture of the pathway from top to bottom, but it might be helpful. For a first impression you can have a look at the following diagram.

Write your own Defi Lending Rate Scraper

Write your own blockchain scraper

Last modified 1yr ago

Export as PDF

Copy link

Contents

Instructions for the addition of a rate scraper

Practical advice