Introduction
Documentation
Contribute
Write your own rate scraper
These instructions concern writing scrapers for single units characterised by a (floating point) number. For scrapers describing the relation between pairs of units, i.e. exchange rates see the instructions in exchangescrapers.md.
Instructions for the addition of a rate scraper
In order to add your own scraper for a new data source, you must adhere to our format. Create the package file
UpdateMYRATE.go in the package /internal/pkg/ratescrapers. The central method is UpdateMYRATE(). This method acts on a RateScraper struct which is defined in RateScraper.go in the ratescrapers package. For instance, for the the Euro Short-Term Rate (ESTER) issued by the ECB, UpdateESTER.go would look likePractical advice
Let's assume you want to scrape a data source that provides floating point information. Create a new file in
exchange-scrapers/ and call it MySourceScraper.go. The main difference between scrapers is the Update() method, where the actual scraping is done.1
func (s *RateScraper) UpdateESTER() error {
2
// scraper code here
3
}
Copied!
Here, the type
RateScraper from the ratescrapers package is something like this:1
type RateScraper struct {
2
// signaling channels
3
shutdown chan nothing
4
shutdownDone chan nothing
5
// error handling;
6
errorLock sync.RWMutex
7
error error
8
closed bool
9
ticker *time.Ticker
10
datastore models.Datastore
11
chanInterestRate chan *models.InterestRate
12
}
Copied!
The scraped data has to be written into a struct of type
InterestRate from pkg/model/types.go1
type InterestRate struct {
2
Symbol string
3
Value float64
4
PublicationTime time.Time
5
EffectiveDate time.Time
6
Source string
7
}
Copied!
and sent to the channel
chanInterestRate of s. In order to write a new scraper, it is not imperative to understand the architecture of the pathway from top to bottom, but it might be helpful. For a first impression you can have a look at the following diagram.Export as PDF
Copy link