In the first part, we've described how to scrape COVID-19 statistics and save extracted data to JSON or CSV files.
We are going to create a simple COVID API service layer to grab live statistics from https://www.worldometers.info/coronavirus/#countries website periodically.
HTML Endpoints serve and share parsed information with third-party applications/services after that.
Endpoints.
Simple HTML server written in Go handles 2 endpoints for us:
GET /v1 - List all COVID-19 cases.
GET /v1/{cntr} - Get COVID-19 cases for a specified country.
Available {cntr} values: world, USA, spain, Slovakia and etc.
We run our open API at
https://covid-19.dataflowkit.com/v1
https://covid-19.dataflowkit.com/v1/world
You may use it absolutely for free.
How does it work?
Some parts of the code are apparent, so we omit it here. Please refer to our public repository to get full code.
//Start HTTP server to handle API endpoints.
func Start(cfg Config) *HTMLServer {
...
// Omitted for brevity
...
//Get COVID-19 cases for specified country.
router.HandleFunc("/v1/{cntr}", covidHandler)
//List all COVID-19 cases
router.HandleFunc("/v1", covidHandler)
...
// Omitted for brevity
...
return &htmlServer
}
// runScheduler launches updateCovidStat func to pull
// updated information periodically (every hour)
func runScheduler(done chan struct{}) {
gocron.Every(1).Hour().From(gocron.NextTick()).Do(updateCovidStat)
for {
select {
case <-gocron.Start():
case <-done:
fmt.Println("Scheduler: Stopped")
return
}
}
}
Let's pass coronaPayload.json to the parse endpoint of Dataflow Scraper Kit API.
{
"name":"COVID-19",
"request":{
"url":"https://www.worldometers.info/coronavirus/",
"actions":[
],
"type":"chrome"
},
"commonParent":"tr[role]",
"fields":[
{
"name":"Country",
"selector":"#main_table_countries_today thead+ tbody td:nth-child(1)",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"Total Cases",
"selector":"#main_table_countries_today .sorting_1",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"New Cases",
"selector":"#main_table_countries_today .sorting_1+ td",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"Total Deaths",
"selector":"#main_table_countries_today .sorting_1~ td:nth-child(4)",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"New Deaths",
"selector":"#main_table_countries_today thead+ tbody td:nth-child(5)",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"Total Recovered",
"selector":"#main_table_countries_today thead+ tbody td:nth-child(6)",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
},
{
"name":"Active Cases",
"selector":"#main_table_countries_today thead+ tbody td:nth-child(7)",
"attrs":[
"text"
],
"type":1,
"filters":[
{
"name":"trim"
}
]
}
],
"format":"json"
}We schedule func updateCovidStat() to launch periodically (every hour) to keep this information up to date.
var (
payloadFile = "coronaPayload.json"
//covidStatistics map keeps data returned by func updateCovidStat()
covidStatistics []map[string]string
)
// updateCovidStat - send requests to DFK API
// DFK API pulls an actual COVID-19 data to covidStatistics map
func updateCovidStat() {
//Load Payload to request live stats from worldometers.info
payload, err := ioutil.ReadFile(payloadFile)
if err != nil {
fmt.Printf("Failed to read payload file: %s", err.Error())
return
}
//Send POST request to Dataflowkit Scraping API.
response, err := http.Post("https://api.dataflowkit.com/v1/parse?api_key="+*apiKey, "application/json", bytes.NewReader(payload))
if err != nil {
fmt.Printf("Failed to post request to DFK Scraper API: %s", err.Error())
return
}
defer response.Body.Close()
if response.StatusCode != 200 {
body, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Printf("Failed to read respose body: %s", err.Error())
return
}
fmt.Printf("Failed to get COVID-19 statistics. Server returned: %s", string(body))
return
}
//StatusOk
body, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Printf("Failed read response body: %s", err.Error())
return
}
err = json.Unmarshal(body, &covidStatistics)
if err != nil {
fmt.Printf("Failed unmarshal response into map: %s", err.Error())
return
}
}
func covidHandler(...) handles API endpoints.
covidStatistics variable keeps actual COVID cases grouped by country.
- If no
{cntr}parameter specified, the full dataset returned. - Specify
{cntr}parameter to extract results for this country only.
// covidHandler handles API /v1 and /v1/{cntr} endpoints
// covidStatistics variable is parsed here according to
// passed {cntr} parameter
func covidHandler(w http.ResponseWriter, r *http.Request) {
...
// Omitted for brevity
...
w.Header().Set("Content-Type", "application/json")
vars := mux.Vars(r)
country, ok := vars["cntr"]
//return results for all coutries
if !ok {
writeResponse(w, covidStatistics)
return
}
//return results for specified country
countryStatistic := map[string]string{}
for _, countryStatistic = range covidStatistics {
if strings.ToLower(countryStatistic["Country_text"]) == strings.ToLower(country) {
writeResponse(w, countryStatistic)
return
}
}
//If specifid country not found return the very first result (world)
fmt.Println("Not Found")
countryStatistic = covidStatistics[0]
writeResponse(w, countryStatistic)
}
How to build COVID-19 API Server?
Clone public repository from GitHub.
git clone https://github.com/slotix/COVID-19.gitRun the following command to build Go binary
cd COVID-19/covid-19-service && go buildGet a Free API Key.
Dataflow Kit API Key is required to Start the server. You can obtain it from the user dashboard after free registration. Once you sign-up, we grant you free 1000 credits.
Go to https://account.dataflowkit.com and either use Facebook/Google login or register with your email.
Click on the "Log in" button to register with your Facebook or Google account. Or press the "Sign Up" link to register with your email.
Now go to dashboard Settings and Copy your API Key to clipboard.
Start Service and send requests.
Now we can run the Service and try to send requests.
./covid-19-service -a DFK-API-KEY
COVID-19 Service : started : Host=0.0.0.0:8008
Open http://0.0.0.0:8008/v1 in your web browser.
As you can see, a extended JSON containing an actual COVID data returned.
Type curl command in another terminal window to check out the Service:
curl 0.0.0.0:8008/v1/world
{"Active Cases_text":"1,311,078","Country_text":"World","New Cases_text":"+67,094","New Deaths_text":"+5,104","Total Cases_text":"1,846,837","Total Deaths_text":"113,883","Total Recovered_text":"421,876"}
You can pipe json_pp at the end, so you have a prettier JSON response:
curl 0.0.0.0:8008/v1/world | json_pp
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 237 100 237 0 0 1338 0 --:--:-- --:--:-- --:--:-- 1338
{
"New Cases_text" : "+62,892",
"Total Cases_text" : "1,915,149",
"Total Deaths_text" : "118,984",
"New Deaths_text" : "+4,790",
"Active Cases_text" : "1,354,933",
"Last Update" : "2020-04-13 20:11",
"Total Recovered_text" : "441,232",
"Country_text" : "World"
}
But we have to consider an even better way to display information.
In the next part, I'm going to show how to bring actual JSON data coming from API endpoints to HTML Widget and share it with others, as shown below.
Resources
Dataflow Kit open COVID-19 Free API:
https://covid-19.dataflowkit.com/v1
https://covid-19.dataflowkit.com/v1/world
Free Coronavirus (COVID-19) Widgets:
https://covid-19.dataflowkit.com/
Public Github repository with COVID-19 open API sources.
GitHub
Source of COVID-19 actual data.
Worldometer
Related posts?
- Part 1. Scrape COVID-19 cases.
- Part 2. Coronavirus Tracker API.
- Part 3. COVID-19 Widgets.