Part 1. In the first part, we've dealt with the way of scraping raw data from the source website and structured data extraction.

Part 2. The second part describes the API Layer that periodically brings the latest data from the source website. Also, the API web server handles HTML endpoints, which output data in JSON format.

JSON became de facto an industry's most advanced data format which is ready to integrate with your apps.


Here is the third part of our series of articles on scraping Coronavirus (COVID-19) statistics. To reveal the full potential of the scraper, we add a web widget to visualize JSON data coming from the API layer.

With a Coronavirus widget, you can track the data on COVID-19 spread all over the world.

Your website visitor's country is determined automatically. So you can display statistics right on your website and provide your visitors with information about statistics from their location.

I've described HTML Server serving API endpoint /v1/{country} in the previous part of the tutorial. We are going to send requests to this service to pull live statistics on the specified country for our widget.

CoronaWidget object holds properties and methods for retrieving live statistics of COVID-19 cases.

//Widget constructor
function Widget() {
    // URL of API server, which serves endpoints for getting live statistics
    this.url = 'http://0.0.0.0:8008/v1';
    this.ui = {
        mainContainer: null,
        country: null,
        tot_cases: null,
        new_cases: null,
        tot_deaths: null,
        new_deaths: null,
        tot_recovered: null,
        active_cases: null,
        updateDate: null
    };
    this.country = '';
    this.init();
}

init function of CoronaWidget calls  functions which associates Widget members with HTML DOM structure elements and determine visitor's country.

Widget.prototype.init = function() {
    // _initUI associates Widget members with HTML DOM structure elements. 
    this._initUI();
    this.__initCountry().then((countryCode) => {
        flag = '<img class="flag-img" src="./flags/' + countryCode.toLowerCase() + '.svg" alt="' + countryList[countryCode] + '">';
        this.ui.country.innerHTML = flag + '&nbsp' + countryList[countryCode];
        this.country = countryList[countryCode];
        this._updateData();
    }).catch((err) => {
        console.log(err);
        this._updateData();
    });
}

The code below determines visitor's country to show its status in the widget. If the county is not determined by some reason, summary information about World COVID-19 statistics returned.

const countryCodeExpression = /loc=([\w]{2})/;
//automatic country determination.
Widget.prototype.__initCountry = function () {
	return new Promise((resolve, reject) => {
		var xhr = new XHR();
		xhr.timeout = 3000;
		xhr.onreadystatechange = function () {
			if (this.readyState == 4) {
				if (this.status == 200) {
					result = countryCodeExpression.exec(this.responseText)
					if (result == null || result[1] === '') {
						console.log('Failed determine country code');
						resolve('world');
					}
					resolve(result[1])
				} else {
					reject(xhr.status)
				}
			}
		}
		xhr.ontimeout = function () {
			reject('timeout')
		}
		xhr.open('GET', 'https://www.cloudflare.com/cdn-cgi/trace', true);
		xhr.send();
	});
}

At first we send requests to COVID-19 API in _updateData() function specifying country determined earlier in  __initCountry() function. Then we parse resulted JSON and fill HTML Widget templates.

// Send requests to COVID-19 API and parse results for a specific country.
Widget.prototype._updateData = function(e) {
    e && e.preventDefault();
    var xhr = new XHR(),
        tot_cases = this.ui.tot_cases,
        new_cases = this.ui.new_cases,
        tot_deaths = this.ui.tot_deaths,
        new_deaths = this.ui.new_deaths,
        tot_recovered = this.ui.tot_recovered,
        active_cases = this.ui.active_cases,
        updateDate = this.ui.updateDate,
        country = this.country,
        resp;
    xhr.timeout = 3000;

    xhr.onreadystatechange = function() {
        if (this.readyState == 4) {
            if (this.status == 200) {
                resp = JSON.parse(this.responseText);
                if (tot_cases != null) {
                    tot_cases.innerHTML = resp['Total Cases_text'] === '' ? '0' : resp['Total Cases_text'];
                }
                if (new_cases != null) {
                    new_cases.innerHTML = resp['New Cases_text'] === '' ? '0' : resp['New Cases_text'];
                }
                if (tot_deaths != null) {
                    tot_deaths.innerHTML = resp['Total Deaths_text'] === '' ? '0' : resp['Total Deaths_text'];
                }
                if (new_deaths != null) {
                    new_deaths.innerHTML = resp['New Deaths_text'] === '' ? '0' : resp['New Deaths_text'];
                }
                if (tot_recovered != null) {
                    tot_recovered.innerHTML = resp['Total Recovered_text'] === '' ? '0' : resp['Total Recovered_text'];
                }
                if (active_cases != null) {
                    active_cases.innerHTML = resp['Active Cases_text'] === '' ? '0' : resp['Active Cases_text'];
                }
                updateDate.innerHTML = resp['Last Update'] === '' ? '' : resp['Last Update'];
            } else {
                console.log(`Failed to retrieve COVID-19 statisctic. Server returned status ${this.status}: ${this.responseText}`);
            }
        }
    }

    xhr.ontimeout = function() {
        console.log('Failed to retrieve COVID-19 statisctic. Timeout');
    }

    xhr.onerror = function(e) {
        console.log('Failed to retrieve COVID-19 statisctic.');
    }

    if (country !== '') {
        this.url += '/' + country;
    }
    xhr.open('GET', this.url, true);
    xhr.send();
}

The full code code is available in our public repository at https://github.com/slotix/COVID-19.git

At the end of the third part of our series of articles, you get a web widget showing live stats on COVID-19, which you can place on your web site.

Find several widgets at https://covid-19.dataflowkit.com/

Resources

Dataflow Kit open COVID-19 Free API:

https://covid-19.dataflowkit.com/v1
https://covid-19.dataflowkit.com/v1/world

Free Coronavirus (COVID-19) Widgets:

https://covid-19.dataflowkit.com/


Public Github repository with COVID-19 open API sources.

slotix/COVID-19
Coronavirus (COVID-19) open API . Contribute to slotix/COVID-19 development by creating an account on GitHub.

Source of  COVID-19 actual data.

Coronavirus Update (Live): 2,552,491 Cases and 177,234 Deaths from COVID-19 Virus Pandemic - Worldometer
Live statistics and coronavirus news tracking the number of confirmed cases, recovered patients, tests, and death toll due to the COVID-19 coronavirus from Wuhan, China. Coronavirus counter with new cases, deaths, and number of tests per 1 Million population. Historical data and info. Daily charts, …