PHP scraperWiki::get_var Examples

Programming Language: PHP

Class/Type: scraperWiki

Method/Function: get_var

Examples at hotexamples.com: 3

Scraperwiki is a free web scraping tool that allows us to extract data from websites easily. The get_var function is one of the many useful functions available in the Scraperwiki library.

The get_var function is used to retrieve a single value from the Scraperwiki datastore. It takes two arguments: the first is the name of the variable you wish to retrieve and the second is the default value if the variable is not found.

Code Example:

$name = scraperwiki::get_var('name', 'Unknown');
echo "Name: " . $name;

In this example, we retrieve the value of the variable 'name' from the Scraperwiki datastore. If the variable does not exist in the datastore, we set the default value to 'Unknown'. The value of the variable is then stored in the $name variable, which is then printed out to the console.

Another example of using the get_var function is to retrieve the date of the last time a scraper was run:

$last_run = scraperwiki::get_var('last_run', 'Never');
echo "Last Run: " . $last_run;

In this example, we retrieve the value of the variable 'last_run' from the Scraperwiki datastore. If the variable does not exist in the datastore, we set the default value to 'Never'. The value of the variable is then stored in the $last_run variable, which is then printed out to the console. The get_var function is part of the Scraperwiki library, which is a package library for PHP.

PHP scraperWiki::get_var - 3 examples found. These are the top rated real world PHP examples of scraperWiki::get_var extracted from open source projects. You can rate examples to help us improve the quality of examples.

Frequently Used Methods

Show Hide

scrape(30)

save_sqlite(18)

save(5)

get_var(3)

save_var(2)

gb_postcode_to_latlng(1)

httpresponseheader(1)

select(1)

sqliteexecute(1)

Example #1

Show file

File: whoint_gap.php Project: flyeven/scraperwiki-scraper-vault

        }
    }
}
function getData($query)
{
    if (!count(scraperwiki::table_info("swdata"))) {
        return false;
    }
    return scraperWiki::select($query);
}
if (!count(scraperwiki::table_info("swvariables"))) {
    // if no stored variables are available, start from beginning
    $start = '';
} else {
    // else, load last disease from database
    $start = scraperWiki::get_var('disease', '');
}
$html = scraperWiki::scrape("http://www.who.int/csr/don/archive/disease/en/index.html");
$dom = new simple_html_dom();
$dom->load($html);
$handleNextEntry = false;
// no previous entries -> start from beginning
if ($start == '') {
    $handleNextEntry = true;
} else {
    print "starting after: {$start}\n\n";
}
foreach ($dom->find("ul[@class='a_z'] li a") as $data) {
    $disease = trim($data->plaintext);
    print "fetching data for disease: {$disease}";
    if ($handleNextEntry) {

Example #2

Show file

File: the_synth_list.php Project: flyeven/scraperwiki-scraper-vault

$synthList3 = file_get_contents("https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=jsondict&name=current_synths&query=select%20DISTINCT%20manufacturer%2C%20url%2C%20name%20from%20%60swdata%60");
if (!empty($synthList3)) {
    $synthList3 = json_decode($synthList3);
}
$synths = array();
$synths = traverseList($synthList1);
$synths = array_merge(traverseList($synthList2), $synths);
$synths = array_merge(traverseList($synthList3), $synths);
$synths = array_map('unserialize', array_unique(array_map('serialize', $synths)));
echo "Total synths: " . count($synths) . "\n";
//var_dump($synths);
if (!empty($synths)) {
    //$dbName = "vintagesynth-scrape-".$today = date("m-d-Y");
    $saveMessage = scraperWiki::save_sqlite(array('manufacturer', 'name', 'url'), $synths);
    //print strval($saveMessage);
    scraperwiki::save_var('total_results', count($synths));
    print scraperWiki::get_var('total_results');
}
function traverseList($list)
{
    $dataList = array();
    foreach ($list as $item) {
        //Clean up the data
        foreach ($item as $key => $value) {
            $item->{$key} = preg_replace("/<*.>/", "", $value);
            //echo $item->$key."\n";
        }
        $dataList[] = $item;
    }
    return $dataList;
}

Example #3

Show file

File: schools_in_the_western_cape_south_africa_1.php Project: flyeven/scraperwiki-scraper-vault

        scraperwiki::save(array('EMISNumber'), $tosave);
    } catch (Exception $e) {
        print 'Caught exception (' . $data->value . '): ' . $e->getMessage() . "\n";
    }
    //break; //uncomment to just process one school
}
scraperWiki::save_var('place', 'none');
$html = scraperWiki::scrape("http://wcedemis.pgwc.gov.za/wced/findschoolO.shtml?2");
require 'scraperwiki/simple_html_dom.php';
$dom = new simple_html_dom();
$dom->load($html);
//scraperwiki::sqliteexecute("CREATE TABLE `swdata` (`EMISNumber` text)");
//scraperWiki::save_var('place','none');
//scraperWiki::save_var('place','0130333352');
//exit();
$place = scraperWiki::get_var('place', 'none');
$resume = true;
if ($place == 'none') {
    $resume = false;
}
foreach ($dom->find("select[name='EMIS_NO'] option") as $data) {
    //loop through the list of ordinary schools
    $d = http_build_query(array('EMIS_NO' => $data->value));
    $dlength = strlen($d);
    if ($resume) {
        //we need to resume, so find the correct place
        if ($place == $data->value) {
            $resume = false;
        } else {
            continue;
        }