This library extract article/news information from a webpage including: title, main image, description, author, keywords, publish date and body (if possible)...
This library supports scrapping using standard structured meta data, like: Microdata, hAtom Microformat ..etc, along with custom selectors that can be specified to support unstructured webpages.
News-Scrapper requires PHP >= 5.4
You can install this library with Composer. Drop this into your composer.json
manifest file:
{
"require": {
"zrashwani/news-scrapper": "1.*"
}
}
Then run composer install
.
Here's a quick how to scrap news data from a webpage:
require 'vendor/autoload.php';
// Initiate scrapper
$scrap_client = new \Zrashwani\NewsScrapper\Client();
print_r($scrap_client->getLinkData($url));
By default, scrapper tries to guess the best structured data adapter and apply it.
You can select a specific adapter to be used for extracting the data as following:
$url = "http://example.com/your-news-uri";
//use microdata standard for scrapping
$scrap_client = new \Zrashwani\NewsScrapper\Client('Microdata');
print_r($scrap_client->getLinkData($url));
Here is the list of supported structured data adapters or scrapping modes:
If the webpage doesn't follow any standard structured data, you can still scrap news information by specifying xpath or css selector for different article parts like: title, description, image and body. as following:
$scrapClient = new \Zrashwani\NewsScrapper\Client('Custom');
/*@var $adapter \Zrashwani\NewsScrapper\Adapters\CustomAdapter */
$adapter = $scrapClient->getAdapter();
$adapter
->setTitleSelector('.single-post h1') //selectors can be either css or xpath
->setImageSelector(".sidebar img")
->setAuthorSelector('//a[@rel="author"]')
->setPublishDateSelector('//span[@class="published_data"]')
->setBodySelector('//div[@class="contents"]');
$newsData = ($scrapClient->getLinkData("http://example.com/your-news-uri"));
print_r($newsData);
Custom scrapping adapter CustomAdapter
supports method chaining for setting the selectors.
If any selector is not specified it will use default selectors based on DefaultAdapter
(which is html adapter that depends of standard meta tags).
To scrap group of news article from certain page containing news links, scrapLinkGroup
method can be used
$listingPageUrl = 'https://www.readability.com/topreads/'; //url containing news listing
$linksSelector = '.entry-title a'; //css or xpath selector for news links inside listing page
$numberOfArticles = 3; //number of links to scrap, use null to get all matching selector
$scrapClient = new \Zrashwani\NewsScrapper\Client();
$newsGroupData = $scrapClient->scrapLinkGroup($listingPageUrl, $linksSelector,$numberOfArticles);
foreach($newsGroupData as $singleNews){
print_r($singleNews);
}
- Fork this repository
- Create a new branch for each feature or improvement
- Send a pull request from each feature branch
It is very important to separate new features or improvements into separate feature branches, and to send a pull request for each branch. This allows me to review and pull in new features or improvements individually.
All pull requests must adhere to the PSR-2 standard.
- PHP 5.4.0+
MIT Public License