This is a link checking robot, that crawls your moodle site following links and reporting on links that are either broken or that link to very large files.
It is a local plugin with a moodle cron task. It logs into your moodle via curl effectively from outside moodle. The cronjob scrapes each page, parses it and follows links. By using this architecture it will only find broken links that actually matter to students.
Since the plugin cronjob comes in from outside it needs to authenticate in Moodle.
The plugin has a dependency on the moodle-auth_basic. To install the dependency plugin as a git submodule:
git submodule add https://github.com/CatalystIT-AU/moodle-auth_basic.git auth/basic
Install plugin moodle-local_linkchecker_robot as a git submodule:
git submodule add https://github.com/central-queensland-uni/moodle-local_linkchecker_robot.git local/linkchecker_robot
When installing the plugins please keep in mind the official Moodle recommendations: installing Moodle plugins
Login to Moodle after you have downloaded the plugin code with git. You will be forwarded to URL http://your_moodle_website.com/admin/index.php with Plugins check. There you should see plugins "Basic authentication" and "Link checker robot".
Click button "Upgrade Moodle database now" which should initiate plugins installation.
Now you should see page "Upgrading to new version" with plugins installation statuses and button "Continue".
**Note! Plugin auth_basic is disabled by default after installation. You will need to enable it manually from
Home ► Site administration ► Plugins ► Authentication ► Manage authentication**
After clicking "Continue" you will get to the page "New settings - Link checker robot". While you may leave other settings default, you might want to setup a custom bot username and make sure to change bot password.
It is recommended that bot user should be kept with readonly access to all the site pages you wish to crawl. You can give the robot similar read capabilities that real students have. Never give your bot user write capabilities.
After verifying all settings click "Save changes".
Enable auth_basic plugin (if you haven't done that earlier) from
Home ► Site administration ► Plugins ► Authentication ► Manage authentication
Now navigate to URL http://your_moodle_website.com/local/linkchecker_robot/index.php". It will show some stats about the Link checker Robot.
Click "Auto create" button against "Bot user". This actually creates the user which username and password you have configured previously on page "New settings - Link checker robot".
Once bot user is created "Bot user" line in status report should be showing "Good".
This is achieved by configuring proper security roles in Moodle and assigning these roles to the robot user on desired categories.
Import role "Robot" from local/linkchecker_robot/roles/robot.xml on
Site administration ► Users ► Permissions ► Define roles ► Add a new role
Add this role to the "Link checker robot" user on
Site administration ► Users ► Permissions ► Assign system roles.
Import role "Robot nofollow" from file local/linkchecker_robot/roles/robotnofollow.xml on
Site administration ► Users ► Permissions ► Define roles ► Add a new role.
To disable crawling of, say "Category ABC", go to
Site administration ► Courses ► Manage courses and categories ► Category ABC
then click on "Assign roles" in the left navigation menu. Click on role "Robot nofollow", click on user "Link checker Robot" under "Potential users" and add him to "Existing users".
The above configuration applies role "Robot" on the whole moodle site and lets crawler to access general content. And "Role nofollow" prohibits crawler from accessing the specific category.
In the same way it is possible to restrict crawler from accessing other moodle contexts such as courses, activities and blocks.
The same effect could be achieved even without role "Robot nofollow" by assigning role "Robot" on the contexts you want to be crawled. But using the combination of two roles gives more flexibility.
##Test basic authentication with curl
Example in bash:
curl -c /tmp/cookies -v -L --user moodlebot:moodlebot http://your_moodle_website.com/course/view.php?id=3
This command should log you in with specified credentials via Basic HTTP Auth. It will dump headers, requests and responses and among the output you should be able to see the line "You are logged in as ".
Once Basic HTTP auth works test running the robot task from the CLI:
php admin/tool/task/cli/schedule_task.php --execute='\local_linkchecker_robot\task\crawl_task'
Scheduled task: Link checker robot
... used 2997 dbqueries
... used 59.828736066818 seconds
Task completed.
If this worked then it's a matter of sitting back and waiting for the robot to do it's thing. It works incrementally spreading the load over many cron cycles, you can watch it's progress in
/local/linkchecker_robot/report.php?report=queued
and
/local/linkchecker_robot/report.php?report=recent
4 new admin reports are available for showing the current crawl status, broken links and URLs and slow links. They are available under:
Administration > Reports > Link checker
Please raise any issues in github:
https://github.com/central-queensland-uni/moodle-local_linkchecker_robot/issues
If you need anything urgently and would like to sponsor it's implemenation please email me: Brendan Heywood.