public function fetchTestBinaries($groupname, $numarticles, $clearexistingbins) { $db = new DB(); $nntp = new Nntp(); $binaries = new Binaries(); $groups = new Groups(); $ret = array(); if ($clearexistingbins == true) { $db->exec('truncate releaseregextesting'); } $nntp->doConnect(); $groupsToFetch = array(); if (preg_match('/^[a-z]{2,3}(\\.[a-z0-9\\-]+)+$/', $groupname)) { $groupsToFetch[] = array('name' => $groupname); } elseif ($groupname === 0) { $groupsToFetch = $groups->getAll(); } else { $newsgroups = $nntp->getGroups(); foreach ($newsgroups as $ngroup) { if (preg_match('/' . $groupname . '/', $ngroup['group'])) { $groupsToFetch[] = array('name' => $ngroup['group']); } } } foreach ($groupsToFetch as $groupArr) { $group = $groupArr['name']; $data = $nntp->selectGroup($group); if ($nntp->isError($data)) { $ret[] = "Could not select group (doesnt exist on USP): {$group}"; continue; } else { $rangeStart = $data['last'] - $numarticles; $rangeEnd = $groupEnd = $data['last']; $rangeTotal = $rangeEnd - $rangeStart; $done = false; while ($done === false) { if ($rangeTotal > $binaries->messagebuffer) { if ($rangeStart + $binaries->messagebuffer > $groupEnd) { $rangeEnd = $groupEnd; } else { $rangeEnd = $rangeStart + $binaries->messagebuffer; } } if ($binaries->compressedHeaders) { $msgs = $nntp->getXOverview($rangeStart . "-" . $rangeEnd, true, false); } else { $msgs = $nntp->getOverview($rangeStart . "-" . $rangeEnd, true, false); } if ($nntp->isError($msgs)) { $ret[] = "Error {$msgs->code}: {$msgs->message} on " . $group; continue 2; } $headers = array(); if (is_array($msgs)) { //loop headers, figure out parts foreach ($msgs as $msg) { if (!isset($msg['Number'])) { continue; } $msgPart = $msgTotalParts = 0; $pattern = '|\\((\\d+)[\\/](\\d+)\\)|i'; preg_match_all($pattern, $msg['Subject'], $matches, PREG_PATTERN_ORDER); $matchcnt = sizeof($matches[0]); for ($i = 0; $i < $matchcnt; $i++) { //not (int)'d here because of the preg_replace later on $msgPart = $matches[1][$i]; $msgTotalParts = $matches[2][$i]; } if (!isset($msg['Subject']) || $matchcnt == 0) { // not a binary post most likely.. continue continue; } if ((int) $msgPart > 0 && (int) $msgTotalParts > 0) { $subject = utf8_encode(trim(preg_replace('|\\(' . $msgPart . '[\\/]' . $msgTotalParts . '\\)|i', '', $msg['Subject']))); if (!isset($headers[$subject])) { $headers[$subject]['Subject'] = $subject; $headers[$subject]['From'] = $msg['From']; $headers[$subject]['Date'] = strtotime($msg['Date']); $headers[$subject]['Message-ID'] = $msg['Message-ID']; $headers[$subject]['Size'] = $msg['Bytes']; } else { $headers[$subject]['Size'] += $msg['Bytes']; } } } unset($msgs); if (isset($headers) && count($headers)) { $groupRegexes = $this->getForGroup($group); $binSetData = array(); foreach ($headers as $subject => $data) { $binData = array('name' => $subject, 'fromname' => $data['From'], 'date' => $data['Date'], 'binaryhash' => md5($subject . $data['From'] . $group), 'groupname' => $group, 'regexID' => "null", 'categoryID' => "null", 'reqID' => "null", 'blacklistID' => 0, 'size' => $data['Size'], 'relname' => "null", 'relpart' => "null", 'reltotalpart' => "null"); //Filter binaries based on black/white list if ($binaries->isBlackListed($data, $group)) { //binary is blacklisted $binData['blacklistID'] = 1; } //Apply Regexes $regexMatches = array(); foreach ($groupRegexes as $groupRegex) { $regexCheck = $this->performMatch($groupRegex, $subject, $data['From']); if ($regexCheck !== false) { $regexMatches = $regexCheck; $binData['regexID'] = $regexCheck['regexID']; $binData['categoryID'] = $regexCheck['regcatid']; $binData['reqID'] = empty($regexCheck['reqID']) ? "null" : $regexCheck['reqID']; $binData['relname'] = $regexCheck['name']; break; } } $binSetData[] = $binData; } //insert 500 bins at a time $binChunks = array_chunk($binSetData, 500); foreach ($binChunks as $binChunk) { foreach ($binChunk as $chunk) { $binParams[] = sprintf("(%s, %s, FROM_UNIXTIME(%s), %s, %s, %s, %s, %s, %d, %d, now())", $db->escapeString($chunk['name']), $db->escapeString($chunk['fromname']), $db->escapeString($chunk['date']), $db->escapeString($chunk['binaryhash']), $db->escapeString($chunk['groupname']), $chunk['regexID'], $chunk['categoryID'], $chunk['reqID'], $chunk['blacklistID'], $chunk['size']); } $binSql = "INSERT IGNORE INTO releaseregextesting (name, fromname, date, binaryhash, groupname, regexID, categoryID, reqID, blacklistID, size, dateadded) VALUES " . implode(', ', $binParams); //echo $binSql; $db->exec($binSql); } $ret[] = "Fetched " . number_format($numarticles) . " articles from " . $group; } else { $ret[] = "No headers found on " . $group; continue; } } else { $ret[] = "Can't get parts from server (msgs not array) on " . $group; continue; } if ($rangeEnd == $groupEnd) { $done = true; } $rangeStart = $rangeEnd + 1; } } } $nntp->doQuit(); return $ret; }
/** * Download a range of usenet messages. Store binaries with subjects matching a * specific pattern in the database. */ function scan($nntp, $groupArr, $first, $last, $type = 'update') { $db = new Db(); $releaseRegex = new ReleaseRegex(); $n = $this->n; $this->startHeaders = microtime(true); if ($this->compressedHeaders) { $nntpn = new Nntp(); $nntpn->doConnect(5, false, true); $response = $nntpn->_sendCommand('XFEATURE COMPRESS GZIP'); if ($nntpn->isError($response) || $response != 290) { $response2 = $nntpn->_sendCommand('XZVER'); if ($nntpn->isError($response2) || $response2 != 412) { $msgs = $nntp->getOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } else { $msgs = $nntp->getXOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } } else { $msgs = $nntp->getOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } } else { $msgs = $nntp->getOverview($first . "-" . $last, true, false); } if ($nntp->isError($msgs) && ($msgs->code == 400 || $msgs->code == 503)) { echo "NNTP connection timed out. Reconnecting...{$n}"; if (!$nntp->doConnect()) { // TODO: What now? echo "Failed to get NNTP connection.{$n}"; return; } $nntp->selectGroup($groupArr['name']); if ($this->compressedHeaders) { $nntpn = new Nntp(); $nntpn->doConnect(5, false, true); $response = $nntpn->_sendCommand('XFEATURE COMPRESS GZIP'); if ($nntpn->isError($response) || $response != 290) { $response2 = $nntpn->_sendCommand('XZVER'); if ($nntpn->isError($response2) || $response2 != 412) { $msgs = $nntp->getOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } else { $msgs = $nntp->getXOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } } else { $msgs = $nntp->getOverview($first . "-" . $last, true, false); $nntpn->doQuit(); } } else { $msgs = $nntp->getOverview($first . "-" . $last, true, false); } } $rangerequested = range($first, $last); $msgsreceived = array(); $msgsblacklisted = array(); $msgsignored = array(); $msgsinserted = array(); $msgsnotinserted = array(); $timeHeaders = number_format(microtime(true) - $this->startHeaders, 2); if ($nntp->isError($msgs)) { echo "Error {$msgs->code}: {$msgs->message}{$n}"; echo "Skipping group{$n}"; return false; } $this->startUpdate = microtime(true); if (is_array($msgs)) { //loop headers, figure out parts foreach ($msgs as $msg) { if (!isset($msg['Number'])) { continue; } $msgsreceived[] = $msg['Number']; $msgPart = $msgTotalParts = 0; $pattern = '|\\((\\d+)[\\/](\\d+)\\)|i'; preg_match_all($pattern, $msg['Subject'], $matches, PREG_PATTERN_ORDER); $matchcnt = sizeof($matches[0]); for ($i = 0; $i < $matchcnt; $i++) { $msgPart = $matches[1][$i]; $msgTotalParts = $matches[2][$i]; } if (!isset($msg['Subject']) || $matchcnt == 0) { $msgsignored[] = $msg['Number']; continue; } if ((int) $msgPart > 0 && (int) $msgTotalParts > 0) { $subject = utf8_encode(trim(preg_replace('|\\(' . $msgPart . '[\\/]' . $msgTotalParts . '\\)|i', '', $msg['Subject']))); if (!isset($this->message[$subject])) { $this->message[$subject] = $msg; $this->message[$subject]['MaxParts'] = (int) $msgTotalParts; $this->message[$subject]['Date'] = strtotime($this->message[$subject]['Date']); } if ((int) $msgPart > 0) { $this->message[$subject]['Parts'][(int) $msgPart] = array('Message-ID' => substr($msg['Message-ID'], 1, -1), 'number' => $msg['Number'], 'part' => (int) $msgPart, 'size' => $msg['Bytes']); $this->message[$subject]['PartNumbers'][(int) $msgPart] = $msg['Number']; } } } unset($msg); unset($msgs); $count = 0; $updatecount = 0; $partcount = 0; $rangenotreceived = array_diff($rangerequested, $msgsreceived); if ($type != 'partrepair') { echo "Received " . sizeof($msgsreceived) . " articles of " . ($last - $first + 1) . " requested, " . sizeof($msgsignored) . " not binaries {$n}"; } if ($type == 'update' && sizeof($msgsreceived) == 0) { echo "Error: Server did not return any articles.{$n}"; echo "Skipping group{$n}"; return false; } if (sizeof($rangenotreceived) > 0) { switch ($type) { case 'backfill': //don't add missing articles break; case 'partrepair': case 'update': default: $this->addMissingParts($rangenotreceived, $groupArr['ID']); break; } echo "Server did not return " . count($rangenotreceived) . " article(s).{$n}"; } if (isset($this->message) && count($this->message)) { $groupRegexes = $releaseRegex->getForGroup($groupArr['name']); //insert binaries and parts into database. when binary already exists; only insert new parts foreach ($this->message as $subject => $data) { //Filter binaries based on black/white list if ($this->isBlackListed($data, $groupArr['name'])) { $msgsblacklisted[] = count($data['Parts']); if ($type == 'partrepair') { $partIds = array(); foreach ($data['Parts'] as $partdata) { $partIds[] = $partdata['number']; } $db->exec(sprintf("DELETE FROM partrepair WHERE numberID IN (%s) AND groupID=%d", implode(',', $partIds), $groupArr['ID'])); } continue; } if (isset($data['Parts']) && count($data['Parts']) > 0 && $subject != '') { //Check for existing binary $binaryID = 0; $binaryHash = md5($subject . $data['From'] . $groupArr['ID']); $res = $db->queryOneRow(sprintf("SELECT ID FROM binaries WHERE binaryhash = %s", $db->escapeString($binaryHash))); if (!$res) { //Apply Regexes $regexMatches = array(); foreach ($groupRegexes as $groupRegex) { $regexCheck = $releaseRegex->performMatch($groupRegex, $subject); if ($regexCheck !== false) { $regexMatches = $regexCheck; break; } } $sql = ''; if (!empty($regexMatches)) { $relparts = explode("/", $regexMatches['parts']); $sql = sprintf("INSERT INTO binaries (name, fromname, date, xref, totalparts, groupID, procstat, categoryID, regexID, reqID, relpart, reltotalpart, binaryhash, relname, dateadded) VALUES (%s, %s, FROM_UNIXTIME(%s), %s, %s, %d, %d, %s, %d, %s, %d, %d, %s, %s, now())", $db->escapeString($subject), $db->escapeString(utf8_encode($data['From'])), $db->escapeString($data['Date']), $db->escapeString($data['Xref']), $db->escapeString($data['MaxParts']), $groupArr['ID'], Releases::PROCSTAT_TITLEMATCHED, $regexMatches['regcatid'], $regexMatches['regexID'], $db->escapeString($regexMatches['reqID']), $relparts[0], $relparts[1], $db->escapeString($binaryHash), $db->escapeString(str_replace('_', ' ', $regexMatches['name']))); } elseif ($this->onlyProcessRegexBinaries === false) { $sql = sprintf("INSERT INTO binaries (name, fromname, date, xref, totalparts, groupID, binaryhash, dateadded) VALUES (%s, %s, FROM_UNIXTIME(%s), %s, %s, %d, %s, now())", $db->escapeString($subject), $db->escapeString(utf8_encode($data['From'])), $db->escapeString($data['Date']), $db->escapeString($data['Xref']), $db->escapeString($data['MaxParts']), $groupArr['ID'], $db->escapeString($binaryHash)); } elseif ($type == 'partrepair') { $partIds = array(); foreach ($data['Parts'] as $partdata) { $partIds[] = $partdata['number']; } $db->exec(sprintf("DELETE FROM partrepair WHERE numberID IN (%s) AND groupID=%d", implode(',', $partIds), $groupArr['ID'])); continue; } if ($sql != '') { $binaryID = $db->queryInsert($sql); $count++; if ($count % 500 == 0) { echo "{$count} bin adds..."; } } } else { $binaryID = $res["ID"]; $updatecount++; if ($updatecount % 500 == 0) { echo "{$updatecount} bin updates..."; } } if ($binaryID != 0) { $partParams = array(); $partNumbers = array(); $totsize = 0; foreach ($data['Parts'] as $partdata) { $partcount++; $totsize += $partdata['size']; $partParams[] = sprintf("(%d, %s, %s, %s, %s)", $binaryID, $db->escapeString($partdata['Message-ID']), $db->escapeString($partdata['number']), $db->escapeString(round($partdata['part'])), $db->escapeString($partdata['size'])); $partNumbers[] = $partdata['number']; } $partSql = "INSERT INTO parts (binaryID, messageID, number, partnumber, size) VALUES " . implode(', ', $partParams); $pidata = $db->queryInsert($partSql, false); if (!$pidata) { $msgsnotinserted = array_merge($msgsnotinserted, $partNumbers); } else { $msgsinserted = array_merge($msgsinserted, $partNumbers); } // update bin size $upsql = sprintf("update binaries set size = size + %d where ID = %d", $totsize, $binaryID); $db->exec($upsql); } } } //TODO: determine whether to add to missing articles if insert failed if (sizeof($msgsnotinserted) > 0) { echo 'WARNING: ' . count($msgsnotinserted) . ' Parts failed to insert' . $n; $this->addMissingParts($msgsnotinserted, $groupArr['ID']); } if ($count >= 500 || $updatecount >= 500) { echo $n; } //line break for bin adds output } $timeUpdate = number_format(microtime(true) - $this->startUpdate, 2); $timeLoop = number_format(microtime(true) - $this->startLoop, 2); if (sizeof($msgsblacklisted) > 0) { echo "Blacklisted " . array_sum($msgsblacklisted) . " parts in " . sizeof($msgsblacklisted) . " binaries" . $n; } if ($type != 'partrepair') { echo number_format($count) . ' new, ' . number_format($updatecount) . ' updated, ' . number_format($partcount) . ' parts.'; echo " {$timeHeaders} headers, {$timeUpdate} update, {$timeLoop} range.{$n}"; } unset($this->message); unset($data); return $last; } else { echo "Error: Can't get parts from server (msgs not array) {$n}"; echo "Skipping group{$n}"; return false; } }
/** * Update a group back to a specified date. */ function backfillGroup($groupArr, $backfillDate = null, $regexOnly = false) { $db = new DB(); $binaries = new Binaries(); $n = $this->n; if ($regexOnly === true) { echo "Only inserting binaries which match regex{$n}"; $binaries->onlyProcessRegexBinaries = true; } $this->startGroup = microtime(true); $nntp = new Nntp(); $nntpc = new Nntp(); //Make sure we actually have a connection going before doing anything. if ($nntp->doConnect(5, false, true)) { echo 'Processing ' . $groupArr['name'] . $n; $data = $nntp->selectGroup($groupArr['name']); if ($nntp->isError($data)) { echo "Could not select group (bad name?): {$groupArr['name']}{$n}"; return; } if ($backfillDate) { $targetpost = $this->daytopost($nntp, $groupArr['name'], $this->dateToDays($backfillDate), TRUE); } else { $targetpost = $this->daytopost($nntp, $groupArr['name'], $groupArr['backfill_target'], TRUE); } //get targetpost based on days target if ($groupArr['first_record'] == 0 || $groupArr['backfill_target'] == 0 && !$backfillDate) { echo "Group " . $groupArr['name'] . " has invalid numbers. Have you run update on it? Have you set the backfill days amount?{$n}"; return; } echo "Group " . $data["group"] . ": server has " . $data['first'] . " - " . $data['last'] . ", or ~"; echo (int) (($this->postdate($nntp, $data['last'], FALSE) - $this->postdate($nntp, $data['first'], FALSE)) / 86400); echo " days." . $n . "Local first = " . $groupArr['first_record'] . " ("; echo (int) ((date('U') - $this->postdate($nntp, $groupArr['first_record'], FALSE)) / 86400); echo " days). Backfill target of " . ($backfillDate ? date('Y-m-d', $backfillDate) : $groupArr['backfill_target'] . " days") . " is post {$targetpost}.{$n}"; if ($targetpost >= $groupArr['first_record']) { echo "Nothing to do, we already have the target post.{$n} {$n}"; return ""; } //get first and last part numbers from newsgroup if ($targetpost < $data['first']) { echo "WARNING: Backfill came back as before server's first. Setting targetpost to server first.{$n}"; echo "Skipping Group {$n}"; return ""; } echo $binaries->onlyProcessRegexBinaries === true ? "Note: Discarding parts that do not match a regex" . $n : ""; //Done with $nntp for now, close it to avoid timeouts. $nntp->doQuit(); $nntpc->doConnect(); $datac = $nntpc->selectGroup($groupArr['name']); if ($nntpc->isError($datac)) { echo "Could not select group (bad name?): {$groupArr['name']}{$n}"; return; } //calculate total number of parts $total = $groupArr['first_record'] - $targetpost; $done = false; //set first and last, moving the window by maxxMssgs $last = $groupArr['first_record'] - 1; $first = $last - $binaries->messagebuffer + 1; //set initial "chunk" if ($targetpost > $first) { //just in case this is the last chunk we needed $first = $targetpost; } while ($done === false) { $binaries->startLoop = microtime(true); echo "Getting " . ($last - $first + 1) . " parts (" . number_format($first - $targetpost) . " in queue)" . $n; flush(); $success = $binaries->scan($nntpc, $groupArr, $first, $last, 'backfill'); if (!$success) { return ""; } $db->exec(sprintf("update groups SET first_record = %s, last_updated = now() WHERE ID = %d", $db->escapeString($first), $groupArr['ID'])); if ($first == $targetpost) { $done = true; } else { //Keep going: set new last, new first, check for last chunk. $last = $first - 1; $first = $last - $binaries->messagebuffer + 1; if ($targetpost > $first) { $first = $targetpost; } } } //Done with $nntpc $nntpc->doQuit(); //Just need $nntp for a quick check on the first_record_postdate $nntp->doConnect(); //$nntp->selectGroup($groupArr['name']); // some users report having this in keeps backfill working $first_record_postdate = $this->postdate($nntp, $first, false); //All done with NNTP. $nntp->doQuit(); if ($first_record_postdate != "") { $db->exec(sprintf("update groups SET first_record_postdate = FROM_UNIXTIME(" . $first_record_postdate . "), last_updated = now() WHERE ID = %d", $groupArr['ID'])); } //Set group's first postdate $timeGroup = number_format(microtime(true) - $this->startGroup, 2); echo "Group processed in {$timeGroup} seconds {$n}"; } else { echo "Failed to get NNTP connection.{$n}"; } }
/** * Get nzpre data from usenet and parse. */ public function nzpreUpdate() { require_once WWW_DIR . "/lib/nntp.php"; $s = new Sites(); $site = $s->get(); if (empty($site->nzpregroup) || empty($site->nzpresubject) || empty($site->nzpreposter) || empty($site->nzprefield) || empty($site->nzprekey)) { return false; } if ($this->echooutput) { echo "Predb : Checking for new pre data "; } $db = new DB(); $nntp = new Nntp(); if (!$nntp->doConnect()) { echo "Failed to get NNTP connection\n"; return false; } $ret = $groupData = $nntp->selectGroup($site->nzpregroup); if ($nntp->isError($ret)) { echo "Predb : Error " . $ret->getMessage() . "\n"; return false; } $ret = $groupMsgs = $nntp->getOverview($groupData['last'] - (!empty($site->nzprearticles) ? $site->nzprearticles : 500) . '-' . $groupData['last']); if ($nntp->isError($ret)) { echo "Predb : Error " . $ret->getMessage() . "\n"; return false; } $added_updated = 0; $nzprekey = $site->nzprekey; while (strlen($nzprekey) < 1024) { $nzprekey = $nzprekey . $nzprekey; } $cnt = !empty($site->nzprearticles) ? $site->nzprearticles : 500; foreach ($groupMsgs as $groupMsg) { if ($cnt % 50 == 0 && $cnt != 0 && $this->echooutput) { echo $cnt . ".."; } $cnt--; if (preg_match('/^' . $site->nzpresubject . '$/', $groupMsg['Subject']) && preg_match('/^' . $site->nzpreposter . '$/', $groupMsg['From'])) { $ret = $msgHeader = $nntp->getHeader($groupMsg['Message-ID']); if ($nntp->isError($ret)) { continue; } for ($i = 0; $i < count($msgHeader); $i++) { if (preg_match('/^' . $site->nzprefield . ': /', $msgHeader[$i])) { if ($nzpreParse = $this->nzpreParse(str_replace($site->nzprefield . ': ', '', $msgHeader[$i]), $nzprekey)) { if ($this->updatePreDB($db, $nzpreParse)) { $added_updated++; } } break; } } } } $nntp->disconnect(); if ($this->echooutput) { echo "\nPredb : Added/Updated " . $added_updated . " records\n"; } }