/** * @param DelegatingSourceCleaner $cleaner * @param OriginInterface $origin * @param ThresholdVoterInterface $voter * * @return boolean */ public function cleanOrigin(DelegatingSourceCleaner $cleaner, OriginInterface $origin, ThresholdVoterInterface $voter) { if (null === ($expireDate = $this->getLastFullImportDate($origin))) { $this->logger->debug(sprintf('Skipping origin %s, because it has no recent imports', $origin->getTitle())); $this->eventDispatcher->dispatch(IoEvents::ORIGIN_CLEANUP_SKIP, new OriginCleanupEvent($origin, 0)); return false; } $this->eventDispatcher->dispatch(IoEvents::PRE_CLEAN_ORIGIN, new OriginEvent($origin)); $this->logger->debug(sprintf('Checking sources of %s that have not been visited since %s', $origin->getTitle(), $expireDate->format('Y-m-d H:i:s'))); // get sources that haven't been visited since $expireDate $sourceRepo = $this->sourceManager->getRepository(); $count = $sourceRepo->countByOriginAndUnvisitedSince($origin, $expireDate); // fail safe: see if percentage of sources to be removed is not too high $total = $sourceRepo->countByOrigin($origin); $max = $this->getThreshold($total); // see if threshold is reached if ($count > $max) { $message = sprintf('Stopping cleanup for origin %s, because %s of %s sources were to be deleted, %s is the maximum.', $origin->getTitle(), $count, $total, $max); if (!$voter->vote($count, $total, $max, $message)) { $this->eventDispatcher->dispatch(IoEvents::ORIGIN_CLEANUP_HALT, new OriginCleanupHaltEvent($origin, $count, $total, $max)); return false; } } $this->logger->debug(sprintf('Cleaning %d sources for origin %s', $count, $origin->getTitle())); $builder = $sourceRepo->queryByOriginAndUnvisitedSince($origin, $expireDate); $numCleaned = $cleaner->cleanByQuery($builder->getQuery()); $this->eventDispatcher->dispatch(IoEvents::POST_CLEAN_ORIGIN, new OriginCleanupEvent($origin, $numCleaned)); return $numCleaned; }
public function testProcessException() { $executor = new SourceProcessExecutor($this->manager, $this->processor, new NullLogger()); $source = new SourceMock(12345); $this->manager->expects($this->once())->method('findById')->will($this->returnValue($source)); $this->processor->expects($this->once())->method('isLinked')->will($this->returnValue(false)); $this->processor->expects($this->once())->method('process')->will($this->throwException(new SourceProcessException('Foobar'))); $this->assertFalse($executor->execute($executor->getObjectPayload($source))); $messages = $source->getMessages(); $this->assertInternalType('array', $messages); $this->assertArrayHasKey('process', $messages); $this->assertArrayHasKey(LogLevel::ERROR, $messages['process']); $this->assertContains('Foobar', $messages['process'][LogLevel::ERROR]); }
/** * @param AbstractQuery $query * * @return integer * * @throws \LogicException */ public function cleanByQuery(AbstractQuery $query) { $numCleaned = 0; /** @var SourceInterface $source */ foreach ($query->iterate() as list($source)) { if (!$source instanceof SourceInterface) { throw new \LogicException(sprintf('Invalid iterator given, encountered %s instead of SourceInterface', is_object($source) ? get_class($source) : gettype($source))); } $this->eventDispatcher->dispatch(IoEvents::PRE_CLEAN_SOURCE, new SourceEvent($source)); $this->sourceManager->remove($source); $this->eventDispatcher->dispatch(IoEvents::POST_CLEAN_SOURCE, new SourceEvent($source)); $numCleaned++; if ($numCleaned % 50 === 0) { $this->sourceManager->flush(); $this->sourceManager->clear(); } } if ($numCleaned > 0) { $this->sourceManager->flush(); $this->sourceManager->clear(); } return $numCleaned; }
/** * @param integer $sourceId * * @return SourceInterface */ protected function findSource($sourceId) { return $this->sourceManager->findById($sourceId); }
/** * @inheritdoc */ public function clear() { $this->sourceManager->clear(); $this->sources = []; $this->originSources = []; }
/** * @inheritdoc */ public function clean(DelegatingSourceCleaner $cleaner, ThresholdVoterInterface $voter) { $builder = $this->sourceManager->getRepository()->createQueryBuilder('s')->select('s')->innerJoin('s.origin', 'o')->where('SIZE(o.feeds) = 0'); return $cleaner->cleanByQuery($builder->getQuery()); }