A repo for news crawling. Then combine similar news. Temporarily used python and Scrapy framework.
Used Jieba and Scrapy

1. Find a proper Chinese segmentation tool
2. Split a JSON file into small files, every file contains only one piece of news
3. Read some articles about SVM (Did not use SVM, but tfidf and cosin similarity)
4. Try to categorize different news
5. Build another IDF dictionary from web news
6. Categorize those similar passages in sina and netease but not in tencent
7. Make a website display the results and show the comments
8. Improve categorization performance
9. Make the website more beautiful!
10. Reconstruct this project with php

http://news.reetsee.com