Affects Version/s: 0.1.0
Fix Version/s: None
1. create a new module in svn, move any crawler specific code to the module.
2. create a JIRA component for crawler
3. define what should be in core and what should be in crawler.
- stuff like task, taskmaster, queue will be in core
- stuff like fetcher/parser maybe in core or crawler
- stuff like link extractor are crawler specific and should go to droids-crawler