How to wrap the process of creating start_urls in scrapy?

Content Index :

How to wrap the process of creating start_urls in scrapy?
Scrapy Getting Start_Urls
start_urls in Scrapy
Scrapy: How to set scrapy start_urls from a setting file?
Python scrapy start_urls
Scrapy start_urls

How to wrap the process of creating start_urls in scrapy?

Tag : python , By : Guyou

Date : November 25 2020, 03:01 PM

it helps some times This is what start_requests method of spider is for. It serves the purpose of creating initial set of requests. Building on your example, it would read as:

class TestSpider(scrapy.Spider):
    def __init__(self, *args, **kw):
        self.timeout = 10

    def start_requests(self):
        url_nasdaq = "ftp://ftp.nasdaqtrader.com/SymbolDirectory/nasdaqlisted.txt"
        s = urllib.request.urlopen(url_nasdaq).read().decode('ascii')
        s1 = s.split('\r\n')[1:-2]
        namelist = []
        for item in s1:
            if "NASDAQ TEST STOCK" not in item : namelist.append(item)
        s2 = [s.split('|')[0] for s in namelist] 
        s3=[]
        for symbol in s2:
            if  "." not in symbol : s3.append(symbol)
        for s in s3:
            yield scrapy.Request("https://finance.yahoo.com/quote/"+s+"/financials?p="+s, callback=self.parse)

Comments

No Comments Right Now !

Boards Message :

You Must Login Or Sign Up to Add Your Comments .

Share :

Tags :

How to wrap the process of creating start_urls in scrapy , Scrapy: How to set scrapy start_urls from a setting file , Scrapy Getting Start_Urls , Scrapy start_urls , scrapy example ,

Scrapy Getting Start_Urls

Tag : python , By : Mena

Date : March 29 2020, 07:55 AM

To fix this issue item['link'], as opposed to item['title'], is just a string, not a list:

self.cursor.execute("INSERT INTO items (title, url) VALUES (%s, %s)",
                    (item['title'][0], item['link']))

start_urls in Scrapy

Tag : python , By : user119413

Date : March 29 2020, 07:55 AM

it should still fix some issue What you could do is set the start_urls to the main page then based on the number of pages shown in the footer pagination (in this case 3), use a loop to create a yield Request for each of the pages:

allowed_domains = ["go-on.fi"]
start_urls = ["http://www.go-on.fi/tyopaikat"]

def parse(self, response):
    pages = response.xpath('//ul[@class="pagination"][last()-1]/a/text()').extract()
    page = 1
    start = 0
    while page <= pages:
        url = "http://www.go-on.fi/tyopaikat?start="+str(start)
        start += 20
        page += 1
        yield Request(url, callback=self.parse_page)

def parse_page(self,response):
    hxs = HtmlXPathSelector(response)
    items = []
    titles = hxs.select("//tr")

    for row in titles:
        item = JobData()
        item['header'] = row.select("./td[1]/a/text()").extract()
        item['link'] = row.select("./td[1]/a/@href").extract()
        items.append(item)

Scrapy: How to set scrapy start_urls from a setting file?

Tag : development , By : Stephen Judge

Date : March 29 2020, 07:55 AM

it helps some times Let's say you put your config files inside the spiders directory and config dir. So overall path scrapy_project -> spiders -> configs-> .txt
Then you can override init of your spiders to populate your start_urls something like this.

def __init__(self, *args, **kwargs):
    script_dir = os.path.dirname(__file__)
    abs_file_path = os.path.join(script_dir, "configs/%s.txt" % self.name)
    with open(abs_file_path) as f:
        self.start_urls  = [line.strip() for line in f.readlines()]

Python scrapy start_urls

Tag : python , By : Ambarish Singh

Date : March 29 2020, 07:55 AM

To fix the issue you can do is it possible to do something like below but with multiple url like below? Each link will have about 50 pages to crawl and loop. The current solution is working but only working if I use 1 URL instead of multiple urls. , I recommend to use start_requests for this:

def start_requests(self):
    base_urls = [

        'https://www.xxxxxxx.com.au/home-garden/page-{page_number}/c18397',
        'https://www.xxxxxxx.com.au/automotive/page-{page_number}/c21159',
        'https://www.xxxxxxx.com.au/garden/page-{page_number}/c25449',
    ]

    for page in range(1, 50):
        for base_url in base_urls:
            url = base_url.format( page_number=page )
            yield scrapy.Request( url, callback=self.parse )