Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[Jules--Winnfield]

Please, Log in or Register to view quotes

got an example?

 

[wolfkenx]

Please, Log in or Register to view quotes

No problem. I appreciate the reply.

 

[User9876543210x]

I am on version 4.0.6 (probably before and now tested on 4.0.21, Jesus you updated a lot in a day) I was getting some GoFile errors during the scraping part. But manually checking the links immediately after getting the error in firefox allowed me to download them without issue, the error'd links also have 'finished' GoFile scrapes for other URLs before and after them and its the same URLs that are always problematic even after restarting it, so im not sure what the issue is with those specific URLs. This is a snippet of the log that should be relevant as it contains a couple of the error'd links.

Please, Log in or Register to view codes content!

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

ClientResponseError: 429

You are getting HTTP status 429, which means: Too Many Requests (you are sending too many requests to gofile).

I should limit it on my side, but are you scraping a forum thread? or just a URLs.txt file full of gofile links?

 

[User9876543210x]

Please, Log in or Register to view quotes

No forum scraping. Just a large URLs folder, 23 URLs for GoFile and a lot more Bunkr. But its weird to me that its always those specific URLs, and doesnt have an issue with GoFile URLs after them, or if I cancel and restart it as soon as I get to the error'd ones, it doesnt error any of the GoFile URLs higher up the list, and the site has no problem letting me access and download them immediately after the error.

--- Just cleared out my URL list, and only added a couple of those problematic URLs, it scrapes them just fine now, but also says they were skipped, and checking my mess of a download folder, they were indeed already downloaded.

TLDR; Guess my problem is simply having too many GoFile URLs in URLs.txt

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Still, I'm gonna manually ratelimit scraping to gofile to help with that.

 

[dsffadsfadsfdasdfsaadf]

is there a way to disable the new gui its using a lot of my cpu

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Please, Log in or Register to see links and images

hide_new_progress does that. Though I very much doubt that's what is using your CPU. If anything, it'd likely be more beneficial to reduce the simultaneous_downloads_per_domain

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I'm about to put out 4.0.25, try it first and see if it improves.

 

[ElGuapito99]

Is there a quick way to see all URLs that failed to download?

 

[whoknows10]

Please, Log in or Register to view quotes

This worked beautifully! Kick-ass new look to the program now that I have it up and running. Thank you so much! You're the best.

 

[Shaggy12777]

hey Jules--Winnfield just wondring if there is a fix to the stalling problem, it get to most of the thread and then stops an it trys to start up but wont move.
thanks for the hard work you put in this!!! keep it up!

 

[whoknows10]

Please, Log in or Register to view quotes

Just realized this is happening to me as well. Tried to rip a thread to test the new gui, and it got most of the way through and has just stopped. Tried changing the rate limit and such, but sadly nothing has worked. Again thanks for all of your work on this!

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Please, Log in or Register to see links and images

by turning on output_errored_urls

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

what thread, let me see the downloader.log

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

can I see the downloader.log

 

[whoknows10]

Please, Log in or Register to view quotes

I actually think it may be still downloading, but it's just really struggling with Loose ShareX files. I'm going to let it run for a bit and see if it's just that before bothering you any longer! Haha I'll post the downloader log if I notice that it's just truly hung up! Sorry for potentially jumping the gun! Haha

 

[Shaggy12777]

2023-02-16 09:02:08,990:DEBUG:base_functions:base_functions.py:46:[red]Error:

Please, Log in or Register to see links and images

2023-02-16 09:02:08,990:DEBUG:Bunkr_Spider:Bunkr_Spider.py:164:404, message='Not Found', url=URL('

Please, Log in or Register to see links and images

')
2023-02-16 09:02:09,123:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,127:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,128:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,129:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,133:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,136:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,298:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,306:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,399:DEBUG:GoFile_Spider:GoFile_Spider.py:76:Error encountered while handling

Please, Log in or Register to see links and images

NoneType: None
2023-02-16 09:02:09,399:DEBUG:base_functions:base_functions.py:46:[red]Error:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,400:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,645:DEBUG:GoFile_Spider:GoFile_Spider.py:76:Error encountered while handling

Please, Log in or Register to see links and images

NoneType: None
2023-02-16 09:02:09,645:DEBUG:base_functions:base_functions.py:46:[red]Error:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,646:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,653:DEBUG:GoFile_Spider:GoFile_Spider.py:76:Error encountered while handling

Please, Log in or Register to see links and images

NoneType: None
2023-02-16 09:02:09,653:DEBUG:base_functions:base_functions.py:46:[red]Error:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,654:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,757:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,758:DEBUG:GoFile_Spider:GoFile_Spider.py:76:Error encountered while handling

Please, Log in or Register to see links and images

NoneType: None
2023-02-16 09:02:09,758:DEBUG:base_functions:base_functions.py:46:[red]Error:

Please, Log in or Register to see links and images

2023-02-16 09:02:09,758:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:10,738:DEBUG:base_functions:base_functions.py:46:[green]Starting:

Please, Log in or Register to see links and images

2023-02-16 09:02:10,739:DEBUG:base_functions:base_functions.py:46:[yellow]Not Supported:

Please, Log in or Register to see links and images

2023-02-16 09:02:10,976:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:10,984:DEBUG:base_functions:base_functions.py:46:[green]Finished:

Please, Log in or Register to see links and images

2023-02-16 09:02:10,985:DEBUG:base_functions:base_functions.py:46:[green]Finished Scrape[/green]
2023-02-16 09:02:16,577:DEBUG:base_functions:base_functions.py:46:Completed Download: xdIz7eeY.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:17,019:DEBUG:base_functions:base_functions.py:46:Completed Download: HTK123f2-pVVxjQbd.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:17,252:DEBUG:base_functions:base_functions.py:46:Completed Download: bSKsssKO.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:17,611:DEBUG:base_functions:base_functions.py:46:Completed Download: oXMoCXNv.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:17,852:DEBUG:base_functions:base_functions.py:46:Completed Download: 0gyvvimggzffjqu30k62d_source.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:18,235:DEBUG:base_functions:base_functions.py:46:Completed Download: ODDNU2lq.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:18,560:DEBUG:base_functions:base_functions.py:46:Completed Download: RESDjhuvh1-hX9O74zZ.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:18,960:DEBUG:base_functions:base_functions.py:46:Completed Download: 9oL8ARen.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:19,609:DEBUG:base_functions:base_functions.py:46:Completed Download: 28kOpXQV.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:19,952:DEBUG:base_functions:base_functions.py:46:Completed Download: AEhDj41f-qg0Pdj3k.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:20,251:DEBUG:base_functions:base_functions.py:46:Completed Download: rAsVhX4Y_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:20,701:DEBUG:base_functions:base_functions.py:46:Completed Download: LAkrMifj.jpeg from

Please, Log in or Register to see links and images

2023-02-16 09:02:21,009:DEBUG:base_functions:base_functions.py:46:Completed Download: wQfh61oS_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:21,352:DEBUG:base_functions:base_functions.py:46:Completed Download: 0REach-RZ8h5NgH.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:21,684:DEBUG:base_functions:base_functions.py:46:Completed Download: ukWkYAdJ_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:22,467:DEBUG:base_functions:base_functions.py:46:Completed Download: 67rpeb4n_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:22,759:DEBUG:base_functions:base_functions.py:46:Completed Download: H41jk2-ehmXP2X3.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:23,075:DEBUG:base_functions:base_functions.py:46:Completed Download: ASD512-ib7B8wkb.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:23,412:DEBUG:base_functions:base_functions.py:46:Completed Download: tYJzK5Rk_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:24,154:DEBUG:base_functions:base_functions.py:46:Completed Download: N322SYD8_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:24,492:DEBUG:base_functions:base_functions.py:46:Completed Download: oakleymiddleton18---fucked-in-the-kitchen-s8ZgJkZt.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:25,250:DEBUG:base_functions:base_functions.py:46:Completed Download: Oakleymiddleton18-bg-anal-6NHlLKba.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:25,758:DEBUG:base_functions:base_functions.py:46:Completed Download: qF2G1Vic_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:26,241:DEBUG:base_functions:base_functions.py:46:Completed Download: oakleymiddleton18-bg-fingering-T7N3LB1X.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:26,900:DEBUG:base_functions:base_functions.py:46:Completed Download: eZKes6pd_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:27,208:DEBUG:base_functions:base_functions.py:46:Completed Download: YtFX3rRF_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:27,541:DEBUG:base_functions:base_functions.py:46:Completed Download: GGeWk99c_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:28,251:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:28,251:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:28,252:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:28,252:DEBUG:downloaders:downloaders.py:49:Retrying (0)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:28,582:DEBUG:base_functions:base_functions.py:46:Completed Download: poN6o16E_720p.mp4 from

Please, Log in or Register to see links and images

2023-02-16 09:02:28,996:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:28,996:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:28,996:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:28,996:DEBUG:downloaders:downloaders.py:49:Retrying (1)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:29,601:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:29,601:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:29,601:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:29,601:DEBUG:downloaders:downloaders.py:49:Retrying (2)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:30,189:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:30,189:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:30,189:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:30,189:DEBUG:downloaders:downloaders.py:49:Retrying (3)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:30,775:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:30,775:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:30,775:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:30,775:DEBUG:downloaders:downloaders.py:49:Retrying (4)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:31,367:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:31,367:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:31,367:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:31,367:DEBUG:downloaders:downloaders.py:49:Retrying (5)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:31,911:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:31,911:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:31,911:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:31,911:DEBUG:downloaders:downloaders.py:49:Retrying (6)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:32,469:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:32,469:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:32,469:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:32,469:DEBUG:downloaders:downloaders.py:49:Retrying (7)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:32,995:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:32,995:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:32,995:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:32,995:DEBUG:downloaders:downloaders.py:49:Retrying (8)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:33,523:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:33,523:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:33,523:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:33,523:DEBUG:downloaders:downloaders.py:49:Retrying (9)

Please, Log in or Register to see links and images

...
2023-02-16 09:02:34,085:DEBUG:downloaders:downloaders.py:265:

Please, Log in or Register to see links and images

(Bunkr under maintenance)
2023-02-16 09:02:34,085:DEBUG:downloaders:downloaders.py:278:Error status code: 503
2023-02-16 09:02:34,085:DEBUG:downloaders:downloaders.py:48:DownloadFailure(503)
2023-02-16 09:02:34,085:DEBUG:downloaders:downloaders.py:49:Retrying (10)

Please, Log in or Register to see links and images

...

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

All good. During scraping, it'll look like it freezes when working through Coomer/Kemono archives. Downloading, downloads popping in and out of the files section are downloads that are failing for whatever reason

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Doesn't seem like anything is wrong here, you're just running into files that you can't access right now.