Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[Jules--Winnfield]

Please, Log in or Register to view quotes

Me forgetting I said I'd do things. I wonder how many of those there has been. After I chill out for a bit I'll start on that later. You got an example link I can test with, so we're on the same page

 

[Cutelo]

Please, Log in or Register to view quotes

No problem, you do a lot already.

Please, Log in or Register to see links and images

Please, Log in or Register to see links and images

Please, Log in or Register to see links and images

 

[Frankorion]

Please, Log in or Register to view quotes

Here are the logs:

2022-05-06 02:29:25,773:DEBUG:proactor_events:proactor_events.py:626:Using proactor: IocpProactor
2022-05-06 02:29:25,807:DEBUG:base_functions:base_functions.py:47:We are running version 2.17.7 of Cyberdrop Downloader
2022-05-06 02:29:25,808:DEBUG:main:main.py:49:Starting downloader with args: {'input_file': WindowsPath('URLs.txt'), 'output_folder': WindowsPath('Downloads'), 'log_file': 'downloader.log', 'db_file': 'download_history_size_based.sqlite', 'threads': 0, 'attempts': 10, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'SimpCity_username': None, 'SimpCity_password': None, 'cyberfile_username': None, 'cyberfile_password': None, 'links': []}
2022-05-06 02:29:25,809:DEBUG:base_functions:base_functions.py:47:Starting Scrape
2022-05-06 02:29:25,810:DEBUG:base_functions:base_functions.py:47:Starting scrape of

Please, Log in or Register to see links and images

2022-05-06 02:29:26,368:DEBUG:ShareX_Spider:ShareX_Spider.py:128:Error encountered while handling

Please, Log in or Register to see links and images

Traceback (most recent call last):
File "C:\Users\locke\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\utils\crawlers\ShareX_Spider.py", line 110, in get_list_links
async with session.get(url, ssl=ssl_context) as response:
File "C:\Users\locke\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 1138, in __aenter__
self._resp = await self._coro
File "C:\Users\locke\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 640, in _request
resp.raise_for_status()
File "C:\Users\locke\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client_reqrep.py", line 1004, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 503, message='Service Unavailable', url=URL('

Please, Log in or Register to see links and images

')
2022-05-06 02:29:26,369:DEBUG:ShareX_Spider:ShareX_Spider.py:129:503, message='Service Unavailable', url=URL('

Please, Log in or Register to see links and images

')
2022-05-06 02:29:26,369:DEBUG:base_functions:base_functions.py:47:Finished scrape of

Please, Log in or Register to see links and images

2022-05-06 02:29:26,370:ERROR:main:main.py:68:ValueError No links
2022-05-06 02:29:26,370:DEBUG:base_functions:base_functions.py:47:No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me.
2022-05-06 02:29:26,371:DEBUG:base_functions:base_functions.py:47:This program does not currently support password protected albums.
2022-05-06 02:29:26,371:DEBUG:sql_helper:sql_helper.py:52:Successfully closed sqlite database connection

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I quoted the relevant message, you are getting a 503 error code, "service unavailable".

"503 Service Unavailable server error response code indicates that the server is not ready to handle the request. Common causes are a server that is down for maintenance or that is overloaded."

Essentially, it's just server issues you are facing.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

So. That was easy. 2.18.5 is out with Coomer support.

 

[Jules--Winnfield]

Would be good if I actually included the file in the git commit. 2.18.6 coming soon that'll actually work.

edit: it's out.

 

[Cutelo]

Please, Log in or Register to view quotes

You are a life saver my friend.
Thanks a lot!

 

[Frankorion]

Please, Log in or Register to view quotes

Ah, okay, thanks.
Kinda weird because all the links are working just fine in the browser - no delay etc. Might be some internal server stuff idk.

 

[ppbizonrekt]

hi, sorry if this question has been answered before, but i'm having a problem with bunkr links not downloading. I used the same link on my browser and it shows the album, but when I input it to the .txt file and start the downloader, it just says scraping [link], finished scraping [link] and proceeds to say link not found. thanks in advance.

Below is the log of the downloader

2022-05-08 02:52:51,739:DEBUG:proactor_events:proactor_events.py:626:Using proactor: IocpProactor
2022-05-08 02:52:51,797:DEBUG:base_functions:base_functions.py:47:We are running version 2.18.6 of Cyberdrop Downloader
2022-05-08 02:52:51,798:DEBUG:main:main.py:49:Starting downloader with args: {'input_file': WindowsPath('URLs.txt'), 'output_folder': WindowsPath('Downloads'), 'log_file': 'downloader.log', 'db_file': 'download_history_size_based.sqlite', 'threads': 0, 'attempts': 10, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'SimpCity_username': None, 'SimpCity_password': None, 'cyberfile_username': None, 'cyberfile_password': None, 'links': []}
2022-05-08 02:52:51,800:DEBUG:base_functions:base_functions.py:47:Starting Scrape
2022-05-08 02:52:51,801:DEBUG:base_functions:base_functions.py:47:Starting scrape of

Please, Log in or Register to see links and images

2022-05-08 02:52:51,907:DEBUG:Chibisafe_Spider:Chibisafe_Spider.py:48:Error encountered while handling

Please, Log in or Register to see links and images

Traceback (most recent call last):
File "C:\Users\jimmy\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\utils\crawlers\Chibisafe_Spider.py", line 27, in fetch
async with session.get(url, ssl=ssl_context) as response:
File "C:\Users\jimmy\AppData\Roaming\Python\Python310\site-packages\aiohttp\client.py", line 1138, in __aenter__
self._resp = await self._coro
File "C:\Users\jimmy\AppData\Roaming\Python\Python310\site-packages\aiohttp\client.py", line 640, in _request
resp.raise_for_status()
File "C:\Users\jimmy\AppData\Roaming\Python\Python310\site-packages\aiohttp\client_reqrep.py", line 1004, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('

Please, Log in or Register to see links and images

')
2022-05-08 02:52:51,909:DEBUG:Chibisafe_Spider:Chibisafe_Spider.py:49:403, message='Forbidden', url=URL('

Please, Log in or Register to see links and images

')
2022-05-08 02:52:51,909:DEBUG:base_functions:base_functions.py:47:Finished scrape of

Please, Log in or Register to see links and images

2022-05-08 02:52:51,911:ERROR:main:main.py:68:ValueError No links
2022-05-08 02:52:51,911:DEBUG:base_functions:base_functions.py:47:No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me.
2022-05-08 02:52:51,911:DEBUG:base_functions:base_functions.py:47:This program does not currently support password protected albums.
2022-05-08 02:52:51,912:DEBUG:sql_helper:sql_helper.py:52:Successfully closed sqlite database connection

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I've quoted the portion that's important. You are getting an HTTP status code 403: Forbidden.

"HTTP 403 is an HTTP status code meaning access to the requested resource is forbidden. The server understood the request, but will not fulfill it."

I'm not sure why it would work in your browser, but not in the downloader, unless your browser is using a VPN by default. Unfortunately that's about all I can tell you.

 

[Jules--Winnfield]

GoFile is currently broken, I'm aware. Not sure if what they did is a permanent change or temporary change/Outage. I'll look into it further in the morning

 

[ppbizonrekt]

Please, Log in or Register to view quotes

ah i understand, yeah it might be because my browser is using VPN by default. Thanks for the answer.

 

[MagicBarca]

how can i download?there is an album and the site does not allow registration or download.how can i download an album from there?(cyberdrop)

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

read the first post in this thread

 

[DoofusMan]

Hey there Jules--Winnfield ! How are ya?

I'm having problem downloading a cyberfile.su link. It's the first time trying and god damn I'm so glad just by the fact you even considered adding it. It's such a pain to download each file one by one, only 6 at the same time and can't quickly open multiple tabs so you keep having to go back and forth X.X ...

So yeah was happy to see that you added support, but it don't work for me

This exact URL:

Please, Log in or Register to see links and images

Is the first one I try as of now.

Just says it doesn't recognize any URL in my txt file (yes I saved it after adding the link hahaha)

Is the link a format not supported? Or is it not just me and a new issue? If it's the format, will I have to go in each file and pick each URL? x.x

As always, huge fan of your work. Can't start even counting the hours you saved me (also just saw you added support for coomer party, and I,m glad cuz I fucking hate the layout of their site and never bothered with them unless it was direct links to their files lmao.)

----------------------------------

EDIT:

2022-05-08 12:51:48,571:DEBUG:proactor_events:proactor_events.py:625:Using proactor: IocpProactor
2022-05-08 12:51:48,631:DEBUG:base_functions:base_functions.py:47:We are running version 2.18.6 of Cyberdrop Downloader
2022-05-08 12:51:48,632:DEBUG:main:main.py:49:Starting downloader with args: {'input_file': WindowsPath('URLs.txt'), 'output_folder': WindowsPath('REDACTED'), 'log_file': 'downloader.log', 'db_file': 'download_history_size_based.sqlite', 'threads': 0, 'attempts': 10, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'SimpCity_username': None, 'SimpCity_password': None, 'cyberfile_username': None, 'cyberfile_password': None, 'links': []}
2022-05-08 12:51:48,637:DEBUG:base_functions:base_functions.py:47:Starting Scrape
2022-05-08 12:51:48,640:DEBUG:base_functions:base_functions.py:47:Not logged into Cyberfile, skipping

Please, Log in or Register to see links and images

2022-05-08 12:51:48,641:ERROR:main:main.py:68:ValueError No links
2022-05-08 12:51:48,641:DEBUG:base_functions:base_functions.py:47:No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me.
2022-05-08 12:51:48,641:DEBUG:base_functions:base_functions.py:47:This program does not currently support password protected albums.
2022-05-08 12:51:48,642:DEBUG:sql_helper:sql_helper.py:52:Successfully closed sqlite database connection

--------------

Wait just thought of checking this out (the log). It skipped it because I'm not logged in? Do I just have to be logged in the website itself and it's detect that? (Just tested that out and that didn't work as suspected ) Or do I have to enter a parameter? I looked at your instructions earlier but didn't see any mention of it though o.o"

==============

EDIT#2: Quick question also on the side! Regarding the history of files downloaded previously from a certain link (thank you soooo much for that omfg) is there any precaution to take to make sure it saves what has been downloaded properly? Like, as soon as it downloaded a file, does it add it to the database/history? Or it has to finish downloading everything from an URL before it registers up to which file it has downloaded already? By that I mean; if I close the process while it's downloading and restart it, move the files away that it just downloaded and then start it again, will it remember that it downloaded those files I just changed location? Or will it download them again?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Doing good, just busy.

It is because you aren't logged in. I added parameters: --cyberfile-username and --cyberfile-password (I forgot to put it in the post/readme). I'm currently using their backend api which requires a login. This might change in the near future, but it'd be moving to selenium most likely and would make chrome a requirement again.

Per your second edit, as soon as a file is properly saved (no longer a .part file) it is saved to the database as downloaded. As well as if the downloader finds the already downloaded file in the directory, it is saved to the DB.

 

[P7w7P]

Could you explain how exactly you download files from the cyberdrop

 

[DoofusMan]

Please, Log in or Register to view quotes

The first post has literally a guide and link to the github that has also the guide itself

 

[DoofusMan]

Please, Log in or Register to view quotes

Ahhh oki thats why haha. I figured. perfect then! I love it that way. I just have to edit the start batch file with the new parameters. I prefer that than fucking around with chrome again.

Ahhh perfect, exactly like I'd hoped! amazing! Thanks for the reply and explanation Jules! Until next time, take care and have a good one!

 

[DoofusMan]

Please, Log in or Register to view quotes

Well poop...I used the parameters and tried with and without having the ones for thotbay at the same time and it gives the same error nonetheless.

"Traceback (most recent call last):
File "C:\Users\damie\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\damie\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\damie\AppData\Roaming\Python\Python310\Scripts\cyberdrop-dl.exe\__main__.py", line 7, in <module>
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\main.py", line 104, in main
asyncio.get_event_loop().run_until_complete(download_all(args))
File "C:\Users\damie\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
return future.result()
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\main.py", line 66, in download_all
content_object = await scrape(links, args.include_id, SimpCity_auth, cyberfile_auth, args.separate_posts)
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\utils\scraper.py", line 22, in scrape
await asyncio.gather(*tasks)
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\utils\scraper_helper.py", line 140, in map_url
await value(url=url_to_map, title=title)
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\utils\scraper_helper.py", line 124, in cyberfile
domain_obj = await self.cyberfile_crawler.fetch(url)
File "C:\Users\damie\AppData\Roaming\Python\Python310\site-packages\cyberdrop_dl\utils\crawlers\Cyberfile_Spider.py", line 40, in fetch
results.extend(await self.get_folder(url))
TypeError: 'NoneType' object is not iterable"

-----------------------------------

and in the log:

2022-05-08 19:57:45,871:DEBUG:proactor_events:proactor_events.py:625:Using proactor: IocpProactor
2022-05-08 19:57:45,927:DEBUG:base_functions:base_functions.py:47:We are running version 2.18.6 of Cyberdrop Downloader
2022-05-08 19:57:45,928:DEBUG:main:main.py:49:Starting downloader with args: {'input_file': WindowsPath('URLs.txt'), 'output_folder': WindowsPath('REDACTED'), 'log_file': 'downloader.log', 'db_file': 'download_history_size_based.sqlite', 'threads': 0, 'attempts': 10, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'SimpCity_username': None, 'SimpCity_password': None, 'cyberfile_username': 'REDACTED', 'cyberfile_password': 'REDACTED', 'links': []}
2022-05-08 19:57:45,929:DEBUG:base_functions:base_functions.py:47:Starting Scrape
2022-05-08 19:57:45,931:DEBUG:connectionpool:connectionpool.py:971:Starting new HTTPS connection (1): cyberfile.su:443
2022-05-08 19:57:46,228:DEBUG:connectionpool:connectionpool.py:452:

Please, Log in or Register to see links and images

"GET /api/v2/authorize?username=REDACTED&password=REDACTEDHTTP/1.1" 200 None
2022-05-08 19:57:46,230:DEBUG:base_functions:base_functions.py:47:Starting scrape of

Please, Log in or Register to see links and images

2022-05-08 19:57:46,234:DEBUG:connectionpool:connectionpool.py:971:Starting new HTTPS connection (1): cyberfile.su:443
2022-05-08 19:57:46,426:DEBUG:connectionpool:connectionpool.py:452:

Please, Log in or Register to see links and images

"GET /api/v2/folder/listing?access_token=REDACTED&account_id=REDACTED&parent_folder_id=ce1d2b9f205434d3e35c6ab574d51e3d HTTP/1.1" 200 None
2022-05-08 19:57:46,434:DEBUG:sql_helper:sql_helper.py:52:Successfully closed sqlite database connection