Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

Satyr · Mar 3, 2023

Some files are uploaded on multiple hosts like bunkr, gofile, saint, all in a single post. Is there anyway to avoid downloading the duplicate files on different hosts or download from only one filehost per post?

Jules--Winnfield · Mar 3, 2023

Satyr said:
Please, Log in or Register to view quotes

What I've been doing to combat this on my own end is changing both remove_bunkr_identifier and block_sub_folders to true in the config (with apply config on). Files will then have the same name, and all look ups for existing files are in the same directory. It cuts down on this very heavily.

(the look ups are based on filesize and name)

But other than that, no.

mynameiscute · Mar 3, 2023

when scraping this thread

https://simpcity.su/threads/eunji-pyoapple.10207/page-7#post-1633437

the gofile links do not get scraped, any idea why?

Jules--Winnfield · Mar 3, 2023

mynameiscute said:
Please, Log in or Register to view quotes

Because whoever posted them put them in quote blocks, like me quoting your message above. The program ignores quote blocks so that it doesn't scrape the same url over and over when someone quotes someone.

alibali123123 · Mar 3, 2023

Please, Log in or Register to see links and images

I just redownloaded python and cyberdrop-dl, and now i'm getting this error when i try to download any links

Jules--Winnfield · Mar 3, 2023

alibali123123 said:
Please, Log in or Register to view quotes

As the error would suggest, your database file is locked (due to it being open, or corrupted, etc). try restarting your computer.

mynameiscute · Mar 3, 2023

Jules--Winnfield said:
Please, Log in or Register to view quotes

is there a way to turn that off? since i probably missed a few things because of that and only just now noticed.

if not then i guess i need to go through every thread again and look if it did not miss any quoted stuff

Jules--Winnfield · Mar 4, 2023

mynameiscute said:
Please, Log in or Register to view quotes

There is not

Meepmoop · Mar 4, 2023

Has anyone tried v4 on mac? i've followed the instructions, did chmod +x "Start Mac.command". but i get

“Start Mac.command” cannot be opened because it is from an unidentified developer.

Not too sure what else to do.

Jules--Winnfield · Mar 4, 2023

Meepmoop said:
Please, Log in or Register to view quotes

Please, Log in or Register to see links and images

BABASS222 · Mar 4, 2023

Please, Log in or Register to see links and images

I've been having this problem lately with almost all types of bunkr links. Whole albums dont work, and now even specific file URLs show me this message. I can download the file normally from the browser without any password or anything too. I've reinstalled and updated to the newest version of the program from Github too.

The downloader log is:

2023-03-04 10:08:40,970:DEBUG:selector_events:selector_events.py:54:Using selector: KqueueSelector
2023-03-04 10:08:40,977:DEBUG:base_functions:base_functions.py:43:We are running version 3.3.29 of Cyberdrop Downloader
2023-03-04 10:08:40,977:DEBUG:main:main.py:64:Starting downloader with args: {'input_file': PosixPath('URLs.txt'), 'output_folder': PosixPath('Downloads'), 'log_file': 'downloader.log', 'db_file': 'download_history.sqlite', 'threads': 0, 'attempts': 10, 'connection_timeout': 15, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'output_last_forum_post': False, 'proxy': None, 'separate_posts': False, 'xbunker_username': None, 'xbunker_password': '!REDACTED!', '*Blacklisted site*_username': None, '*Blacklisted site*_password': '!REDACTED!', 'simpcity_username': None, 'simpcity_password': '!REDACTED!', 'skip_hosts': [], 'ratelimit': 50, 'throttle': 0.5, 'links': []}
2023-03-04 10:08:40,982:DEBUG:base_functions:base_functions.py:43:Starting Scrape
2023-03-04 10:08:40,982:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Mommy-Gets-Disturbed-FfNkx2vT.mp4
2023-03-04 10:08:40,983:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sister-Fucks-your-dog-QNH8y28o.mp4
2023-03-04 10:08:40,983:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/sister-fucks-you-next-to-wife-HHWfR7Ni.mp4
2023-03-04 10:08:40,983:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sydney-Harwin-Fucking-Mommy-Next-Dad-mp4-ZQJqfipw.mp4
2023-03-04 10:08:40,983:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sydney-Harwin----Punishment-Pegged-By-Mom-4rv6B0YY.mp4
2023-03-04 10:08:40,984:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sydney-Harwin-Joes-House-mp4-DKaAsT75.mp4
2023-03-04 10:08:40,984:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sydney-Harwin-Babysitter-Lets-You-Put-mp4-PX163wzn.mp4
2023-03-04 10:08:40,984:DEBUG:base_functions:base_functions.py:43:Starting scrape of https://bunkrr.su/v/Sydney-Harwin-Wife-Fucking-Perv-mp4-UPDvG9CT.mp4
2023-03-04 10:08:41,684:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sydney-Harwin-Fucking-Mommy-Next-Dad-mp4-ZQJqfipw.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,684:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sydney-Harwin-Fucking-Mommy-Next-Dad-mp4-ZQJqfipw.mp4
2023-03-04 10:08:41,684:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,684:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sydney-Harwin-Fucking-Mommy-Next-Dad-mp4-ZQJqfipw.mp4
2023-03-04 10:08:41,687:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/sister-fucks-you-next-to-wife-HHWfR7Ni.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,687:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/sister-fucks-you-next-to-wife-HHWfR7Ni.mp4
2023-03-04 10:08:41,687:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,687:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/sister-fucks-you-next-to-wife-HHWfR7Ni.mp4
2023-03-04 10:08:41,690:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sydney-Harwin-Joes-House-mp4-DKaAsT75.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,690:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sydney-Harwin-Joes-House-mp4-DKaAsT75.mp4
2023-03-04 10:08:41,690:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,691:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sydney-Harwin-Joes-House-mp4-DKaAsT75.mp4
2023-03-04 10:08:41,730:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Mommy-Gets-Disturbed-FfNkx2vT.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,730:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Mommy-Gets-Disturbed-FfNkx2vT.mp4
2023-03-04 10:08:41,730:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,730:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Mommy-Gets-Disturbed-FfNkx2vT.mp4
2023-03-04 10:08:41,763:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sydney-Harwin----Punishment-Pegged-By-Mom-4rv6B0YY.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,763:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sydney-Harwin----Punishment-Pegged-By-Mom-4rv6B0YY.mp4
2023-03-04 10:08:41,763:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,763:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sydney-Harwin----Punishment-Pegged-By-Mom-4rv6B0YY.mp4
2023-03-04 10:08:41,765:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sydney-Harwin-Wife-Fucking-Perv-mp4-UPDvG9CT.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,766:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sydney-Harwin-Wife-Fucking-Perv-mp4-UPDvG9CT.mp4
2023-03-04 10:08:41,766:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,766:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sydney-Harwin-Wife-Fucking-Perv-mp4-UPDvG9CT.mp4
2023-03-04 10:08:41,768:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sister-Fucks-your-dog-QNH8y28o.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,768:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sister-Fucks-your-dog-QNH8y28o.mp4
2023-03-04 10:08:41,768:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,768:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sister-Fucks-your-dog-QNH8y28o.mp4
2023-03-04 10:08:41,770:DEBUG:Bunkr_Spider:Bunkr_Spider.py:54:Error encountered while handling https://bunkrr.su/v/Sydney-Harwin-Babysitter-Lets-You-Put-mp4-PX163wzn.mp4
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
    build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,771:DEBUG:base_functions:base_functions.py:43:Error scraping https://bunkrr.su/v/Sydney-Harwin-Babysitter-Lets-You-Put-mp4-PX163wzn.mp4
2023-03-04 10:08:41,771:DEBUG:Bunkr_Spider:Bunkr_Spider.py:56:'NoneType' object has no attribute 'get_text'
2023-03-04 10:08:41,771:DEBUG:base_functions:base_functions.py:43:Finished scrape of https://bunkrr.su/v/Sydney-Harwin-Babysitter-Lets-You-Put-mp4-PX163wzn.mp4
2023-03-04 10:08:41,771:ERROR:main:main.py:107:ValueError No links
2023-03-04 10:08:41,771:DEBUG:base_functions:base_functions.py:43:No links found duing scraping, check passwords or that the urls are accessible
2023-03-04 10:08:41,771:DEBUG:base_functions:base_functions.py:43:This program does not currently support password protected albums.
2023-03-04 10:08:41,772:DEBUG:sql_helper:sql_helper.py:93:Successfully closed sqlite database connection

Anything I can do to fix this? Thanks for your time.

Jules--Winnfield · Mar 4, 2023

BABASS222 said:
Please, Log in or Register to view quotes

Pretty simply, you need to update cyberdrop-dl. We are on version 4.0.65. You are hundreds of iterations behind at this point.

https://github.com/Jules-WinnfieldX/CyberDropDownloader/wiki/Frequently-Asked-Questions#how-to-update

BABASS222 · Mar 4, 2023

Jules--Winnfield said:
Please, Log in or Register to view quotes

Oh ok, sorry then. But I re-downloaded cyberdrop-dl from the github release just a few days ago. And the folder version I have says the CyberDrop version is V4, and I have the start mac alternate terminal file which you only made in the newer versions. I'll visit the FAQ page you mentioned to see if I need to update something seperatley, and if Im being stupid by just deleteing the old folder and expecting to update it when I replace it with the newer version. Thanks for your contribution and reply.

BABASS222 · Mar 4, 2023

Jules--Winnfield said:
Please, Log in or Register to view quotes

I ran the command line in the page you mentioned. It seems updated now, sorry for being a bit stupid. Still getting used to Mac.

Jules--Winnfield · Mar 4, 2023

BABASS222 said:
Please, Log in or Register to view quotes

We all start somewhere

Jules--Winnfield · Mar 4, 2023

BABASS222 said:
Please, Log in or Register to view quotes

Depending on command line input worked for you will tell you which start file (normal or alt) to keep using to stay up to date.

The normal is pip3, the alternate is pip. If the pip worked, use the alternate file.

morat2004 · Mar 4, 2023

can anyone explain to me what am i doing wrong, i tried to use this program for some time now, but every time i open the "start windows.bat" it says "press any key to continue" and then just close and doing nothing, i tried redownloading several times and i followed the instructions in the first page but it still doesn't seems to work, is there anything i missed?

shareholder · Mar 4, 2023

morat2004 said:
Please, Log in or Register to view quotes

paste links in URLs.txt
open Command Prompt, type cyberdrop-dl

Jules--Winnfield · Mar 4, 2023

morat2004 said:
Please, Log in or Register to view quotes

It should say somthing before the "press any key to continue". Post a screenshot / post the downloader.log

Sanfour · Mar 5, 2023

Hello, can you please guide me as I can't find the right way to only download post and not the whole thread? Thanks

Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

The GOAT

Cyberdrop-DL Creator

Casual

Cyberdrop-DL Creator

Bathwater Drinker

Cyberdrop-DL Creator

Casual

Cyberdrop-DL Creator

Bathwater Drinker

Cyberdrop-DL Creator

Lurker

Cyberdrop-DL Creator

Lurker

Lurker

Cyberdrop-DL Creator

Cyberdrop-DL Creator

Casual

Bathwater Drinker

Cyberdrop-DL Creator

Tier 3 Sub