Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads
- Thread starter Jules--Winnfield
- Start date
I'm tring to download a pixl album. I put the url in the txt file but it's saying that
"No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me."
Now, the album is not protected by a password but is private and the url works on browser, so can this method download private album? If the answer is yes, what did I do wrong?
"No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me."
Now, the album is not protected by a password but is private and the url works on browser, so can this method download private album? If the answer is yes, what did I do wrong?
Hi Jules,
Been reading through most recent posts to find a solution to this, redownloaded latest version, and tried some of the fixes(including adding my simpcity login/pass to the downloader file) but no luck. I keep getting this on a Mac:
"Finished scrape of
No links found duing scraping, check passwords or that the urls are accessible
This program does not currently support password protected albums."
and this from the log:
2023-02-10 08:14:13,057:DEBUG:selector_events:selector_events.py:54:Using selector: KqueueSelector
2023-02-10 08:14:13,064:DEBUG:base_functions:base_functions.py:41:We are running version 3.3.14 of Cyberdrop Downloader
2023-02-10 08:14:13,065:DEBUG:main:main.py:60:Starting downloader with args: {'input_file': PosixPath('URLs.txt'), 'output_folder': PosixPath('Downloads'), 'log_file': 'downloader.log', 'db_file': 'download_history.sqlite', 'threads': 0, 'attempts': 10, 'connection_timeout': 15, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'leakednudes_username': None, 'leakednudes_password': '!REDACTED!', '*Blacklisted site*_username': None, '*Blacklisted site*_password': '!REDACTED!', 'simpcity_username': 'amirogozhin@protonmail.com', 'simpcity_password': '!REDACTED!', 'skip_hosts': [], 'ratelimit': 50, 'throttle': 0.5, 'links': []}
2023-02-10 08:14:13,089:DEBUG:base_functions:base_functions.py:41:Starting Scrape
2023-02-10 08:14:13,091:DEBUG:base_functions:base_functions.py:41:Starting scrape of
2023-02-10 08:14:13,496:DEBUG:Bunkr_Spider:Bunkr_Spider.py:51:Error encountered while handling
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-02-10 08:14:13,497:DEBUG:base_functions:base_functions.py:41:Error scraping
2023-02-10 08:14:13,498:DEBUG:Bunkr_Spider:Bunkr_Spider.py:53:'NoneType' object has no attribute 'get_text'
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:Finished scrape of
2023-02-10 08:14:13,498:ERROR:main:main.py:95:ValueError No links
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:No links found duing scraping, check passwords or that the urls are accessible
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:This program does not currently support password protected albums.
2023-02-10 08:14:13,499:DEBUG:sql_helper:sql_helper.py:93:Successfully closed sqlite database connection
Been reading through most recent posts to find a solution to this, redownloaded latest version, and tried some of the fixes(including adding my simpcity login/pass to the downloader file) but no luck. I keep getting this on a Mac:
"Finished scrape of
No links found duing scraping, check passwords or that the urls are accessible
This program does not currently support password protected albums."
and this from the log:
2023-02-10 08:14:13,057:DEBUG:selector_events:selector_events.py:54:Using selector: KqueueSelector
2023-02-10 08:14:13,064:DEBUG:base_functions:base_functions.py:41:We are running version 3.3.14 of Cyberdrop Downloader
2023-02-10 08:14:13,065:DEBUG:main:main.py:60:Starting downloader with args: {'input_file': PosixPath('URLs.txt'), 'output_folder': PosixPath('Downloads'), 'log_file': 'downloader.log', 'db_file': 'download_history.sqlite', 'threads': 0, 'attempts': 10, 'connection_timeout': 15, 'disable_attempt_limit': False, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'separate_posts': False, 'leakednudes_username': None, 'leakednudes_password': '!REDACTED!', '*Blacklisted site*_username': None, '*Blacklisted site*_password': '!REDACTED!', 'simpcity_username': 'amirogozhin@protonmail.com', 'simpcity_password': '!REDACTED!', 'skip_hosts': [], 'ratelimit': 50, 'throttle': 0.5, 'links': []}
2023-02-10 08:14:13,089:DEBUG:base_functions:base_functions.py:41:Starting Scrape
2023-02-10 08:14:13,091:DEBUG:base_functions:base_functions.py:41:Starting scrape of
2023-02-10 08:14:13,496:DEBUG:Bunkr_Spider:Bunkr_Spider.py:51:Error encountered while handling
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cyberdrop_dl/crawlers/Bunkr_Spider.py", line 34, in fetch
build_id = json.loads(soup.select_one("script[id=__NEXT_DATA__]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-02-10 08:14:13,497:DEBUG:base_functions:base_functions.py:41:Error scraping
2023-02-10 08:14:13,498:DEBUG:Bunkr_Spider:Bunkr_Spider.py:53:'NoneType' object has no attribute 'get_text'
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:Finished scrape of
2023-02-10 08:14:13,498:ERROR:main:main.py:95:ValueError No links
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:No links found duing scraping, check passwords or that the urls are accessible
2023-02-10 08:14:13,498:DEBUG:base_functions:base_functions.py:41:This program does not currently support password protected albums.
2023-02-10 08:14:13,499:DEBUG:sql_helper:sql_helper.py:93:Successfully closed sqlite database connection
It is currently already supported, you just need to update cyberdrop-dl.
2023-02-10 13:07:52,673:DEBUG:proactor_events:proactor_events.py:625:Using proactor: IocpProactor
2023-02-10 13:07:52,713:DEBUG:base_functions:base_functions.py:61:We are running version 3.6.61 of Cyberdrop Downloader
2023-02-10 13:07:52,713:DEBUG:main:main.py:73:Creating Config
2023-02-10 13:07:52,714:DEBUG:base_functions:base_functions.py:146:Creating Config File
2023-02-10 13:07:52,719:DEBUG:base_functions:base_functions.py:61:Getting Run Args
2023-02-10 13:07:52,727:DEBUG:base_functions:base_functions.py:205:Gathering Args
2023-02-10 13:07:52,727:DEBUG:main:main.py:97:Starting Cyberdrop-DL
2023-02-10 13:07:52,728:DEBUG:main:main.py:98:Using authorization arguments: {'pixeldrain_api_key': '!REDACTED!', 'simpcity_username': None, 'simpcity_password': '!REDACTED!', '*Blacklisted site*_username': None, '*Blacklisted site*_password': '!REDACTED!', 'xbunker_username': None, 'xbunker_password': '!REDACTED!'}
2023-02-10 13:07:52,728:DEBUG:main:main.py:99:Using file arguments: {'input_file': WindowsPath('URLs.txt'), 'output_folder': WindowsPath('Downloads'), 'output_last_forum_post_file': WindowsPath('URLs_last_post.txt'), 'db_file': WindowsPath('download_history.sqlite')}
2023-02-10 13:07:52,728:DEBUG:main:main.py:100:Using jdownloader arguments: {'jdownloader_enable': False, 'jdownloader_username': None, 'jdownloader_password': '!REDACTED!', 'jdownloader_device': None}
2023-02-10 13:07:52,728:DEBUG:main:main.py:101:Using runtime arguments: {'threads': 0, 'attempts': 10, 'required_free_space': 5, 'disable_attempt_limit': False, 'connection_timeout': 15, 'ratelimit': 50, 'throttle': 0.5, 'include_id': False, 'exclude_videos': False, 'exclude_images': False, 'exclude_audio': False, 'exclude_other': False, 'ignore_history': False, 'output_last_forum_post': False, 'separate_posts': False, 'mark_downloaded': False, 'proxy': None, 'skip_hosts': []}
2023-02-10 13:07:53,211:DEBUG:base_functions:base_functions.py:61:No links found, check the URL.txt
If the link works in your web browser, please open an issue ticket with me.
2023-02-10 13:07:53,212:DEBUG:base_functions:base_functions.py:61:Starting Scrape
2023-02-10 13:07:53,212:ERROR:main:main.py:128:ValueError No links
2023-02-10 13:07:53,212:DEBUG:base_functions:base_functions.py:61:No links found during scraping, check passwords or that the urls are accessible
2023-02-10 13:07:53,212:DEBUG:base_functions:base_functions.py:61:This program does not currently support password protected albums.
2023-02-10 13:07:53,213:DEBUG:sql_helper:sql_helper.py:133:Successfully closed sqlite database connection
Thanks for your time!
make sure you include the https:// at the start of the link in the URLs.txt file.
Yeah, I did it. The first time I did it with the complete url, "http://...", and it gave me the error so I've tried whitout the http part. But now while I was running cmd again like the first time with the http part it worked, so problem solved
Hello, i'm new on this. And i got this message on my termux
~ $ cyberdrop-dl
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 399, in wrapper
return cache[key]
~~~~~^^^^^
KeyError: (('/proc',), frozenset())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_pslinux.py", line 285, in <module>
set_scputimes_ntuple("/proc")
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 401, in wrapper
ret = cache[key] = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_pslinux.py", line 268, in set_scputimes_ntuple
with open_binary('%s/stat' % procfs_path) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 728, in open_binary
return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/proc/stat'
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/bin/cyberdrop-dl", line 5, in <module>
from cyberdrop_dl.main import main
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/cyberdrop_dl/main.py", line 20, in <module>
from cyberdrop_dl.client.downloaders import get_downloaders
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/cyberdrop_dl/client/downloaders.py", line 9, in <module>
import aiofiles.os
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/aiofiles/os.py", line 31, in <module>
link = wrap(os.link)
^^^^^^^
AttributeError: module 'os' has no attribute 'link'. Did you mean: 'unlink'?
~ $
How can i solve this?
~ $ cyberdrop-dl
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 399, in wrapper
return cache[key]
~~~~~^^^^^
KeyError: (('/proc',), frozenset())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_pslinux.py", line 285, in <module>
set_scputimes_ntuple("/proc")
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 401, in wrapper
ret = cache[key] = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_pslinux.py", line 268, in set_scputimes_ntuple
with open_binary('%s/stat' % procfs_path) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/psutil/_common.py", line 728, in open_binary
return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/proc/stat'
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/bin/cyberdrop-dl", line 5, in <module>
from cyberdrop_dl.main import main
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/cyberdrop_dl/main.py", line 20, in <module>
from cyberdrop_dl.client.downloaders import get_downloaders
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/cyberdrop_dl/client/downloaders.py", line 9, in <module>
import aiofiles.os
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/aiofiles/os.py", line 31, in <module>
link = wrap(os.link)
^^^^^^^
AttributeError: module 'os' has no attribute 'link'. Did you mean: 'unlink'?
~ $
How can i solve this?
jpg.fish is incredibly slow now, if it's scraping those files. But I'd need to see a downloader.log to know anything.
I'd recommend just waiting a while.
I'm up to around 65 hours on the rewrite, just handed beta 1 to the beta team... and subsequently beta 2 an hour afterwards. And Beta 3 as I was trying to get a screenshot for this....
Lots of changes coming in V4:
- New Scrapers
- Optimization of older scrapers where possible
- Hopefully better error handling
- Changes to existing options
- Entirely new options
But the part I'm most excited about is:
- Simultaneous downloads from different file hosts (domains)
- A spiffy paint job
Lots of changes coming in V4:
- New Scrapers
- Optimization of older scrapers where possible
- Hopefully better error handling
- Changes to existing options
- Entirely new options
But the part I'm most excited about is:
- Simultaneous downloads from different file hosts (domains)
- A spiffy paint job
That looks amazing! You're doing wonders and can't imagine the effort you've put in, well done man.
Edit; Is it possible to code the "putting all X files into 1 folder" at the present time while you're redoing everything?
Last edited:
You're an absolute mad man, cheers mate. Do you have an ETA for V4?
Last edited:
I ain't gonna fall trap to that. But hopefully within a week or two. Basically everything is done, it's just hunting down what doesn't work, or doesn't work as it's supposed to and fixing it.
Last edited: