Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads
- Thread starter Jules--Winnfield
- Start date
So Python comes pre-installed on the newer Mac OS'es. I followed the directions on page to create the .zshrc and add the path to it from the Cyberdrop installation that errors out. Works perfect now, thank you for your help.
Also wanted to add that scraping large amounts of posts on coomer.party gets you rate limited which results in failed downloads, and you'll hit the rate limiter again when retrying resulting in more failed downloads.
I've started the rewrite, and I'm excited for it. I won't give an ETA on how long it'll take to complete. Here is my WIP To-Do/Wishlist:
- Parallel downloads between filehosts
- Cache albums and associated links
- Cache coomer posts (to help reduce rate limit hitting)
- Migrate old download history to new format if one is needed
- Rewrite all scrapers (there is one for each host/forum/etc)
- Use less SQL Lookups and generally try and reduce SQL overhead
- Attempt to fix database lock issues when running off a NAS
- New arguments (such as allow_insecure_connections (important lately for Cyberdrop), ignore_cache, etc)
- Split the config file into more headings (Files, Authentication, JDownloader, Forum_Options, Ignore, Ratelimiting, Runtime)
- Improve error handling for files, not having a generic FailureException in the code.
- When files already exist, instead of looking up individually, look up based on folder.
- Changing Unsupported_URLs to be a CSV (excel file essentially) that will tell you where said file came from
- Output error-ed files alongside reason and where they came from
- Fix header out of range issue for gofile and others
- Expand site support
- Remove TQDM and Colorama and replace both with
(this one I'm excited for assuming I can make it do what I want)
- This means better looking output and new (hopefully) better progress bars
This is exciting, Jules! Rich should be a huge upgrade; it's crazy what those guys have been doing with the terminal.
Also very much appreciate the communication and would love to follow along as much as you're willing to share. Keep up the great work!
Seems like a good time for me to actually properly use github projects now.
Assuming I remember to update it, you can follow along here:
And I'll give periodic updates as posts here.
It's telling you what the issue is. If you add that directory to PATH, it should work just fine.
I am having problems downloading videos within certain posts, seemingly within Cyberdrop posts. The image files download fine. Here is a sample:
Helly von Valentine - Saber Nero Boudoir: 100%|█| 22/22 [00:00<00:00, 5606.68FIL
HVV - Ann Takamaki: 100%|███████████████████| 21/21 [00:00<00:00, 4402.70FILE/s]
GB 18+: 100%|███████████████████████████████| 47/47 [00:00<00:00, 3834.89FILE/s]
Helly Valentine - See Through Lingerie: 0%| | 0/38 [00:00<?, ?FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 95%|▉| 36/38 [00:28<00:01, 1.26FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 97%|▉| 37/38 [00:29<00:00, 1.27FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 100%|█| 38/38 [00:31<00:00, 1.21FILE/s]
Helly von Valentine - Saber Nero Boudoir: 100%|█| 22/22 [00:00<00:00, 5606.68FIL
HVV - Ann Takamaki: 100%|███████████████████| 21/21 [00:00<00:00, 4402.70FILE/s]
GB 18+: 100%|███████████████████████████████| 47/47 [00:00<00:00, 3834.89FILE/s]
Helly Valentine - See Through Lingerie: 0%| | 0/38 [00:00<?, ?FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 95%|▉| 36/38 [00:28<00:01, 1.26FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 97%|▉| 37/38 [00:29<00:00, 1.27FILE/s]
Error attempting
Helly Valentine - See Through Lingerie: 100%|█| 38/38 [00:31<00:00, 1.21FILE/s]
It’s because the website’s certificate is expired, which prevents Cyberdrop-DL from connecting securely.
It's on the website, it's not an issue with the downloader.
If you are a religious man, pray cyberdrop gets their shit together and fixes their website. Or just download the ones that error out by hand.
I'll probably do written updates every few days as to the progress I'm making on the rewrite. You can follow along more closely
.
I'm about 5.5 hours in, actual development time so far, I've got a time log going for myself as I'm curious how long it will take.
So far, I've re-written majority of Cyberdrop-DL's director, telling the program what portions to work on (argument gathering, scraping, downloading, checking for completeness, ... sorting (not written yet)... I've rewritten the structure of the SQL database that will keep track of already completed downloads, however I'm changing it to keep track of what is scraped, as well as building the coomer db to reduce hits to the website and lower ratelimit hits. I'm unsure yet on whether or not I'll be able to effectively migrate over the old DB, but I will worry about that after everything else is finished.
The SQL rewrites so far will both slow down and speed up scraping (as well as having the option to ignore said history). It will catalogue what has been scraped and whether it was completed or not, Scraping an album again will first check whether or not it has a history, it will use the history to selectively scrape links from the album, or skip scraping the album entirely if everything was completed. (this will also make finishing .part files a heck of a lot easier).
The config file has also been re-written to include more options, as well as being sectioned more appropriately. Old configs won't work, but that's not that big of an issue. The old one will be purged and the new one will be in it's place. The config is also loaded before logging starts, which means all files can be relocated using the config (and the config itself can be specified with command line argument).
I've decided to change Unsupported_URLs into a CSV of two columns, referrer and the actual link itself. I'm not sure how feasible that is, but I'll play around with it and see.
I'm just wrapping up the internal dataclass restructuring, and from here I can go on to rewrite the scraping and continue down the rest of the program.
Needless to say, there is a long way to go. It is a long way off, but before I push it to release, I will be trying to gather beta testers to test it out and put it through the ringer. There's a lot the program does, and it'd be difficult for me to test it all by myself... Plus it seems like it's you all who seem to find a way to break things more than myself.
I'm about 5.5 hours in, actual development time so far, I've got a time log going for myself as I'm curious how long it will take.
So far, I've re-written majority of Cyberdrop-DL's director, telling the program what portions to work on (argument gathering, scraping, downloading, checking for completeness, ... sorting (not written yet)... I've rewritten the structure of the SQL database that will keep track of already completed downloads, however I'm changing it to keep track of what is scraped, as well as building the coomer db to reduce hits to the website and lower ratelimit hits. I'm unsure yet on whether or not I'll be able to effectively migrate over the old DB, but I will worry about that after everything else is finished.
The SQL rewrites so far will both slow down and speed up scraping (as well as having the option to ignore said history). It will catalogue what has been scraped and whether it was completed or not, Scraping an album again will first check whether or not it has a history, it will use the history to selectively scrape links from the album, or skip scraping the album entirely if everything was completed. (this will also make finishing .part files a heck of a lot easier).
The config file has also been re-written to include more options, as well as being sectioned more appropriately. Old configs won't work, but that's not that big of an issue. The old one will be purged and the new one will be in it's place. The config is also loaded before logging starts, which means all files can be relocated using the config (and the config itself can be specified with command line argument).
I've decided to change Unsupported_URLs into a CSV of two columns, referrer and the actual link itself. I'm not sure how feasible that is, but I'll play around with it and see.
I'm just wrapping up the internal dataclass restructuring, and from here I can go on to rewrite the scraping and continue down the rest of the program.
Needless to say, there is a long way to go. It is a long way off, but before I push it to release, I will be trying to gather beta testers to test it out and put it through the ringer. There's a lot the program does, and it'd be difficult for me to test it all by myself... Plus it seems like it's you all who seem to find a way to break things more than myself.
I dont have a github account, but it would be nice and fairly simple (?) if the rewrite has a feature that spits out failed download URLs. Like 'Failed Downloads.log', which tells you the parent URL (the one you put in the URLs.txt), the direct URL to the content, and the fail reason, letting you easily save that URL for retry later or manual downloads.
Because while some of that data exists in the "Downloader.log", it isnt exactly user friendly to sort through, and honestly unless I catch the failure while the application is running, I dont check the downloader.log, I simply re-run the script a few times and hope it got everything without issue.
Then if the script downloads the previously failed content, its entry is removed from the log. Ideally the log would never be purged, just constantly updated with failures or edited when a successful retry occurs.
Because while some of that data exists in the "Downloader.log", it isnt exactly user friendly to sort through, and honestly unless I catch the failure while the application is running, I dont check the downloader.log, I simply re-run the script a few times and hope it got everything without issue.
Then if the script downloads the previously failed content, its entry is removed from the log. Ideally the log would never be purged, just constantly updated with failures or edited when a successful retry occurs.
I'm trying to figure out how the Jdownloader integration works
my batch file to start cyberdrop has the arguments
with replacements for the actual values
what I'm getting confused is the jdownloader-device, I've tried both Jdownloader@PC and also the actual deviceID from the tab that loads while testing my account on jd
the log file shows this
this is after a run where I replaced "Jdownloader@PC" for the ID
and as you can see, it fails on the connection
Maybe I'm using wrong values on the batch file?
my batch file to start cyberdrop has the arguments
with replacements for the actual values
what I'm getting confused is the jdownloader-device, I've tried both Jdownloader@PC and also the actual deviceID from the tab that loads while testing my account on jd
the log file shows this
this is after a run where I replaced "Jdownloader@PC" for the ID
and as you can see, it fails on the connection
Maybe I'm using wrong values on the batch file?
It's whatever you put as the Device Name in MyJDownloader. I'm not seeing a failure from what you've given though. As a reminder, the program only sends unsupported links to jdownloader.
Hello friends, Is it normal to get this error at the end?
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:21<00:00, 21.63s/FILE]
Purging empty directories
Traceback (most recent call last):
File "/home/master/.local/bin/cyberdrop-dl", line 8, in <module>
sys.exit(main())
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/main.py", line 212, in main
loop.run_until_complete(director(args))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/main.py", line 191, in director
await purge_dir(file_args['output_folder'])
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/base_functions/base_functions.py", line 88, in purge_dir
list(map(os.rmdir, deleted))
PermissionError: [Errno 13] Permission denied: 'Downloads/Compressed/compa/CyberDropDownloader/.git/refs/tags'
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:21<00:00, 21.63s/FILE]
Purging empty directories
Traceback (most recent call last):
File "/home/master/.local/bin/cyberdrop-dl", line 8, in <module>
sys.exit(main())
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/main.py", line 212, in main
loop.run_until_complete(director(args))
File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/main.py", line 191, in director
await purge_dir(file_args['output_folder'])
File "/home/master/.local/lib/python3.9/site-packages/cyberdrop_dl/base_functions/base_functions.py", line 88, in purge_dir
list(map(os.rmdir, deleted))
PermissionError: [Errno 13] Permission denied: 'Downloads/Compressed/compa/CyberDropDownloader/.git/refs/tags'
Depends on what you have in / what you use your CyberDropDownloader folder for. To be clear, you shouldn't be downloading the source code to run it.