Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

axej

(◕‿◕✿) STYLE (◠‿◠)✌ START (◠‿◠✿)PORNHOARD
Mar 11, 2022
276
3,828
1,299

neolith

egirl shill
Mar 11, 2022
599
38,351
1,793
0fya082315al84db03fa9bf467e3.png
Getting these SSLCertVerificationError again. :/
think it has something to do with my VPN or it's specifically anonfiles blocking my vpn.
(because in browser the https version of the link will have a certificate error, but the http link will not.)

If I just restart the script it'll progress a little, but then get caught up again.

It doesn't seem to respect the --attempts argument as well, which is my main motivation for posting this. Since I've tried setting that to 5, but it'll still loop on the same broken links forever.

Code:
Please, Log in or Register to view codes content!
 

Major Tom

Tier 1 Sub
Mar 13, 2022
33
125
389
0fya082315al84db03fa9bf467e3.png
Any plans on adding support for zz.fo files? I'm having issues seeing them on the site. Same way I had issues with cyberdrop. Altho with cyberdrop folks told me to ad fs-01 to the links and that helped. No idea if there's a solution like that for zz.fo
 

Xem

Bathwater Drinker
Mar 25, 2022
102
2,195
1,252
0fya082315al84db03fa9bf467e3.png
Is it normal to take a long time to download videos from the "bunkrr.su" website even with 1T internet?
 
Last edited:

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,504
1,427
0fya082315al84db03fa9bf467e3.png
Hi Jules! I hope your vacations are doing some good to you! :pepoLove:

But uh yeah when you'll be back or have the chance I've go an ongoing issue I'm not able to resolve by myself (if there is a way at all, which I don't know), so in the past week or two there's been an issue where it often ends up getting stuck on specific albums either indefinitely from what it seems OR sometimes for a long time before finally continuing to download and get through overnight. Usually it's largely the last few files it gets stuck on or right at the beginning. Or sometimes it gets stuck on the last files and then if I close it and restart it it gets to that album again and stays on 0 (probably because it tries to download the missing files first and/or the ones already partially downloaded to continue from where it was). A good example is the Disharmonica/Helly Valentine huuuge Patreon album linked in many posts from that one specific dude.

Or a bunch of albums does the same thing in the Mikomi Hokina thread here: https://simpcity.su/threads/mikomi-hokina.9998/ .

Been at it on that link for a good while now and every time I come back to it I get a bit further but it always ends up getting stuck somewhere.

I also tried putting the command "--attempts 3" in the start file so that at least it skips the problematic files but that doesn't change anything either :T.

Here's the beginning and the end of the logs (there was like 43000+ lines so...yeah ^^":


PHP:
Please, Log in or Register to view codes content!


I hope there's a way to fix/work around this :peepoShy:

EDIT: LMAO did expect it to pop pictures from the album. makes sense though. I uh, don't know if there is a way to undo that so it's less of a mess? xD
EDIT#2: AH HA! Found a way uwu

EDIT#3: Here's a pic of how it looks when it's stuck.

Please, Log in or Register to see links and images
 
Last edited:

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,504
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
bunkrr.su currently down right now for maintenance. More accurately they are finalizing the update mentioned on it since a few weeks now that they've been working on for some months. After that if everything goes well, when it's back up we should be able to upload files with a size up to 5GB and the speeds should be better among other things. So it's just a question of being patient.
 

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,198
5,235
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
If you open a github feature request for it I'll look into it.
Please, Log in or Register to view quotes
Vacation was fabulous. It seems I've come back to a ton of work though, both at work and with this.

Bunkr not working right now is a result of a change on their end, when you first open a page you will see "loading..." in the top left corner before the archive actually loads in. That's all the program is seeing currently.

Current to do list stands:
- Change how the program looks for .part files on completion to be more accurate
- Implement support for saint.to
- Rewrite Bunkr (unsure if I will wait for the update or not).
- Figuring out this ghost freezing issue that happens with some archives it seems.

Unfortunately this will come sporadically as I need to prioritize the deadline I have for my actual job. But I'll get going on it all soon.
 

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,504
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Nice! I'm glad you could get at least some good rest in and had a great vacation :pepoLove:

And yes, with Bunkr still working out some big update release issues (and the many probable changes that'll bring) and me dumping a (from what I seem to understand from what you said) pretty tricky bug, there's stuff to keep you busy haha!

Glad to know it's not me that fucked up or that there's not an easy fix I forgot to do or something. Then again it's a bummer it won't be easy to fix at the same time but hey, that's life. I shall restart the program from time to time on those few URLs when I don't have anything new to feed your wonderful app. It shall much on these URLs until a fix comes.

And no worries of course any IRL stuff is much more important than anything here. You certainly have your priorities in the right places. Looking forward to the next updates! Until then take care Jules! :peepoHappy: :PeepoBlanket:
 
  • Like
Reactions: Jules--Winnfield

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,198
5,235
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
I realize now what this is likely. This is most likely the checks for already-downloaded happening (and would be confirmed by that snippet of the log). The progress bar doesn't currently update on each one it skips. So it's skipping everything in the background, while it appears to be doing nothing. It's just taking a fair amount of time to do the skipping because there are a lot of files.
 
  • pikaOMG
Reactions: DoofusMan

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,198
5,235
1,127
0fya082315al84db03fa9bf467e3.png
So 2.22.1 is out now featuring:

- Rewrite of bunkr.to scraping
- More relaxed bunkr restrictions
- Saint.to support
- SimpCity scraping update for saint.to
- Improved partial downloads detection upon exit (for message alert).
 

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,198
5,235
1,127
0fya082315al84db03fa9bf467e3.png
I'm aware there's a bunch of regular ass domains being scraped from cyberdrop right now. I'll get to it later.
 
  • Like
Reactions: neolith

APujazon

Tier 1 Sub
Mar 15, 2022
6
101
383
0fya082315al84db03fa9bf467e3.png
Thank you for the downloader, it's really great!

Don't know if anyone would find this useful, but I created a simple python script to extract all the files from the subfolders into the main folder, just in case you, like me, want to see all the thumbnails at once and don't care about the subfolders' names

import os from os.path import join from shutil import move links = [r"your folder location "] #if you have more than one folder, add it after a comma and preceded by 'r' for link in links: root=link new_directory = "everything" destination = os.path.join(root, new_directory) os.mkdir(destination) subdirectories = os.listdir(root) for subdirectory in subdirectories: if subdirectory != new_directory: for filename in os.listdir(join(root, subdirectory)): move(join(root, subdirectory, filename), join(destination, filename)) os.rmdir(join(root, subdirectory)) for filename in os.listdir(join(root, 'everything')): move(join(root, 'everything', filename), join(root, filename)) os.rmdir(join(root,'everything'))
 
  • Like
Reactions: DoofusMan

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,504
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Interesting! :eek:

For people that don't want to have to enter the folder path manually every time in between uses with different models folders and have a GUI with a bunch of options I recommend "Folder Merger". Super light and simple app. Works super fast and you can drag and drop stuff in it.

Please, Log in or Register to see links and images
 
  • Like
Reactions: APujazon

suecharger

Lurker
Mar 18, 2022
1
0
46
0fya082315al84db03fa9bf467e3.png
Been using your scraper for a while and when scraping Kemono, the program gets stuck when it comes across duplicate files for some reason. The downloader log states "File already exists and matches expected size" for the post. It takes a long time to skip over the file and continue so I've been manually downloading the images to force the program to continue which is a major pain when there's a lot of posts to scrape.