Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,178
5,117
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
I'm doing some major revisions to the downloading portion currently, then I'll debug that. Then toss up the update. Then I have to rewrite the pixeldrain scraper, then figure out how to fix the rate limiting with coomer.party, then rewrite cyberfile to allow for direct downloads... The list goes on. So much shit has broken or needs fixing currently it's driving me a bit nutty. Not to mention the bunkr update coming down the pipeline I'll likely have to fix things for.

I gotchu though.
 

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,204
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Yeah and now it's back. Also for a good while it was on ".me" and for the past two days it wouldn't connect at all anymore for some reason, and they went back to ".to" without me noticing and wouldn't redirect there automatically. Anyways I figured it out when I opened a new link of cyberdrop in a thread and it worked I was like "wait what!?...omfg is it really just that!?" *looks at the URL* ...BRUH. Also if you click on the "cyberdrop" logo in the top left corner it stills points toward the ".me" version and doesn't load :hahaa: :hahaa: :hahaa:
 

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,204
1,427

Pussy Hunter_x

Thicc Lover
Mar 11, 2022
440
9,582
1,302
0fya082315al84db03fa9bf467e3.png
Jules--Winnfield a doubt. When I download an entire topic it downloads all the files, does that include the files from the links too? For example, there are many links to bunkr albums, GoFile for example that we use in topics. Will the Scraper identify them and download the content of these links or just what is in the site's topic?
 

xylophix

Bathwater Drinker
Mar 11, 2022
116
4,332
1,252
0fya082315al84db03fa9bf467e3.png
Played around with trying to download an entire SimpCity thread earlier, and a few questions arose while looking at the logfile.

Why does it start by opening a connection to gofile.io
Code:
Please, Log in or Register to view codes content!

Why does it not give any indication there was problems with some downloads in the cli?
I had several dls fail due to 503 errors from jpg.church (most likely due to their maintenance) but no message at all about it.​

Why does it post username & password in cleartext in the logs?

if i run it with "--include-id" i get the below error, it works if i remove that argument.
2022-05-09 13:35:05,423:DEBUG:base_functions:base_functions.py:93:This program does not currently support password protected albums.

This was done on version 2.14.12
 

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,178
5,117
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
You don't sound ungrateful, I'm just a bit overwhelmed at the moment.

Please, Log in or Register to view quotes
It will.

Please, Log in or Register to view quotes
This was fixed in a later version, do a pip update of the program. We're on 2.18.7 now I believe.

Please, Log in or Register to view quotes
I should probably address that.

Please, Log in or Register to view quotes
This I'm unsure about. I'll look into it, but try again on the latest version and see if it persists.
 
  • PeepoLove
Reactions: DoofusMan

Lil' Sissy Amy

Bathwater Drinker
Mar 11, 2022
52
1,706
1,242
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Hi Jules! Big fan of your work, been using your scraper since 2 years back :) Thank you! By the way is it possible to get coomer.party files to have a date or the post caption posted in the file name? I know a kemono.party scraper can do that, but the downloading speed is too slow..

example: [20210308] [119911831] A little bonus content I filmed after the throatfucking

Please, Log in or Register to see links and images
 

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,178
5,117
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Yea. Can definitely do that. I need to take another look at coomer.party either way. The downloader runs into their DDOS protection and I've not found a way around it yet (I honestly haven't tried as of yet, this might be why that one is slow in all honesty).

Coomer is 3rd on my block to revisit.

I'm just finished of reverse engineering the cyberfile ajax calls and building my crawler so that the scraper will actually support that site (fuck this site).
Next I need to rebuild the pixeldrain scraper from scratch, and then I can do that when I try and fix coomer.

Love to hear you've been using it for so long though. It's been a wild ride since then.
 

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,178
5,117
1,127
0fya082315al84db03fa9bf467e3.png
So. Very large update.

Pixeldrain is working again, Cyberfile is fully supported.

Changes to downloader somewhere in 2.18.X fixed a bug related to download history. With it being fixed some files may redownload (apologies).

There will probably be some bugs, as to be expected. Let me know how its works for you all. Put a ton of work into these last few updates.

After the incoming bunkr changes and hopefully fixes to coomer throttling on the downloading end, I'll probably take a bit of a break from the massive updates for a hot minute unless something spurs on motivation.
 

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,204
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Yo seriously I hate cyberfile so much. The way you have to download things individually, can't open stuff in tabs and have to go back and forth constantly, and the limit of 6 concurrent DLs (not too bad though, considering the speeds are pretty fast in exchange). The DLs just giving errors constantly and having to restart them (less recently though. Also must've been because of high traffic initially) Anyways, thank you so much for putting up with that clunky as hell website to make everyone's life better and save us crazy amounts of time. :peepoShy:

Please, Log in or Register to view quotes
True, it must make for great learning experiences in the end though. Always a good thing if you're looking to expand your knowledge and skillsets for sure.
--------------
On an unrelated note...can't get enough of your pfp dude hahaha! God I love that movie (and actor :pepoLove:). Makes me giggle every time I take a good look at it lmao.
 

DoofusMan

Unintentional Forum Speedrunner
Mar 11, 2022
547
12,204
1,427
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
Oh shit! missed that one, didn't notice the new page in the thread! imma try it out right now with the link I kept on the side! :peepoClap::ppOverheat:


EDIT: It works! Speeds are slow but IDGAF I'm just happy to not have to deal with that hot mess of a website. And was surprised when it said the arguments for the login weren't recognized. Double checked my start batch file and then I tried without it and it worked hahahaha! So that's also awesome :peepoHappy: :PeepoBlanket:

Oh shit, somehow it's downloading right not 13 files (keeps going up, was at 11 a few seconds ago. I wonder if it'll go to 24 to fill all my threads :ddHuh:) at the same time! I thought their hard limit for concurrent downloads was 6! :wowers:

20 concurrent DLs now :monkaX:... 24!!! NOW! :hyperz:

Welp, wasn't expecting that out of this one with the limiter through the browser. Compensates for the speed too hehehe. :PeepoBlanket:
 
Last edited:

Jules--Winnfield

Cyberdrop-DL Creator
Mar 11, 2022
2,178
5,117
1,127
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
I honestly didn't know they had a concurrent cap lmao. I probably would've limited it if I had known. If it doesn't cause any issues... no need to put one in now. :WeSmart:

Please, Log in or Register to view quotes
This was my first time replicating ajax calls. The scraper itself isn't fast, and can only run through one URL at a time... (they tie some of the requests to the last accessed URL). But it works.

You aren't the first to say downloads are slow though which confuses me. The downloading section of the program (once it actually starts grabbing the file) is about as simple as can possibly be. Don't think there's any bottlenecks in there that I can do anything about.

I could speed things up but it would sacrifice the partial download functionality. That's not a trade off I want to make currently with how often files error out while downloading.
 
  • Wesmart
Reactions: DoofusMan

Shaggy12777

Lurker
Mar 11, 2022
21
3
52
0fya082315al84db03fa9bf467e3.png
Please, Log in or Register to view quotes
hey im trying to scrape a whole thread but its only picking up 4 photos. this is the thread

this is the download out come. running 2.19.1


2022-05-09 21:56:40,484:DEBUG:base_functions:base_functions.py:47:Starting Scrape
2022-05-09 21:56:40,485:DEBUG:base_functions:base_functions.py:47:Starting scrape of https://simpcity.su/threads/julieta-allegretti-girlofnox.14843/
2022-05-09 21:56:41,547:DEBUG:base_functions:base_functions.py:47:Starting scrape of
Please, Log in or Register to see links and images

2022-05-09 21:56:41,548:DEBUG:base_functions:base_functions.py:47:Starting scrape of
Please, Log in or Register to see links and images

2022-05-09 21:56:41,548:DEBUG:base_functions:base_functions.py:47:Starting scrape of
Please, Log in or Register to see links and images

2022-05-09 21:56:41,549:DEBUG:base_functions:base_functions.py:47:Starting scrape of
Please, Log in or Register to see links and images

2022-05-09 21:56:41,773:DEBUG:base_functions:base_functions.py:47:Finished scrape of https://simpcity.su/threads/julieta-allegretti-girlofnox.14843/
2022-05-09 21:56:41,811:DEBUG:downloaders:downloaders.py:280:Working on
Please, Log in or Register to see links and images

2022-05-09 21:56:41,811:DEBUG:downloaders:downloaders.py:280:Working on
Please, Log in or Register to see links and images

2022-05-09 21:56:41,811:DEBUG:downloaders:downloaders.py:280:Working on
Please, Log in or Register to see links and images

2022-05-09 21:56:41,811:DEBUG:downloaders:downloaders.py:280:Working on
Please, Log in or Register to see links and images

2022-05-09 21:56:42,100:DEBUG:downloaders:downloaders.py:168:GIRLOFNOX-293.jpg Already Downloaded
2022-05-09 21:56:42,207:DEBUG:downloaders:downloaders.py:168:GIRLOFNOX-282.jpg Already Downloaded
2022-05-09 21:56:42,324:DEBUG:downloaders:downloaders.py:168:GIRLOFNOX-233.jpg Already Downloaded
2022-05-09 21:56:42,432:DEBUG:downloaders:downloaders.py:168:GIRLOFNOX-231.jpg Already Downloaded
2022-05-09 21:56:42,433:DEBUG:main:main.py:81:Finished
2022-05-09 21:56:48,893:DEBUG:base_functions:base_functions.py:47:Purging empty directories
2022-05-09 21:56:49,165:DEBUG:base_functions:base_functions.py:47:Finished downloading. Enjoy :)
2022-05-09 21:56:49,166:DEBUG:base_functions:base_functions.py:47:There are still partial downloads in your folders, please re-run the program.
2022-05-09 21:56:49,172:DEBUG:sql_helper:sql_helper.py:97:Successfully closed sqlite database connection