Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[Jules--Winnfield]

CyberDropDownloader

Please, Log in or Register to see links and images

Supported Cyberdrop-DL Development​

If you want to support me and my effort you can buy me a coffee or send me some crypto:

BTC: bc1qzw7l9d8ju2qnag3skfarrd0t5mkn0zyapnrcsn


ETH: 0xf36ef155C43Ed220BfBb2CBe9c5Ae172A8640e9B


XMR: 46vMP5MXVZqQeGzkA1mbX9WQKU8fbWRBJGAktDcjYkCMRDY7HMdLzi1DFsHCPLgms968cyUz1gCWVhy9cZir9Ae7M6anQ8Q

---

Read about the program here:

Please, Log in or Register to see links and images

Requires Python 3.11

---

Currently supports: Bunkrr, Coomer, Cyberdrop, Cyberfile, E-Hentai, *Blacklisted site*, GoFile, HotPic, ImageBan, ibb (imbb), ImgBox, Imgur, Jpg.Church/Jpg.Pet/Jpeg.Pet/Jpg1.su/Jpg2.su/Jpg3.su, Kemono, MediaFire, OmegaScans, PimpAndHost, PixelDrain, PostImg, RedGifs, Rule34.xxx/Rule34.xyz, Saint, Scrolller, Toonily

 

[Jules--Winnfield]

For those following along, I've found a fix for the "Reactor already installed error". However, I don't know if this is going to break things for other people. Going to test as much as I can before putting it out, I already knew I was stretching the use case of one of the libraries I'm using, and I think I found a solution, but it's dirty and I'm not a massive fan.

After I finish testing that, I'm going to add support for jpg.church and *Blacklisted site*. Should be out in a couple of hours unless I pass out.

 

[Jules--Winnfield]

After 75 hours of development time...

With my 1000th message, I announce,
Cyberdrop-DL 4.0.1 is out now.

Read the damn wiki.

Please, Log in or Register to see links and images

 

[Jules--Winnfield]

4.2.177 is out, it fixes SimpCity scraping. Here is what you need to do:

Run the program once, at least long enough that the config is updated (if it starts scraping links, you can shut it down).

Follow this

Please, Log in or Register to see links and images

to obtain the XF_Session token for SimpCity

Put that value into the appropriate spot in the config.

 

[Jules--Winnfield]

Please, Log in or Register to see links and images

was just released.
This should fix the "Reactor Already Installed" error some users were facing.
This also adds support for jpg.church and *Blacklisted site*.

Consider this fix and support to be in beta. Only so much I can test, and every machine acts differently.

 

[Jules--Winnfield]

I've started the rewrite, and I'm excited for it. I won't give an ETA on how long it'll take to complete. Here is my WIP To-Do/Wishlist:

• Parallel downloads between filehosts
• Cache albums and associated links
• Cache coomer posts (to help reduce rate limit hitting)
• Migrate old download history to new format if one is needed
• Rewrite all scrapers (there is one for each host/forum/etc)
• Use less SQL Lookups and generally try and reduce SQL overhead
  • Attempt to fix database lock issues when running off a NAS
• New arguments (such as allow_insecure_connections (important lately for Cyberdrop), ignore_cache, etc)
• Split the config file into more headings (Files, Authentication, JDownloader, Forum_Options, Ignore, Ratelimiting, Runtime)
• Improve error handling for files, not having a generic FailureException in the code.
• When files already exist, instead of looking up individually, look up based on folder.
• Changing Unsupported_URLs to be a CSV (excel file essentially) that will tell you where said file came from
• Output error-ed files alongside reason and where they came from
• Fix header out of range issue for gofile and others
• Expand site support
• Remove TQDM and Colorama and replace both with
  Please, Log in or Register to see links and images
  (this one I'm excited for assuming I can make it do what I want)
  • This means better looking output and new (hopefully) better progress bars

If you have other things you want to see, post a

Please, Log in or Register to see links and images

. Thank you neoliberal for using it.

 

[Jules--Winnfield]

I'm up to around 65 hours on the rewrite, just handed beta 1 to the beta team... and subsequently beta 2 an hour afterwards. And Beta 3 as I was trying to get a screenshot for this....

Lots of changes coming in V4:
- New Scrapers
- Optimization of older scrapers where possible
- Hopefully better error handling
- Changes to existing options
- Entirely new options

But the part I'm most excited about is:
- Simultaneous downloads from different file hosts (domains)
- A spiffy paint job

Please, Log in or Register to see links and images

 

[errananna]

- Where is the new urls.txt folder?
C:\Users\username\AppData\Local\Cyberdrop-DL\Cyberdrop-DL\Configs\Default

- Where are my downloads?
C:\Users\username\Downloads\Cyberdrop-DL Downloads

- How can I change my downloads directory?
Edit the file located in C:\Users\users\AppData\Local\Cyberdrop-DL\Cyberdrop-DL\Configs\Default\settings.yaml
Under files, paste in the download folder you want it to download to

- How can I roll back to continue my pending downloads?
In the start windows.bat, change the first line to read
pip install cyberdrop-dl==4.2.231


These should answer most questions. As for resuming partial downloaded files, I don't know if that's going to work.
I currently have several things in some named pixeldrain folders (with the random URL on it), and the new version has started to download partial files in a folder under c\users\downloads\%archivename% (PixelDrain). It already downloaded some things which were already done.
But now that I have already downloaded a huge portion of that archive, I don't know if it is going to be smart enough to see that they were already downloaded, or if I'm going to need to spend another week downloading all this stuff again. Once I've tested some more I'll update or post again.. but really, this kind of thing shouldn't had happened... or at least it should had been a stand alone upgrade with no option to save preferences. Right now it's a mess. Let people use the old version, let it be abandoned so it'll eventually stop working. That's fine, but messing with settings like this isn't fun

 

[Geekyfire]

Just throwing an appreciation post out there, thank you for your dedicated time and I'm sure everyone appreciates everything you do.

 

[Jules--Winnfield]

Version 5.0 is officially released!

You will likely want to grab the new start files located here:

Please, Log in or Register to see links and images

You can also read the wiki here on transitioning and use of V5:

Please, Log in or Register to see links and images

 

[slitchampion]

Please, Log in or Register to view quotes

Lol. DDoS-Guard provides protection against denial of service attacks, and distributed denial of service attacks. A side effect of that is that it will block scraping activity pretty frequently as well, especially (1) activity from a VPN that is pretty easy for DDoS-Guard to identify and (2) that which doesn't come from a legitimate browser. DDoS-Guard will block a lot of CDL scraping, but also any other program anyone writes to scrape from bunkr via a non-browser program.

You could, alternatively, be thankful that Jules spends his free time writing, maintaining, and updating CDL for all of us, with little thanks and no revenue. You're free to write your own scraper to bypass DDoS-Guard; good luck.

 

[Jules--Winnfield]

Some teasers. (Not even close to half done)
27 Hours of development logged though, progress has been made.

 

[Jules--Winnfield]

Valtrox grabo

It's because SimpCity added a captcha to login. I have a way around it, but it'll probably take me a few weeks to add it.

 

[Jules--Winnfield]

So. Very large update.

Pixeldrain is working again, Cyberfile is fully supported.

Changes to downloader somewhere in 2.18.X fixed a bug related to download history. With it being fixed some files may redownload (apologies).

There will probably be some bugs, as to be expected. Let me know how its works for you all. Put a ton of work into these last few updates.

After the incoming bunkr changes and hopefully fixes to coomer throttling on the downloading end, I'll probably take a bit of a break from the massive updates for a hot minute unless something spurs on motivation.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

yes. I'll be trying to get an intermediate solution done tonight

 

[Jules--Winnfield]

This will get buried fast I'm sure.

But I'm in the planning phase of building out a full fledged app with the same functionality as the current program. Both versions will stay in production and be updated simultaneously. Trying to decide how best to lay it out and the frameworks to use. It won't be out any time soon, but it's on the way.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

It's simple as is. If you can't figure it out, that's your own fault. It's quite literally download a zip and run a file already, which you'd know if you read the main post. Don't whine and complain about something that is free, doesn't even take donations. It's people like you that cause good utilities like this to become abandoned.

I offer constant help through this forum, DMs and issues on github. I update it within minutes usually of identifying problems. Instead of being salty, crawl back under the bridge you came from.

While your there you're more than welcome to put in the work and effort to make the program how you think it should be. Put a pull request up on github after and I'll look at it to possibly merge it.

 

[Jules--Winnfield]

3.6.53 is out.

It should fix bunkr issues.
pvtrich baddog-11 backwards

 

[DrFapGood]

Anyone else getting a lot of 502 and 503 errors from Coomer? And are there any known workarounds?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

3.6.51
- Removed Kemono support.

Kemono has implemented DDOS-Guard, which I'm not going to attempt to get around. So as of now, it's no longer supported by Cyberdrop-DL