Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[Jules--Winnfield]

Please, Log in or Register to view quotes

As for part one, the new DB works based on URL, not filename and filesize anymore. If the URL changes, things will redownload, as will re-ups. I'm aware this isn't incredibly desirable, but it's a lot easier on my end and less buggy.

For part 2, that was intentional, and I have no intention of changing it. Things being separated into folders for each post will likely become optional.

 

[axej]

hmmmmm, i thought that might be the case with the url change

but why would the entire album change urls on jpg church?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

As an example,

jpg4.su/img/xyz.jpg
jpg4.su/images/xyz.jpg
jpg4.su/img/xyz.jpg

these would be counted as three separate files. I know they've been changing things around as of late, but I'm unsure if endpoints have changed like that. I'll have to keep better watch.

 

[axej]

regex error when using non english characters in url.txt, it seems

Please, Log in or Register to view codes content!

got my links labeled in the txt and some of them have non english chars
example:

Please, Log in or Register to view codes content!

i could just not label them if it's not an easy fix.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Pushed a fix. 2.21.6

 

[axej]

uyup that worked

i hate when regex breaks, shit is never intuitive to me

thanks!

 

[axej]

looks like scraping the party sites isn't like scraping SimpCity threads?

in particular wondering about posts that don't have content as attachments, i think this is most prevalent on patreon stuff

instead the content is linked offsite.

example

Please, Log in or Register to see links and images

so for SimpCity threads, those links end up in unsupported urls.txt (accessing and sifting through that is a pain btw, but i doubt there's any improvements to be made). or getting followed and downloaded. not the case for the party sites

could the change be made to follow those links where possible?

seeing all sorts of links. google drive, catbox,mega,nextcloud(seedbox),dropbox among others. nextcloud,catbox, look easy to scrape i guess. dropbox maybe? meh


gah i wish i could get back all the time i spent on the party sites this weekend

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I honestly didn't know they did that. While it's for sure possible, I'd have to write scrapers for dropbox or nextcloud in this case. Not sure what other sites they would link to. I'll add it to my to-do list, but it's not extremely high priority for me. I'll try and do it sometime this week or next weekend. Can you open a feature request on the github to remind me?

 

[axej]

done

also added a request for a rate limit argument

 

[losthesodium]

Hey yall, anyone else experiencing it not scraping SimpCity threads properly? I'm trying to scrape this thread atm: https://simpcity.su/threads/amanda-elise-lee.11183 and it only finds the content that was posted by StrangeSupreme on the second page

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

it's scraping it fine on my end. You probably didn't give the program your login credentials for SimpCity

 

[doomed_elf]

is it possible to download from jpg church from specific page? like :

Please, Log in or Register to see links and images

because the user's images is a lot and the user doesn't use album.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

It is not currently. It'll scrape the entire profile.

 

[axej]

fully admitting that idk if this is an issue or not because i haven't tested the current behaviour

dms seem to be a separate page on party sites

Please, Log in or Register to see links and images

Please, Log in or Register to see links and images

and uh idk if dms ever have attachments or not

looks like sometimes it's used to give passwords for posts too, idk

patreon in general seems like a huge mess and it's just reflected in kemono


actual known issue

individual posts are not recognized by downloader.

example

Please, Log in or Register to see links and images

 

[losthesodium]

Please, Log in or Register to view quotes

I was adding my username and password using -- like your instructions say. It works for some threwds and not for others. When you scraped that link, how many files did it download? Mine downloaded 25 pics.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

around 150. if you look at the downloader.log it'll probably show that it isn't logging in.

 

[axej]

no support for downloading a single SimpCity post and stopping there, right? it has to do everything after the specified post in the url?

i think i saw an answer to that somewhere before, it's gotten lost in the thread.

throw it in the faq?

also, how necessary are the cyberfile credentials. seems to work for me without them. mostly. haven't really been paying attention to bigger links, but single links work great and that's the only time i really pay attention

 

[bangchang]

Question here, not sure if someone asked or answered already, but is a way to exclude downloading from certain domain when using the SimpCity link to parse threads?

Sample:
https://simpcity.su/threads/aqua-shuimiaoaqua.12262/

this thread have combination of JPG Church and Cyberdrop links. In some good days, JPGChurch will download fast for me, but sometime it goes super slow. Cyberdrops always download fine (decent speed) for me. So i am currently looking if there is a function that can exclude certain domain, in this case jpgChurch, from the parse/download.

 

[DoofusMan]

Please, Log in or Register to view quotes

cyberfile doesn't require a login at all anymore. He mentioned it in the last few posts I think.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Correct, no support for single posts at this time. And cyberfile doesn't need credentials anymore.

Please, Log in or Register to view quotes

There is not currently.