Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads
- Thread starter Jules--Winnfield
- Start date
I just realized I had broken the SimpCity scraper. I'm pushing an update. Make sure you read the post below this.
appears to be down.
Last edited:
*IMPORTANT*
I had to change the DB structure because of bugs that were impossible to fix bugging the shit out of me. There were so many bugs and the code was getting extremely complicated.
What this means:
The old history is wiped. It is also no longer based off file size. If you have downloads in the download folder and run the program again, it *should* rebuild history.
What this means going forward:
Checking for things that have already downloaded is going to be a hell of a lot faster and should be bug free. Should being key word.
I had to change the DB structure because of bugs that were impossible to fix bugging the shit out of me. There were so many bugs and the code was getting extremely complicated.
What this means:
The old history is wiped. It is also no longer based off file size. If you have downloads in the download folder and run the program again, it *should* rebuild history.
What this means going forward:
Checking for things that have already downloaded is going to be a hell of a lot faster and should be bug free. Should being key word.
Last edited:
yeah exactly, at that point if it works with no limiter then we'll just see how long that'll last hehehe.
Oh dang ok I kinda see what you mean.
Yeah and also their servers had also at least that on their side. Albeit having limited concurrent downloads and giving errors while downloading pretty often, at least it was blazing fast. But in the app for some reason it stays between 25-100Kb most of the time per download. No matter how many are running concurrently be it 1 or 24. but hey, can download up to 24 at the same time so that's something for sure at least hehe. I mostly find that peculiar, not complaining here
Oh, yeah no no. Way better to have partial downloads to be able to shut the app mid download to add links or need to do a reboot or etc than faster downloads. Reliability first and foremost for sure!
Oki, so if I understand correctly, previous DB code was a cesspit so you exterminated it, and made a new one that is much faster. So we still have the download history function to not redownload the same files twice from the same album BUT the old DB had to be scraped so we are starting from scratch in regard to our downloads history. Is that all correct?
You got it. Not an easy decision. But it's probably the best way forward.
DoofusMan Cutelo Exhis Shaggy12777
Tagging you with an update because you reacted to the above.
I put in some DB rebuilding. If the download files still exist in the proper download location it should rebuild most of it. Then continue. (rebuild after running the link again)
Tagging you with an update because you reacted to the above.
I put in some DB rebuilding. If the download files still exist in the proper download location it should rebuild most of it. Then continue. (rebuild after running the link again)
consequence of using the new DB schema. I had to wipe previous history. So that's the expected result.
Welp, something weird happened but now seems to work like a charm and even better than ever? So since yesterday (basically when launching the app before going to sleep and now since I woke up)...any link I would put in it would give an error for each file of the link saying it couldn't download them. And in the log would talk about an optional error with a missing column or etc.
BUT since we are supposed to restart from scratch with our history DB I went ahead and deleted the file out of suspicion. And of course now not only it works but the link of Cyberfile is not only downloading on all of my 24 threads but also anywhere between 1MB/s up to 6MB/s EACH!!! So yup, we are un business
BUT since we are supposed to restart from scratch with our history DB I went ahead and deleted the file out of suspicion. And of course now not only it works but the link of Cyberfile is not only downloading on all of my 24 threads but also anywhere between 1MB/s up to 6MB/s EACH!!! So yup, we are un business
Last edited:
Weird. But noted. The old DB should've been completely deleted, and a new DB with a different name created. So not sure what happened there. But glad to hear things are going stellar.
Oh, just thought of something when seeing a Kemono party link being posted in 2 threads today. When you'll end up having done the code for coomer party, since it seems to use the same website infrastructure, do you think you could support Kemono also?
Yea. Can do. Coomer currently works. When it hits ddos protection is up in the air. Don't know how to handle it yet. Gotta make some test scripts and try and sus it out.
Awesome! And yea true you mentioned that before. Sounds tricky as hell. I guess you'll have to find out how many request you can do within X amount of time before it hits a threshold that triggers the DDoS protection? And then play around to see how far you can optimize that so it's stable while getting as much as you can? I'm no expert though of course. But yeah wtv it'll take doesn't sound simple.
Yea. That's what I was planning on doing, but both myself and the person that brought it to my attention tried again the next day and neither of us hit it. Given the exact same links. So I'm confused.
the new db file should just have a single table with 3 fields right? path, filename, completion?
does this mean that if you use a different path, it'll download the file again?
it'll also mean that mirrors are ignored when scraping threads as long as it's marked completed? which is nice. bunkr's adding of a "unique identifier" to the filename still fucks with this tho lol, but then again does differing naming schemes on rips and just people naming stuff differently. ah well
does this mean that if you use a different path, it'll download the file again?
it'll also mean that mirrors are ignored when scraping threads as long as it's marked completed? which is nice. bunkr's adding of a "unique identifier" to the filename still fucks with this tho lol, but then again does differing naming schemes on rips and just people naming stuff differently. ah well
Path is misleading here. It's not filepath, it's url path.
As an example:
https://jpg4.su/img/img-20220511-042012.E4t0Ktthe path for this url would be
img/img-20220511-042012.E4t0KtAs a result, no, mirrors won't be ignored unfortunately. However, this will be a lot more accurate of tracking than the previous filesize implementation. That implementation was based off of HTTP headers content length. Which in some cases, returned 0, in other cases (chunked downloads) returned a fraction of the total filesize. This always led to files never being marked as completed, and would always redownload no matter what. The new implementation is a lot more fool proof and resistant to errors.
are cyberdrop (.me) and (.to) different? whenever i tried tomacces my dashboard on (.me) it wont show, but when i change to (.to) it works just fine, do i have to reupload ll of my stuff? if it yess then omg.......because all of my links are dead, help
They are the same website. The .me domain is just down right now seemingly. Also, I'm not affiliated with cyberdrop, nor am I a support person for them.