Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[ausugnd]

Hey Jules--Winnfield, as always, thanks for your support!
Quick question... Is there a way I can have the config file setup, set to 'true', but, on occasion call Cyberdrop-DL and have it ignore the config file?

Example is where I run a batch over time, and have it pull from multiple txt files, and place into multiple folders for easier sorting (The solution you proposed many posts ago which was a massive help).
I noticed when I run the custom .bat file (example below) and config.yaml is Apply_Config: true, the -i and -o arguments don't work.

Please, Log in or Register to view codes content!

It only opened the default URLs.txt and placed it in the base /Downloads folder from the Config file.

In the options, i noticed there is a --config-file CONFIG_FILE option, is there a way to have something like --config-file -ignore and then just throw that in my folder batch file for that occasion only? Current workaround is I just change the flag in the config file to false manually.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I see what you're talking about. That's a bit of a pain in the ass for ya. my URLs.txt file has grown too much and I should probably do something similar at this point. I don't have time to do this now, could you copy paste that as a "feature" issue on the github? I'm going to try and work on things this weekend, and probably start a rewrite of the project.

 

[ausugnd]

Please, Log in or Register to view quotes

Can do! Will do!
Workaround works for now, and mainly I wanted to check I wasn't missing something in the existing options.

Personally, have a massive list of urls in a master queue/notes txt file. Sometimes I run small batches using just the standard settings, but when I have time, I copy them to individual txt files by name and run a batch of say 20 overnight. When done, I combine it with a simple EXE I found MoveFilesUpFromSubfolders.exe - Run it in the directory, and it flattens it out. So if you have multiple Bunkr, GoFile and other folders, removes the sub-folder structure for easier filing.
I also use FreeCommander as a replacement file browser, and I have added the exe as a custom tool so with 2 clicks, the folder is clean.

Can't remember where I found it...
May have been a link on Media🔥🔥🔥🔥 but have zipped and dumped into a Bunkr for you if you're interested:

Please, Log in or Register to see links and images

I did try writing my own version in PowerShell and failed/gave up, then in looking at a Python attempt, found the exe instead and didn't have time to keep trying to figure it out. Ultimately I wanted a 'Flatten then move image files to a /Pics/ sub-folder but that's a project for another day when I have more time.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I've got you covered, I've written myself a sorting function that does essentially that. I'll include it in the rewrite.

 

[grsycfed]

Thanks for putting in the time!

Suggestions:
- parallelize downloads from different hosts. The download speeds can be fairly slow and there is rate limiting as well. I'm on a gigabit connection but using only ~8MB/s of bandwidth right now.
- cache scrapes from coomer.party posts since they're almost guaranteed to be static. rescraping even one creator page with a whole lot of posts is quite slow and can happen a lot when tweaking configurations

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

parallelizing is one of the main reasons I'm going to do it.

I'll think about the caching, but I don't think it'll be changing for the time being. I also don't know how long the rewrite will take me.

 

[ausugnd]

Ok... Sorry... Somehow I broke it... And then fixed it... But I'm not sure exactly what I did. So I'm posting this just in case it helps in the future!
I THINK... The errors were caused because I was saving config.yaml with Apply_Config: false after having run it with true earlier, then using my custom command cyberdrop-dl -i urls01.txt -o "./CDDownloads/0001" --ignore-history. It would not work, display some errors in the prompt, and not download anything or even create the custom director. But for the life of me I couldn't figure out why.

Setting it to false, and then running cyberdrop-dl normally through PowerShell seems to have then 'updated' the settings somewhere else, and now cyberdrop-dl -i urls01.txt -o "./CDDownloads/0001" --ignore-history works as intended.

Because it's now working, unfortunately I have lost the downloader.log file... But I do have the output messages if they help.
cyberdrop-dl -i urls01.txt -o "./CDDownloads/0001" --ignore-history
It would not work and just spit out the below for each line and not download anything.

Spoiler: CMD Output Example 1

Please, Log in or Register to view spoilers

Spoiler: CMD Output Example 2

Please, Log in or Register to view spoilers

Spoiler: CMD Output Example 3

Please, Log in or Register to view spoilers

 

[treidon]

I've got a new Mac that I downloaded Cyberdrop-dl onto and I get the following message after installing it:

Please, Log in or Register to view codes content!

Is there a command line that I can use to fix this? Thanks

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I'm not gonna lie, I'm at a pretty big loss here. I'm not sure what is really going on with any of those outputs. But noted, thanks! I'll come back to these later.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Open the python folder in your applications folder and run the update shell profile file and see if it resolves anything.

 

[treidon]

Please, Log in or Register to view quotes

I am guessing you mean the "/Users/treidon/Library/Python/3.9/lib/python/site-packages/pip" folder? Because when I go under /Users/treidon/Applications" I dont see anything in terminal.

What is the exactly update shell profile file? Sorry for my ignorance with CLI.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I mean just go to the python folder in the applications folder in finder

 

[treidon]

Well this is embarrassing, there isn't a Python folder in Applications. I can verify its installed, though. I'm running Ventura 13.0

 

[destroyerofpussies]

Sorry to bother, but is there a reason as to why the most files stop downlaoding halfway? It happens in all albums, I have checked lightly on git issues and this thread but I have found no reference to this issue :/
The file download just skips to the next one at random seemingly

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Does cyberdrop work if you try and run it?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

It happens when a download fails. Just let the program run, and if there are any .part files in the folder after it's complete, run the program again. You should give a read through of what all the potential arguments are for the program, specifically --attempts.

 

[treidon]

Please, Log in or Register to view quotes

Well when I try to run it I get:

Please, Log in or Register to view codes content!

 

[treidon]

Please, Log in or Register to view quotes

Ok I know this a temporary fix until I can get it into bash or zsh but it works for now:

Please, Log in or Register to view codes content!

 

[dsffadsfadsfdasdfsaadf]

i noticed kemono.party doesnt work anymore so is because of kemono?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Apologies, but I don't have a mac new enough to really test out ventura. What you want though is to find a permanent way to do essentially what you did. How did you install python 3.9?