Tools Ripandtear - An asynchronous file archival program

[glandalf]

So I got the script written and executed! Thank you again for your help.

I split the usernames from the tags, then passed the usernames individually as generics to RAT (I will clean up later and assign them correctly to right sites in the rat file) and the tags individually to the tags. I also added a step to rename the folder to strip the tags from it before passing the usernames and tags into RAT so that the rat file and the directories were more clean.
I did notice an oversite with my script though, as there were a couple folders which didn't have the same naming convention/format at the rest of them, and actually had a '(' before the tags. But I took a backup of the directory names before running, so I don't mind a little manual clean up.

One thing I've noticed after running this is that RAT doesn't seem to like UTF-8 characters. So models with non-english characters in the usersnames (e.g. ø, ö, 呆, etc, etc). It replaces the actual character with the unicode number, for example 'ø' becomes '\u00f8' inside the rat file.

So it took me all day to write 70 lines of code, most of which are comments or empty lol. Though I'm not to disheartened by that, as this is literally my first python script I've written outside of basic beginner follow along learning videos.

If you interested, here is what I came up with:

Please, Log in or Register to view codes content!

Then ran it via:

Please, Log in or Register to view codes content!

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Congrats!!! That is awesome! I remember the first python script I wrote was to rename files based off the directory they were in so they would be uniform. I did the same thing and commented almost every line to remind me what it did just in case I had to come back later to use it as a reference. It took me all day aswell, but was 100 lines of codes (lots of comments and spaces though). Looking at your code it looks like it is simple enough, but with a few complexities that you should feel accomplished that you got it done in a day.

Please, Log in or Register to view quotes

The edge cases always get you. That is the one thing I have learned with making RAT. You get it working perfectly, then all of the sudden there is a wrench that is thrown into your plans. For me it has happened a lot with the bunkr extractor. I get everything right then, take a breath, run it on a new link and all of the sudden there is an odd ball name, or a change in the site layout, or how they configure their backend that fucks everything up and you have to go fix. I understand how spaghetti code gets written now because you just want shit to work.

Please, Log in or Register to view quotes

Ya that should be fine. That is how all UTF-8 characters are displayed and stored, it's just that your terminal and browser convert the bytes into the emoji/character they represent when they print them to the screen. When you look at the raw json it is showing you everything literally so that you why you see the bytes. If you print them out to the screen they will be converted.

Please, Log in or Register to see links and images

Please, Log in or Register to view quotes

The first organic project is always the hardest. At least it was for me. I spent way too long in tutorial hell before I worked up the courage to actually try. It also took me a day to write that first file renamer script. After I did it though there was a click for me and realized the power of programming.

Thanks for the update. I am honored that I got to play a small part in your first script.

 

[glandalf]

Please, Log in or Register to view quotes

Yeah I thats why I added so many comments, so that when I come back to it, or just refer to it, I can quickly see what each part is doing. Also helped me whilst I was scripting it to compartmentalise each section. A lot of the day was spent reading documentation and watching some videos on specific functions and methods.

Please, Log in or Register to view quotes

I tried to avoid the edge cases by building a bunch of test scenarios of dummy directories I set up. Tried coming up with as many different combinations of characters/number of tags and usernames/etc. Some having ( a bit earlier in the directory than the tags didn't cross my mind until it was too late though. But you live and learn!
I can imagine scripting around bunkr can be tricky, what with them constantly changing the configuration and limits, than and their servers going in and out of maintenance a lot as of late.

Please, Log in or Register to view quotes

Ah ok cool, thats fine then. This came from a case of me again not using the rat file as intended and just reading it in a text editor rather than calling the print function.

Please, Log in or Register to view quotes

I come from the world of databases thus I know SQL quite well, so the programmatic thinking is/was there for me already. Although some similarities, programming is a whole different world though, so was still quite the jump and a lot to get my head around. It helped having something I wanted specifically to do/achieve, that I could apply myself to, rather than "lets build a calculator", or see how many different ways you can get terminal to spit out "hello world". Don't get my wrong, that stuff is very useful and i'm glad i've done it (several times lol).

Please, Log in or Register to view quotes

Well thank you again mate, its much appreciated!


On a different note, can I ask how about your workflow for how you typically go about setting up a new model and rat file? Do you just type of the commands and arguments each time manually or do you have other ways of doing it? I was thinking about setting up a template of all the different arguments in a file, then I can just copy paste the various usernames/tags/urls/etc, and then when its populated with everything, execute it. The only issue with that if an argument is blank because the model doesn't have an Instagram or something, you can't execute the command.

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Here is the fish function I have for mkcd

Please, Log in or Register to view codes content!

This is how I would make a folder for https://simpcity.su/threads/greatmoongirl-danibunnygirl.21/

Please, Log in or Register to view codes content!

mkcd to create and move into the new directory

then use all of the different flags to fill in the names within the .rat file. -sa to sync everything (reddit, redgifs, coomer) and -d to download specific files I find from the thread that I want. This is how I would make a new user for the first time. I use a lot of keyboard shortcuts so I can switch between monitors and windows without having to move my hands so I am pretty fast.

After creation I would just need to go into the directory and run ripandtear -sa to sync everything, or ripandtear -d if I wanted to download specific files.

If you want to hash and clean up everything then you could throw in a -H -S (or -SH to combine and shorten)

 

[glandalf]

Please, Log in or Register to view quotes

ok cool, so you do just build the full command each time with only the arguments needed.

Might try and create a separate script that looks at a template and then builds the RAT command only if the argument contains a value.

Rough example:
template file

Please, Log in or Register to view codes content!

Then have a script that would turn this into something like:

Please, Log in or Register to view codes content!

So whilst I'm browsing a thread I can just copy what I want into the template as I go, so when I'm finished I can just execute it.

 

[glandalf]

johnny.barracuda

Can I ask if you'd consider adding ManyVids as a native site to store in the rat file? No worries if not.

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Ya I could do that. I will add it under the links section because it seems like you need every part of the link for it to work. I am probably going to be pretty busy this week so it might take me some time to get around to it.

 

[johnny.barracuda]

Released an update last night that fixed an issue with the file extensions after files were downloaded. RAT now makes sure the file has the correct file extension, in lower case, after it has been successfully downloaded.

 

[glandalf]

Please, Log in or Register to view quotes

No rush at all mate, in your own time, I'm just happy your considering it. Currently I'm adding them to the rat file as a generic value (e.g. -g 'ManyVids:username')

Please, Log in or Register to view quotes

Nice one. I've still got to try and write a script to go through my files and update existing uppercase extensions to be lowercase, so the sorter picks them up. If you run hash a second time in a folder that's already been hashed and sorted, the sorter removes the existing files as it sees them as dupes because the hash is in the rat file. I've got a command that will do it, just need to iterate over my directories and run it.


Sorry to keep asking for stuff, but would you mind adding .ts to the video_suffix variable in the File_Sorter.py util? Again no worries if not.

Another thing is that after upgrading to 0.9.17, I got errors about the magic module. So you might want to add python-magic to the dependencies.

 

[johnny.barracuda]

Please, Log in or Register to view quotes

I am an archivist at heart. The original idea was if you end up with identical files in the folder, but they are separated (the newly downloaded one being in the root folder and the older one being in a sub directory) RAT would keep the older one and delete the new one. The thought was keeping the older file would be more important to track the first instances of when file was uploaded. It seems like this might just be too complicated and overly anal so I changed it so it will always keep the new file and delete the old one. This should fix the problem described and I think it will be better overall. Plus the fix was as easy as changing one variable.

Please, Log in or Register to view quotes

Thanks for the callout. I added it to the setup.cfg so when you update it should download the module. I don't think too many people are using RAT so sometimes I get stuck in the mentality of "it works on my machine" and forget about others.

Please, Log in or Register to view quotes

Done. If you ever want to get the feel of working with git and developing a project feel free to make a pull request, especially for something as simple as adding "ts" to the end of a regex expression. I won't be offended.

Please, Log in or Register to view quotes

Added. Use -mv to add a manyvids link and -pmv to print them out. Ideally you should save them as the full url. The reason I store the names as just names is those websites have a simple url that you can plug the name into one spot and go to the destination. Manyvids you need both a number and the models name so I feel it would be easier and cleaner to just store the full url.

So that's it for now. Update has been pushed so feel free to download it. Please keep letting me know what bugs and quirks you find. I appreciate it.

 

[DaFapper]

Will it continue to work after reddit turns off the nsfw api feeds?

Please, Log in or Register to view quotes

source:

Please, Log in or Register to see links and images

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Using the API, like I am right now, most likely not. I was just reading about this earlier and MAYBE I might be able to use playwright to pretend I am a user to gather the links, but who knows if they will block that method. Reddit seems like it is going to become a dead end in terms of downloading content which is really unfortunate. We just need to keep an eye out for where all the girls migrate to.

 

[DaFapper]

Please, Log in or Register to view quotes

That is too bad. Actually, just saw on reddit a user mentioned this and i wonder if it could be used as a work around in the future. Since the official reddit app will have access to the full api, maybe this is feasible. I'm not a dev though so no way to know if it is or this dude is just talking out of his ass. Each user would probably have to extract their own key but if its doable than someone will probably come up with a guide

Please, Log in or Register to view quotes

 

[johnny.barracuda]

Please, Log in or Register to view quotes

I think it's one of those situations of time will tell. I don't know how it is going to be implemented so I can't start working on a work around right now. I know with their API access they say that you will only get 100 requests per minute if you use OAuth. RAT currently does that, but instead of waiting for the cool down like you are supposed to, RAT just asks for a new API as if it were a different user to reset that API limit. In theory this would mean that the decrease in API requests shouldn't effect RAT in that sense. The only problem is them saying they are going to not show NSFW content in the API. That is what will break RAT.

As long as we can still download direct file links (i.redd.it/asdf123.png) we should be fine. I should be able to build a generic scrapper to find the images/video. It will just cost them a lot more data because I will need to load the whole website, but fuck them.

 

[glandalf]

Please, Log in or Register to view quotes

To be honest I agree with you on the archivist mentality and keeping the first file, but I guess this works too. I've gone through my entire library now updating uppercase file extensions to lower, so I shouldn't have the issue I was having with -H removing files that were hashed already but not sorted due to them having uppercase extensions. Have also now finished re-sorting and hashing every file in my library. It took a few days to iterate over everything, but its done now!

Please, Log in or Register to view quotes

Nice I will test installing it again, because when I installed it on my windows system I had to install a different package than the one I install in Linux. Yeah I get the "it works on my machine" mentality, though this tool is great, so I can see it gaining traction :)

Please, Log in or Register to view quotes

Awesome thank you! I've dabbled a little with git for my job, but nothing collaborative, so my usage is basically just pushing and pull scripts from a directory only I work in, so pull requests is not something I've dealt with. I will do some research on it.

Please, Log in or Register to view quotes

Thank you again. That makes sense, I will store the full URL then.

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Part of the reason I suggested you do a pull request is I have never used git in a collaborative way as well and I want to learn too. The only thing I have done is pushed code to my repo. That's it.

 

[glandalf]

Got my script that builds the rat command from a template working.

I created a json file like this:

Please, Log in or Register to view codes content!

I then will populate this with values as I browse a thread.

Then I run:

Please, Log in or Register to view codes content!

So for example, if a model doesn't have a instagram, thus I don't populate anything in that key in the json, -i just won't be appended to the command.

 

[johnny.barracuda]

Please, Log in or Register to view quotes

Ahhhh ok. I understand what you are doing now. Earlier when you first mentioned it, I thought you were going to build a script to input the commands into RAT for a brand new model. I thought you were being a bit redundant, but you were talking about for the models you already have and made templates for.

Looks really good. Pretty simple and I hope it works well. Hopefully after this you will truly be able to harness the power of RAT and it will be helpful for you in it's natural state.

 

[johnny.barracuda]

Pushed a new update that allows for direct imgur media to be downloaded again. Ever since Imgur banned NSFW content they seemed to turn off their API (it just gives you 429 error codes), at least with the API auth code I am using. I realized that direct links to imgur media still works (i.imgur was here once, but it's now gone/asdf123.png). RAT no longer can download albums, galleries or media pages (imgur was here once, but it's now gone/asdaf123), but can still download the direct media. I mainly implemented this so I can still get pictures from reddit user profiles that are still using imgur.

 

[johnny.barracuda]

Pushed a new update. Fixed a minor bug in the hasher where I forgot to add new files to the rat.