I'll probably do written updates every few days as to the progress I'm making on the rewrite. You can follow along more closely
.
I'm about 5.5 hours in, actual development time so far, I've got a time log going for myself as I'm curious how long it will take.
So far, I've re-written majority of Cyberdrop-DL's director, telling the program what portions to work on (argument gathering, scraping, downloading, checking for completeness, ... sorting (not written yet)... I've rewritten the structure of the SQL database that will keep track of already completed downloads, however I'm changing it to keep track of what is scraped, as well as building the coomer db to reduce hits to the website and lower ratelimit hits. I'm unsure yet on whether or not I'll be able to effectively migrate over the old DB, but I will worry about that after everything else is finished.
The SQL rewrites so far will both slow down and speed up scraping (as well as having the option to ignore said history). It will catalogue what has been scraped and whether it was completed or not, Scraping an album again will first check whether or not it has a history, it will use the history to selectively scrape links from the album, or skip scraping the album entirely if everything was completed. (this will also make finishing .part files a heck of a lot easier).
The config file has also been re-written to include more options, as well as being sectioned more appropriately. Old configs won't work, but that's not that big of an issue. The old one will be purged and the new one will be in it's place. The config is also loaded before logging starts, which means all files can be relocated using the config (and the config itself can be specified with command line argument).
I've decided to change Unsupported_URLs into a CSV of two columns, referrer and the actual link itself. I'm not sure how feasible that is, but I'll play around with it and see.
I'm just wrapping up the internal dataclass restructuring, and from here I can go on to rewrite the scraping and continue down the rest of the program.
Needless to say, there is a long way to go. It is a long way off, but before I push it to release, I will be trying to gather beta testers to test it out and put it through the ringer. There's a lot the program does, and it'd be difficult for me to test it all by myself... Plus it seems like it's you all who seem to find a way to break things more than myself.
I'm about 5.5 hours in, actual development time so far, I've got a time log going for myself as I'm curious how long it will take.
So far, I've re-written majority of Cyberdrop-DL's director, telling the program what portions to work on (argument gathering, scraping, downloading, checking for completeness, ... sorting (not written yet)... I've rewritten the structure of the SQL database that will keep track of already completed downloads, however I'm changing it to keep track of what is scraped, as well as building the coomer db to reduce hits to the website and lower ratelimit hits. I'm unsure yet on whether or not I'll be able to effectively migrate over the old DB, but I will worry about that after everything else is finished.
The SQL rewrites so far will both slow down and speed up scraping (as well as having the option to ignore said history). It will catalogue what has been scraped and whether it was completed or not, Scraping an album again will first check whether or not it has a history, it will use the history to selectively scrape links from the album, or skip scraping the album entirely if everything was completed. (this will also make finishing .part files a heck of a lot easier).
The config file has also been re-written to include more options, as well as being sectioned more appropriately. Old configs won't work, but that's not that big of an issue. The old one will be purged and the new one will be in it's place. The config is also loaded before logging starts, which means all files can be relocated using the config (and the config itself can be specified with command line argument).
I've decided to change Unsupported_URLs into a CSV of two columns, referrer and the actual link itself. I'm not sure how feasible that is, but I'll play around with it and see.
I'm just wrapping up the internal dataclass restructuring, and from here I can go on to rewrite the scraping and continue down the rest of the program.
Needless to say, there is a long way to go. It is a long way off, but before I push it to release, I will be trying to gather beta testers to test it out and put it through the ringer. There's a lot the program does, and it'd be difficult for me to test it all by myself... Plus it seems like it's you all who seem to find a way to break things more than myself.
