I'm confused on what that means. it will skip which files and mark them as completed in the database?
right i am familiar with all the log files and the different errors. but once i have a list of URLs i dont want it to try and download anymore, what do i do with them. I was thinking you were saying i can add them to the unsupported_urls_filename txt file, but that didnt work. it just overwrote it and those urls still were in the download/scrape error logs.
Exactly. With that option enabled, running the program will just add URLs to history without actually attempting to download them. Then you can disable that option again, and every time you run the program from then on, it will ignore those links because they are now in your history.
Ahh ok thanks... Seems dangerous if you were to forget to toggle it back off then you basically fuck up your history database. Unless I misunderstand how it works. Would be nice if you just add links you want to ignore to some file.
I'm not sure if i did something wrong, but it only worked with some of the urls for some reason. Also, does it work with the scraped errors as well? I put the dead links from the scrape log in there as well and it didnt seem to do anything.
Some links are in a format that the scraper won't accept, it'll need the input URL not the output url.
not a CDL Problem, their DL Servers are Slow since about the end of the first / Beginning of second week of March.
You are running an ancient version of CDL that needs to be updated.
for example this one
My guess is that
skip_download_mark_completed probably only applies to links that are about to be downloaded. That if the scraping fails for a link, it won't proceed to attempt downloading the link, so that option won't kick in at all.
yeah seems like that may be the case. seems like we should somehow have a way to "mark downloaded" urls that fail in scraping. so there isnt so much noise in the logs of it failing each time.
