Tools Cyberdrop-DL: Mass Downloader for most forum sites and SimpCity threads

[ezraarya]

Anybody facing DNS resolution issues? When I try to use bunkr or cyberfile, the main pages work fine but the video doesn't load, returns 404 not found. After inspection, when turning on the VPN the IP address resolved for the media file domain (ex: media-files12.bunkrr.su) is different from when the VPN was turned off. Is this some caching issue with the popular DNS providers or something else which I'm not aware of?

After this, I tried manually setting the IP address for the media file domain resolved when VPN was turned on in the hosts file in my system and ran the cyberdrop-dl program. It still didn't work. Note, I flushed DNS and pinged the domain to confirm the address resolution before trying.

What am I missing here? Please help.

 

[My0dude]

Please, Log in or Register to view quotes

thank you legend

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

When given media-file links, the program takes them, transforms them back into stream links and rescrapes them.

 

[ezraarya]

Please, Log in or Register to view quotes

The problem is not with the scraping, the scraping works fine and the link for the media in the album are found, but the issue is when downloading them. The program terminates when the download link it scraped returns 404

2023-04-21 22:14:53,453:DEBUG:base_functions:base_functions.py:51:We are running version 4.1.12 of Cyberdrop Downloader
2023-04-21 22:14:53,683:DEBUG:base_functions:base_functions.py:51:Starting Scrape
2023-04-21 22:14:53,684:DEBUG:base_functions:base_functions.py:51:Starting:

Please, Log in or Register to see links and images

2023-04-21 22:14:54,950:DEBUG:base_functions:base_functions.py:51:Finished:

Please, Log in or Register to see links and images

2023-04-21 22:14:54,951:DEBUG:base_functions:base_functions.py:51:
2023-04-21 22:14:54,951:DEBUG:base_functions:base_functions.py:51:Finished Scrape
2023-04-21 22:15:01,559:DEBUG:downloaders:downloaders.py:260:

Please, Log in or Register to see links and images

(Not Found)
2023-04-21 22:15:01,559:DEBUG:downloaders:downloaders.py:265:We ran into a 400 level error: 404
2023-04-21 22:15:01,559:DEBUG:base_functions:base_functions.py:51:Failed Download: Tonights.Girlfriend.2017.11.17.India.Summer.mp4
2023-04-21 22:15:01,592:DEBUG:base_functions:base_functions.py:51:| [green]Files Complete: 0[/green] - [yellow]Files Skipped: 0[/yellow] - [red]Files Failed: 1[/red] |
2023-04-21 22:15:01,593:DEBUG:base_functions:base_functions.py:51:
2023-04-21 22:15:01,593:DEBUG:base_functions:base_functions.py:51:Checking for incomplete downloads
2023-04-21 22:15:03,337:DEBUG:base_functions:base_functions.py:51:Purging empty directories
2023-04-21 22:15:03,349:DEBUG:base_functions:base_functions.py:51:Finished downloading. Enjoy :)
2023-04-21 22:15:03,349:DEBUG:base_functions:base_functions.py:51:There are partial downloads in the downloads folder.
2023-04-21 22:15:03,349:DEBUG:base_functions:base_functions.py:51:
2023-04-21 22:15:03,350:DEBUG:base_functions:base_functions.py:51:If you enjoy using this program, please consider buying the developer a coffee :)

Please, Log in or Register to see links and images

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

this download works for me through the program. Probably just a bug in cyberfile.

404 is what it is, it's just not found returned by the server. Might've been too new of a file when you tried.

 

[ezraarya]

Please, Log in or Register to view quotes

Hi Jules, sorry the issue is not with the program in itself but with the new DDoS-Guard migration of all bunkr and cyberfile files. All domains resolve to the same IP "84.239.49.207" hosted by DDoS server.
That particular service does not allow IP address belonging to some regions to pass through. Analyzed the raw response and the HTML states

DDoS-Guard
Service is not available in your region
Sorry for the inconvenience caused

So I guess if you have this issue, you might have to use a VPN and there's no other workaround unless the server configuration is changed to allow said region to send requests.

 

[hkei]

Bunkr PSA: bunkr's cdn10/server5 is yet again having issues; so keep an eye on

Please, Log in or Register to see links and images

and save your thread URL for later when its back up.

 

[mussie]

I'm getting an error trying to download something from ehentai. The program outputs this:

Starting:

Please, Log in or Register to see links and images

C:\Users\<username>\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\client\client.py:71: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup.
return BeautifulSoup(text, 'html.parser')
Error:

Please, Log in or Register to see links and images

Finished:

Please, Log in or Register to see links and images

When I check the log, I see this:
2023-04-21 22:27:29,301:DEBUG:base_functions:base_functions.py:51:We are running version 4.1.12 of Cyberdrop Downloader
2023-04-21 22:27:29,314:DEBUG:base_functions:base_functions.py:51:Starting Scrape
2023-04-21 22:27:29,315:DEBUG:base_functions:base_functions.py:51:Starting:

Please, Log in or Register to see links and images

2023-04-21 22:27:29,513:DEBUG:EHentai_Spider:EHentai_Spider.py:52:Error encountered while handling

Please, Log in or Register to see links and images

Traceback (most recent call last):
File "C:\Users\<username>\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\crawlers\EHentai_Spider.py", line 31, in get_album
title = await make_title_safe(soup.select_one("h1[id=gn]").get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'
2023-04-21 22:27:29,513:DEBUG:base_functions:base_functions.py:51:Error:

Please, Log in or Register to see links and images

2023-04-21 22:27:29,514:DEBUG:EHentai_Spider:EHentai_Spider.py:54:'NoneType' object has no attribute 'get_text'
2023-04-21 22:27:29,514:DEBUG:base_functions:base_functions.py:51:Finished:

Please, Log in or Register to see links and images

2023-04-21 22:27:29,514:DEBUG:base_functions:base_functions.py:51:
2023-04-21 22:27:29,515:DEBUG:base_functions:base_functions.py:51:Finished Scrape

It seems like I'm running into this issue with any url on their site

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

You'd need to inspect element of the title on the page and show me what the element is called to fix it. That is proper on my computer and works fine.

 

[Sadada123]

Sorry if this is has already been answered, but I can't find an answer: Redgif not supported? Is it because of the site redesign or?

 

[uNderdog_101]

I think there may be some kind of bug with the history functionality, as CDD keeps downloading the same couple of files again and again.

I don't have "ignore_history" active, so previously downloaded files should not be redownloaded, and the thread they're from hasn't been updated in a month, so it's not like they're being reposted, yet they're downloaded every time I run the script. And it's just those two files, from the same the thread, among hundred of scraped threads and thousands of ignored previously downloaded files.

 

[thatsmyfetishnosrsly]

So I actually used URL notepad list as a good way to not download any dupes by keeping the actual links in there too. I know the system also recognises that the URL has been done before and skips it but now because of the DDOS shit recently it thinks its done a bunch of URLS when it definitely hasn't yet.

Switching the those two things to True definitely pushes through the new URLs but now also redownloads everything else.

Are we able to delete the History file completely, to start a-new?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

Yes it is.

Please, Log in or Register to view quotes

I'd need an example of a file that does this.

Please, Log in or Register to view quotes

You are, but the DDOS-Guard won't have marked anything as complete. It all failed in the scraping phase. Things are only ever marked as complete after a successful download.

 

[uNderdog_101]

Please, Log in or Register to view quotes

The "using a glass dildo..." vid from https://simpcity.su/threads/kiara-moon.12949/post-799927 and the "imagine me smothering you" from https://simpcity.su/threads/kiara-moon.12949/post-820636 , but specifically the ones from the pixeldrain links, since they end up in subfolders named "4Y7hNRwb" and "VwbWrEkj" respectively.

 

[elchinohouse]

Really would love to try this program but cannot ever get it to work. I know it is something on my end...im on a mac. Is there a mac tutorial...like very basic i could watch/read?

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

I can't recreate this, it's skipping as it's supposed to for me. You could open the DB file and search for the URL path and see if there's a duplicate that isn't marked as complete and change it or delete the record.

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

There isn't anything more basic than what is on the readme here:

Please, Log in or Register to see links and images

You may also need this though:

Please, Log in or Register to see links and images

 

[uNderdog_101]

Please, Log in or Register to view quotes

I opened download_history.sqlite in

Please, Log in or Register to see links and images

and went looking, changed a couple of zeroes to ones and now it stopped redownloading.

 

[snowfox925]

Any help with this simpcity is the only issue *Blacklisted site* and others work fine been like this for 2 week i thought waiting fix it because it was on simpcity side passwords are fine thanks for the help

2023-04-23 14:54:01,698:DEBUG:base_functions:base_functions.py:51:Starting: https://simpcity.su/threads/mia-khalifa.10885/post-1934858
2023-04-23 14:54:01,933:DEBUG:Xenforo_Spider:Xenforo_Spider.py:123:Error encountered while handling https://simpcity.su/threads/mia-khalifa.10885/post-1934858
Traceback (most recent call last):
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\crawlers\Xenforo_Spider.py", line 113, in fetch
await self.simpcity.login(session, url, self.quiet)
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\crawlers\Xenforo_Spider.py", line 54, in login
text = await session.get_text(domain)
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\client\client.py", line 37, in wrapper
return await func(self, *args, **kwargs)
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\cyberdrop_dl\client\client.py", line 87, in get_text
async with self.client_session.get(url, ssl=self.client.ssl_context) as response:
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 1141, in __aenter__
self._resp = await self._coro
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client.py", line 643, in _request
resp.raise_for_status()
File "C:\Users\name\AppData\Local\Programs\Python\Python310\lib\site-packages\aiohttp\client_reqrep.py", line 1005, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://simpcity.su/login')
2023-04-23 14:54:01,934:DEBUG:base_functions:base_functions.py:51:Error: https://simpcity.su/threads/mia-khalifa.10885/post-1934858
2023-04-23 14:54:01,935:DEBUG:Xenforo_Spider:Xenforo_Spider.py:125:400, message='Bad Request', url=URL('https://simpcity.su/login')
2023-04-23 14:54:01,936:DEBUG:base_functions:base_functions.py:51:

 

[Jules--Winnfield]

Please, Log in or Register to view quotes

4.1.14 should hopefully fix that issue.