Tools Scripting help for crawling a website for content

BobDobbs86 · Jan 28, 2024

I know this is a longshot but I am looking for some help creating a script that will test a possible web address to see if it is a valid page and then drop the valid links in a file for me to then grab content from. Basically, I know that there is a website of URL+random six character string that has content. So can someone help me figure out how to iteratively test all possible variables of that URL-"random six character string" and then write out the one that isn't 404 to a file so I can then grab the content from it (I know there is no password protection or anything past the random six character string)?

yticpmis234 · Jan 28, 2024

how about this

Please, Log in or Register to see links and images

not entirely what you want, because it will not test the url+random_six_character requirement , just follow all existing links on a page.

Try Bing Copilot GPT-4 , it will usually write a solution for you, as long as the task is broken up in to highly descriptive parts, you will have to debug it , by pasting all errors and asking it to correct the script., but it will work.

I fed it your question and this is what it spat out,

Question

help creating a script that will test a possible web address to see if it is a valid page and then drop the valid links in a file for me to then grab content from. Basically, I know that there is a website of URL+random six character string that has content.
how to iteratively test all possible variables of that URL-"random six character string" and then write out the one that isn't 404 to a file so I can then grab the content from it
use python, thanks chat gpt

Answer

Python:

Please, Log in or Register to view codes content!

Or search for an existing project on github
Please, Log in or Register to see links and images
👀

sexyegirlsaccount · Feb 10, 2024

Did you work it out or do you need help still?

BobDobbs86 · Feb 10, 2024

sexyegirlsaccount said:
Please, Log in or Register to view quotes

It got me most of what I needed. Once I tweaked it it is doing what I needed. Thanks.

Tools Scripting help for crawling a website for content

BobDobbs86

Tier 3 Sub

yticpmis234

Bathwater Drinker

Try Bing Copilot GPT-4 , it will usually write a solution for you, as long as the task is broken up in to highly descriptive parts, you will have to debug it , by pasting all errors and asking it to correct the script., but it will work.

sexyegirlsaccount

Bathwater Drinker

BobDobbs86

Tier 3 Sub

Tools Scripting help for crawling a website for content

BobDobbs86

Tier 3 Sub

yticpmis234

Bathwater Drinker

Try Bing Copilot GPT-4 , it will usually write a solution for you, as long as the task is broken up in to highly descriptive parts, you will have to debug it , by pasting all errors and asking it to correct the script., but it will work.​

sexyegirlsaccount

Bathwater Drinker

BobDobbs86

Tier 3 Sub

Try Bing Copilot GPT-4 , it will usually write a solution for you, as long as the task is broken up in to highly descriptive parts, you will have to debug it , by pasting all errors and asking it to correct the script., but it will work.