saversites Posted January 16, 2018 Report Posted January 16, 2018 Folks, Why is cURL failing to fetch the page ? All this time it worked. Echoes "Page fetching problem!" <?php //Required PHP Files. include 'config.php'; include 'header.php'; //1). Set Banned Words. $banned_words = array("asshole", "nut", "bullshit"); $url = "http://devshed.com"; // 2). $curl is going to be data type curl resource. $curl = curl_init(); // 3). Set cURL options. curl_setopt($curl, CURLOPT_URL, "$url"); curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); // 4). Run cURL (execute http request). $result = curl_exec($curl); if (curl_errno($curl)) { echo 'Error:' . curl_error($curl); } $response = curl_getinfo( $curl ); //If page is fetched then replace banned words found on page. if($response['http_code'] == '200' ) { $regex = '/\b'; $regex .= implode('\b|\b', $banned_words); $regex .= '\b/i'; $substitute = 'BANNED WORD REPLACED'; $clean_result = preg_replace($regex, $substitute, $result); //Present the banned words filtered webpage. echo $clean_result; } else { //Show error if page fetching failed. echo "Page fetching problem!"; exit(); } ?> Quote
saversites Posted January 17, 2018 Author Report Posted January 17, 2018 I added this line today ... echo "$response[http_code]"; Result I get is this: 403. Googling, I see that the page devshed.com is forbidden. Since I can view the page on y browser and not manage to get cURL to fetch it then I'm guessing devshed.com has put a measurement in place to foil proxies. Switched the url to another site and that got fetched. last night, even google did not get fetched and so I suspected maybe I accidently deleted something from my code but could not spot the deletion. Hence, opened this thread to see if anyone else spots it. Now, I understand, both devshed.com and google were foiling the cURL fetch or proxy fetch. How well did I do ? Quote
saversites Posted January 17, 2018 Author Report Posted January 17, 2018 I now get error that variable $url is undefined on line 72. Notice: Undefined variable: url in C:\xampp\htdocs\...... If you check line 72, it says between double quotes: $url = "http://devshed.com"; Even if I change the url to a url who's page cURL is able to fetch, I still get the same error. This does not work either, with single quotes: $url = 'http://devshed.com'; This is very very strange! If the $url variable has not been defined then how is cURL able to fetch the page who's url is on the $url variable value ? Even though page gets fetched, I still see the error! Weird! Quote
administrator Posted January 18, 2018 Report Posted January 18, 2018 Hi, If you are getting HTTP 403 Forbidden status , it could be the website blocking unknown bots - curl would be perceived as such. Stef Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.