Receive wrong Html response rather than xml/json in curl request

I am sending curl request to an api. for first 20 requests I get the correct xml/json response. but after that I receive HTML response. I logged the urls that caused that, the urls work fine on the browser. when I send the request again with curl it receives the correct response. in my local machine I do not receive such error at all, no matter what I do. so it's really hard to recreate the issue to be able to debug it. I contacted the provider of the api. they logged our requests and ther responses. they send the correct response. we just don't get them apparently.

this is the content of the static function that I call:

$curl = curl_init();
curl_setopt_array($curl, array(
    CURLOPT_URL => $url,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => "",
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 60,
    CURLOPT_FOLLOWLOCATION => false,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
    CURLOPT_CUSTOMREQUEST => "GET",
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
    self::$message = 'CURL ERROR : '. $err;
    return false;
} else {
    try {
        $xml_res = simplexml_load_string($response, "SimpleXMLElement", LIBXML_NOCDATA);
        self::$response = $xml_res->asXML();
        return true;
    } catch (\Throwable $th) {
        self::$message = $th->getMessage();
        Log::error($th->getMessage());
        Log::error($url);
        Log::error($response);
        return false;
    }
    
}

I get this error in my log


simplexml_load_string(): Entity: line 28: parser error : Opening and ending tag mismatch: meta line 26 and head  
production.ERROR: url that we send to
production.ERROR: <!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <style type="text/css">
        html, body, #partner, iframe {
            height: 100%;
            width: 100%;
            margin: 0;
            padding: 0;
            border: 0;
            outline: 0;
            font-size: 100%;
            vertical-align: baseline;
            background: transparent;
        }

        body {
            overflow: hidden;
        }
    </style>
    <meta content="NOW" name="expires">
    <meta content="index, follow, all" name="GOOGLEBOT">
    <meta content="index, follow, all" name="robots">
    <!-- Following Meta-Tag fixes scaling-issues on mobile devices -->
    <meta content="width=device-width; initial-scale=1.0; maximum-scale=1.0;
            user-scalable=0;" name="viewport">
</head>
<body>

<div id="partner"></div>
<script type="text/javascript">
    function getParam() {
        var query = window.location.search.substring(1);
        var vars = query.split("&");
        for (var i = 0; i < vars.length; i++) {
            var pair = vars[i].split("=");
            if (pair[0] == 'domain') {
                return pair[1];
            }
        }
        return  window.location.host;
    }
    ;

    document.write(
            '<script type="text/javascript" language="JavaScript"'
                    + 'src="//sedoparking.com/frmpark/'
                    + getParam() + '/'
                    + 'sedopark'
                    + '/park.js">'
                    + '<\/script>'
    );
</script>
</body>
</html>

I am using laravel 7. the script mentiones sedoparking! we don't have anything to do with sedoparking.

Tray2

5 years ago

Level 73

Most likely you have hit the max number of requests within a certain time span. Thus giving you the error.

amirkamizi

5 years ago

Level 3

so it's based on the max number that the API's provider has set? they informed us that they have not set any limits on the number of requests we can send.

Tray2

5 years ago

Level 73

Most likely yes. They don't have any limit on the amount of requests but rather the amount of request within a certain time limit. They might have set it to 20 requests per minute or somesuch.

Not throttling a API can cause performance issues for the provider not to mention opening for DDOS. Even unintentional ones.

I suggest setting a sleep time between the calls to keep them outside the threshold.

Please or to participate in this conversation.