NGINX $host block doesn't work?

So I tried to make an exception for my ssl clients verify. I tried something like this

set $client_allowed 0;

# Check if client cert verification is successful
if ($ssl_client_verify = "SUCCESS") {
    set $client_allowed 1;
}

# Allow requests to the specific host
if ($host = "mydomain.com") {
    set $client_allowed 1;
}

# Return 403 if access is not allowed
if ($client_allowed = 0) {
    return 403;
}

But my question is, Why I still can hit the app using my localhost? I expect it to block me by $host is not "mydomain.com"