Virtual Name Server Compatibility

The sitemap.class in the Curl multi sitemap package is fast and potentially very useful except that it doesn't work with named-based virtual hosts, because it doesn't send proper HTTP Host headers. This is required by HTTP/1.1 and is implemented by all modern HTTP/1.0 browsers as an extension.

Would it be difficult to implement HTTP Host header transmission to make the package compatible with virtually all web server configurations?

Hi,

I don't have an option to try it out on virtual hosts, but I've included Host header in each class HTTP request.

Let me now if that works for you, or if you have any other suggestions.

I am sorry to have misled you about your original code not supporting name-based virtual web servers. Actually it did. After more investigation I determined that I wasn't able to access my name-based virtual servers because curl wasn't connecting to a proxy server that has to be used from my LAN where public IP addresses obtained from DNS are not valid. After adding proxy-server support to your original class library I could access my name-based virtual servers with no problems. However, the modified class library you posted doesn't work, so you should revert back to what you had in the beginning.

To implement proxy-server support I added this function near the top of your sitemap class:

//set a proxy host and port (such as someproxy:8080 or 10.1.1.1:8080
public function set_proxy($host_port){
$this->proxy = $host_port;
}

Then I inserted proxy option setting code below each of the three CURLOPT_URL setting lines further down in the class, like this:

url_setopt($curl, CURLOPT_URL, $url);

if (isset($this->proxy) && !$this->proxy == '') {
curl_setopt($curl, CURLOPT_PROXY, $this->proxy);
}

Then I added a set_proxy() function call near the top of your generate.php script, like this:

//declaring class instance
include("./sitemap.class.php");
$sitemap = new sitemap();

//optionally set proxy server name and port or ip and port
//comment-out or set to an empty string to disable proxy use
$sitemap->set_proxy('10.1.1.1:8080');

Those changes make it easy to optionally enable proxy-server compatibility.

-Bob

Thank you very much for cooperation.

I removed Host header modification and added your proxy method. ;)