INTRODUCTION

You should already know that a web page displayed in your web browser is described by an ordinary HTML text file, which is typically located on an Internet server (also called a host). In order to locate the file, the browser uses the URL in the form http://servername/filepath.
http stands for Hypertext Transfer Protocol and refers to the process of how the file is transferred from the server to your PC browser, called a client. The server name, also called its IP address (IP: Internet Protocol), is either in the "dotted" form, e.g. 192.41.150.141, or it is an alias, e.g. www.tigerjython.com. The file path of the HTML file begins with a slash, but it is relative to a specific document path on the server.

The exchange of data consists of the following four phases:

Phase 1: The client creates a socket object and connects to the server. The server accepts the connection and remembers the client address.
Phase 2: The client sends a request to the server and passes it the path to the desired file.
Phase 3: The server processes the request and sends the file to the client in response.
Phase 4: The client receives the response and processes it (displayed in the browser window).

REQUESTING A WEBSITE WITH HTTP

Your client program performs the phases 1, 2, and 4 and fetches the file welcomex.html located in the subdirectory py of the server document path.

import socket
 
host = "www.tigerjython.ch"
port = 80

# Phase 1
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host , port))

# Phase 2
request = "GET /py/welcome.html HTTP/1.1\r\nHost: " + host + "\r\n\r\n"
s.sendall(request)

# Phase 4
reply = s.recv(4096)
print("\nReply:\n")
print(reply)

MEMO

USING HTTPS

Instead of HTTP, more and more websites require the HTTPS protocol, where the data is encrypted using SSL (Secure Socket Layer). For such a server, an HTTP request on port 80 could result in an error message, as in the following program for the server github.com:

import socket 
 
host = "www.github.com"
port = 80

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host , port))
request = "GET /py/welcome.html HTTP/1.1\r\nHost: " + host + "\r\n\r\n" 
s.sendall(request)
reply = s.recv(4096)
print("\nReply:\n")
print(reply)

The reply is:

HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://www.github.com/py/welcome.html
  

But it is very easy to change the request to HTTPS. To do this, set the port to 443 and insert a line with ssl.wrap_socket ().

import socket 
import ssl
 
host = "github.com"
port = 443

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host , port))
s = ssl.wrap_socket(s)
request = "GET / HTTP/1.1\r\nHost: " + host + "\r\n\r\n" 
s.sendall(request)
reply = s.recv(4096)
print("\nReply:\n")
print(reply)

HTTP HEADER AND CONNECT

The response consists of a header with status information and the content with the requested file. In order to represent the website, you cut off the header and deliver the contents to a HtmlPane.

import socket
from ch.aplu.util import HtmlPane

host = "www.tigerjython.ch"
port = 80

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host , port))
request = "GET /py/welcome.html HTTP/1.1\r\nHost: " + host + "\r\n\r\n"
s.sendall(request)
reply = s.recv(4096)

index = reply.find("<html")
html = reply[index:]

pane = HtmlPane()
pane.insertText(html)

MEMO

READING THE WEATHER FORECASTS

You will probably wonder why you should apply such a complicated procedure to display a web page, when you can do the same thing using a single line insertUrl() from HtmlPane. What you have just learned makes perfect sense, however, if you do not want to display the content of the website in a browser window, for example, but if you are rather interested in only certain information embedded there. As a sensible application your program fetches the current weather forecast from the website of Australian's Bureau of Meteorology and writes it out as text.

To find out where the desired information is located, you can create an analysis program that simultaneously represents the page as a text in the console window and as a web page in your default browser.

import urllib2
from ch.aplu.util import HtmlPane

url = "http://www.weatherzone.com.au"
HtmlPane.browse(url)
html = urllib2.urlopen(url).read()
print html

MEMO

PARSING OF TEXTS

You now have the interesting and challenging task of fetching the relevant information from a long string of text, which is the parsing of a text.

The procedure is typical and the applied algorithm can be described as follows:

To extract the information of interest you analyze the text after removing the HTML tags by copying it into a text editor and looking for a token that is unique for the beginning of the information. With the string method find(), you can get the index start of this token.  You then look for a token that characterizes the end of the information and search its index end beginning at the start index. For this website the start token is "National Summary" and the end token is "State Weather".  The text in between is extracted by a slice operation [start:end].

import urllib2

def remove_html_tags(s):
    inTag = False
    out = ""

    for c in s:
        if c == '<':
            inTag = True
        elif c == '>':
          inTag = False
        elif not inTag:
            out = out + c
    return out

url = "http://www.weatherzone.com.au"
html = urllib2.urlopen(url).read()
html = remove_html_tags(html)

start = html.find("National Summary")
end = html.find("State Weather", start)
html = html[start:end].strip()
print html

MEMO

VOICE/SPEECH SYNTHESIS OF WEATHER FORECASTS

With your knowledge from the previous chapter Sound, you can have the text of the weather forecast read by a synthetic voice with just a few extra lines of code. You can simply add the following lines to the program:

from soundsystem import *

initTTS()
selectVoice("german-man")
sound = generateVoice(html)
openSoundPlayer(sound)
play() 

EXERCISES