Ahoy Programming World

Sockets Programming Guide for Python
22y key-ets-hyu

Created: April 29, 2004
Last Updated:

---

INTRODUCTION

The use of sockets started during the development of the BSD(Berkeley Software Distribition) flavor of UNIX. There are many other forms of IPC(inter process communication), but sockets are the most widely used mechanism.

A socket is just like an ordinary file descriptor object. I/O operations could be done with it... but with sockets, you'd be able to do I/O across networks. So instead of reading and writing from an ordinary file, you'll be sending and receiving network data.

So for this thing to work(like any other mode of communication), there should be the concept of source and destination. This also holds true for those of us who are into "self-talk" (wherein the source and destination are one).

- person "A" tells something to person "B".
- mother bear sends i-miss-u postal mail to baby bear
- an employee calls in sick to the company nurse

Some things could be inferred here.

First, the means of communication. There is the concept of "language". The mechanism in which thoughts and ideas are represented and carried in the form of spoken words or actions(in the case of sign/body language) or to something more concrete or easier to understand(unless if we have telepathic abilities). This concept is important in the exchange of information. For two non-linguistic people to understand each other, they should be both conversing using a common language.

Second, the mode of addressing, the way the source makes it known to the destination that it is the target of the message. This is where the concept of "names" come in. For example, when baby bear wants to say something to mother bear, he usually calls her attention by calling her name:
"Mama bear, I want to have some of that bear brand you keep in our storage room".
This concept is also employed by the postal service thru the use of postal codes and location identifiers:
Mr. Ctrl A. Del
Qewrty Kyeboardz
Asiatown, IT Park, Apas
6000 Cebu City

Now, we'll just have to stick to these two concepts... we will eventually stumble upon the other issues as we go along with this tutorial, so there's no need to worry. Let's just work on these two concepts first since they're the basic tools we'll need to get started on sockets programming.

(Where were we? Oh...) The phone system implemented the use of the source-destination concept thru the use of telephone wires, satellites, and of course, phone devices.

And just as the telephone system makes use of telephones in order for the exchange of information(otherwise known as communication) to take place, sockets programming makes use of... what'dya know, SOCKETS!

The source and destination should have sockets! Whoohooo! Eureka!

---

SOCKETS

To create a socket, you need to specify what addressing family to use and the type of the socket which usually indicates what kind of actions you could do with the socket.

    socket( AF_INET, SOCK_STREAM )

Now now. No need to worry about the other things you could plug into the two parameters. But if you wanna know some iddy biddy things about them, then here goes...

For the first parameter:
	AF_INET	- internet style of addressing is to be used
	AF_UNIX	- unix domain addressing is to be used
	AF_PACKET	- low level packet interface
For the second parameter:
	SOCK_STREAM	- connection-oriented; provides sequenced, reliable, two-way,connection based streams.
	SOCK_DGRAM	- connection-less; supports datagrams (connectionless,unreliable messages)
	SOCK_RAW	- Provides raw network protocol access

*Actually, I got most of the info from the man pages of socket(C language). And if you're familiar with sockets programming in C, then understanding python sockets will be very easy.

The discussion in this "tutorial" is limited only to the internet style of addressing and to the SOCK_STREAM and SOCK_DGRAM socket types. And now, you might ask "What is this 'internet style of addressing'?" and I'll tell you it's the use of IP addresses(the ones that resemble some form of far-out social security numbers).

And for the sake of simplicity (and for my sanity), we will not venture out into the real dirt of IP addressing and on the how, when and why this mode of addressing came into being. They're there and let's just use them for our Mama-bear-lemme- have-some-of-that-bear-brand code snippets. (I am not a sockets guru nor Brian "Beej" Hall, although I adopted the tone he used in his C sockets tutorial and I shall always be grateful to him and his doc; And if you have the time you may visit his site www.ecst.csuchico.edu/~beej/guide/net)

So instead of "Mama Bear", postal codes and nicknames, IP addresses will be used in sockets communication. So say hello to localhost and 127.0.0.1, they'll be your best friends from now on.

As for the difference between SOCK_STREAM and SOCK_DGRAM, it is 'synonymous' to the difference between the telephone system and the postal service. In the telephone system, before any communication could take place, a connection should first be established. While in the postal service, there would be no need for an actual connection between the sender and the receiver... it would be up to the post man/mail delivery trucks to dispatch the mail. So there really would be no guarantee that the mail will arrive at the right destination point or whether it will arrive on time.

It is also worth noting that the person at the receiving end of a call is not strictly the "receiver" since once the connection is established, both the caller and the receiver of the call may converse at the same time. While in the postal system, if the receiver wants to make a response, a mail needs to be sent to the sender (in this case the receiver becomes the sender and vice versa).

Here's a code snippet for creating a socket object in python:

    from socket import *
sock=socket( AF_INET, SOCK_STREAM )
sock2=socket( AF_INET, SOCK_DGRAM )

Now, that was easy. But we are nowhere close to doing network I/O, so let's proceed to the next method.

    bind( ( host, port ) )

The bind method "binds" or associates the socket to a specific port in the local machine. The port where the socket is to be associated with must be "unused" by other sockets, or else you'll get an error or an exception. Let's continue our code snippet(eventually this will no longer be a snippet but a functional program code)

    from socket import *
try:
sock=socket( AF_INET, SOCK_STREAM )
sock2=socket( AF_INET, SOCK_DGRAM )

# We'll use the SOCK_STREAM socket
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )
except:
print "error detected"
sock.close()

As you can see ''(emptry string) is the value used for the host. This is quite similar to using the localhost address or 127.0.0.1. But using '' leaves it to the python interpreter or the system(i dont know which) to assign the IP address of your local machine to the bind method (aint that nice?).

As for the port number, you can safely specify any value from 49152 to 65535, although in some code the values starting from 1025 are used as port numbers(like our sample code). It should be noted that some numbers are reserved. For example port number 80 is for HTTP, 21 is for FTP, 23 for telnet and 22 for SSH remote login. You can refer to the website of IANA(The Internet Assigned Numbers Authority), it is the organization responsible for assigning port numbers for certain types of communication (here's the url: http://www.iana.org).

Here's the first 12 lines of the "port-numbers" file from IANA:

    PORT NUMBERS

(last updated 2004-02-26)

The port numbers are divided into three ranges: the Well Known Ports,
the Registered Ports, and the Dynamic and/or Private Ports.

The Well Known Ports are those from 0 through 1023.

The Registered Ports are those from 1024 through 49151

The Dynamic and/or Private Ports are those from 49152 through 65535

Next is the listen method, it is used to wait for connection requests from clients. It only accepts one parameter, the backlog.

    listen( backlog )

The backlog indicates the number of pending connection requests to be queued. Thus, the minimum acceptable value is 1 and the maximum is system-dependent. The values often used are 1, 5 and SOMAXCONN.

    from socket import *

sock=socket( AF_INET, SOCK_STREAM )
try:
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )

# Listen for incoming connection requests
sock.listen( 5 )
except:
print "error detected"
sock.close()

Client applications are now able to connect to our server. But that would be useless since our server needs to invoke another method for accepting connection requests, and that method is accept.

The accept method has no input parameters but it returns two values.

     
 accept()

First is the client socket that would be created once the connection request is accepted. This is the socket needed to communicate with the client app. Second is the address information about the client. This is a tuple consisting of the client's IP address and the port number that was assigned for the connection session.

    from socket import *

sock=socket( AF_INET, SOCK_STREAM )
try:
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )

# Listen for incoming connection requests
sock.listen( 5 )

# Accept a connection request
client, address = sock.accept()
print "Connection request received from", address[0]
except:
print "error detected"
sock.close()

*You can test this code by doing a telnet on the port 9000(given of course, that u ran our code via the python interpreter) ex: telnet 127.0.0.1 9000

Now that you have a socket for the client, you can now send and receive data from the client with the methods send and recv(Yes! Finally, we're getting somewhere.)

To send the text "Baby bear, here's your milk." you need to do the following

    client.send( "Baby bear, here's your milk." )

The method returns the actual bytes sent. So you might want to do some checking on the return value if you want to be sure that the message was sent in one go. But then this error scenario seldom occurs, so if you're not sending a HUGE chunk of data and if there is only light network traffic, then you have nothing to worry about.

And to receive incoming data from the client:

    client.recv(  )
ex: client.recv( 100 )

The recv method is by default blocking, which means if the other side of the connection doesnt send any data, your application will block/wait until there is data to receive. There are many mechanisms that you can use to go around this blocking issue, one common way is by using non-blocking sockets and performing non-blocking receive/read attempts. Another common technique is thru the use of the select and poll methods. Threading also could be employed. Only the select mechanism will be discussed in this tutorial (but only later on and not at this point, so you'll just have to wait).

*The recv method is also used for the detection of disconnection events. If the data from recv has a length of zero(or if recv returns a blank) , it means that the remote side of the connection has already disconnected(An example for this will be shown later on). This is also true for the send method. If you tried to send the message "hello world" and got a return value of 0, then it could be an indication of network trouble, network buffer overload or socket disconnection(u might as well close ur side of the connection).

So here's our modified server code(yep, that's right, the code we've been modifying all this time is of the server-type category):

    from socket import *

sock = socket( AF_INET, SOCK_STREAM )
try:
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )

# Listen for incoming connection requests
sock.listen( 5 )

# Accept a connection request
client, address = sock.accept()
print "Connection request received from", address[0]

# Send data to client
client.send( "Baby bear, here's your milk." )

# Receive data from client
data = client.recv( 100 )
print "Baby talk:", data
except:
print "error detected"
sock.close()

You can test this code again with the telnet command (And if you compare the results of windows' telnet.exe and the telnet command of linux, you'll find some intereseting differences).

A final note for the sending and receiving data to the client: they only work when the socket is active or currently connected. Doing a send and recv on an inactive socket will cause an error or an exception (use the socket object created for the client and never the server socket object... or else you'll get an error/exception).

Ooops... Did i mention that send sometimes may not send the exact number of bytes that you had specified and that even recv may not always retrieve the exact number of bytes that you had intended/expected it to? And if the implications of what I've just stated bothers you in any way, then here are some functions that may help ease your pain =):

    
def sendAll( sock, msg ):
  len, totalSent,tmp = len( msg ), 0, 0
  while ( totalSent < len ): 

Of course, just like the other code samples here, the functions are not perfect nor free from errors. You may have to add some error handling for the return value of recv and send in case of disconnection events.

---

CLOSING A SOCKET

And before we proceed in creating our client code, let's examine the close() method. I know you've been itching to ask why in the world are we doing a close() on our socket(it's practically in all of our sample code versions!). Here's why:

Sockets, like almost everything else, is a resource. There's a limit to how many descriptors a user can open or how many open descriptors the operating system will allow. Port numbers are also an issue, yep, there's also a limit and port numbers cant be associated to two different servers(considering also that some port numbers are reserved).

Having learned that, you shouldnt forget to do a close() on your sockets. An exception will not be thrown or raised when you perform multiple close() operations on a socket, so dont worry whether the socket was properly bound to the target port, JUST DO IT. This thing also goes for ordinary file descriptors, close them after using or else your system will ran out of file descriptors.

The close() method is also used, aside from the save-the-sockets-save-the-world reasons, to indicate that your side of the connection(regardless of the perspective, the server's or client's) is no longer interested and wants "out". Further sending or receiving on a closed socket will result to an error or an exception(so please spare urself from the pain).

There are also merits in doing a shutdown() on the socket before actually closing it.

---

HOW-TO FOR CLIENT (SOCK_STREAM)

For the client code, you'll also need to create a socket object:

    from socket import *

# Create socket object
client = socket( AF_INET, SOCK_STREAM )

And afterwards, you should invoke the connect() method to issue a connection request to the server:

    connect( ( HOST, PORT ) )

The first parameter is the IP address or hostname of the machine where the server is running. And the second parameter is the port where the server is listening for incoming connection requests.

    from socket import *

# Create socket object
client = socket( AF_INET, SOCK_STREAM )

try:
# Connect to server
client.connect( ( '127.0.0.1', 9000 ) )
except:
print "error detected"
client.close()

Connecting to a server that doesnt exist or not currently running, or even to a server where there is no route, will result to an error or an exception. So please check the status of the connect operation before proceeding to send and receive data from the server.

If the connect operation goes well, your client app can now send and receive data from the server.

    from socket import *

# Create socket object
client = socket( AF_INET, SOCK_STREAM )

try:
# Connect to server
client.connect( ( '127.0.0.1', 9000 ) )

# Receive a message from the server
data = client.recv 100 )
print "From mama bear:", data

# Send a message to the server
client.send( "Thank you mama" )

except:
print "error detected"
client.close()

You can test this code by running the server code and the client code(In this order.. or else connect() will return an error).

---

SELECT (A POLLING MECHANISM)

You might have noticed that the communication between the server and client is a little bit scripted in the sense that there's some sort of switching... kinda like a snakes-and-ladders or monopoly board game wherein each player has his turn.

Server	Client
send	recv
recv	send
send	recv
recv	send

Such an arrangement is boring. It's a good thing human dialogue is much more interactive... (although there are some rules like "that it's best to have women say the last word", "let the parents speak first", "let the mafia boss have the first say or else...", "teachers talk while the students listen", etc...)

An extreme opposite of this "boring" scenario is when both applications keep on sending data to each other (with no recv's ):

Server	Client
send	send
send	send
send	send
send	send
send	send

The result is chaos... no real communication takes place. What we must do is to employ a concept of send-what-u-want-and-receive-when-there-is-incoming-data thru the use of select. There is, however, also the concept of "read-when-must- and-send-when-u-can", but we will be using the former concept and not this one.

    select( readlist, writelist, exceptlist [, timeout] )
returns: ( modif readlist, modif writelist, modif exceptlist )

The select methods take 3 to 4 parameters. This method will simply wait/block until one or more descriptors are ready for reading/writing(or some other form of I/O).

The first parameter is the list of descriptors that should be monitored if there is data to be read. Here select is used to see if read will block or not

The second parameter is the list of descriptors that should be monitored if they are ready for writing.

The third parameter is a list of decriptors that will be watched for exceptions. (I dont know much about the exceptlist...)

The fourth parameter is optional. It indicates the timeout or the maximum time that select will have to monitor the 3 lists. Once the timeout expires, select will immediately return letting the application do other tasks. If no value is specified(or the value None ), the select method will block forever until any of the descriptors are ready.

For the first parameters, a blank list [] could be specified. So if you only want to monitor which descriptors have data to read, you'll do this:

    select( [ fd1, fd2 ], [], [] )

*Take note that in windows, only sockets can be placed inside the lists while in UNIX/Linux, all kinds of file descriptors could be used.

The select method will return 3 lists which are subsets of the original lists. For example, if you want to monitor for data to read in the descriptors fd1, fd2 (with a timeout of 1 second):

    r, w, e = select( [ fd1, fd2 ], [], [], 1 )

The possible contents of r would be:

[]		- no descriptors have data to be read
[ fd1, fd2 ]		- both fd1 and fd2 are ready for reading
[ fd1 ]		- only fd1 has data to be read
[ fd2 ]		- only fd2 has data to be read

Here's our modified server code:

    from socket import *
from select import *

sock = socket( AF_INET, SOCK_STREAM )
try:
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )

# Listen for incoming connection requests
sock.listen( 5 )

# Accept a connection request
client, address = sock.accept()
print "Connection request received from", address[0]

# Send data to client
client.send( "Baby bear, here's your milk.\r\n" )

while 1:
rList, wList, eList = select( [ client ], [], [] )

# examine readable descriptors
for fd in rList:

  # Receive data from client
  data = fd.recv( 100 )

  # Check if NOT disconnection event
  if ( len( data ) > 0 ):
      # print data received
      print "Baby talk:", data

      # send message to client
      fd.send( "How sweet baby bear: " + data +"\r\n")

  # Disconnection event
  else:
      # raise a string-exception just to exit from while loop
      raise "disconnection"

except "disconnection":
# print message regarding disconnection event
print "client has disconnected"
except:
print "error detected"
sock.close()

Run the modified server code and test it with the telnet command. You'll notice that every time you enter a string followed by a carriage return(for linux/unix) or a single character(windows), the server will intercept the message and send back the original message "prefixed" with the text "How sweet baby bear".

If you examine our latest server code version, you will notice that the length of the data coming from the recv method is always checked against zero. This is done to see if the remote side of the connection has disconnected. Detection of disconnection events are very important, and if you'll remove the checking for disconnection events and run the code, you'll find out why. =)

There is also another benefit of using select. In all of the previou s versions of our server, you might have noticed that our server only accepts a single client. Which is again another boring behavior that we need to fix. And to "remedy" this situation, we will just simply modify our server code(DUH! sample code modification is what we've always been doing. Wow, where am i getting these we'll-just-simply-modify talk?):

    from socket import *
from select import *

sock = socket( AF_INET, SOCK_STREAM )
try:
# Associate the socket with port 9000
sock.bind( ( '', 9000 ) )

# Listen for incoming connection requests
sock.listen( 5 )

# Create list of fds
fdList=[sock]

while 1:
rList, wList, eList = select( fdList, [], [] )

# Examine readable descriptors
for fd in rList:

  # Check if server socket
  if ( fd == sock ):
      # Accept a connection request
      client, address = sock.accept()
      print "Connection request received from", address[0]

      # Send data to client
      client.send( "Baby bear, here's your milk.\r\n" )

      # Add client socket to list
      fdList.append( client )

  # For the other descriptors
  else:
      # Receive data from client
      data = fd.recv( 100 )

      # Check if NOT disconnection event
      if ( len( data ) > 0 ):
          # Print data received
          print "Baby talk:", data

          # Send message to client
          fd.send( "How sweet baby bear: " + data +"\r\n")

      # Disconnection event
      else:
          # Print message regarding disconnection event
          print "client has disconnected"

          # Remove client from list of fds
          fdList.remove( fd )
except:
print "error detected"
sock.close()

Now, it is possible for a thousand baby bears to connect to our mama bear server... I know, I know this mama-bear-baby-bear thingee is getting too cute for comfort(unless if you're into bears). So lemme rephrase that...

(Ahem)It is now possible for our server to have multiple client connections (There! R u happy now?). Each client upon "admittance" will receive the text "Baby bear, here's your milk." (I bet you're saying this welcome message needs changing too). The server application is now also able to receive messages coming from all of its clients. You can test all these by making multiple telnet sessions to the server and you would also then be able to observe that our new server will no longer terminate once a disconnection event is caught.

---

ABOUT CONNECTION-LESS SOCKETS

The previous code examples were on SOCK_STREAM sockets, also called TCP sockets since the Transmission Control Protocol is connection-oriented. We'll now move on to SOCK_DGRAM sockets or UDP sockets(for User Datagram Protocol).

It was mentioned earlier in this tutorial that the use of connection-less sockets is synonymous or akin to the postal system. With connection-less sockets, there is no actual connection between two remote applications, just like there is no need for a physical connection to be established between two mailboxes before their owners cound send letters to each other. You will begin to get a clearer picture of this concept by examining the following:

    Peer A:
(yep, "Peer A" since the mama-bear and baby-bear-routines have been scratched out)
sock = socket( AF_INET, SOCK_DGRAM )
sock.bind( ( '', 9000 ) )
data, address = sock.recvfrom( 100 )
print "from: " + address[0]
print "data: " + data
sock.close()

Peer B:
sock = socket( AF_INET, SOCK_DGRAM )
sock.sendto( "Hi Peer A!", ( '127.0.0.1', 9000) )
sock.close()

Or the example...

    Mailbox B:
sock = socket( AF_INET, SOCK_DGRAM )
sock.bind( ( '', 9001 ) )
sock.sendto( "Hello MailboxA owner!", ( '127.0.0.1', 9000 ) )
data, address = sock.recvfrom( 100 )
print "from: " + address[0]
print "data: " + data
sock.close()

Mailbox A:
sock = socket( AF_INET, SOCK_DGRAM )
sock.bind( ( '', 9000 ) )
data, address = sock.recvfrom( 100 )
print "from: " + address[0]
print "data: " + data
sock.sendto( "Hello to you too!", ( '127.0.0.1', 9001 ) )
sock.close()

As you might've noticed, the code for the UDP-based client is much shorter than the code for the TCP-based client. In the client code, connect is no longer needed (Ow... so that's why they termed it 'connection-less'... because there's no connection!!) just like that there's no listen and accept in the UDP-based server.

---

FOR ADDITONAL READING AND REFERENCES

OFFICIAL DOCUMENTATION
The Python Docs
url: http://www.python.org/doc/

BOOK/S CREATED BY THE NETWORK/UNIX GURU:
Unix Network Programming
by W. Richard Stevens and published by Prentice Hall.

THE BEEJ GUIDE
Beej's Guide to Network Programming
url: www.ecst.csuchico.edu/~beej/guide/net

PORT-NUMBER REFERENCE
The Internet Assigned Numbers Authority
url: www.iana.org

FOR THE MUCH DIRTIER STUFF, YOU MAY CONSULT THE
RFC(REQUEST FOR COMMENTS) DOCUMENTS:
RFC-791 The Internet Protocol (IP)
url: http://www.rfc-editor.org/rfc/rfc791.txt

RFC-768 The User Datagram Protocol(UDP)
url: http://www.rfc-editor.org/rfc/rfc768.txt

RFC-793 The Transmission Control Protocol (TCP)
url: http://www.rfc-editor.org/rfc/rfc793.txt

RFC-854--The Telnet Protocol
url: http://www.rfc-editor.org/rfc/rfc854.txt

*Although the beej guide and the books by Richard Stevens are used mostly by C programmers, they will also serve and prove to be good sources of information for python socket programmers. Much of the python socket library was derived from C, you could just simply discard the weird-looking structs because the system calls(functions/methods) used are identical.

# posted by SleepyHead : 11:53 AM  6 comments