Simple is better than complex.

sudo NOPASSWD in ubuntu

2010-11-30T20:16:00.000-08:00

In Ubuntu, do

1. sudo visudo
2. Go to the last line and add
username ALL=(ALL) NOPASSWD: ALL

I could not change the default editor to use vim by exporting the EDITOR variable. I had to use nano only, which has some quirks like it asks you the save in DOS Format or Mac Format ( I choose Mac). Btw, while using uxterm, you cannot use nano editor, because the Meta key is bound by uxterm and you wont be able to save it. I had to xfce4-terminal to do the first step. Phew!

Installing VirtualBox Guest additions on Ubuntu Guest

2010-11-13T23:18:00.000-08:00

1. Mount the cdrom from within the Guest OS, so that the guest additions will be available.

sudo mount /dev/cdrom /mount/cdrom

2. There is a VBoxGuestAdditions.bin script which you will have to execute.

Accessing the Shared Folder. From the VirtualBox Interface, a select folder in your Host Operating System as the folder which you would like to share on the Guest OS.

Remember the name of the shared folder and access it from within the Ubuntu Guest OS like this.

sudo mount -t vboxsf sharedfoldername mountpoint

Twice I made a mistake of typing vboxfs instead of vboxsf , so be careful with the spelling.

Virtualbox Tip (Ubuntu Host, Windows Guest)

2010-10-31T23:58:00.001-07:00

Accessing a Shared Folder from a Windows Guest

On Windows guests, the shared folder should appear as a standard Windows network share listed under VirtualBox Shared Folders, accessible from the Network option in the Start menu. If the shared folder is not visible within the Network window (as can sometime be the case), it may be mapped to a disk drive in a command prompt window by issuing the following command:

net use : \\vboxsvr\

where : represents the drive letter to be mapped to the shared folder and > is the name assigned to the shared folder during the creation process outlined above. For example, to map a shared folder named TempFiles to the S: drive on the guest operating system:

C:\Users\vbox> net use s: \\vboxsrv\TempFiles
The command completed successfully.

My Ubuntu Install

2010-10-29T00:07:00.000-07:00

Install Ubuntu using Ubuntu mini iso
sudo aptitude install xorg
sudo aptitude install ratpoison
sudo aptitude install build-essential checkinstall
sudo aptitude install vim
sudo aptitude install subversion
sudo aptitude install chromium-browser
sudo aptitude install mutt
sudo aptitude install ssh
sudo aptitude install socat
sudo aptitude install msmtp
sudo aptitude install fetchmail
sudo aptitude install virtualbox-ose
sudo aptitude install flashplugin-nonfree
sudo aptitude install openssl ca-certificates
sudo aptitude install procmail and follow http://ubuntuforums.org/showthread.php?t=565326
sudo aptitude install alsa and alsactl init
sudo aptitude install xfonts-terminus-oblique xfonts-terminus-dos xfonts-terminus console-terminus
sudo aptitude install irssi
sudo aptitude install ttf-inconsolata
sudo aptitude install fortune fortunes
search and run google-repo-setup.sh
sudo aptitude install picasa
sudo aptitude install cowsay xcowsay
sudo visudo and then write senthil ALL=NOPASSWD: ALL
sudo aptitude install urlview
sudo aptitude install xscreensaver xscreensaver-data

Making xterm beautiful

2010-10-27T18:41:00.000-07:00

I thought it was not possible. I was pleasantly surprised to be proved wrong.

Do the following steps.

sudo aptitude install ttf-inconsolata

Open your .Xresources file and type

XTerm*faceName: Inconsolata
XTerm*faceSize: 16

Do an

xrdb -merge .Xresources

Thanks to levin, this answered my question on "What's there in a font and why are so many of them?" very well.

Remote empty directories

2010-10-26T23:47:00.000-07:00

find -depth -type d -empty -exec rmdir {} \;

Jump in and Code

2010-10-26T12:18:00.000-07:00

Complete the Projects which you want to do.

Cyanogen Mod Upgrade on ADP2

2010-10-23T23:52:00.000-07:00

Android 1.6 OS
Dimension (LxWxTmm) 113 x 55.56 x 14.65
Display Size 3.17
HVGA Resolution
Capacitive touch screen
512MB FLASH Memory
192MB RAM
MSM7200A,528MHz chipset
GSM/GPRS/EDGE Quad band
850/900/1800/1900 MHz
WCDMA 1700/2100 MHz : BC4
2100 MHz : BC1
HSPA Speed HSDPA 7.2 Mbps
HSUPA 2 Mbps
Blutooth 2.0 with EDR
WiFi 802.11b/g
3MP Auto Focus Camera
1340 Battery(mAh)
microSD Memory Slot
USB 2.0
GPS/AGPS
Mailing list
----
ADP2 is SAPPHIRE PVT 32B DEV S-ON G
HBOOT-1.33.3007 (SAPP30000)
CPLD-10
RADIO-2.22.19.26I
Sep 25 2009, 16:22:56
----
PVT32B handsets have 192MB total RAM (96MB usable by OS) and use the Qualcomm MSM7201A CPU.
----

Follow the instructions here:
http://wiki.cyanogenmod.com/index.php?title=Full_Update_Guide_-_HTC_Magic_(32B)

startopenbox

2010-10-19T23:35:00.001-07:00

When using Ratpoison, I have F8 set to openbox Window Manager.
This is what my settings are. I do a startopenbox.sh to load the second window manager.

File: openbox.xinitrc

xrdb -load $HOME/.Xresources
xsetroot -solid gray &
feh --bg-scale /home/senthil/sentosa.jpg &
xclock -g 50x50-0+0 -bw 0 &
xterm &
openbox

File: startopenbox.sh

#!/bin/bash
xinit ./openbox.xinitrc -display :1 -- :1 2>/dev/null &

Any Advice for young programmers ?

2010-10-17T09:46:00.001-07:00

Wired.com: Any advice for young programmers?

Stroustrup: I guess giving advice is easy compared to taking it. Know your fundamentals (algorithms, data structures, machine architecture, systems) and know several programming languages to the point where you can use them idiomatically.

Know some non-computer field of study well — math, biology, history, optics, whatever. Learn to communicate effectively in speech and in writing. Spend an unreasonable amount of time on some difficult topic to really master it. Try to do something that might make a difference in the world.

Bjarne Stroustrup's old interview

2010-10-15T04:54:00.000-07:00

Computer: I know you've been heavily involved in the C++ standards process, but what other projects are you involved in currently and how will you continue working to evolve C++?

Stroustrup: I'm suffering from having successfully completed a very large project. The ISO C++ standard is done. The third edition of The C++ Programming Language is there to acquaint serious programmers with what Standard C++ has to offer and how best to use it. And The Design and Evolution documents the design decisions that shaped C++. There is more to be done, but nothing that requires a full-time language designer. Now is the time to enjoy the flexibility and power of C++ by writing code, rather than focusing on possible changes.

Apart from that, I'm taking the opportunity to learn a few things-a couple of new languages and something about how software is used in a large business-and planning some experiments in distributed computing.

C++ Standard Library

2010-10-15T04:27:00.001-07:00

Computer: How was the Standard Library defined and what impact will it have on the C++ community?

Stroustrup: The most novel and interesting part of the standard library is the general and extensible framework for containers and algorithms. It is often called the STL and is primarily the work of Alex Stepanov (then at Hewlett-Packard Labs, now at Silicon Graphics).

Alex worked on providing uncompromisingly general and uncompromisingly efficient fundamental algorithms which, for example, find an element in a data structure, sort a container, or count the occurrences of a value in a data structure. Such algorithms are fundamental to much computing.

recording script using recordmydesktop command

2010-10-14T02:06:00.001-07:00


#!/bin/bash
#
# Warning: this does not have robust error checking!

bad=67
if [ $# -lt 1 ]
then
echo "Usage: $0 [filename] [[optional time]]"
exit $bad
fi

if [ $# -eq 1 ] #check for arguments
then
time=3 #if one (only filename) exists, sleep for 3 seconds
filename=$1 #set filename
else
time=$2 #else, we'll sleep for $2 seconds
filename=$1 #set filename
fi

recordmydesktop -windowid `xwininfo |grep "Window id:"|sed -e "s/xwininfo\:\ Window id:\ //;s/\ .*//"` -o $filename.ogv -delay $time --no-sound

ffmpeg -i $filename.ogv -b 384000 -s 640x480 -pass 1 -passlogfile log-file $filename.flv

SSH - Swiss Army Knife

2010-10-01T01:38:00.000-07:00

Doing remote file operations using SSH is covered well in this link: http://ultra.ap.krakow.pl/~bar/DOC/ssh_backup.html

Download Picasaweb Album on Linux

2010-09-29T07:50:00.001-07:00

Using this project: http://sourceforge.net/projects/picasaalbumdown/

Direct download to Picasa does not work. It works on Windows.

James Gosling on Development of Java

2010-09-29T00:25:00.000-07:00

James Gosling:So long as they do that. The development of Java is not an inexpensive thing. It takes a fair amount of funding. It's not just about writing code. Learning the code is two or three percent of the expense. You're shipping fifteen million copies a week, just the bandwith is horrible. The QA when you have to worry about something that has thirty issues. When you've got everything, every stock exchange, every phone company on the planet. Their security depends on Java. So it's not a causual piece of testing.

You know, when it comes to open source contributions, our history with contributions over the years have been kinda snarky. We'd get lost of people sending code and fixes. But on average, we'd get a submission that fixed the bug but it caused three or four more. And it probably didn't fix the bug for everybody. It probably only fixed the bug for their one case. And trying to get people in the community to actually think about the whole code base and not just their particular issue today. Doing one line of change means an immense amount of testing.

Most open source projects are way too casual for that. Sometimes when you get bugs that are potential security issues, you have to move fast, you have to put immense resources on getting it done. Maybe it's just one engineer fixing one character in one line, but then testing it and making sure you didn't introduce a bug. The harder stuff is if you have a bug, there are probably people out there who have worked around that bug, so how many of the workarounds are you going to break. And when you've got nine or ten million in the developer community you have enormous applications, trivial fixes are not trivial. And open source projects, the way the average open source projects are constituted. IT's easy to get people to do the fun stuff. It's hard to get people to do the hard stuff.

Like QAing the math libraries. Like doing QA on sine and cosine, you absolutely have to have a PHd in Mathematics. Sine and cosine: it sounds really simple, but there is unbelievable amount of depths of subtlety in there. There are extraordinarily few people on the planet qualified to QA that type of stuff.

http://www.basementcoders.com/transcripts/James_Gosling_Transcript.html

Quote HTML Function in python

2010-09-27T08:36:00.001-07:00

def _quote_html(html):
return html.replace("&", "&").replace("<", "<").replace(">", ">")

Android Dev Phone 1

2010-09-26T23:29:00.001-07:00

For all purposes, it is same as HTC Dream or G1

Additional X Server on Ubuntu

2010-09-26T23:04:00.001-07:00

xinit -display :n+1 -- :n+1

-bash: /bin/ls: Argument list too long

2010-09-22T00:21:00.000-07:00

In bash, doing a

ls *-output.log

on directory containing 7000 files fails with the error

-bash: /bin/ls: Argument list too long

The best way to go about with this is:

find . -name '*-output.log' | xargs ls -l

Nice join methods in python

2010-09-21T20:13:00.000-07:00

_nulljoin = ''.join
_semispacejoin = '; '.join
_spacejoin = ' '.join

using imapfilter

2010-09-21T00:06:00.000-07:00

http://moiristo.wordpress.com/2008/11/18/sorting-imap-mail-with-imapfilter/

I struggled a bit to use imapfilter to delete the mails at the server.
The following was configuration

--------------
-- Options --
---------------

options.timeout = 120
options.subscribe = true

----------------
-- Accounts --
----------------

-- Connects to "imap1.mail.server", as user "user1" with "secret1" as
-- password.
account1 = IMAP {
server = 'myserver.com',
port = 993,
username = 'username',
password = 'password.',
ssl = 'tls1',
}

results = account1.INBOX:contain_subject('Subject')
-- Delete messages. This DOES NOT Work.
-- results:delete_messages()

-- This Works

account1.INBOX:delete_messages(results)

urlopen with multiple retries

2010-09-20T18:35:00.000-07:00

A good example from test_urllib2net

def _retry_thrice(func, exc, *args, **kwargs):
   for i in range(3):
   try:
   return func(*args, **kwargs)
   except exc, last_exc:
   continue
   except:
   raise
   raise last_exc

def _wrap_with_retry_thrice(func, exc):
   def wrapped(*args, **kwargs):
   return _retry_thrice(func, exc, *args, **kwargs)
   return wrapped

# Connecting to remote hosts is flaky. Make it more robust by retrying
# the connection several times.
_urlopen_with_retry = _wrap_with_retry_thrice(urllib2.urlopen, urllib2.URLError)

Constructing a command for subprocess execution

2010-09-15T00:26:00.000-07:00

Python module, subprocess's Popen call takes a list as its first argument, with name of the command as the first element of the list. Most often, we do "command line sentence".split() to get the list for the subprocess.Popen call. It is very suboptimal way, especially when you have double-quotes characters which served as shell escape characters.

Consider the following command and see how shlex module makes it more suitable for subprocess.Popen

/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"

import shlex
cmd = """/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'" """

list_cmd1 = cmd.split(' ')
list_cmd2 = shlex.split(cmd)

print list_cmd1
print list_cmd2

$python prog.py
['/bin/vikings', '-input', 'eggs.txt', '-output', '"spam', 'spam.txt"', '-cmd', '"echo', '\'$MONEY\'"', '']
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]

Isn't the second version much simpler?

Least used Modules from Python Standard Library

2010-09-14T21:22:00.000-07:00

Geremy Condra Wrote in the Python-dev mailing list.

I (Geremy Condra) ran some statistics (over pypi) on the number of
times modules out of the stdlib got imported a few months ago and came
up with a reasonably comprehensive list of the least-used things in
the stdlib. For the record, since I wound up parsing import statements
and know some garbage data got in, its reasonable to assume that a few
otherwise valid imports aren't recorded here. But enough with the
disclaimers.

0 AL
0 ColorPicker
0 DEVICE
0 FL
0 FrameWork
0 Nav
0 PixMapWrapper
0 SUNAUDIODEV
0 aepack
0 aetypes
0 al
0 applesingle
0 autoGIL
0 buildtools
0 cd
0 cfmfile
0 dbhash
0 dl
0 dummy_threading
0 findertools
0 flp
0 fm
0 fpectl
0 gensuitemodule
0 icopen
0 imageop
0 imgfile
0 jpeg
0 macerrors
0 macostools
0 macresource
0 nis
0 posixfile
0 spwd
0 sunaudiodev
0 symtable
0 videoreader
0 winsound
1 Tix
1 audioop
2 ic
3 Bastion
3 binhex
3 dumbdbm
3 dummy_thread
3 fractions
3 future_builtins
3 mailcap
3 ossaudiodev
3 tabnanny
3 xdrlib
4 ScrolledText
4 macpath
4 stringprep
5 DocXMLRPCServer
5 GL
5 aifc
5 mimify
5 sunau
6 fl
6 pickletools
6 statvfs
6 turtle
7 W
8 codeop
8 multifile
8 nntplib
8 poplib
8 sndhdr
9 EasyDialogs
9 pipes
9 pyclbr
10 dbm
10 gdbm
10 imputil
11 MiniAEFrame
11 fpformat
11 numbers
14 CGIHTTPServer
14 pty
16 rexec
18 netrc
19 msvcrt
19 uu
20 rlcompleter
21 compileall
22 tty
24 lib2to3
24 mutex
25 chunk
25 mhlib
27 whichdb
28 robotparser
29 ssl
30 dircache
32 gl
33 runpy
34 posix
36 aetools
36 wave
37 termios
42 bdb
44 imaplib
46 ast
47 bsddb
47 imghdr
50 crypt
50 smtpd
53 Carbon
57 MimeWriter
57 msilib
60 cmath
66 filecmp
67 syslog
68 MacOS
73 cProfile
74 asynchat
74 repr
75 ftplib
76 htmllib
83 abc
91 quopri
93 pkgutil
98 anydbm
98 telnetlib
99 trace
102 formatter
104 __main__
104 readline
105 colorsys
110 _winreg
111 curses
113 plistlib
115 modulefinder
116 UserString
121 cookielib
125 mailbox
126 cgitb
128 bz2
128 sched
134 io
146 mimetools
147 pydoc
148 SimpleXMLRPCServer
154 mmap
155 user
156 site
157 symbol
159 zipimport
166 pstats
172 fileinput
173 encodings
179 py_compile
180 SimpleHTTPServer
181 profile
183 cmd
198 Tkinter
200 fcntl
206 copy_reg
225 linecache
226 hotshot
234 multiprocessing
262 dis
273 UserList
273 resource
287 SocketServer
289 shelve
297 sqlite3
317 grp
322 asyncore
335 timeit
339 keyword
345 sgmllib
363 token
367 test
383 parser
386 shlex
421 wsgiref
451 contextlib
458 unicodedata
471 tokenize
472 pwd
487 webbrowser
526 hmac
529 heapq
542 platform
573 gettext
594 pdb
597 popen2
607 json
608 marshal
619 smtplib
621 bisect
637 difflib
647 commands
657 BaseHTTPServer
677 Cookie
688 locale
695 zlib
708 HTMLParser
710 code
721 rfc822
748 compiler
759 gzip
759 select
771 ctypes
788 gc
796 binascii
812 getpass
822 __builtin__
854 htmlentitydefs
857 tarfile
869 decimal
872 xmlrpclib
903 csv
933 atexit
943 functools
946 exceptions
976 array
979 sha
1044 thread
1056 calendar
1064 zipfile
1070 UserDict
1078 new
1102 uuid
1148 Queue
1159 sets
1172 signal
1213 hashlib
1242 getopt
1276 email
1310 imp
1321 fnmatch
1328 mimetypes
1348 collections
1442 httplib
1469 cPickle
1505 md5
1614 weakref
1618 textwrap
1654 pickle
1722 errno
1729 stat
2020 pprint
2060 struct
2389 codecs
2391 ConfigParser
2406 operator
2578 math
2626 base64
2925 inspect
3013 cgi
3105 itertools
3250 xml
3318 glob
3402 __future__
3505 warnings
3549 socket
3722 urlparse
4014 traceback
4142 subprocess
4194 threading
4198 cStringIO
4224 string
4501 copy
4696 random
5088 shutil
5392 tempfile
5426 doctest
5642 optparse
5913 types
6185 StringIO
6522 urllib
7346 distutils
7930 datetime
8416 urllib2
9567 logging
9906 time
13019 re
14714 unittest
18334 sys
27468 os

How to charge canon fs 200 camcoder.

2010-08-24T17:34:00.000-07:00

I did not know how to charge my canon fs200 camcoder, the charging button is hidden in the side. This video helped me.

perforce tip - p4 describe changelist

2010-08-19T16:57:00.000-07:00

We usually attach the perforce changelist numbers in our bug reports. I used to wonder as how do we use those changelist numbers and to see the diff of whats changed.

The command is:


p4 describe -du changenumber

-du stands for unified diff which is often more helpful than others.

Look for p4 help describe for more information.

CD Creation in Ubuntu

2010-08-07T11:12:00.000-07:00

There is a this program by name wodim. Just use that as

wodim name_of_iso

. I have used it thrice and everytime I forget the name. Speak of poorly named programs, I would give this a high rank.

If you want to create an iso, there is genisoimage program too. I have not used it yet, but I would be using it now. Spent 15 minutes today to findout the name of 'wodim' program.

PubSubHubBub

2010-08-06T01:45:00.000-07:00

Linux Tip - ^

2010-07-30T10:36:00.000-07:00

Ubuntu Server Tip: If you need to perform a command a second time on a different file, you can use command replacement with the ^ symbol. e.g. "cp foo.txt /to/some/directory" then "^foo^bar", expanding to: "cp bar.txt /to/some/directory".

Simple Countdown Clock - Python and xdaliclock

2010-07-18T07:03:00.000-07:00

Here is a simple countdown clock script which you can use. It is simply
sending -countdown argument properly to the xdaliclock binary.

I find having this script in my /usr/local/bin pretty useful.




#!/usr/bin/python
# A Simple Countdown Clock using xdaliclock
# usage: countdown [time]
#        time defaults to 1 hour.

import time
import sys
import subprocess

try:
    hours = float(sys.argv[1])
except IndexError:
    hours = 1
hours = int(time.time()) + int(hours * 3600)
command = 'xdaliclock -countdown %d' % hours
subprocess.Popen(command.split())

power 2 in python using lshift

2010-06-26T17:36:00.000-07:00

left shifting 1 in python does a pow 2 operation. I was just doing some performance comparison and here is an interesting result.


$ python -mtimeit 'pow(2,64)'
1000000 loops, best of 3: 1.18 usec per loop
12:31 AM:senthil@:~
$ python -mtimeit 'getattr(1,"__lshift__")(64)'
1000000 loops, best of 3: 0.356 usec per loop
12:32 AM:senthil@:~


I don't know the reason for this difference, I shall update the post when i find out why.

How to find if a port is free or occupied

2010-06-17T06:57:00.000-07:00

One of the ways is:


netstat -tulnap

Py3k PEPS at APAC PyCon

2010-06-15T04:02:00.000-07:00

I went to Singapore to attend APAC PyCon and also to meet Shalini. It was a good 4 days for me. I presented a talk on Py3K PEPS at APAC Pycon.

The Conference Experience was very good. I had a chance to meet Mark Hammond and discuss a lot of Windows Related things with him with. His presentation on raindrop, couch db and and his perspective on Windows development was very good. It was good to meet in person Liew Beng Keat, the organizer of the Conference who had done a lot of hard work to put this together. I also met Steve Holden, who was present along with his wife, I assume they had a good holiday time at SG, as it is a shopper's paradise. It was good to the warmth in Steve's welcome and this makes him a very good organizer, I guess.

On the first day, the talks I enjoyed the most were Mark's Couch DB talk, wherein I gained good knowledge of JSON based Non-Relational database and a javascript based map-reduce framework. The concept was very interesting. The final talk by Wesley Chun drew in a huge enthusiasm. Wesley did a very nice presentation on Py3k and explaining its features and kind of potrayed the picture that it would take long time for people to move to Py3k. Well, it could be true, but as I debated with him, it would be very good to just potray positively how Py3k is a more symmetrical in many ways that Py2k. His point of view was pragmatism and training for corporations. I tend to agree with him to an extent, but I still believe that for some "good programmer" to start learning Python, Python 3 is a very well designed a symmetrical one rather than Python 2. The libraries and packages will catch up soon.

I also enjoyed Graham Dumpleton's short pitch on mod_wsgi and flask. I still use mod_python and plan to move to more web-development related libraries soon.

My presentation on Py3k PEPS was on next day, it went well and was attended by a small interested audience. Following Presentation by Martin Faassen was very good too. He presented a perspecive on creating libraries, the creative aspect of development of software. Steve Holden's Metaclass Madness talk was enlightening too. It was short presentation and a consise one. It would good to write an article based on his presentation if its not already there. Because Python Metaclasses are something which does not have a lot of literature around in the web. The Q&A in the Metaclass'es talk was good one, as one person asked as when do the metaclasses take effective if we were to wrap teh private methods ( which was denoted by startswith('__') and and it turned out that Class mangles them to _Classname__privatemethod and the metaclass wrapper comes to affect later only). If I write an article, I shall discuss this in more detail. There are some interesting studies which can be done on Metaclass vs Class decorators.

I also attended Noufal's game related talk. It was good one which was attended by a sizable audience. He walked through the code and explained the physics of the game. It was good to see that if developing games we can use real world physics in games using libraries, Interesting. Also, I am not sure, how effective showing a lot of code in the talk is effective. It is very difficult to follow through. Somethings slides with less bullets and easily chew-able points make the presentation more grasping and provide useful inputs to the audience.

The singpass coder's tournament was good one too. I managed to come into Second round. My trials some with functional programming and mis-reading of problem statements cost me some time. But it was a very enjoyable game. A person by name 'Che' from China won the iPad and Noufal got the $100 Amazon Web-services coupon.

On Both days, Shalini came to pick me up from the conference and we went for a stroll in the nearby park of Fort Canning. It was good to talk, walking in the park and saw a lot of people practicing tai-chi. On Saturday we visited a lot of friends, inviting them for our marriage reception at Singapore, we also went to Jackie Chan movie, "The Karate Kid", it was fun. I liked the part where the kid shows his dancing skills on DDR to his girlfriend and she in turn amazes him after acting very shy. The concept of kung-fu as a way of life is also good. On Sunday we did some purchasing, spent a wonderful afternoon at home and in the evening I headed back to India.

Latin Characters in xterm

2010-06-01T05:56:00.000-07:00

Pressing the ALT key and pressing the key gives the Latin characters



A - á  Á
B - â  Â
C - ã  Ã
D - ä  Ä
E - å  Å
F - æ  Æ
G - ç  Ç
H - è  È
I - é  É
J - ê  Ê
K - ë  Ë
L - ì  Ì
M - í  Í
N - î  Î
O - ï  Ï
P - ð  Ð
Q - ñ  Ñ
R - ò  Ò
S - ó  Ó
T - ô  Ô
U - õ  Õ
V - ö  Ö
W - ÷  ×
X - ø  Ø
Y - ã  Ù
Z - ú  Ú

Martin Vön Löwis
Résumé

float.as_integer_ration

2010-05-02T07:52:00.000-07:00

From python2.6, you can get a the integer ratio of a float.


$ ./python 
Python 2.7b1+ (trunk:80674M, May  1 2010, 08:23:48) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 3.14.as_integer_ratio()
(7070651414971679, 2251799813685248)
>>> x*1.0/y
3.14
>>>

Posting via Jabber

2010-04-09T03:44:00.000-07:00

Posting this via Jabber, that is LJ talk. Noticed today that Live journal user page has a forceful ad serving. You got to wait and press on the top right x to close it and then access your account page.

Video Chat on Ubuntu

2010-04-08T17:27:00.000-07:00

If you would like to do a Video Chat on Ubuntu, then
1. Install Skype 2.0 on Ubuntu.
2. Follow the instructions mentioned here to make the Video option work fine.

How to Use Math Symbols in gnome-terminal

2010-02-15T02:48:00.000-08:00

HOLD CTRL+SHIFT + U + CODEPOINT. That's it.

For e.g. CTRL + SHIFT + U + 2200 will output ∀ After typing CTRL + SHIFT + U,
when the terminal identifies that what follows is unicode, you might leave the
keypresses and just type the codepoint.

∀ U2200
∁ U2201
∂ U2202
∃ U2203
∄ U2204
∅ U2205
∆ U2206
U2207
U2208
∉ U2209
∊ U220A

C++0x support in gcc

2009-11-01T23:02:00.000-08:00

http://gcc.gnu.org/projects/cxx0x.html

Some of the features are already available in gcc.

Python Strings as Comments

2009-10-20T23:29:00.000-07:00

The question was:

In Python we can emulate multiline comments using triple-quoted
strings, but conceptually strings and comments are very different.
I.e. strings are objects, comments are auxillary text discarded at
compile time. Strings are objects created at runtime, comments are
not.

The answer from Steven D'Aprano:

Guido's time-machine strikes again.



>>> import dis
>>> def test():
...     x = 1
...     """
...     This is a triple-quote comment.
...     """
...     return x
...
>>> dis.dis(test)
  2           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  6           6 LOAD_FAST                0 (x)
              9 RETURN_VALUE

String literals -- not just triple-quoted strings, but any string
literal -- that don't go anywhere are discarded by the Python compiler,
precisely so they can be used as comments.

ln explained best

2009-10-12T15:58:00.000-07:00

At this article titled Demystifying the Natural Logarithm

His entire series of get an intuition articles on Maths is very good.

Pell's Equation

2009-10-09T13:56:00.000-07:00

x^2 - n y^2 = 1

(Pell's equation) which is named after the English mathematician John Pell. It was studied by Brahmagupta in the 7th century, as well as by Fermat in the 17th century.

http://en.wikipedia.org/wiki/Pell%27s_equation

8 Bit Byte

2009-10-01T13:45:00.000-07:00

Factors behind the ubiquity of the eight bit byte include the popularity of the IBM System/360 architecture, introduced in the 1960s, and the 8-bit microprocessors, introduced in the 1970s. The term octet unambiguously specifies an eight-bit byte (such as in protocol definitions, for example)

http://en.wikipedia.org/wiki/Byte

Otherwise, people have tried with 12 bit byte. Varying byte length in PDP 10. 6, 7 and 9 bits in Univac computers.

Brainfuck

2009-09-28T12:03:00.000-07:00

+++ +++ +++ +[>
+++ +++ +
>
+++ +++ +++ +
<++ .
>+++ ++ .

Brainfuck is easy and interesting.
We work on cells, like if you know c, cell is like ptr = unsigned char *
+ stands for increment like ++*ptr
- stands for decrement like --*ptr
> goes one cell right like ++ptr
< goes one cell left like --ptr
. spits out output like putchar(*ptr)
, expects input. like *ptr = getchar(stdin)
[ is the start of while block test condition (for non-zero) which is like while(*ptr) {
] is the end of the block }
And everything else is comment, like the these sentences.
The above program is designed to output "Hi" and if you include these
sentenses too, it will print "Hii" and wait for some input and enter
presses till the cell becomes zero.

Isn't it cool? :)

BTW, you can run this program using an interpretor called (bf) just an apt-get away.

Kavka's toxin puzzle

2009-08-01T23:23:00.000-07:00

In its original form, it is:
An eccentric billionaire places before you a vial of toxin that, if you drink it, will make you painfully ill for a day, but will not threaten your life or have any lasting effects. The billionaire will pay you one million dollars tomorrow morning if, at midnight tonight, you intend to drink the toxin tomorrow afternoon. He emphasizes that you need not drink the toxin to receive the money; in fact, the money will already be in your bank account hours before the time for drinking it arrives, if you succeed. All you have to do is. . . intend at midnight tonight to drink the stuff tomorrow afternoon. You are perfectly free to change your mind after receiving the money and not drink the toxin.

Kavka's theory is that, one cannot intend to do something which one won't do.

There is an analysis of payoffs from different scenarios and the real-world example of this puzzle is:

...the Political Manifesto. Before an election, a political party will release a written document outlining their policies and plans should they win office. Many of these promises may be difficult or impossible to implement in practice. Having won, the party is not obligated to follow the manifesto even if they would have lost without it.

X Window joke

2009-07-30T09:47:00.000-07:00

X window: The ultimate bottleneck. Flawed beyond belief. The only thing you have to fear. Somewhere between chaos and insanity. On autopilot to oblivion. The joke that kills. A disgrace you can be proud of. A mistake carried out to perfection. Belongs more to the problem set than the solution set. To err is X windows. Ignorance is our most important resource. Complex nonsolutions to simple nonproblems. Built to fall apart. Nullifying centuries of progress. Falling to new depths of inefficiency. The last thing you need. The defacto substandard. Elevating brain damage to an art form. X window.

Password-less root shell in ubuntu

2009-07-10T03:02:00.000-07:00

If you forgot you password for your ubuntu system you can recover using the following steps

1. Turn your computer on.
2. Press ESC at the grub prompt.
3. Press e for edit.
4. Highlight the line that begins kernel ………, press e
5. Go to the very end of the line, add rw init=/bin/bash
6. Press enter, then press b to boot your system.
7. Your system will boot up to a passwordless root shell.

Source

Signature in Evolution

2009-07-06T02:30:00.000-07:00

You can have evolution to execute a script as your signature. I had set it so, but whatever I do:


#!/usr/bin/sh
echo "-- "
echo "Senthil"
exec fortune -s

Resulted in something like this:


-- Senthil BOFH excuse #261: The Usenet news is out of date

Finally figured out with the help of this post(thanks man!), that evolution is actually storing every compose action as HTML so

I would need to wrap it in HTML the output of my script to make it a legible signature.

There is a web interface for CUPS

2009-06-24T15:52:00.000-07:00

http://127.0.0.1:631

Stumbled upon this one today.

_GLOBAL_DEFAULT_TIMEOUT in socket.py

2009-06-20T21:51:00.001-07:00

In python 2.6, the timeout attribute of the socket from various higher level modules (like urllib2) is passed as socket._GLOBAL_DEFAULT_TIMEOUT.
If you look at socket.py in Lib, you will find that


_GLOBAL_DEFAULT_TIMEOUT = object()

a) This kind of a construct was new to me, why should _GLOBAL_DEFAULT_TIMEOUT be set to an object() than say None.

Update: Because None for a timeout is a special value that would set the socket in blocking mode. That is the reason None is not used.

b) If this is just for a global value holding, why not just


global _GLOBAL_DEFAULT_TIMEOUT

Here is my understanding, the reason for having _GLOBAL_DEFAULT_TIMEOUT is to have a global default timeout value that can be accessible via various application layer modules (like ftplib, httplib, urllib).

Update: Correct

But the default value of the timeout is written at the lower layer socket module and is returned by
socket.getdefaulttimeout()
and may be set via socket.setdefaulttimeout or via timeout parameters at the application opening functions (like urllib2.urlopen('url',timeout=42).

Update: So what?. Wrong.

Get socket.getdefaulttimeout() is implemented in C; and to be preserve the value between C methods and the python library methods, an object() would be essential instead of a global.

object() might essentially provide an address location for the variable which would be global.

Am I confused? The way the C implementation works is the value is set to a global variable inside of C and its accessed by interfaces like getdefaulttime and setdefaulttime.

Nothing to maintain global values between Python and modules written in C. Are they sharable? I dont know.

This kind of construct is used to have a object to maintain a global state.

global myobj is just a declaration to the compiler that it is global. declared somewhere else and start using the same value. It is not setting things or even defining for the myobj to be used as global. OKay?


a = 100
print id(a)

def foo():
    global a
    print a, id(a)

foo()


[12:17:45 senthil]$python a1.py 
23379248
100 23379248

So, in effect:

_GLOBAL_DEFAULT_TIMEOUT = object() is a just creating a empty object whose value can be shared.

Thats it and None is not a good option because, None in this case means that socket be a blocking one.

Here are some experimentations with urllib2 and new socket._GLOBAL_DEFAULT_TIMEOUT



Python 2.7a0 (trunk:72879M, May 24 2009, 12:51:19) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> print socket.getdefaulttimeout()
None
>>> socket.setdefaulttimeout(42)
>>> print socket.getdefaulttimeout()
42.0
>>> import urllib2
>>> obj = urllib2.urlopen("http://www.google.com")
>>> dir(obj)
['__doc__', '__init__', '__iter__', '__module__', '__repr__', 'close', 'code', 'fileno', 'fp', 'getcode', 'geturl', 'headers', 'info', 'msg', 'next', 'read', 'readline', 'readlines', 'url']
>>> dir(obj.fp)
['__class__', '__del__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_close', '_getclosed', '_rbuf', '_rbufsize', '_sock', '_wbuf', '_wbuf_len', '_wbufsize', 'bufsize', 'close', 'closed', 'default_bufsize', 'fileno', 'flush', 'mode', 'name', 'next', 'read', 'readline', 'readlines', 'softspace', 'write', 'writelines']
>>> dir(obj.fp._sock)
['__doc__', '__init__', '__module__', '_check_close', '_method', '_read_chunked', '_read_status', '_safe_read', 'begin', 'chunk_left', 'chunked', 'close', 'debuglevel', 'fp', 'getheader', 'getheaders', 'isclosed', 'length', 'msg', 'read', 'reason', 'recv', 'status', 'strict', 'version', 'will_close']
>>> dir(obj.fp._sock.fp)
['__class__', '__del__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_close', '_getclosed', '_rbuf', '_rbufsize', '_sock', '_wbuf', '_wbuf_len', '_wbufsize', 'bufsize', 'close', 'closed', 'default_bufsize', 'fileno', 'flush', 'mode', 'name', 'next', 'read', 'readline', 'readlines', 'softspace', 'write', 'writelines']
>>> dir(obj.fp._sock.fp._sock)
['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'accept', 'bind', 'close', 'connect', 'connect_ex', 'dup', 'family', 'fileno', 'getpeername', 'getsockname', 'getsockopt', 'gettimeout', 'listen', 'makefile', 'proto', 'recv', 'recv_into', 'recvfrom', 'recvfrom_into', 'send', 'sendall', 'sendto', 'setblocking', 'setsockopt', 'settimeout', 'shutdown', 'timeout', 'type']
>>> dir(obj.fp._sock.fp._sock.gettimeout)
['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> print(obj.fp._sock.fp._sock.gettimeout)

>>> print(obj.fp._sock.fp._sock.gettimeout())
42.0

Redirect business again

2009-06-20T02:45:00.000-07:00

.htaccess file has


Redirect 302 /index.html "http://localhost/new file.html"

But still when I do:

>>>obj = urllib2.urlopen("http://localhost/index.html")
>>>print obj.code()
200
>>>
Funny, what am I doing and why am I getting it this way? Figuring that out. Because the direct is happening transparently, one is not able to capture the redirect code.

If one needs to the capture redirect code, here is how it is done.


import urllib2

class SmartRedirectHandler(urllib2.HTTPRedirectHandler):
    def http_error_302(self, req, fp, code, msg, headers):
        result = urllib2.HTTPRedirectHandler.http_error_302(self,  req, fp,code, msg,headers)

        result.status = code
        return result

request = urllib2.Request("http://localhost/index.html")
opener = urllib2.build_opener(SmartRedirectHandler())
obj = opener.open(request)
print 'I capture the http redirect code:', obj.status
print 'Its been redirected to:', obj.url

And the output from the session will be:


[06:59:14 senthil]$python smartredirecthandler.py 
I capture the http redirect code: 302
Its been redirected to: http://localhost/new%20file.html

The pasta theory of design

2009-06-16T13:29:00.000-07:00

The pasta theory of design:

* Spaghetti: each piece of code interacts with every other piece of code [can be implemented with GOTO, functions, objects]

* Lasagna: code has carefully designed layers. Each layer is, in theory independent. However low-level layers usually cannot be used easily, and high-level layers depend on low-level layers.

* Ravioli: each part of the code is useful by itself. There is a thin layer of interfaces between various parts [the sauce]. Each part can be usefully be used elsewhere.

* ...but sometimes, the user just wants to order "Ravioli", so one coarse-grain easily definable layer of abstraction on top of it all can be useful.

An application design using Twisted's Aspect Oriented Programming Design is that special Ravioli.

http://twistedmatrix.com/projects/core/documentation/howto/tutorial/library.html

Dismal Google Search

2009-06-14T01:57:00.000-07:00

Every technical query returns, results from:

www.daniweb.com/forums/thread97401.html
www.daniweb.com/forums/thread110548.html
www.gidforums.com/t-3560.html
www.velocityreviews.com/forums/t280081-floating-point-exception.html

And those pages are invariably filled with advertisements and and sometimes asks me to register to view the result too. I am dismayed, either got to have custom google search to exclude these or try other search engine.

All your (http,https) traffic belongs to squid

2009-05-24T04:49:00.000-07:00

phoe6: hello, I have setup a squid proxy running at port 3128; I want all my HTTP as well as HTTPS to proxy to through this one only. what should I do? (the tutorials are too wide and I think I am impatient too :( )

rob0: um, hire someone to do it for you?

phoe6: :) well, it is just two lines rob0. :) I am trying it out.,

phoe6: Sorry, if I sounded offending with the question. Let me try it and ask specific quesitons.

rob0: np, that is how to do it: try things, read manuals, ask specific questions when you get stuck.

phoe6: well I just had to write a filter rule to accept packets through squid-user and any traffic to port 80, pot 443 to reject. by this way my browsing is through squid proxy only.


$iptables -t filter -A OUTPUT -m owner --uid-owner 13 -j ACCEPT
$iptables -t filter -A OUTPUT -p tcp --dport 80 -j REJECT
$iptables -t filter -A OUTPUT -p tcp --dport 443 -j REJECT

Helpful entry: http://stuvel.eu/transproxy

Manually removing a corruped .deb install

2009-05-10T02:07:00.000-07:00

I was getting this error, on a corruped install of a debian package.
""The package 'printers-bangalore' is in an inconsistent state and needs to be reinstalled, but no archive can be found for it. Please reinstall the package manually or remove it from the system"

To solve this issue, I followed these steps.

1) I edited /var/lib/dpkg/status and remove all references of that package. (Note that this is a tricky step; in my case it was simple)

2) sudo dpkg --configure -a

World's longest domain name

2009-05-06T18:32:00.000-07:00

Some Trivia. I liked the pi value domain name.

BTW, it goes like this:
1) Each domain name can be maximum upto 63 octets. (Octets is 8bits; can safely be considered 1 English Character).
2) The sub-division of domains like (cs.univ.edu;math.univ.edu) can go upto 127 levels.
3) But the entire domain name may not exceed 253 octets.

So, if we are to search for the longest domain name (including sub-domains), we will end up with 253 english character long string. ( That would be like a page of characters). :)

Upgraded Ubuntu from 8.10 to 9.04

2009-05-05T11:40:00.000-07:00

This was on my Thinkpad T400 laptop, where Ubuntu is installed through Wubi.

After upgrade, X could not start. No GUI No Window Manager.

Followed the steps mentioned here.

The problem was with fglrx driver.

1) Go to Recovery mode during boot.
2) Choose to drop into Shell with networking support

and do


sudo apt-get remove xorg-driver-fglrx

Reboot. It started working for me.

First thing you will notice in 9.04 is the notification system it has. Nice eye candy stuff.

IOError: decoder jpeg not available

2009-05-02T20:42:00.000-07:00

Following these steps fixed the issue.

Upgraded to Ubuntu 8.10 from 8.04

2009-04-28T02:31:00.000-07:00

Used the Update Manager from the System-> Administration menu.

* Attempts to update through Indian Servers failed.
* There were constant interview dialogues. I got to look out for a configuration file, which can provide an interrupted/quiet upgrade.

Setting up http redirect

2009-04-15T17:22:00.001-07:00

I was trying to setup a http redirect on ubuntu. Spent quite some time.
1) Ubuntu has a

/etc/apache2/sites-available/default

file where you will have to change the AllowOverride option from None to All.
Only this step will enable you to use .htaccess file.
2) In the .htaccess file, I was doing


Redirect 302 ./index.html http://localhost/new.html

Spent more than a hour to figure out, why the redirect is not happening. The problem was, I was doing

./index.html

instead of just

/index.html

Bytes and String in Py3k

2009-04-11T18:26:00.000-07:00

Martin's Explaination:

It's really very similar to 2.x: the "bytes" type is to used in all
interfaces that operate on byte sequences that may or may not represent characters; in particular, for interface where the operating system deliberately uses bytes - ie. low-level file IO and socket IO; also for cases where the encoding is embedded in the stream that still needs to be processed (e.g. XML parsing).

(Unicode) strings should be used where the data is truly text by
nature, i.e. where no encoding information is necessary to find out
what characters are intended. It's used on interfaces where the
encoding is known (e.g. text IO, where the encoding is specified
on opening, XML parser results, with the declared encoding, and
GUI libraries, which naturally expect text).

- base64.encodestring expects bytes (naturally, since it is supposed to
encode arbitrary binary data), and produces bytes (debatably)
- binascii.b2a_hex likewise (expect and produce bytes)
- pickle.dumps produces bytes (uniformly, both for binary and text
pickles)
- marshal.dumps likewise
- email.message.Message().as_string produces a (unicode) string
(see Barry's recent thread on whether that's a good thing; the
email package hasn't been fully ported to 3k, either)
- the XML libraries (continue to) parse bytes, and produce
Unicode strings
- for the IO libraries, see above

Backport this issue http://bugs.python.org/issue5542

2009-03-31T04:15:00.001-07:00

update urlparse to RFC 3986

2009-03-31T04:09:00.001-07:00

Targetting to complete this by sprint. Issue1591035.

Does parse_header really belong to cgi?

2009-03-30T16:33:00.001-07:00

http://bugs.python.org/issue3609

Barry's thought is it can be moved to email module.

Bugs Fixed

2009-03-30T15:22:00.000-07:00

Fixed Bugs:
http://bugs.python.org/issue4675
http://bugs.python.org/issue4962

Simple CGIHTTPServer and Client in Python 3k

2009-03-28T23:17:00.000-07:00

Python 3k - CGIHTTPRequestHandler example.


import http.server
import http.server

class Handler(http.server.CGIHTTPRequestHandler):
    cgi_directories = ["/cgi"]

server = http.server.HTTPServer(("",8000),Handler)
server.serve_forever()

Run this server.

Create a Directory 'cgi' in the directory you have the above server script and place the following python cgi code.

filename: test.py


#!/usr/local/bin/python3.1

import cgitb;cgitb.enable()

print("Content-type: text/html")
print()
print("<title>CGI 101</title>")
print("<h1>First CGI Example</h1>")
print("<p>Hello, CGI World!</p>")

Open the Browser and point to http://localhost:8000/cgi/test.py

file opening modes.

2009-03-20T11:58:00.000-07:00

Possible values for the mode open are

"r" or "rt"
Open for reading in text mode.
"w" or "wt"
Open for writing in text mode.
"a" or "at"
Open for appending in text mode.
"rb"
Open for reading in binary mode.
"wb"
Open for writing in binary mode.
"ab"
Open for appending in binary mode.
"r+", "r+b"
Open for reading and writing.
"w+", "w+b"
Open for reading and writing, truncating file initially.
"a+", "a+b"
Open for reading and appending.

Yahoo Hack Day India 2009

2009-02-15T08:03:00.000-08:00

I developed an application using Y! BOSS Search APIs, Linguistics, Machine Learning and Google App engine.

Here is the Application Live. Use it anytime:

Elementary, My dear Watson!

Here is the Source code for the Project:

yhackday-elementary

unix_time_1234567890

2009-02-13T23:58:00.000-08:00

unix_time_1234567890

This was fun. I could manage to capture it again. This time it was with Vijay.

Previously, while at Dell, along with another friend, Raj, had captured 1111111111.

Sent Form 8233

2009-02-12T17:26:00.000-08:00

For the PyCon Tutorial Compenstation.


I request you to print the attached PDF. Complete the Necessary
details (lower half) from PSF and please FAX

a) My Form 8233  with PSF details.
b) PSF Authorization Letter.

To my FAX Number: 91 80 30501090.

Bugs to Work

2009-02-08T17:40:00.000-08:00

http://bugs.python.org/issue5040
http://bugs.python.org/issue5054

Bug-queue - http.client.HTTPMessage.getallmatchingheaders() always returns []

2009-02-08T17:31:00.001-08:00

> Further investigation ( grep -r getallmatchingheaders Lib/* ) reveals
> that in addition to having no tests, and being implemented incorrectly
> in http.client, getallmatchingheaders() is called only once, in
> http.server; that code is also broken (I reported this yesterday in
> #5053).
>
> Maybe Python 3 is where getallmatchingheaders can make a graceful
> goodbye (and a 2to3 conversion added?).
>
> Python tracker http://bugs.python.org/issue5053

PyCon Tax information

2009-02-03T17:17:00.001-08:00

On a go forward basis, we are taking the position that the PSF won't issue tutorial payments without a W-9 for US persons (US citizen or resident alien) or an 8233 with ITIN for foreign persons (but see below).

Non-US Taxpayers
============

For non-US persons ('non-resident aliens', in the IRS lingo) we need to withhold 30% of the payment unless they are citizens of a country which
has a tax treaty with the USA which allows a lesser rate on personal service income. e.g. for Canada, UK, DE the withholding rate is currently zero. The PSF has to file a 1042-S information return (which is equivalent to a 1099) regardless; there is no lower limit on the compensation.

http://www.irs.gov/publications/p515/ar02.html NRA Withholding
http://www.irs.gov/pub/irs-pdf/p519.pdf US Tax Guide for Aliens
http://www.irs.gov/publications/p901/ar02.html Countries with treaties

To avoid withholding we need to get a Form 8233 from each presenter each year, assuming they are citizens of a treaty country.

http://www.irs.gov/pub/irs-pdf/f8233.pdf On-line Form 8233
http://www.irs.gov/instructions/i8233/ Instructions

This is very easy to complete, except that it requires an ITIN ('individual taxpayer identification number) the first time, and that is a bother:

http://www.irs.gov/pub/irs-pdf/fw7.pdf On-line Form W-7
http://www.irs.gov/taxtopics/tc857.html Instructions - short
http://www.irs.gov/pub/irs-pdf/p1915.pdf Instruction Booklet

The catch on the W-7 is the US Social Security Administration has to deny a Social Security Number application and that denial has to be provided along with the W-7. That's a separate application to the SSA.

There is an exception to this: if the tutorial presentation compensation can be considered a honorarium, then a letter from the PSF to that effect will allow
the SSA denial process to be skipped, per Exception 2a (page 25 of p1915). The PSF has discussed this with the IRS, and they agree that the tutorial
compensation can be considered an honorarium. This means that the applicant would need a letter from the PSF requesting the tutorial presentation, and the applicant would fill in 'Python Software Foundation' on line 6g as the 'company' and provide a copy of the letter along with his/her W-7 application.

So the process would be as follows, assuming the presenter doesn't already have a US ITIN:

Fax the PSF an 8233 (without an ITIN) and a fax number. The PSF will complete the lower half and FAX a copy back along with the authorizing letter. Submit a W-7 with the 8233, acceptable identification documentation (see Pub 1915, typically a certified copy of a passport), and the authorizing letter to the IRS to get the ITIN. This should take 4 - 6 weeks. Send the ITIN back to the PSF on an updated 8233. This will allow the PSF to avoid the withholding. If the presenter has no other US source income, there would be no taxes owed on amounts lower than $3300, and no 1040NR need be filed in that case.

http://www.irs.gov/localcontacts/article/0,,id=101292,00.html

Without the ITIN, the PSF is required to withhold 30%. To get it back by filing a 1040NR plus Schedule C, the presenter would need an ITIN, anyway. So
if he is a citizen of a treaty country and cares about the 30%, he should get an ITIN, send the 8233, and avoid both withholding and filing (assuming no
other US income).

The PSF will offer the option of 30% withholding if the presenter chooses not to supply the document, or cannot do so in time. The presenter could later
acquire an ITIN and file for a refund up to three years after the due date of the return. At least, that is the filing deadline for a refund on a US 1040;
the foreign filer should confirm that deadline for a 1040NR. However, it is probably simpler for all concerned to delay payment until the ITIN is
available for the 8233.

Python for Haiku

2009-01-15T08:43:00.001-08:00

This bug tracks the effort to include Python on Haiku
<http://bugs.python.org/issue4933>
I find this effort interesting as people are working to bring up an
interesting OS up again! I would try Haiku at some-point, now that
Python should be available.

My musts for an OS are:
1) sh
2) vim
3) net
4) mutt
5) python
6) gcc
7) X Window System
8) firefox

Guido's response on supporting Python on Haiku:
>
> I'm with Martin. In these days of distributed version control systems, I
> would think that the effort for the Haiku folks to maintain a branch of
> Python in their own version control would be minimal. It is likely that
> for each new Python version that comes out, initially it is broken on
> Haiku, and then they have to go in and fix it. Doing that in their own
> version control has the advantage that they don't have to worry about
> not breaking support for any other minority operating systems, so I
> expect that all in all the cost will be less for them than if they have
> to submit these patches to core Python; and since that will definitely
> be less work for core Python, I would call that a win-win solution.
>

Py3K, what to look for in brief

2009-01-08T02:05:00.000-08:00

This article was referenced by Guido in the Python-Dev indicating certain interests from academic community.

Actually thats a good article summarizing what to look for in Py3k.

1) Simpler built-in types.
Instead of having a built-in type for int and then for long as in Py26, have a single built-in type int in py3k, which will behave like long and serve for int as well.

2) In py26, you and str and unicode. Now Its just str, which is internally unicode. So all strings are unicode in py3k.
There is a separate bytes type for bytes of characters.

3) 1/5 will be 0.2 in Python3k. If you want the result to be 0, like in Python26, do 1//5

4) No comparisons supported between incompatible types. Documents from time immemorial advised the users to not to rely and it can change anytime. Well the change has happened.

5) its print() function for output now, just like input() function input.
Two things here. Previously in py2x, input() expected an object and raw_input() was actually used to input. Now in py3k, its just input() which will behave just like raw_input() before.

And in py2x, print was a statement, now its a function.
The article mentioned at the start gives a good rationale.

All you have do is, as soon as you type print, your fingers should automatically now get used to typing ( and proceed with what you were doing. More details on print function in py3k can be found in docs.

6) Everything is a new style class. All classes that you define will implicitly be derived from the object class.

7) There is refactoring tool developed by python hackers which can be used to port your py2x code to py3k. Everyones resounding advice is Use It!. This will help in migration as well fix any issues with the refactoring tool.

Greedy vs Non-Greedy in Re - Good Example

2008-12-28T00:47:00.000-08:00

Here is a good example to explain greedy vs, non-greedy search using module re in Python.

*?, +?, ??

The '*', '+', and '?' qualifiers are all greedy; they match as much text as possible. Sometimes this behaviour isn’t desired; if the RE is matched against '<H1>title</H1>', it will match the entire string, and not just '<H1>'. Adding '?' after the qualifier makes it perform the match in non-greedy or minimal fashion; as few characters as possible will be matched. Using .*? in the previous expression will match only '<H1>'.

Once in a blue moon

2008-12-23T11:30:00.000-08:00

A Blue moon is a name given to an irregularly timed full moon. Most years have twelve full moons which occur approximately monthly, but each calendar year contains those twelve full lunar cycles plus about eleven days to spare. The extra days accumulate, so that every two or three years there is an extra full moon (this happens every 2.72 years). (Source: Wikipedia)

So, its frequency of occurrence would be:


1/ 2.72 * 365 * 24 * 60 * 60) = 1.16580118 × 10 ^ -8

Now, this one is good. Excellent humor!

But don't get how they arrived at that number. Average considering the leap year does not result in that either.

Python Types and Obects

2008-12-19T08:24:00.000-08:00

http://www.cafepy.com/article/python_types_and_objects/python_types_and_objects.html

Good article to understand new style classes.

Discussing English Grammar in the bug report.

2008-12-18T13:02:00.000-08:00

Follow the discussion in this Python Documentation bug report.
Its for correct usage of English and grammar.

Terry J. Reedy added the comment:

Benjamin: I thank you too for verifying that I was not crazy.

Martin: I noticed native/non-native split too, and chalked it up to a subtle difference between German and English.

For future reference, the problem with the original, as I see it now, is a subtle interaction between syntax and semantics. The original sentence combined two thoughts about has_key. The two should be either
coordinate (parallel in form) or one should be clearly subordinate. A subordinate modifier should generally be closer to the subject, especially if it is much shorter. Making that so was one of my suggestions. The coordinate form would be 'but it is deprecated'. But this does not work because 'it' would be somewhat ambiguous because of the particular first modifier.

The following pair of sentences illustrate what I am trying to say. Guido was once a Nederlander, but he moved to America. Guido was once a student of Professor X, but he moved to America.
In English, the second 'he' is ambiguous because of the particular first modifier.

So, to me, 'but deprecated' at the end of the sentence reads as either a misplaced subordinate form or as an abbreviated coordinate form that is at least somewhat ambiguous because of the meaning of the other modifier.

git

2008-11-17T13:42:00.000-08:00

Linus Torvalds has quipped about the name "git", which is British English slang for a stupid or unpleasant person:
“ I'm an egotistical bastard, and I name all my projects after myself. First Linux, now git.”

This self-deprecation is certainly tongue-in-cheek, insofar as Torvalds did not, in fact, name Linux after himself (see History of Linux).

http://en.wikipedia.org/wiki/Git_(software)

RFC Hierarchy for Related URL parsing

2008-11-13T20:11:00.001-08:00

RFC Hierarchy for Relative URL formats
RFC3986(STD066) - This is the current and is the standard.
|
RFC2396 - This was previous one.
|
RFC2368
|
RFC1808 - urlparse header says, it follows this. But this has been upgraded a lot times.
|
RFC1738 - It started with this.

console font and resolution in ubuntu (Hardy Heron)

2008-11-12T07:08:00.000-08:00

When ubuntu (Hardy Heron) booted by default, the console resolution and font ( ALT-F[1-6]) were really awful. Searched a lot to arrive at this solution.

To change your Console Resolution:

sudo vim /boot/grub/menu.lst

Go to your default kernel line and remove splash and add vga=791
The vga argument can take values from 791 to 795, google for the resolutions.
You have to remove splash term from that line, otherwise it wont get set.

To change your console font.

sudo dpkg-reconfigure console-setup

Follow the interview screens and choose the font to your preference.
For me, VGA with Size 16 worked best.

Proposed Python 3.0 schedule

2008-11-11T23:48:00.001-08:00

> 19-Nov-2008 3.0 rc 3
> 03-Dec-2008 3.0 final

TODO [issue4191] urlparse normalize URL path

2008-11-11T23:14:00.001-08:00

> <http://bugs.python.org/issue4191>

By Nov 12.

TODO: [issue3991] urllib.request.urlopen does not handle non-ASCII characters

2008-11-11T21:55:00.001-08:00

<http://bugs.python.org/issue3991>

TODO: [issue3932] HTMLParser cannot handle '&' and non-ascii characters in attribute names

2008-11-11T21:35:00.001-08:00

http://bugs.python.org/issue3932

Test Patch - [issue3924] cookielib chokes on non-integer cookie version, should ignore it instead

2008-11-11T20:58:00.001-08:00

Test Patch - [issue3924] cookielib chokes on non-integer cookie
version, http://bugs.python.org/issue3924

Bogosort

2008-10-24T11:47:00.000-07:00

"In computer science, bogosort (also random sort, shotgun sort or monkey sort) is a particularly ineffective sorting algorithm. Its only use is for educational purposes, to contrast it with other more realistic algorithms. If bogosort were used to sort a deck of cards, it would consist of checking if the deck were in order, and if it were not, one would throw the deck into the air, pick up the cards up at random, and repeat the process until the deck is sorted."

http://en.wikipedia.org/wiki/Bogosort

Conversations with Jeremy while working on urllib for py3k

2008-09-29T09:56:00.001-07:00

(01:08:39) Senthil: Hi Jeremy!
(01:14:33) Jeremy: hi
(01:16:00) Senthil: The patch applied properly.. while executing the tests,
it was not able to view the parse.py
(01:16:10) Senthil: File "/usr/local/lib/python3.0/email/utils.py", line 28,
in <module>
import urllib.parse
ImportError: No module named parse
(01:16:40) Jeremy: Be sure you delete urllib.py and urllib.pyc
(01:16:45) Senthil: python3.0 regrtest.py and python3.0 test_urllib.py would
give the same.
(01:16:47) Senthil: I did that.
(01:17:12) Jeremy: Can you import them interactively?
(01:17:18) Senthil: deleted the older urllib.py ( oops. let me check with
.pyc :) )
(01:17:42) Jeremy: >>> import urllib.parse >>> urllib.parse.__file__
Ê¼/usr/local/google/home/jhylton/py3k/Lib/urllib/parse.pyÊ¼
(01:19:21) Senthil: [root@goofy python3.0]# python3.0
Python 3.0a5+ (py3k:64342M, Jun 18 2008, 00:51:20)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.parse
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named urllib.parse
>>>
(01:20:43) Senthil: Okay, just a min, Jeremy. I shall get back.. I think, I
know the prob, it with patch application...
(01:22:57) Jeremy: ok
(01:57:57) Senthil: Hi Jeremy, two (minor) changes in the patch.
(01:58:42) Senthil: 1) Add in the Makefile LIBSUBDIRS = urllib\ ( so that
make install creates that package dir).
(01:59:14) Senthil: 2) __init__.py in the urllib ( that was reason parse.py
was not getting imported. :) )
(02:02:25) Jeremy: oops
(02:03:17) Jeremy: Is the __init__.py not in the patch?
(02:03:22) Senthil: nope,
(02:03:40) Senthil: It is not. Atleast in one you sent me.
(02:04:13) Senthil: test_urllib.py has 1 failure.
(02:04:19) Senthil: FAIL: test_geturl (__main__.urlopen_FileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_urllib.py", line 103, in test_geturl
self.assertEqual(self.returned_obj.geturl(), self.pathname)
AssertionError: 'file:///tmp/@test' != '/tmp/@test'
(02:04:36) Jeremy: ItÊ¼s in my local client, so IÊ¼m not sure why some of the
new files get added and some donÊ¼t.
(02:04:51) Senthil: ohh ok.
(02:05:00) Jeremy: That really baffling error I mentioned is in
test_urllibnet.py
(02:07:44) Senthil: yup. I see that in my box too.
(02:08:13) Jeremy: It would really help to get a second set of eyes on that
one. I think many of the other ones are shallow.
(02:09:05) Senthil: Jeremy, codereview.appspot.com seems to be giving 500
Server Error.
(02:09:18) Jeremy: IÊ¼ve been getting a lot of errors, too.
(02:09:19) Senthil: Have you checkin your code in the sandbox or any other
location?
(02:09:29) Jeremy: I guess I can check it in on a branch.
(02:10:22) Senthil: Would be a very good idea. Coz, to submit patches,
verify the changes etc etc.
(02:16:47) Senthil: Jeremy, both the test_urllibnet.py are due to urlopen
method not returning the file-object ( which previous urllib.py's urlopen ()
) returned.
(02:17:49) Jeremy: The HTTPResponse ought to behave like a file-object, I
thought.
(02:18:12) Senthil: the fix IMO, may not just be in the tests, but in what
to be decided upon for 2 urlopen from urllib.py and urllib2.py
(02:19:07) Senthil: In fact, both errors. One is test_fileno ( testing the
fileno attribute of file object and other test_readlines testing the
readlines method of the file object).
(02:20:49) Jeremy: Looking again, the both return an addinfourl() instance,
but...
(02:21:32) Jeremy: old urllib wraps a http client response and urllib2 wraps
an io.BufferedReader
(02:23:22) Jeremy: Doh! So a very small change seems to fix this particular
behavior.
(02:23:39) Jeremy: resp = urllib.response.addinfourl(r.fp, r.msg,
req.get_full_url()) resp.code = r.status resp.msg = r.reason return resp
(02:23:53) Jeremy: in urllib2 instead of wrapping it in a BufferedReader
(02:31:45) Senthil: am looking into it, digging through, trying to
understand/check it..
(02:32:10) Jeremy: I checked in my working copy in py3k-urllib branch!
(02:40:11) Jeremy: IÊ¼m heading home now. IÊ¼ll check in with you later. Feel
free to fix stuff on the branch and weÊ¼ll see if we can get it checked in
first thing in the morning (US time :-).
(02:40:14) Senthil: Jeremy, where do you plan to add this code:
(02:40:16) Senthil: resp = urllib.response.addinfourl
(02:40:29) Senthil: :)
(02:40:30) Senthil: sure.
(02:40:37) Senthil: I am at India time :D
(02:47:08) Jeremy: At the end of AbstractHTTPHandler.do_open
(02:58:52) Jeremy logged out.
(02:58:56) Senthil: Okay, I get it. :) such a minor change.
(02:59:20) Senthil: btw, it solved the fileno test, but readlines still
giving issues and new issues cropping up.
(02:59:38) Senthil: I shall look into it, Jeremy, Bye.
--
O.R.Senthil Kumaran
http://uthcode.sarovar.org

Notes while working on urllib patches

2008-09-29T09:54:00.001-07:00

Following are some of the notes I took, while working on urllib patches.
It should be a handy reference when working on bugs again.

RFC 3986 Notes:
Mon Aug 25 10:17:01 IST 2008

- A URI is a sequence of characters that is not always represented as a
sequence of octets. ( What are octets? OCTETS means 8 Bits. Nothing else!)

- Percent-encoded octets may be used within a URI to represent characters
outside the range of the US-ASCII coded character set.

- Specification uses Augmented Backus-Naur Form (ABNF) notation of [RFC2234],
including the followig core ABNF syntax rules defined by that specification:
ALPHA (letters), CR ( carriage return), DIGIT (decimal digits), DQUOTE
(double quote), HEXDIG (hexadecimal digits), LF (line feed) and SP (space).

Section 1 of RFC3986 is very generic. Undestand that URI should be transferable
and single generic syntax should denote the whole range of URI schemes.

- URI Characters are, in turn, frequently encoded as octets for transport or
presentation.
- This specification does not mandate any character encoding for mapping
between URI characters and the octets used to store or transmit those
characters.
pct-encoded = "%" HEXDIG HEXDIG

- For consistency, uri producers and normalizers should use uppercase
hexadecimal digits, for all percent - encodings.
reserved = gen-delims / sub-delims

gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"

sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="

unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

When a new URI scheme defines a component that represents textual
data consisting of characters from the Universal Character Set [UCS],
the data should first be encoded as octets according to the UTF-8
character encoding [STD63]; then only those octets that do not
correspond to characters in the unreserved set should be percent-
encoded. For example, the character A would be represented as "A",
the character LATIN CAPITAL LETTER A WITH GRAVE would be represented
as "%C3%80", and the character KATAKANA LETTER A would be represented
as "%E3%82%A2".

Section 2, was on encoding and decoding the characters in the url scheme. How
that is being used encoding reservered characters within data. Transmission of
url from local to public when using a different encoding - translate at the
interface level.

URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

hier-part = "//" authority path-abempty
/ path-absolute
/ path-rootless
/ path-empty

Many URI schemes include a hierarchical element for a naming
authority so that governance of the name space defined by the
remainder of the URI is delegated to that authority (which may, in
turn, delegate it further).

userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
host = IP-literal / IPv4address / reg-name

In order to disambiguate the syntax host between IPv4address and reg-name, we
apply the "first-match-wins" algorithm:

A host identified by an Internet Protocol literal address, version 6
[RFC3513] or later, is distinguished by enclosing the IP literal
within square brackets ("[" and "]"). This is the only place where
square bracket characters are allowed in the URI syntax.

IP-literal = "[" ( IPv6address / IPvFuture ) "]"

IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )

IPv6address = 6( h16 ":" ) ls32
/ "::" 5( h16 ":" ) ls32
/ [ h16 ] "::" 4( h16 ":" ) ls32
/ [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
/ [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
/ [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
/ [ *4( h16 ":" ) h16 ] "::" ls32
/ [ *5( h16 ":" ) h16 ] "::" h16
/ [ *6( h16 ":" ) h16 ] "::"

ls32 = ( h16 ":" h16 ) / IPv4address
; least-significant 32 bits of address

h16 = 1*4HEXDIG
; 16 bits of address represented in hexadecimal

IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet

dec-octet = DIGIT ; 0-9
/ %x31-39 DIGIT ; 10-99
/ "1" 2DIGIT ; 100-199
/ "2" %x30-34 DIGIT ; 200-249
/ "25" %x30-35 ; 250-255

reg-name = *( unreserved / pct-encoded / sub-delims )

Non-ASCII characters must first be encoded according to UTF-8 [STD63], and then
each octet of the corresponding UTF-8 sequence must be percent- encoded to be
represented as URI characters.

When a non-ASCII registered name represents an internationalized domain name
intended for resolution via the DNS, the name must be transformed to the IDNA
encoding [RFC3490] prior to name lookup.

Section 3 was about sub-components and their structure and if they are
represented in NON ASCII how to go about with encoding/decoding that.

path = path-abempty ; begins with "/" or is empty
/ path-absolute ; begins with "/" but not "//"
/ path-noscheme ; begins with a non-colon segment
/ path-rootless ; begins with a segment
/ path-empty ; zero characters

path-abempty = *( "/" segment )
path-absolute = "/" [ segment-nz *( "/" segment ) ]
path-noscheme = segment-nz-nc *( "/" segment )
path-rootless = segment-nz *( "/" segment )
path-empty = 0<pchar>
segment = *pchar
segment-nz = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
; non-zero-length segment without any colon ":"

pchar = unreserved / pct-encoded / sub-delims / ":" / "@"

relative-ref = relative-part [ "?" query ] [ "#" fragment ]

relative-part = "//" authority path-abempty
/ path-absolute
/ path-noscheme
/ path-empty

Section 4 was on the usage aspects and heuristics used in determining in the
scheme in the normal usages where scheme is not given.

- Base uri must be stripped of any fragment components prior to it being used
as a Base URI.

Section 5 was on relative reference implementation algorithm. I had covered
them practically in the Python urlparse module.

Section 6 was on Normalization of URIs for comparision and various
normalization practices that are used.

========================================================================
Python playground:

>>> if -1:
... print True
...
True
>>> if 0:
... print True
...
>>>

Use of namedtuple in py3k branch for urlparse.

========================================================================

Dissecting urlparse:

1) __all__ methods provides the public interfaces to all the methods like
urlparse, urlunparse, urljoin, urldefrag, urlsplit and urlunsplit.

2) then there is classification of schemes like uses_relative, uses_netloc,
non_hierarchical, uses_params, uses_query, uses_fragment
- there should be defined in an rfc most probably 1808.
- there is a special '' blank string, in certain classifications, which
means that apply by default.

3)valid characters in scheme name should be defined in 1808.

4) class ResultMixin is defined to provide username, password, hostname and
port.

5) from collections import namedtuple. This should be from python2.6.
namedtuple is pretty interesting feature.

6) SplitResult and ParseResult. Very good use of namedtuple and ResultMixin

7) The behaviour of the public methods urlparse, urlunparse, urlsplit and
urlunsplit and urldefrag matter most.

urlparse - scheme, netloc, path, params, query and fragment.
urlunparse will take those parameters and construct the url back.

urlsplit - scheme, netloc, path, query and fragment.
urlunsplit - takes these parameters (scheme, netloc, path, query and fragment)
and returns a url.

urlparse x urlunparse
urlsplit x urlunsplit
urldefrag
urljoin

Date: Tue Aug 19 20:40:46 IST 2008

Changes to urlsplit functionality in urllib.
As per the RFC3986, the url is split into:
scheme, authority, path, query, frag = url

The authority part in turn can be split into the sections:
user, passwd, host, port = authority

The following line is the regular expression for breaking-down a
well-formed URI reference into its components.

^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
12 3 4 5 6 7 8 9

scheme = $2
authority = $4
path = $5
query = $7
fragment = $9

The urlsplit functionality in the urllib can be moved to new regular
expression based parsing mechanism.

>From man uri, which confirms to rfc2396 and HTML 4.0 specs.

- An absolute identifier refers to a resource independent of context, while a
relative identifier refers to a resource by describing the difference from
the current context.

- A path segment while contains a colon character ':' can't be used as the
first segment of a relative URI path. Use it like this './file:path'

- A query can be given in the archaic "isindex" format, consisting of a word or
a phrase and not including an equal sign (=). If = is there, then it must be
after & like &key=value format.

Character Encodings:

- Reserved characters: ;/?:@&=+$,
- Unreserved characters: ALPHA, DIGITS, -_.!~*'()

An escaped octet is encoded as a character triplet consisting of the percent
character '%' followed by the two hexadecimal digits representing the octet
code.

HTML 4.0 specification section B.2 recommends the following, which should be
considered best available current guidance:

1) Represent each non-ASCII character as UTF-8
2) Escape those bytes with the URI escaping mechanism, converting each byte to
%HH where HH is the hexadecimal notation of the byte value.

One of the important changes when adhering to RFC3986 is parsing of IPv6
addresses.

===============================================================================
1) Bug issue1285086: urllib2.quote is too slow

For the short-circuit path, the regexp can make quote 10x as fast in
exceptional cases, even comparing to the faster version in trunk. The
average win for short-circuit seems to be twice as fast as the map in my
timings. This sounds good.

For the normal path, the overhead can make quote 50% slower. This IMHO
makes it unfit for quote replacement. Perhaps good for a cookbook recipe?

Regarding the OP's use case, I believe either adding a string cache to
quote or flagging stored strings as "safe" or "must quote" would result
in a much greater impact on performance.

Attaching patch against trunk. Web framework developers should be
interested in testing this and could provide the use cases/data needed
for settling this issue.

Index: urllib.py
===================================================================
--- urllib.py (revision 62222)
+++ urllib.py (working copy)
@@ -27,6 +27,7 @@
import os
import time
import sys
+import re
from urlparse import urljoin as basejoin

__all__ = ["urlopen", "URLopener", "FancyURLopener", "urlretrieve",
@@ -1175,6 +1176,7 @@
'abcdefghijklmnopqrstuvwxyz'
'0123456789' '_.-')
_safemaps = {}
+_must_quote = {}

def quote(s, safe = '/'):
"""quote('abc def') -> 'abc%20def'
@@ -1200,8 +1202,11 @@
cachekey = (safe, always_safe)
try:
safe_map = _safemaps[cachekey]
+ if not _must_quote[cachekey].search(s):
+ return s
except KeyError:
safe += always_safe
+ _must_quote[cachekey] = re.compile(r'[^%s]' % safe)
safe_map = {}
for i in range(256):
c = chr(i)

----------------------------------------------------------------------
What does this construct imply?

x = lambda: None

----------------------------------------------------------------------
How can we differentiate if an expression used is a general expression or a boolean expression.

----------------------------------------------------------------------
Having a construct like:
def __init__(self, *args, **kwargs):
BaseClass.__init__(self, *args, **kwargs)

But in the base class, I find that it is not taking the tuple and dict as
arguments.

I dont understand the
assert(proxies, 'has_key'), "proxies must be mapping"

----------------------------------------------------------------------

* What is an addrinfo struct.

The getaddrinfo() function returns a list of 5-tuples with the following structure:
(family, socktype, proto, canonname, sockaddr)

family, socktype, proto are all integer and are meant to be passed to the socket() function. canonname is a string representing the canonical name of the host. It can be a numeric IPv4/v6 address when AI_CANONNAME is specified for a numeric host.

socket.gethostbyname(hostname)
Translate a host name to IPv4 address format. The IPv4 address is returned as a string, such as '100.50.200.5'. If the host name is an IPv4 address itself it is returned unchanged. See gethostbyname_ex() for a more complete interface. gethostbyname() does not support IPv6 name resolution, and getaddrinfo() should be used instead for IPv4/v6 dual stack support.

We need to replace the gethostbyname socket call. Because it is only IPv4 specific. using the getaddrinfo() function can include the IPv4/v6 dual stack support.

import socket
print socket.gethostbyname(hostname)

def gethostbyname(hostname)
family, socktype, proto, canonname, sockaddr = socket.getaddrinfo(hostname)
return canonname

----------------------------------------------------------------------

Compare dates for cache control

RFC 1123 date format:
Thu, 01 Dec 1994 16:00:00 GMT

<pre>
>>> datereturned = "Thu, 01 Dec 1994 16:00:00 GMT"
>>> dateexpired = "Sun, 05 Aug 2007 03:25:42 GMT"
>>> obj1 = datetime.datetime(*time.strptime(datereturned, "%a, %d %b %Y %H:%M:%S %Z")[0:6])
>>> obj2 = datetime.datetime(*time.strptime(dateexpired, "%a, %d %b %Y %H:%M:%S %Z")[0:6])
>>> if obj1 == obj2:
print "Equal"
elif obj1 > obj2:
print datereturned
elif obj1 < obj2:
print dateexpired
</pre>

* Now you can compare the headers for expiry in cache control.

Header field definition:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Adding Header information to Apache:
http://webmaster.info.aol.com/apache.html

To add header:
Go to the /etc/httpd/conf/httpd.conf
For e.g:
Add the information on headers
Header set Author "Senthil"

Q) Question is can I add a test for cache.
A) Its not a functionality, but its merely an internal optimization.

Q) If test for cache needs to be written, how will you write it?
A) request an url and redirect and request it again and verify that it is
coming frm a dictionary or the dictionary value is stored.

from_url = "http://example.com/a.html"
to_url = "http://example.com/b.html"

h = urllib2.HTTPRedirectHandler()
o = h.parent = MockOpener()

req = Request(from_url)
def cached301redirect(h, req, url=to_url):
h.http_error_301(req, MockFile(), 301, "Blah", MockHeaders({"location":url}))

# Why is Request object taking two parameters?

req = Request(from_url, origin_req_host="example.com")
count = 0
try:
while 1:
redirect(h, req, "http://example.com")
count = count + 1
if count > 2:
self.assertEqual("http://example.com",
urllib2.HTTPRedirectHandler().cache[req].geturl())
except urllib2.HTTPError:
self.assertEqual(count, urllib2.HTTPRedirectHandler.max_repeats)

CacheFTPHandler testcasesare hard to write.

--
O.R.Senthil Kumaran

Ariane 5 Flight 501

2008-09-24T04:02:00.000-07:00

While listening to an introductory programming class, came across the mention of this this "Ariane 5 Flight 501" failure incident, which was caused by Arithmetic Exception and Integer overflow resulted from automatic type casting float to integer in the ADA program. Very costly software bug.

This German article discusses the issue. Below is the English translation of the same with the help of translate.google.com with some comments.

Ariane 5 - 501 (1-3) Ariane 5 - 501 (1-3)

4th June 1996, Kourou / FRZ. Guyana, ESA Guyana, ESA

Maiden flight of the new European launcher (weight: 740 tons, payload 7-18 t) with 4 Cluster satellites
Development costs in 10 years: DM 11 800 million I am not sure about 11 space 800 million. If it is, then its 4 trillion Rupees roughly.

Ada program of the inertial navigation system (excerpt):


 ... 
 declare 
   vertical_veloc_sensor: float; 
   horizontal_veloc_sensor: float; 
   vertical_veloc_bias: integer; 
   horizontal_veloc_bias: integer; 
   ... 
 begin 
   declare 
     pragma suppress(numeric_error, horizontal_veloc_bias);
   begin 
     sensor_get (vertical_veloc_sensor); 
     sensor_get (horizontal_veloc_sensor); 
     vertical_veloc_bias := integer(vertical_veloc_sensor); 
     horizontal_veloc_bias := integer(horizontal_veloc_sensor); 
     ... 
   exception exceptionnelle 
     When numeric_error => calculate_vertical_veloc (); 
     when others => use_irs1 (); 
   end; 
  irs2 end;

Effect:
37 seconds after ignition of the rocket (30 seconds after Liftoff) Ariane 5 reached 3700 m in altitude with a horizontal velocity of 32768.0 (internal units).This value was about five times higher than that of Ariane 4th

The transformation into a whole number led to an overflow, but was not caught.

The replacement computer (redundancy!) Had the same problem 72 msec before and immediately switched from that.

This resulted in that diagnostic data to the main computer were sent to this interpreted as trajectory data.Consequently, nonsensical control commands to the side, pivoting solid engines, and later to the main engine, to the deviate Flight no large (over 20 degrees) to correct them.

The rocket, however, threatened damage control and tested all himself (39 sec). I guess auto-destruct.

An intensive test of the navigation and main computer had not been undertaken since the software was in tested Ariane 4.

Damage:
DM 250 million start-up costs (~ 8.5 billion INR)
DM 850 million Cluster satellites (~ 29 billion INR)
DM 600 million for future improvements (~ 20 billion INR)
Loss of earnings for 2 to 3 years

The next test flight was only 17 months later carried out - 1 Stage ended prematurely firing.

The first commercial flight took place in December 1999.

Tragedy:

The problematic part of the program was only in preparation for the launch and the launch itself needed.
It should only be a transitional period to be active, for security reasons: 50 sec, to the ground station at a launch control over the interruption would have.

Despite the very different behavior of the Ariane 5 was nothing new about value.

Optimization:
Only 3 of 7 for a variables overflow examined - for the other 4 variables evidence existed that the values would remain small enough (Ariane 4).

This evidence was not for the Ariane 5 and this was not even understood.
Problem was with the of reuse of software!

Incredible - after 40 years of software-defect findings:

It was during program design assumes that only hardware failure may occur!
Therefore, the replacement computers also identical software. The system specification established that in case of failure of the computer is off and the replacement computer einspringt. A restart of a computer was not useful, since the redefinition of the flying height is too expensive.

PS: The attempt to establish new 4 Cluster satellites to launch, succeeded in July and August 2000 with two Russian launchers.

Master Vim Help system

2008-09-16T02:15:00.000-07:00

This article
details steps on how to create your own help file in the vim help system.
Should be very helpful.

fortune from pyljvim

2008-09-15T14:37:00.000-07:00

Education is an admirable thing, but it is well to remember from time to
time that nothing that is worth knowing can be taught.
-- Oscar Wilde, "The Critic as Artist"

[issue600362] relocate cgi.parse_qs() into urlparse

2008-08-14T13:29:00.001-07:00

This issue/comments somehow escaped from my noticed, initially. I have
addressed your comments in the new set of patches.

1) Previous patch Docs had issues. Updated the Docs patch.
2) Included message in cgi.py about parse_qs, parse_qsl being present
for backward compatiblity.
3) The reason, py26 version of patch has quote function from urllib is
to avoid circular reference. urllib import urlparse for urljoin method.
So only way for us use quote is to have that portion of code in the
patch as well.

Please have a look the patches.
As this request has been present from a long time ( 2002-08-26 !), is it
possible to include this change in b3?

Thanks,
Senthil

Added file: http://bugs.python.org/file11116/issue600362-py26-v2.diff

issue2756: urllib2 add_header fails with existing unredirected_header

2008-08-13T23:56:00.000-07:00


>>> import urllib2
>>> url = 'http://www.whompbox.com/headertest.php'
>>> request = urllib2.Request(url)
>>> request.add_data("Spam")
>>> f = urllib2.urlopen(url)
>>> request.header_items()
[]
>>> request.unredirected_hdrs.items()
[]
>>> f = urllib2.urlopen(request)
>>> request.header_items()

[('Content-length', '4'), ('Content-type', 'application/x-www-form-urlencoded'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]


>>> request.unredirected_hdrs.items()

[('Content-length', '4'), ('Content-type', 'application/x-www-form-urlencoded'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]

Comment: This is fine. What is actually happening is do_request_ method in the http_open() is setting the unredirected_hdrs to above items.


>>> request.add_header('Content-type','application/xml')
>>> f = urllib2.urlopen(request)
>>> request.header_items()

[('Content-length', '4'), ('Content-type', 'application/xml'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]



Comment: When we add_header() the headers are indeed changed. Correct behavior. 
>>> request.unredirected_hdrs.items()

[('Content-length', '4'), ('Content-type', 'application/x-www-form-urlencoded'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]

Comment: add_header() has not modified the unredirected_hdr.
Is this the whole purpose of issue2756? If yes, then better understanding of unredirected_hdr is needed and in the do_request_ method of AbstractHTTPHandler, where it changes unredirected_hdrs based on the logic of "not request.has_header(...)" what is actually aimed for checking that.

If add_header() is not supposed to change unredirected_hdrs but, add_unredirected_header() is the call to change unredirected_hdrs then, it is working fine and as expected.

(This is an undocumented interface, items() call was used for viewing the headers, tough actual code might not be using it.


>>>request.add_unredirected_header('Content-type','application/xml')
>>> request.unredirected_hdrs.items()

[('Content-length', '4'), ('Content-type', 'application/xml'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]
>>>
Comment: add_unredirected_header() has correctly affected.

After application of the attached patch in issue report which modifies the add_header() and add_unredirected_header() method to remove the existing header of the same name. We will observe that the unredirected_hdr itself is removed and it is never added back.

After application of attached patch:


>>> url = 'http://www.whompbox.com/headertest.php'
>>> request = urllib2.Request(url)
>>> request.add_data("Spam")
>>> f = urllib2.urlopen(request)
>>> request.header_items()

[('Content-length', '4'), ('Content-type', 'application/x-www-form-urlencoded'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]


>>> request.unredirected_hdrs.items()

[('Content-length', '4'), ('Content-type', 'application/x-www-form-urlencoded'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]


>>> request.add_header('Content-type','application/xml')
>>> f = urllib2.urlopen(request)
>>> request.header_items()

[('Content-length', '4'), ('Content-type', 'application/xml'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]


>>> request.unredirected_hdrs.items()

[('Content-length', '4'), ('Host', 'www.whompbox.com'), ('User-agent', 'Python-urllib/2.6')]
>>>
Comment: Notice the absense of Content-type header.

Issue 2756 - urllib2 add_header fails with existing unredirected header

2008-08-13T10:28:00.000-07:00

The issue noticable here:

[ors@goofy ~]$ python
Python 2.6b2+ (trunk:65482M, Aug 4 2008, 14:26:01)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> url = 'http://hroch486.icpf.cas.cz/formpost.html'
>>> import urllib2
>>> req_obj = urllib2.Request(url)
>>> req_obj.unredirected_hdrs
{}
>>> req_obj.add_data("Spam")
>>> req_obj.unredirected_hdrs
{}
>>> response = urllib2.urlopen(req_obj)
>>> req_obj.unredirected_hdrs
{'Content-length': '4', 'Content-type': 'application/x-www-form-urlencoded',
'Host': 'hroch486.icpf.cas.cz', 'User-agent': 'Python-urllib/2.6'}
>>> req_obj.add_data("SpamBar")
>>> req_obj.add_header("Content-type","application/html")
>>> response = urllib2.urlopen(req_obj)
>>> req_obj.unredirected_hdrs
{'Content-length': '4', 'Content-type': 'application/x-www-form-urlencoded',
'Host': 'hroch486.icpf.cas.cz', 'User-agent': 'Python-urllib/2.6'}
>>> req_obj.get_data()
'SpamBar'
>>>

In the final req_obj call, the unredirected_hdrs had not changed in neither
Content-length nor in Content-type.

[issue2776] urllib2.urlopen() gets confused with path with // in it

2008-08-12T09:24:00.000-07:00

I played around with pdb module today to debug this issue. pdb is really
helpful.
Here's how the control goes.
1) There is an url with two '//'s in the path.
2) The call is data = urllib2.urlopen(url).read()
3) urlopen calls the build_opener. build_opener builds the opener using (tuple)
of handlers.
4) opener is an instance of OpenerDirector() and has default HTTPHandler and
HTTPSHandler.
5) When the Request call is made and the request has 'http' protocol, then
http_request method is called.
6) HTTPHandler has http_request method which is
AbstractHTTPHandler.do_request_

Now, for this issue we get to the do_request_ method and see that

7) host is set in the do_request_ method in the get_host() call.
8) request.get_selector() is the call which is causing this particular issue
of "urllib2 getting confused with path containing //".
.get_selector() method returns self.__r_host.
Now, when proxy is set using set_proxy(), self.__r_host is self.__original (
The original complete url itself), so the get_selector() call is returns the
sel_url properly and we can get teh host from the splithost() call on teh
sel_url.

When proxy is not set, and the url contains '//' in the path segment, then
.get_host() (step 7) call would have seperated the self.host and self.__r_host
(it pointing to the rest of the url) and .get_selector() simply returns this
(self.__r_host, rest of the url expect host. Thus causing call to fail.

9) Before the fix, request.add_unredirected_header('Host', sel_host or host)
had the escape mechanism set for proper urls wherein with sel_host is not set
and the host is used. Unfortunately, that failed when this bug caused sel_host
to be set to self.__r_host and Host in the headers was being setup wrongly (
rest of the url).

The patch which was attached appropriately fixed the issue. I modified and
included for py3k.

>
> I could reproduce this issue on trunk and p3k branch. The patch attached
> by Adrianna Pinska "appropriately" fixes this issue. I agree with the
> logic. Attaching the patch for py3k with the same fix.
>
> Thanks,
> Senthil
>
> Added file: http://bugs.python.org/file11103/issue2776-py3k.diff
>

Python urllib bugs roundup

2008-08-11T17:25:00.000-07:00

From the previous post of bugs round up, finished activities (bugs which are closed now) include:

* http://bugs.python.org/issue1432 - Strange Behaviour of urlparse.urljoin
* http://bugs.python.org/issue2275 - urllib2 header capitalization.
* http://bugs.python.org/issue2916 - urlgrabber.grabber calls setdefaulttimeout
* http://bugs.python.org/issue2195 - urlparse() does not handle URLs with port numbers properly. - Duplicate issue.
* http://bugs.python.org/issue2829 - Copy cgi.parse_qs() to urllib.parse - Duplicate of issue600362.
* http://bugs.python.org/issue2885 - Create the urllib package. (but the tests are still named test_urllib2, test_urlparse etc. I shall discuss with py-dev if names changes in tests is okay before beta3 and give a attempt).

Activities TODO:

High- Priority:

Now, the list of bugs which are partially completed and requires addressing of some issues mentioned in the patches.These would take up some higher priority, as addressing would result in closure sooner.

* http://bugs.python.org/issue600362 - relocate cgi.parse_qs() into urlparse
* http://bugs.python.org/issue2776 - urllib2.urlopen() gets confused with path with // in it
* http://bugs.python.org/issue2756 - urllib2 add_header fails with existing unredirected_header Patch attached.
* http://bugs.python.org/issue2464 - urllib2 can't handle http://www.wikispaces.com

Plan:
I shall attempt to address all these issues before the release of Beta3 (That should be either on August 15 / August 23)

The following were some of the main issues to be taken up during G-SOC.
I see that as I have understood RFC 3986 better, I can work on issue1591035. I shall work on it on the branch and then discuss for inclusion in the trunk.

Feature Requests:

* http://bugs.python.org/issue1591035 - update urlparse to RFC 3986.Plan: By August 23.
* http://bugs.python.org/issue1462525 - URI parsing library - This will depend upon the previous issue, so we can assume Aug 23 for closure.
* http://bugs.python.org/issue2987 - RFC2732 support for urlparse (e.g. http://[::1]:80/) This is related bug again and will conclude by the same time-line.

I shall take up the following listed bugs after completion of the above.

* http://bugs.python.org/issue1448934 - urllib2+https+proxy not working.
* http://bugs.python.org/issue1424152 - urllib/urllib2: HTTPS over (Squid) Proxy fails
* http://bugs.python.org/issue1675455 - Use getaddrinfo() in urllib2.py for IPv6 support. Patch provided.

Low priority.
* http://bugs.python.org/issue1285086 - urllib.quote is too slow

Issue 3300

2008-08-11T17:13:00.000-07:00

There is a good amount of discussion going around with
http://bugs.python.org/issue3300, I had been following from the start and had
an inclination towards quote and quote_plus to support UTF-8. But as the
discussion went further, without strong point on which stance to take, I had to
refresh and improve my knowledge of unicode support in Python and espcially
Unicode Strings in Python 3.0. Hopefully this will come handy in other issues.

Here are some notes on Unicode and Python.

What is Unicode?
In Computing, Unicode is an Industry Standard allowing Computers to
consistently display and manipulate text expressed in most of the world's
writing systems.

Unicode provides a unique number for every character,
no matter what the platform,
no matter what the program,
no matter what the language.

What is Unicode Character Set?

What is character encoding?

What is Encoding?
Converting a Character (or Something) to Number, because Computer internally
store numbers only.

Unicode Strings are a set of Code Points represented from 0x000000 to 0x10FFFF.
This sequence needs to be represented as a set of bytes ( meaning, values from
0-255) in memory. The rules for translating the Unicode String into sequence of
bytes is called encoding.

The representation in the number format is required for homogenuity, otherwise
it will be difficult to convert to and from.

What is Unicode Transformation Format?

What is UTF-8?

Unicode can be implented using a many character encodings. The most commonly
used one is utf-8, which uses 1 byte for all ASCII characters, which have the
same code values as in the standard ASCII encoding, and up to 4 bytes for other
characters.

When it \u the remaining the Unicode Code points which you will find defined
internationally from unicode.org

Now, how to represent them in BINARY (Coz: Computer!), is the trick and you
will have different encodings to do so.
So UTF-8 is one encoding and UTF-16,ASCII are all different encodings.

So you construct a unicode string
mystr = u'\u0065\u0066\u0067\u0068'

mystr is a unicode object. It does not make sense to print it.
But if you wish to see the object, use repr
print repr(mystr)

Now, the unicode object can be coverted to Binary using encoding, and let us
use 'ascii' and 'utf-8'
so you would do
asciistr = mystr.encode('ascii')
utf8str = mystr.encode('utf-8')

Now, it is string object in BINARY

let us print asciistr, and utf8str

STILL NEED MORE UNDERSTANDING.

http://boodebr.org/main/python/all-about-python-and-unicode

A Unicode string holds characters from the Unicode character set.