I start this post with an apology. I usually don’t rant or vent here, which are feelings that I usually reserve to my personal blog.

I don’t use BioPython, never used it. I have it installed in my systems, but I never wrote a piece of code importing BioPython routines. But I subscribe to their mailing lists, both user and developer. I maybe have written once to the list, and I just follow the discussions there.

Since last year one of the main topics has been the possibility of moving BioPython from CVS to another version control system. Yes, you read it right. It’s 2009 and BioPython uses CVS and their version control system. Soon, CVS will be like typewriters and LPs to young developers. Last stable release of CVS was sometime in 2005, what in interwebs time is equivalent to something like 1972. Since 2005, Subversion has taken the world of version control by storm, and Git is getting also very strong, not to mention Bazaar, Darcs, Mercurial and some others that I might not be aware of.

This is a discussion that have been dragging for sometime in the list. And it’s a shame, a clear lack of leadership from whoever is (not) leading the project. BioRuby is Git, BioPerl SVN and BioPython is CVS, because they “need to care for the legacy developers”. It’s like MSFT keeping two copies of the Notepad executable because they needed to cater to legacy applications, but with a different scale of course. With the current Python steam in the non-bioinformatics and bioinformatics community is very sad to see BioPython not evolving (before you ask me, no, I’m not interested in helping, not the way things are now). Perl which is language forever-in-waiting for its holy grail (Perl 6) has a strong community behind it, and more important an excellent leadership, that’s not scare of making decisions.

So, if you’re still using CVS, it’s 2009!

Next is the built-in method (type?) super, which was new to me. Basically super gives you access a method or attribute of a class by calling its parent directly. This is a segue into understanding the Method Resolution Order in Python, which is understanding which class has precedence over the others. For me, I haven’t dealt with such structures before it was a good and straight explanation, especially when he explains about possible pitfalls of using super. A short list of best practices helps:

• Multiple inheritance should be avoided:
• super usage has to be consistent: Mixing super and classic calls is a confusing practice.
• Don’t mix old-style and new-style classes
• Class hierarchy has to be looked over when a parent class is called

After dealing with MRO, comes what I think is one of the best sections of the book so far, where Tarek explains about object descriptors and gives a little bit of the Python’s approach to introspection. This short section is basically all code, but it’s good to have a good best practices reference, including here properties and slots.

The last part of the chapter covers meta programming, and as Chris pointed in the comments, that’s a difficult area of Python (maybe for the ones like me that don’t have a CS formation). I would have to try the examples by hand and maybe define areas in my code where I can use it, so to take fully advantage and fully understand it.

Overall, the chapter gives a good series of topics about Python classes and I enjoyed learning a little bit more things that I couldn’t understand previously. Next we will see a review of chapter 4, that deals with PEP 8 and naming best practices.

So we’re up to the second chapter of Tarek’s book. A short disclaimer before diving into it. I started this blog, basically one year after I had started programming with Python. The initial idea was to “convert” the Beginning Perl for Bioinformatics book to Python and see what were the advantages and disadvantages of both languages. I was far from being a advanced Python programmer, and the inception of the blog helped me getting close to that, even though I consider myself far from being an expert programmer in Python. I learned a lot working on converting the Perl and learned a lot from the comments and interaction with other programmers and visitors of the blog. As anything in life one’s path is long and tortuous and there’s nothing better than daily learning and exercise.

So, as I mentioned in the previous post, this book was tailored for someone like me. I needed a boost on advanced Python techniques and the second chapter just gave me that. Tarek writes in this chapter about good syntax practices below the class level, functions and methods that are common in daily usage. He starts with list comprehensions, that we have seen in this site. It’s a short and concise section and gives you exactly what you need about this functionality.

Next, iterators and generators. I had a little bit of background on iterators, and have used them here and there, but not a lot on generators. I learned a bit from this section, what you expect from a book like this, things like the close and throw. Although this was good first step on generators, I wished the section could be longer, but that maybe not the focus of the book.

Coroutines was a completely new subject for me. Maybe I haven’t been diving into Python as much as I needed to, but time is short these days and programming Python is not the first objective of my work. The example is complete and easy to understand, but again I wish it was a tad bit longer. Tarek then explains a bit of generator expressions (list comprehension for generators) and enters the itertools module. So far so good, it’s a nice summary (at least for me) of simple techniques that can be incorporated into daily coding. And then … Decorators.

I blame on my poor CS skills or maybe my whole background on programming, but I still cannot get decorators. In my short-sighted view of the programming world I cannot see a place, at least on the things I’m doing, where I can use a decorator. And here comes the first criticism of the book: I still cannot get after reading the section. One thing that would help a bit would be to have colours on the examples and maybe go over them explaining some code lines. But at the same time, I admit that this might be a personal problem, where the concept of decorators don’t fit into my brain, and maybe the focus of the book is to show this advanced technique to someone that has a better grasp of the concept.

Overall, it’s a very good chapter and a good pointer to some expert/advanced techniques in Python. Tomorrow, chapter 3, and we’re a going to see classes.

I’m a a biologist, self-taught programmer/coder/you-name-it. I only had a brief course on programming logic with Pascal in 1993 (I think). I first learned Basic on Apple ][, then on PC, then moved to Visual Basic, Pascal, C and C++, most of them with the help of books. About three and a half years ago, I got tired of compiling things and decided to learn a different language that would be more agile to code with. Not liking Perl, made me check Python. And I got hooked. Of course as a lay programmer, I won’t discuss why it’s better or worse than any other language using technical terms, but I can say that Python fits my needs in fast and efficient programming and I’m quite happy with the choice I’ve made. So, this review will not be technical, but will try to expose the book’s strengths and the weak parts.

Chapter one gives a good introduction on how to install Python and some nice pointers on how to program Python, such as IDEs and initial settings you can add to it. Also there is a short overview of the modern Python implementations. Is it a necessary chapter? Yes and no. No, because the schooled Python user won’t need it, his or her programming environment will already be installed, configured, set and ready to go. Yes, because this chapter works as a disclaimer for the not-so-experienced Python programmer, and shows everyone of what is expected of this book and what standards will be used. In my opinion, it’s a necessary starting point, so the author knows that everyone is at the same level. This chapter is also a good short summary of good practices of installing and setting up Python.

Tomorrow, chapter 2.

Edit: problem solved. Thanks everyone!

Anyone with Git experience the link to the repository is here (not many commits yet, but it is coming).

Apart from that I already have some suggestions from friends and colleagues, but I am still looking for some subjects that the readers would like to see covered here. Leave me a message or a comment below with some Bioinformatics subject that would be interesting to write about.

On another note, the blog now is syndicated on Planet Python and Unofficial Planet Python. Welcome.

I browsed the web searching for a sed/awk solution to split FASTA files something that I thought could be done in one line. And I found it. With a little tweak I ended up with a one-liner using awk that split the sequences in multiple files:

awk '/^>/{f="fasta."++d} {print > f}'

Awk breaks each line of its input into fields that can be checked, printed or ignored. In the command line above. awk’s commands are in between the single quotes. The first command is a regex that matches initial >. Inside the curly braces we assign a variable f to store the output filename we want and also have an integer that is incremented every time we pass through it. The second curly braces are for the actual output, where we use awk’s print command and redirect output to a file named after the variable we created in the first commands in between curly braces. Finally, we assign an input file, outside the region delimited by single quotes. And that’s it. As output we will have a series of files called fasta.# each one containing one sequence.

#!/usr/bin/env python

import random
import sys

def simulate_sequence(length):
    dna = ['A', 'C', 'G', 'T']
    sequence = ''
    for i in range(length):
        sequence += random.choice(dna)
    return sequence

setsize = int(sys.argv[1])
minlength = int(sys.argv[2])
maxlength = int(sys.argv[3])

rlength = []
sequenceset = []
for i in range(setsize):
    rlength.append(random.randint(minlength, maxlength))

for length in rlength:
    sequenceset.append(simulate_sequence(length))

for sequence in sequenceset:
    print sequence

I don’t know if the lack of formatting in the comments might have generated a different result to me, but let’s see how we can generate multiple sets of random DNA sequence. Getting back to section 4, we can start with the original script that creates only one set

#!/usr/bin/env python

import random
import sys

def simulate_sequence(length):
    dna = ['A', 'C', 'G', 'T']
    sequence = ''
    for i in range(length):
        sequence += random.choice(dna)
    return sequence

setsize = int(sys.argv[1])
minlength = int(sys.argv[2])
maxlength = int(sys.argv[3])

sequenceset = []
for i in range(setsize):
    rlength = random.randint(minlength, maxlength)
    sequenceset.append(simulate_sequence(rlength))

for sequence in sequenceset:
    print sequence

There is a very easy way of changing this code to make it output a definite number of sets. WE add a new input parameter and also a loop to it, and that should take care of everything.

#!/usr/bin/env python

import random
import sys

def simulate_sequence(length):
    dna = ['A', 'C', 'G', 'T']
    sequence = ''
    for i in range(length):
        sequence += random.choice(dna)
    return sequence

setsize = int(sys.argv[1])
minlength = int(sys.argv[2])
maxlength = int(sys.argv[3])
nsets = int(sys.argv[4])

for i in range(nsets):
    sequenceset = []
    for i in range(setsize):
        rlength = random.randint(minlength, maxlength)
        sequenceset.append(simulate_sequence(rlength))

    for sequence in sequenceset:
        print sequence

    print

A test with 10 as the number of sequences, 20 as minimum and 30 as maximum length and 2 sets output somthing like this

GAGGACGACGTGTCGGATTAGAT
GCAGATAAACGGGTAAGATGA
ATTAGGAAGGGTCTAAGAGATCCCGGTAA
ATCAGCCTCCGCACGCCTCGTGTA
AGTTTAGCTAGCCGTAGTACTTTC
TTTGAAGCTCGGGGCTAAACTCCGACCAC
GACAGGGTAAGGTGACCTTCTCAG
TGGTTGACTCTGTTCTCTGGT
CGCAATACGTCGTGGTGCCGCTCCACT
TCCTGCAGGGAGGCAGCAGC

TTCGGAGGTATTGAGTACTGTTGCATCG
CCCTTATAATACCTAGTGTAACATTC
TAGTCATGTCATATAACCTGTGTCG
AGCGCGAACCCTCCAGTTTTTTT
AGGGCGTGTAAGCGACGTTGAACAAG
CCATGCGTCTTTTTGGCGATCCGT
ACTCAGATCGGCTAGTATCCTTCGCACG
CTATCGTCGTTCCCAGCAACGGCAGCAGA
CGTGCGATGGGGCGAGATTTTT
ACGCTTTGGCTAGTGGGGAGTTTCGCCACT

what is basically what we need. Quick ‘n’ easy.

Of course a top notch bioinformatician that uses Linux 100% of the time would be able to create a simple bash script and use the initial script to generate multiple random sets. bash is not a primary scope of the blog, but let’s see how to do that. Saying the our initial script was saved as randomset.py we would have a very short script in bash to generate multiple sets. We want to generate 10 sets of sequences of length between 20 and 30 nucleotides, and each set would need to have 10 different sequences (remeber that our initial script only accepts 3 parameters).

Off topic

for ((i=0;i<10;i++))
do
min=20
max=30
nseqs=10
python randomset.py $nseqs $min $max
echo
done

That should do the trick. You can even type it from the command line. The for syntax in bash is very similar to C/C++ but it does not accept spaces, as all the variable assignments. When you assigning a variable value in bash you omit the dollar sign and when you are reading the variable value you access it with the dollar sign, like in the line that runs the script. A for loop in bash starts with a do command and ends in a done command, not brackets or parenthesis involved. The echo at the end helps adding a separating blank line after each random set.

off topic

Of course we could use "makenucseq from the EMBOSS package", we can even use Dawg from my friend Reed Cartwright, which I believe has a shorter learning curve (and faster compilation) than EMBOSS, but that will be away from the scope of this blog, which is to show how to use Python in simple bioinformatics scripts. I allowed myself to post off topic subjects (including this last paragraph) because a (very) small percentage of the commenter(s) seems to misunderstand the real focus of this tutorial. We don't need fancy words or complex topics to show how simple things can achieve good results in almost no time. My focus is not give the reader the fish and chips ready to eat. My focus is to show how to fish and peel, cut and fry the potato in order to cook the dish. This way, next time he or she might be cooking better things to eat.

I admit that some of the things you have addressed in the past, whilst ‘reinventing’ the wheel, offer important insights into the general problems of programming in general – this particular example was not one of them. From following the course which I assume was outlined by ‘Beginning Perl for Bioinformatics’ (which I do not think is a good reference for beginning Perl at all) a user should already be able to perform the following operations – open a file, iterate over and parse the contents, open a file for writing, and apply some rules for formatting the output. Thus you have already given them the tools to complete the operation should they wish to do so in a Python context.

The problem is that you are trying to educate potentially early career/transitional bioinformaticians here – and showing them how to complete tasks in X lines of Python when there is a single command line application to do the trick I think is disingenuous. We wouldn’t dream, for instance, of educating our MRes students in this way.

I also would take issue that diversity is a key. You can be as diverse as you like but no-one is seriously going to suggest that you write your next script to parse something out of a GenBank file in assembler. One of the hallmarks of any computer professional in any field is the ability to use the right tool for the job. You should be technology agnostic and be prepared to string together applications or scripts from anywhere to reach your goal – take a look at something like Taverna for instance.

I notice in the other post that you mention there is a high proportion of Windows users to this site and therefore command line *nix comments are of no interest. Although I know a number of bioinformaticians who use Windows (generally to integrate with an institutions existing Microsoft heavy infrastructure) they all have logins to dozens of *nix machines, run *nix in a VM or at the very least have a Cygwin installation with which to work with. Basing the availability of technology on the OS of your page imprints is not necessarily indicative.

I hope you don’t mind me addressing these points here rather in my blog, as my blog is personal (hence the lack of bioinformatics content on there). I have actually been following these tutorials for a while (as someone who came into Bioinformatics some time ago – my first forays in programming were in Perl) it is nice for me to see tasks that I find intuitive in one language written in another. Consequently the fact that I felt need to comment, merely suggests that I felt strongly on this issue, rather than having any negative feelings towards what you are trying to achieve.

I am sorry for the usual readers of this tutorial, for posting his comments as an article here. We will be back to our uneducated, simple and basic “tutorial for bioinformatics in X lines for the noobie in Windows” soon.

Thanks Daniel.