You’ll learn about several of Python’s automated testing tools, and you’ll learn about the philosophies and methodologies that they were designed to support, like unit testing and test-driven development. When you’re done, you’ll be able to produce thoroughly tested code faster and more easily than ever before, and you’ll be able to do it in a way that doesn’t distract you from your “real” programming.

Packt also supplied a preview/sample chapter that you can download here.

I hope to get a review ready by the end of the week. before the Ontario Institute of Cancer Research retreat, otherwise I will try to post a full review next week.

The initial section of the book is well written (I’m not going criticize the book in terms of good/poor English, as I’m not well qualified to do that), and gives a clear perspective on how to program Python for scientists, who are the main target demographic of the book. Of course, it always help to have some basic knowledge of command line shells, but the book also includes some explanations of IDLE and other Python-capable IDEs. I cannot say that I read this section with the enough care and attention, but what I can say is that you won’t miss a beat with PfB, as it has more material than I expected. I still have to start with the more advanced topics, like BioPython and so forth, what I plan to do in the coming month, and as I don’t have a lot of experience with BioPython, I’m looking forward to it.

On the other hand I have a small-ish complaint, that maybe is more about style than substance. I don’t like the design of the book, the way the code interleaves with the text and the way the code explanations are presented. Most of the code blocks are followed by a careful explanation, but this explanation works as a figure label for the code block. That is quite annoying because there are too many stops in the text fluidity as one tends to lose attention to it (my case, not exactly everyone’s).

Another minor detail is the use of “he” every time scientists are referred (one example is on page 3 on the second phrase of the introduction). The (politically) correct would be to use “he or she” or “she or he” (but that’s OK with me).

I will try to post more complete reviews of the sections that I don’t master. I would also like to thank Sebastian for sending me a copy of the book.

I’m slowly moving the posts from the blog to a wiki. It makes easier to display post series and allows people to modify/enhance/discuss.

The wiki address is http://wiki.genedrift.org.

Thanks to the comments and suggestions to the last post, it’s possible to make now a more pythonic and clearly generic database update class. Let’s check how the “generic” update/edit entry function is currently:

def update_data(self, values_list):
    '''edits and updates fields'''

    if sys.platform == 'darwin':
        (cursor, database) = link_db(self.db_path)
    else:
        (cursor, database) = link_db()

    cursor.execute("UPDATE bac SET  projects = ?, comments = ?, temperature = ?, cell = ?, box = ?, tubes = ?, chromosome = ?, sdate = ?, clone = ?, source
	= ?, location1 = ?, startpos = ?, endpos = ?,
	gene = ?, genelink = ?, dnaex = ?, validation = ?, pcr = ?, refs = ?, antibiotic = ? WHERE idbac = ?",
    values_list['projects'], values_list['comments'], values_list['temperature'], values_list['cell'], values_list['box'], values_list['tubes'],
    values_list['chromo'], values_list['date'], values_list['clone'], values_list['source'], values_list['location'], values_list['start']
    values_list['end'], values_list['gene'], values_list['genelink'], values_list['dna'], values_list['validation'], values_list['pcr'],
    values_list['refs'], values_list['antibiotic'], values_list['idbac']))

    database.commit()
    database.close()

which is really ugly and, although it works, is not really useful outside this small project. Based on the comments the best option was to use placeholders and a dictionary, similar to the approach used on the insert data function. Pre-formatting a string to have both the field name to be updated and a placeholder (for instance :idbac) that will receive the values

update = ','.join(['%s=:%s' % (y, y) for y in values_list])

where update is the string we want and values_list is the dictionary with all the key-value pairs. I tried this approach, using this structure in the generic function, but then I decided that the best alternative was to put this join in the derived class function and pre-populate the string with the values and then send this string directly to the update function. In the end I opted to use this

update = ','.join(['%s=\"%s\"' % (y, values_list[y]) for y in values_list])

The latter is slightly different to what was suggested. The original one would create a tuple with the keys from the dictionary, making for instance sdate:sdate. With all these place holders just pass the dictionary and you have all the values inserted. This would be handy if the insert string was being created on the “generic” function. If we move this to the derived class, we can use the the alternative, keeping in mind that the values parsed should be surrounded by quotes, otherwise the SQL UPDATE statement will have problems with spaces and other foreign characters that should not be there. So instead of placeholders we will have gene:"PTEN" and we can attache this joined string to the actual commands. We then can move all the machinery from the “generic” function that can be written as

def update_data(self, update_string):
    '''edits and updates fields'''

    if sys.platform == 'darwin':
        (cursor, database) = link_db(self.db_path)
    else:
        (cursor, database) = link_db()
    cursor.execute(update_string)

    database.commit()
    database.close()

That’s it, very elegant (we will see the derived class in the next post). And finishing our generic class, we would need a delete function, so the user can eliminate entries that he/she doesn’t want anymore. It’s also a very simple function

def delete_data(self, delete_string):
    '''deletes one field'''

    if sys.platform == 'darwin':
        (cursor, database) = link_db(self.db_path)
    else:
        (cursor, database) = link_db()
    cursor.execute(delete_string)

    database.commit()
    database.close()

We will check the delete string next time. Again, I would like to thank for all the comments, it has been really helpful for me.

Previously in the series:
Part 1
Part 2
Part 3
Part 4
Part 5
Part 6
Part 7

def insert_data(self, values_list, insert_string):
    '''inserts data in the database'''

    if sys.platform == 'darwin':
        (cursor, database) = link_db(self.db_path)
    else:
        (cursor, database) = link_db()

    cursor.execute(insert_string % self.table_name, values_list)

    database.commit()
    database.close()

Basically no changes, apart from the obvious check for the current running operating system, which was explained in the last post. The other function to check is the update_data. This function is new and it wasn’t in the first version, but as it can be seen it has a problem being a “generic” function, because it contains information pertained to the table and database being used in the interface. This function basically received information that needs to be updated in the table’s fields and by using the SQL UPDATE ... SET edits and updates data in the changed fields. I have tried several different syntaxes to make the execute generic, mainly trying to pre-format the string without success. IF anyone reading this can help, I’d appreciate.

def update_data(self, values_list):
    '''edits and updates fields'''

    if sys.platform == 'darwin':
        (cursor, database) = link_db(self.db_path)
    else:
        (cursor, database) = link_db()

    cursor.execute("UPDATE bac SET  projects = ?, comments = ?, temperature = ?, cell = ?, box = ?, tubes = ?, chromosome = ?, sdate = ?, clone = ?, source = ?, location1 = ?, startpos = ?, endpos = ?,
	gene = ?, genelink = ?, dnaex = ?, validation = ?, pcr = ?, refs = ?, antibiotic = ? WHERE idbac = ?",
    values_list['projects'], values_list['comments'], values_list['temperature'], values_list['cell'], values_list['box'], values_list['tubes'],
    values_list['chromo'], values_list['date'], values_list['clone'], values_list['source'], values_list['location'], values_list['start'],  values_list['end'],
    values_list['gene'], values_list['genelink'], values_list['dna'], values_list['validation'], values_list['pcr'],
    values_list['refs'], values_list['antibiotic'], values_list['idbac']))

    database.commit()
    database.close()

Anyway, I will explain the logic of the command (OK for a stop gap, but not as a definite solution). values_list is a dictionary that is passed to the function and contains the field names as keys and the new/changed information as values. The execute method simply parses the values from each key in the update string which is then sent to the database and table to be changed. Everything is committed and the database is closed.

As this is a “generic” function from a “generic” class the ideal scenario would be to the function to receive a pre-formatted string with all the information, as in the insert data function, and update the information in the database.

I would like to thank in advance anyone that can comment on this. Next time we will continue checking the generic class and finalize this part in order to start with the interface build process.

Previously in the series:
Part 1
Part 2
Part 3
Part 4
Part 5
Part 6

Let’s get back to our SQLite and wxPython project. We haven’t seen anything on wxPython yet, and we will check the interface only on the next post. For now, let’s see some extra code added to the SQLite access class. Remember that we have a generic class and one class derived from it that would work on accessing specific tables in our database file.

When we last covered the db access routines, there was no search for an entry (the function returned everything in the table no matter what), there was no update function in case someone would want to modify an entry and there was no delete method if you wanted to delete something. In the meantime, I added all of this functionality (and some other) to the generic class and extended it to the class derived from it. Let’s check how the generic class is now (you will notice that there is an issue in one of the methods, if someone can help me I’d appreciate. More details later.)

class DB_Generic():
    '''generic class to add DB functionality'''
    def __init__(self, table_name, db_path = ''):
        #par= name of the table to be used
        self.table_name = table_name
        if len(db_path) > 0:
            self.db_path = db_path
            print db_path

    def get_data_generic(self, range = 1, bac_to_get = 0):
        '''gets the data from the database'''       

        if sys.platform == 'darwin':
            (cursor, database) = link_db(self.db_path)
        else:
            (cursor, database) = link_db()

        if range == 1:
            cursor.execute("""SELECT * FROM %s""" % self.table_name)
        elif range == 2:
            cursor.execute("""SELECT * FROM %s where idbac = %d""" % (self.table_name, bac_to_get))

        table_data = cursor.fetchall()
        raw_data = []
        for i in table_data:
            raw_data.append(list(i))

        self.table_data = raw_data
        database.close()

    def insert_data(self, values_list, insert_string):
        '''inserts data in the database'''

        if sys.platform == 'darwin':
            (cursor, database) = link_db(self.db_path)
        else:
            (cursor, database) = link_db()

        cursor.execute(insert_string % self.table_name, values_list)

        database.commit()
        database.close()

    def update_data(self, values_list):
        '''edits and updates fields'''

        if sys.platform == 'darwin':
            (cursor, database) = link_db(self.db_path)
        else:
            (cursor, database) = link_db()

        #change this to generic!!!!!!!!!!!!
        cursor.execute("UPDATE bac SET  projects = ?, comments = ?, temperature = ?, cell = ?, box = ?, tubes = ?, chromosome = ?, sdate = ?, clone = ?, source = ?, location1 = ?, startpos = ?, endpos = ?,
		gene = ?, genelink = ?, dnaex = ?, validation = ?, pcr = ?, refs = ?, antibiotic = ? WHERE idbac = ?",
        (values_list['projects'], values_list['comments'], values_list['temperature'], values_list['cell'], values_list['box'], values_list['tubes'],
         values_list['chromo'], values_list['date'], values_list['clone'], values_list['source'], values_list['location'], values_list['start'], values_list['end'],
         values_list['gene'], values_list['genelink'], values_list['dna'], values_list['validation'], values_list['pcr'],
         values_list['refs'], values_list['antibiotic'], values_list['idbac']))

        database.commit()
        database.close()

    def delete_data(self, delete_string):
        '''deletes one field'''

        if sys.platform == 'darwin':
            (cursor, database) = link_db(self.db_path)
        else:
            (cursor, database) = link_db()
        cursor.execute(delete_string)

        database.commit()
        database.close()

In the next couple of posts we’ll dissect each function and see what’s going on. The class definition wasn’t changed, so we start with get_data_generic

def get_data_generic(self, range = 1, bac_to_get = 0):
	'''gets the data from the database'''       

	if sys.platform == 'darwin':
		(cursor, database) = link_db(self.db_path)
	else:
		(cursor, database) = link_db()

	if range == 1:
		cursor.execute("""SELECT * FROM %s""" % self.table_name)
	elif range == 2:
		cursor.execute("""SELECT * FROM %s where idbac = %d""" % (self.table_name, bac_to_get))

	table_data = cursor.fetchall()
	raw_data = []
	for i in table_data:
		raw_data.append(list(i))

	self.table_data = raw_data
	database.close()

The first difference we notice here is the sys.platform usage. This is required if we intend to package our application as an OS X app, using py2app. When a Python/wxPython application is packaged in OS X, the actual application executable is inside the a directory named after the application (or whatever you set up). In our case here we don’t provide a way for the Python script to receive the path and name for the database on a command line, as we expect it to be in the executable’s current directory. Because of that we need to provide a “config” file (in our case here a one-line text file with the database path) inside the application wrapper, something we will see in the end of the series.

Another modification here is the range parameter and the addition of the bac_to_get parameter. Notice that both parameters have a value assigned to it. This means that they are optional, the function’s call can pass them or not. If it doesn’t pass, their value will be the one assigned on the function declaration. So, here if we are interested in getting all bacs, range will have the value of 1 and we don’t need to worry about it. If we want an specific bac we will pass range as 2 and then pass the bac_to_get ID to be returned.

A final change/addition is that we added a new select statement for the cases when range equals 2. This time we are adding the bac ID to be returned.

Previously in the series:
Part 1
Part 2
Part 3
Part 4
Part 5

Ruby on Rails from Ilya Grigorik on Vimeo.

Sorry for the non-Python post.

Basically for the type of application we’re developing here, SQLite seems ideal. It eliminates a lot of infrastructure that would be needed if we were working with MySQL or postgresql. We don’t need a server or know how to configure users or manage the databases and tables. All we need is contained in a single file that can be transported from system to system and can be accesed from the computers used in the lab, mainly XP and OS X. Also some web frameworks (Rails and Django, for instance) can use SQLite, so in the end we can have a desktop application and a web application accessing the same file without extra configuration.

Now the database created for this application has 8 tables and almost no relationships among them. SQLite allows the creation of relationships but in our case only a couple of cases were required. For the table we are using at the moment (bac) there is no need for relationships, although there are some fileds that can benefit from a more relational structure. Also SQLite don’t have the same data types that are found on the bigger SQL engines. All values can be stored as text, integer, real (floating point numbers), null and blob (verbose type, what you store is what you get). As actual types, you can set columns as Boolean and Data for instance and SQLite will understand them. If you have no experience in creating databases, let’s check again the table we are using in this small project. First, I would recommend the use of some SQLite database editor. You can find pretty good ones for any computer system and there is even a Firefox extension that allows you to edit some files. Editors make it easier to generate the SQL table creation scripts and make easier to visualize what we are doing.

So, the table bac looks like

CREATE TABLE bac
(idbac INTEGER PRIMARY KEY,
clone Text,
sdate Date,
source Text,
gene Text,
chromosome Text,
startpos Integer,
endpos Integer,
antibiotic Text,
location1 Text,
temperature Integer,
tubes Integer,
box Integer,
cell Integer,
dnaex Boolean,
validation Boolean,
pcr Boolean,
projects Text,
comments Text,
genelink Text,
refs Text);

If you go back to our last post, you will see that in the insert statement there is no mention of the idbac field. We don’t actually insert ay value there, the values that populate this field are created automatically. And idbac is our primary key, meaning it’s the unique identifier of each bac we insert in this table. And in SQLite a integer primary key is automatically incremented whenever values are inserted in the table. So our first insertion will create idbac 1, the second will create idbac 2 and so on.

I’m not going to enter in details about database development and administration, but it’s usual and safe to create tables with an auto-incremental integer primary keys. These fields, apart from make it easier t identify records, make access to such records faster and are great when relationships among tables are set. Let’s say that we had a column user in our bac table. And let’s say we had an user table with two columns: user_id and name, user_id being a auto-increment primary key. The user column in back could be linked with the user_id column in the user table, in what we call a one-to-many relationship (one user can insert as many bacs as he wants). One day we want to know who is actually working in the lab and we want to check how many bacs were catalogued by each user. We can easily search the user table and extract information from bacs at the same time thanks to the relationship between the tables. And the result should be returned quite quickly, as we are only searching integers.

All the other fields/columns in our table are straightforward to understand. They are basically related to the type of data they need to store. validation is a boolean because the bac might have been validated or not, just as danex (DNA extraction). At the same time, the number of tubes stored in the freezer will always be an integer. So, why does temperature is an integer? Because we can only store bacs in two type of freezers: -80 (ultra freezers) or -20 (regular freezer that we can have at home), and we don’t need to worry about fractional numbers.

Well, this is a very short and limited explanation of tables and SQLite. The web is full of resources about it, so next time we will get back to Python.

Previously in the series:
Part 1
Part 2
Part 3
Part 4

Let’s continue building our small db app. As mentioned in the previous post we need now to instantiate a specific class from our generic SQLite access class. In order to do this we just have to declare a new class and its type will be DB_Generic.

class Bac(DB_Generic)

This new class is called Bac because it’s linked to the bac table in our database file. A side note, bacs are Bacterial Artificial Chromosomes and are used in different molecular biology techniques. Mainly in our case bacs have incorporated human DNA segments and are used as probes for deletion, duplication, etc studies.

Now, back to our Python code, as soon as we instantiate our generic class, the object (class) we create has access to all methods and functions from the parent class (by using self), but we still need to create functionality and expose other methods that can be accessed from a class object derived from Bac.

Our instantiated class will be

class Bac(DB_Generic):
    def __init_(self):
        self.bac_data = []
        DB_Generic.__init__(self, 'bac')

    def get_data(self):
        return self.get_data_generic()

    def load_data(self):
        pass

    def add_data(self, values_list):
        insert_string = """INSERT INTO %s (projects, comments, temperature, cell, box, tubes, chromosome, sdate, clone, source,
        location1, startpos, endpos, gene, genelink, dnaex, validation, pcr, refs, antibiotic)
        VALUES (:projects, :comments, :temperature, :cell, :box, :tubes, :chromo, :date, :clone, :source, :location, :start,
        :end, :gene, :genelink, :dna, :validation, :pcr, :refs, :antibiotic)"""
        self.insert_data(values_list, insert_string)

Pretty simple so far, as we don’t have a lot of declared methods. Let’s check one by one

def __init_(self):
    DB_Generic.__init__(self, 'bac')

The only line is the initialization required by the parent class, and we’re passing the value that is the table to be accessed.

def get_data(self):
	self.get_data_generic()
	return self.table_data

The get_data function returns the all elements in our table (So far, we still don’t have an elegant range option) and has one too many lines in it. We will get rid of some useless code here in the future, but it’s OK the way it is. Basically this code access the get_data_generic from the parent class and gets all the values stored in the table.

There is a function not yet complete that will load data, and will be used in the future. And the last one is the function that actually adds the data to the table with a SQL insert statement

def add_data(self, values_list):
	insert_string = """INSERT INTO %s (projects, comments, temperature, cell, box, tubes, chromosome, sdate, clone, source,
	location1, startpos, endpos, gene, genelink, dnaex, validation, pcr, refs, antibiotic)
	VALUES (:projects, :comments, :temperature, :cell, :box, :tubes, :chromo, :date, :clone, :source, :location, :start,
	:end, :gene, :genelink, :dna, :validation, :pcr, :refs, :antibiotic)"""
	self.insert_data(values_list, insert_string)

In this function, we have a large string with all the SQL insert options. A SQL insert statement is divided into two parts, one where you point where to insert the values and another where you input the values. Usually simple insert statements will have this structure

INSERT INTO my_table_name (table_column1, table_column2) VALUES (value1, value2);

So, we have the table we want to insert values into, its columns and the values we set for each column. After executed this will put value1 into table_column1 and value2 into table_column2. The actual syntax can vary a bit for different SQL engines but the structure is identical in most cases. Pretty simple.

For our insert string above, there are some aspects to call for attention. Again note the triple quote around the statement. This make sure that it’s not changed and parsed correctly. We also have a %s for the table name, which will be parsed by the parent class function that insert values, then a list of all the tables in the database and then a list of values to insert. And why the values to be inserted have this :value syntax? Because we are previously storing the values in a dictionary, and the “:” indicates that we need to get the dictionary value for the correspondent key.

The insert string, and the list of values (actually a dictionary, not the best variable/object name I must admit) is then sent to the parent class to be inserted. Storing the values to be inserted in a dictionary is OK for a one time insert case, where the values are obtained from a form. If you are parsing a large CSV or TSV file, ideally it’s better to put it in a list, and dump them at the same time.

We’re progressing. Next we will take a look on some simple SQL table structure and then move to create the form to insert the values and check the table.

Previously in the series:
Part 1
Part 2
Part 3

evi

Mirroring the post that appeared on Blind.Scientist

Some time ago there was a meme about science blogging and one of the questions were “why do you blog”. Well, I do it because of the “Nada Surf effect”. You don’t know the “Nada Surf effect”? Pity you weren’t in Washington, DC 2001.

In March or April of 2001, Nada Surf played a concert there. It was a small bar on 14th Street W, close to the more famous Black Cat. It was a spring night, I was with a couple of Dutch friends that had told me about the concert, if I’m not wrong, a couple of days before. It was also mid-week, so you wouldn’t expect big crowds in most concerts. We left ISH around 7 pm, with spare time for the 9 pm concert. We didn’t know the venue, we got there and it was empty, just a couple of souls at the bar. We sat and for about an hour we were pondering if we were in the right place, until a guy came and asked if we were staying for the concert. We said yes, paid th US$ 7.50 of the admittance and sipped our beers waiting for the opening act. Soon after we paid, a van parked outside and some guys started bringing music equipment inside. At that time there must have been around 20 people there. The van guys set the instruments, wasted 5 minutes soundchecking, and started. IT was Ashtray Babyhead.

They played for 40 minutes and as fast as they arrived they left. Another van parked outside and this time Nada Surf members started unloading and setting up the stage. Now roadies. OK, maybe one guy helped, but I’m getting old and the memory sometimes falters. At that point in time, almost 9 pm, the number of brave souls was at 50. They played as they were playing for 50.000 people in Wembley. Nice set, great songs, unforgettable night. After the show, they sold CDs at the usual after-show gathering, we talked about New York, Brazil and feijoada.

And why do I call it the “Nada Surf effect”? A band that used to play for thousands of people in festivals and stadiums, had a number one video on MTV (Popular), played in a midweek night in a small bar in Washington, DC as it was the band farewell. Every fan that night felt that they were the most important one, maybe even the only one.

And this type of example is the one that brings me to write this, and to write Beginning Python for Bioinformatics. Especially the latter (as I spend too much time here, writing about non-important stuff). If I can make one person have an idea, one person there to use Python, or least to learn something extra in their lives, I’m happy. I don’t care if I have a huge audience or if I’m famous. I care for the undergrad that is starting today, the high school kid that is hacking at night, or the Java coder that is looking for some better language (Ok, not really, but I couldn’t resist). And this is one of the things that I learned in Science, always give back and don’t expect anything in return. Just add to the pile of knowledge.

So, my advice for the three people that read this site is: let Nada Surf in!

class DB_Generic():
    '''generic class to add DB functionality'''
    def __init__(self, table_name):
        #par= name of the table to be used
        self.table_name = table_name

    def delete_entry(self):
        pass

    def get_data_generic(self):
        '''gets the data from the database
        generic so far, needs to be updated to include range'''        

        range = 1
        (cursor, database) = link_db()

        if range == 1:
            cursor.execute("""SELECT * from %s""" % self.table_name)

        table_data = cursor.fetchall()
        raw_data = []
        for i in table_data:
            raw_data.append(list(i))

        self.table_data = raw_data

        database.close()

    def insert_data(self, values_list, insert_string):
        '''inserts data in the database'''

        (cursor, database) = link_db()
        cursor.execute(insert_string % self.table_name, values_list)

        database.commit()
        database.close()

There are different functions in this class, we will take a look at each one individually. We can see that the class is far from being complete, something that we’ll do in the next posts. We start with the class initialization:

def __init__(self, table_name):
        #par= name of the table to be used
        self.table_name = table_name

Very simple and direct, it receives the table name that is being used by the interface (in this case). The table name is then stored in a object that can be accessed by other functions in the class. The function to delete entries is not finished as we only have a pass in it. We’ll will do it soon. Next we have a function that gets the data from the table.

    def get_data_generic(self):
        '''gets the data from the database
        generic so far, needs to be updated to include range'''        

        range = 1
        (cursor, database) = link_db()

        if range == 1:
            cursor.execute("""SELECT * from %s""" % self.table_name)

        table_data = cursor.fetchall()
        raw_data = []
        for i in table_data:
            raw_data.append(list(i))

        self.table_data = raw_data

        database.close()

So far it grabs everything, there is no range selection based on any of the table fields, so it’s a generic SQL SELECT. Let’s dissect it. The range object is a dummy variable that at the moment is there only to remind us that we need to include a range select. The next line is the most important in this function: it will call the link_db function and start the connection. Remember that link_db returns a tuple with the cursor and database connection. Basically we will work with cursor methods to get the data, and the first action is to execute a SQL SELECT stetement where we select everything in the table

cursor.execute("""SELECT * from %s""" % self.table_name)

Notice that the statement is a regular string and we use string formating % in ordert o add the table that we are searching, which was defined when we initialized the class object in the first place. Also, notice the triple quotes around the select statement: this will avoid any problems in parsing it when sending to the database, making it a string literal.

So this line executes the statement we pass to the method, but it does not actually get the data per se. We need to use another method and fetch everything returned by the select. This is done by

table_data = cursor.fetchall()

A couple of things here. The data fetched will be always (or in most cases) in unicode, when it’s a string field. And the data returned will be in a list of tuples, with the actual number of fields from the table. We know that it’s easier to work with lists than tuples, so we code something to convert types

table_data = cursor.fetchall()
raw_data = []
for i in table_data:
    raw_data.append(list(i))

self.table_data = raw_data

There are extra lines here that are not needed, and we will get rid of them in a code refactoring soon. This short function is able to connect to database, execute a SQL statement on a specified table and grab the data selected, returning a list of lists with every field and value available. We need to add a better selection statement later, and we will do as soon as we have a good structure set.

The last function in this generic class is the one that inserts data into the table.

def insert_data(self, values_list, insert_string):
    '''inserts data in the database'''

    cursor, database) = link_db()

    cursor.execute(insert_string % self.table_name, values_list)

    database.commit()
    database.close()

Identical procedure: connect, get a cursor, execute a statement. But in this case the extra step is not to get the data, but to commit the data to the table, which is done by the commit method. We will explain later how the execute method works here and what are the insert_string and values_list. Notice at the end that we need to close the connection to the database, so we know that the data has been inserted properly.

Next, we will instantiate a class from this generic one and see how easy is to manipulate the data. Stay tuned.

Previously in the series:
Part 1
Part 2

It has two class declarations, one called DB_Generic and another one called Bac. Remember in the last post where I mentioned that the idea was to have different simple tables in the same SQLite database and each table would have a simple input/output interface (If I didn’t mention that, I just did!). So, we can create a generic DB access class and we can subtype from it for every table that we will be using. In the db_obj.py file we have at the moment the generic database management class, a class derived from the generic to access the Bac database and an initialization function, that opens the access to the SQLite file. Let’s take a look at it:

def link_db():
    '''initializes the database file'''
    try:
        db = sqlite3.connect("samples.db")
    except sqlite3.Error, errmsg:
        print 'DB not available ' + str(errmsg)
        sys.exit()

    db_cursor = db.cursor()
    return db_cursor, db

In order to access a SQLite database file we need initially the name of the file. After importing sqlite3 (we’re using the latest version of SQLite here) Python has everything it needs to access, change and manipulate data in a SQLite database. Just to be sure the database file is there and we don’t get an error, we have the initialization code inside an exception. We have seen exceptions before and in this case we use it to be sure the database file has been accessed with no problems. The exception structure looks like

try:
	#here we try to do something
	#the code placed here would be executed
	#if no error reported it will go until the end and exit
	#if not, some error (exception) raised
except:
	#the code under except will be executed

So, the first step is to connect to the database file

db = sqlite3.connect("samples.db")

In our case it’s a fixed file, but the connect method receives any kind of string. It could have been some parameter obtained from the command line or a string from a configuration file. If the connect is successful, no error will be raised and we are safe to continue. We connected to database, now what? We need a cursor, an object that will actually access the data and allow us to execute SQL commands on it.

db_cursor = db.cursor()

cursor method works on the database connection object that we created previously. We’re set. This function returns the cursor and database connection objects that we created, in a tuple and this function can be called from the classes we are going to work. The classes will then have connection to the database and a cursor that would manage, select, delete and add data to it.

Next time we’ll see how our generic table class works.

Previously in the series:
Part 1

A little break from reviewing the book, let’s check some database topics in Python. I was asked to create a simple database to organize wet-lab stuff. No relationships needs, no relational tables required. Just a simple table with determined columns, and a nice GUI to go with it so people can edit, search and use.

My first idea was to use SQLite database, and I stuck with it. After the initial phase of “interviews” to check database requirements, I ended up with a list of tables and decided to start working on the table that organizes the BACs used in the lab. BAC is a DNA vector into which large DNA fragments can be inserted and cloned in a bacterial host, and are used mainly in cytogenetics around here. In the end the table had this structure

CREATE TABLE bac
(idbac INTEGER PRIMARY KEY,
clone Text,
sdate Date,
source Text,
gene TEXT,
chromosome Text,
startpos Integer,
endpos Integer,
antibiotic Text,
location1 Text,
temperature Integer,
tubes Integer,
box Integer,
cell Integer,
dnaex Boolean,
validation Boolean,
pcr Boolean,
projects Text,
comments Text,
genelink Text,
refs Text);

I won’t explain in detail each of the fields, but we can see that there is a mix of different types. SQLite doesn’t allow many different field types, so we stick to the basics.

And why SQLite? The module to access it comes with Python 2.5, the whole database is stored in one file that can be moved around and it allows a full SQL query language, which is perfect for these simple cases. So we will going to use Python, SQLite and wxPython to create a simple application to manage our simple database.

I already created a GitHub repository for the upcoming code.

The changes were mainly regarding variable/object names, but they were important as make the code available here more Pythonic following the rules of the Benevolent Dictator for Life.

I would like to thank Robin for spending his time doing this. Much appreciated.

Now, just a quick git tutorial on how to apply patches:

git apply __patch_file__
git commit -a -m “patch applied”
git push

That’s it. Apply, commit, push and you’re done. The repository is already updated.

And … we’re back. The long and cold winter is still out there and January 2009 is almost in the books. After a long period without updating I’ll try to “rush” some posts this week, trying to get back on track. So, a little bit of what’s up and coming:

- a patch provided by Robin Stocker to make all scripts published here (at least the ones on GitHub) PEP 8 compliant.

- using SQLite databases in Python

- developing an interface to access the database

- anything that you might suggest, just leave a comment.

Let’s start 2009 then.

nuin.

Thanks again and I wish an excellent holiday season and a great 2009 to everyone!

See you in 2009.

Image via Wikipedia

This close another phase in the blog and soon we will check some different aspects of Python programming in Bioinformatics.

def get_quorums(seqs, mlen):
    """
    add seq id_no to a set
    use explicit counter to create seq_no
    """
    quorum = defaultdict(int)
    for seq in seqs:
        for n in range(len(seq) - mlen):
            quorum[seq[n:n + mlen]] += 1
    return quorum

His modifications were small but improved the code a bit, as you remove one variable/object from the function. At the same time there is need to change a bit our output section of the code, as we don’t use a defaultdict initialized with a set, but with an integer.

for i in foreground:
    term1 = choose(background[i], foreground[i])
    term2 = choose((N - background[i]), len(input_seqs)-1)
    term3 = choose(N, len(input_seqs))
    p = (float(term1) * float(term2)) / term3
    if 0 < p <= 0.0001:
        print i, foreground[i], background[i], p

Notice that in the term1 line we don’t check for the set length anymore and just use the integer stored in foreground and background. Again a small change, that can make the code a little bit more clear. But we need to modify this section so the output is a little bit more clear, maybe ordered by motif sequence.

But as we are reading the sequences as they are our results are not ordered. It would be great to have a final list starting with AAAAAAAA and ending with TTTTTTTTT. There is an easy way to do that, and very inexpensive regarding code and final performance. Basically we append each one of the motifs (and their extra information) to a list and use the sort method for lists. So our output section of the code will be

res_motifs = []
for i in foreground:
    term1 = choose(background[i], foreground[i])
    term2 = choose((N - background[i]), len(input_seqs)-1)
    term3 = choose(N, len(input_seqs))
    p = (float(term1) * float(term2)) / term3
    if 0 < p <= 0.0001:
        res_motifs.append(i + '\t' + str(foreground[i]) + '\t' + str(background[i]) + '\t' + str(p))

res_motifs.sort()
for i in res_motifs:
    print i

Putting everything together our final motif determination script is (batteries included):

#!/usr/bin/env python

import fasta
import sys
from collections import defaultdict

def choose(n, k):
    if 0 <= k <= n:
        ntok = 1
        ktok = 1
        for t in xrange(1, min(k, n - k) + 1):
            ntok *= n
            ktok *= t
            n -= 1
        return ntok // ktok
    else:
        return 0

def get_quorums(seqs, mlen):
    """
    add seq id_no to a set
    use explicit counter to create seq_no
    """
    quorum = defaultdict(int)
    for seq in seqs:
        for n in range(len(seq) - mlen):
            quorum[seq[n:n + mlen]] += 1
    return quorum

input_seqs = fasta.read_seqs(open(sys.argv[1]).readlines())
input_seqs2 = fasta.read_seqs(open(sys.argv[2]).readlines())

foreground = get_quorums(input_seqs, 10)
background = get_quorums(input_seqs2, 10)

N = len(input_seqs) + len(input_seqs2)

res_motifs = []
for i in foreground:
    term1 = choose(background[i], len(foreground[i])
    term2 = choose((N - background[i]), len(input_seqs)-1)
    term3 = choose(N, len(input_seqs))
    p = (float(term1) * float(term2)) / term3
    if 0 < p <= 0.0001:
        res_motifs.append(i + '\t' + str(foreground[i]) + '\t' + str(background[i]) + '\t' + str(p))

res_motifs.sort()
for i in res_motifs:
    print i

Next we will see some basic Python methods. And maybe start a new series and phase.