Wiki Navigation

Page History

Warning: this tutorial is still under construction and currently contains some errors

Refer to the discussion on the Elixir Google Group while the proper method is being worked out: http://groups.google.com/group/sqlelixir/browse_thread/thread/b725250f0a223c99

Split a Turbogears Model Across Multiple Files -- Proper Technique and Pitfalls

While creating a Turbogears project with Elixir/SQLAlechemy, I ran into some problems splitting a model across multiple files. Not being able to find any good documentation on this, I finally figured out the 'right way' and decided to document common pitfalls I ran into.

Intended Audience

Large Project: The intended audience of this document is someone using Elixir/SQLAlchemy to create a Turbogears project large enough to necessitate splitting the model.py file across multiple model files.
Prerequisite: The reader should have read the Elixir Diving In tutorial and be familiar with how Elixir works at a basic level.
Python Experience: Common pitfalls involve Python or Elixir subtleties, so this will be most useful to relatively newer Pythonistas or people not very experienced with Elixir. The pitfalls here should be pretty obvious to more experienced programmers.
Turbogears vs. Pylons: Concepts might also apply to Pylons projects, but be aware this tutorial is written with a Turbogears project in mind.

Project and Environment Overview

First an overview of what the project looks like. We're going to create a project with two Entities -- a Man and a Dog. Dogs can be pets of a Man (a ManyToOne relationship).

The directory tree (of importance to this tutorial -- other Turbogears files omitted) is going to look like this:

+ tg_project
  | 
  + create_tables_script.py
  + model.py
  + submodels
    |
    + animals.py
    + humans.py

Our Man model is going into humans.py and our Dog model is going into animals.py. model.py will essentially be a wrapper for submodels. create_tables_script.py is a script that just creates your tables and is designed to be run externally from your start-tg_project.py file.

The Right Way of Doing It

This lists 'the right way' of splitting your model up into other files.

submodels/humans.py:

from elixir import using_options, Entity, Field, Unicode, OneToMany

class Man(Entity):
    using_options(shortnames=True, tablename="my_man_table")
    
    name = Field(Unicode(10))
    pets = OneToMany('Dog')
    
    def __repr__(self):
        return '<Man \'%s\'>' % (self.name)

submodels/animals.py:

from elixir import using_options, Entity, Field, Unicode, ManyToOne

class Dog(Entity):
    using_options(shortnames=True, tablename="my_dog_table")
    
    name = Field(Unicode(10))
    owner = ManyToOne('Man')
    
    def __repr__(self):
        return '<Dog \'%s\'>' % (self.name)

As you can see, the Man and Dog entities are very straightforward. The using_options line is included to show that changing names of tables doesn't have any impact.

model.py:

from elixir         import setup_all


from submodels.animals import *
from submodels.humans  import *


setup_all()

model.py is simply a wrapper for including the submodels. Note the way we are importing the submodels -- this will be key to doing it 'the right way'. Also note how we setup_all() after we have imported all the submodels and how we don't setup_all() inside any of the submodels.

create_tables_script.py:

from elixir                 import create_all
from turbogears.database    import metadata, session

metadata.bind = "mysql://user:supersecretpw@localhost:3306/mydb"
metadata.bind.echo = True

from model import *

create_all()

Note how we first create the metadata object (using Turbogears' metadata wrapper) and then we import the model. Only after the model has been imported do we create_all().

This concludes the 'proper' way of doing it. You should be able to use this pattern to extend your model is whatever way necessary. Next up: subtle mistakes that will start giving you weird errors.

Common Mistakes

Importing / Mapper Subtleties: AttributeError

This section should be obvious to an experienced Pythonista, but as a relatively new Pythonista, the major mistakes I was encountering involved how I was importing files. It is important that when importing your model and submodels, you do it in the form: from xxx import *

For example, if your model.py file used the lines:

from submodels import animals
from submodels import humans

then your create_tables_script.py would create the file fine, but anything that would try to use the models would not work as as one would (naively) expect. For example, say that your model.py file looked as above and you had a file called do_stuff.py:

from turbogears.database    import metadata, session
metadata.bind = "mysql://user:supersecretpw@localhost:3306/mydb"

from model import *

fido = Dog(name=u"Fido")
dan = Man(name=u"Dan")
dan.pets.append(fido)
session.flush()

You would get the error NameError: name 'Dog' is not defined. Of course, the proper way of doing it would be do call animals.Dog and humans.Man instead of Dog and Man, respectively. Say, however, that instead, your thought process was "Well, lets just import those missing classes!" and you changed your do_stuff.py file to:

from turbogears.database    import metadata, session
metadata.bind = "mysql://user:supersecretpw@localhost:3306/mydb"

from model import *

from submodels.animals import Dog  #added this line
from submodels.humans  import Man  #added this line

fido = Dog(name=u"Fido")
dan = Man(name=u"Dan")
dan.pets.append(fido)
session.flush()

The error you get now is:

Traceback (most recent call last):
  File "create_tables_script.py", line 16, in <module>
    dan.pets.append(fido)
AttributeError: 'Man' object has no attribute 'pets'

because the Dog and Man classes you imported were imported outside the scope of the setup_all() in model.py and thus are not associated with the proper metadata.

The lesson is this:

Never import submodels directly; always import using the top-level wrapper model.py, which has the setup_all() mapper function. (read the next error to see why it is a bad idea to have a setup_all() line in your submodels.)

setup_all() Subtleties: KeyError

Another common error recieved is a KeyError along the lines of:

Traceback (most recent call last):
  File "create_tables_script.py", line 9, in <module>
    import model
  File "model.py", line 8, in <module>
    setup_all()
  File "__init__.py", line 117, in setup_all
    
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/entity.py", line 754, in setup_entities
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/entity.py", line 228, in setup_relkeys
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/entity.py", line 438, in call_builders
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/relationships.py", line 373, in create_non_pk_cols
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/relationships.py", line 523, in create_keys
  File "/Python25/Lib/site-packages/Elixir-0.5.1-py2.5.egg/elixir/relationships.py", line 434, in target
KeyError: 'Man'

This is caused by calling setup_all() prematurely.

setup_all() must be called after all dependent Entities have been imported'''

The above error, for example, is called by putting a setup_all() at the end of your animals.py model file and is triggered by the owner = ManyToOne('Man') line. With regards to OneToOne, OneToMany, ManyToOne, and ManyToMany relationships, Elixir uses the class name as a string so that it can resolve circular load dependencies at run-time, after all the classes have been imported into the namespace. When splitting your model across multiple files, it is tempting to put a setup_all() at the end of each individual model file, especially if debugging large model files one at a time. Don't. If you get a KeyError, check for stray setup_all()'s.

Download in other formats:

Plain Text