Python + PySide + logging = headache

Wait what?

I’ve been working with Python and PySide for quite a while, but only just a couple of months ago as my full-time job. As it has happened to me before, I start to think “well, I really know this thing from head to toes”, and then I realize I don’t.

As a part of the LEAP Client a friend of mine started developing a log viewer for the app. Since we were already using logging, what made most sense was to have a logging.Handler subclass that harvested all the logs and then later on display those in a nice dialog. While the dialog is open, it needs to receive new logs, and for that of course we used the mighty signals.

Here’s where it got tricky. Being good devs, we are using PySide’s new style signal handling, but when the time came to actually emit it, here’s the error message we got:

  File "blah.py", line 18, in emit
    self.new_log.emit(log_item)
TypeError: emit() takes exactly 2 arguments (3 given)

new_log is a QtCore.QSignal(dict) class member. self.new_log on the other hand gets you a SignalInstance, huh.

Looking at it quickly it’s kind of funny, using one arg in the code, it says that emit() takes 2 args and we gave it 3, WTF?.

Calling conventions

Whenever you do self.method(somearg) in Python it gets converted into something like ClassName.method(self, somearg), so there’s always one more arg than what you use, but two more args? That’s a new one.

PySide and Signals

So, PySide is great, it does some nice tricks to get you to write nicer code, but when that fails you are drowning in error messages that you cannot really figure out from the python side, there’s no pdb, inspect or traceback that can give you any more info about those 3 args given, and where they came from.

The deal with signals is this: their concept was born with C++. There, you have macros that can convert whatever you write into a string literal, so when you write:

connect(somebtn, SIGNAL(clicked()), this, SLOT(myslot()));

The SIGNAL and SLOT macros, among other tricks, convert clicked() to “clicked()” and myslot() to “myslot()”, and then Qt with its MOC take it from there. That’s why the Qt documentation intructs you to write signals’ and slots’ parameters with just their type, not a parameter name. You never see this, and you don’t need to.

When you have to do that in Python, there are no macros, so you’ve got to go through the pain of typing quotes and praying that you don’t have any typos otherwise stuff might not work and who knows why. So PySide gives you a way out of this, you can use the strings and be as close to the C++ counter part of your Python code, or you can follow a
minimal series of rules and do it the PySide-way.

These rules are:

  • Always declare signals as a part of a QObject derived class
  • Always declare signals as a class member with type QtCore.Signal

And that’s about it. The trick is that PySide knows the difference between when you say MyClass.my_signal, and self.my_signal, and those mean different things even though you haven’t defined self.my_signal anywhere.

The idea once you’ve got that, is that you use signals like this:

self.my_signal.emit(my_whatever_data)

and do things like:

self.my_signal.connect(self._my_slot)

So this is really neat when you are working with Qt’s widgets, you can do:

my_pushbutton.clicked.connect(self._btn_clicked)

The bug

Back to the bug:

  File "blah.py", line 18, in emit
    self.new_log.emit(log_item)
TypeError: emit() takes exactly 2 arguments (3 given)

But lets see some context now, here’s a small app that reproduces this error:

The idea is that we want to implement a logging.Handler that emits Qt signals, but logging.Handler makes you reimplement emit(self, logRecord), while QObject has its own emit(self, signal, data).

The first thing I thought is that we are emitting a signal before we create the QApplication and run its exec_() (we log things before that), but that wasn’t it (and looking at this from the other side it also sounds like a really stupid idea). Since we cannot gather any new info about the error, lets dive into PySide’s code.

As a part of the code generator, the function that generates the constructor for the Python module that will wrap a Qt class adds the following call:

PySide::Signal::updateSourceObject(self)

What this does is initialize all the signals that were declared as a class member for the given object being constructed (rule number 1). This initialization involves creating an object member called exactly the same as the signal but as a SignalInstance object instead of a Signal.

The SignalInstance is what implements the methods emit, connect and disconnect (but not quite so).

So when you run self.my_signal.emit(my_data), the SignalInstance code kicks in:

and converts that Python line to the following:

self.emit("my_signal(PyObject)", my_data)

self is expected to be a QObject, so it’ll implement that emit and we are all happy, BUT we have our own emit that we have to define because of the logging.Handler implementation.

So the next step for Python is to do:

Obj.emit(self, "my_signal(PyObject)", my_data)

Well, well, 3 parameters, but our implementation only takes 2 (self and logRecord), and that’s the error we’ve got.

To work around it we need to change our line from:

self.sig.emit(my_dict)

to:

QtCore.QObject.emit(self, "sig(PyObject)", my_dict)

The PyObject is needed because we used a signal that uses a dict, which is represented in Python’s source as a PyObject, if we were using data types from Qt, that would be different.

The bug itself is that PySide doesn’t use the QObject to look for the emit implementation but the current self. It’s an edge case, I guess, but with just a couple of years of using this I’ve bumped into it, so may be not that edgy.

The bug has been reported here, we’ll see if upstream agrees with me.

Component Oriented Design

Prelude

I’ve been introduced (against my will… kind of) to game programming. Games have the peculiarity that some of the problems you need to solve are really interesting the first time you tackle them. Although most or all problems have already been solved, I like to reinvent the wheel and, just like the reason why I started this blog, lets re-solve a design problem.

This is basically the post I would’ve wanted to read while learning about this, not really much new here. So if you know what the title means, may be you can just skip to the code.

The problem

The “nice thing” about the design problem I’m about to comment on is that even with small games it’s likely that you bump into it.

To illustrate the problem, lets introduce two types of sprite representation that can be found in Cocos2D: CCSprite and CCSpriteBatchNode. CCSpriteBatchNode is not technically a sprite in itself, but it represents a group of sprites that use the same texture so they are drawn “at the same time”. That way you can have thousands of copies of the same object in screen and you’ll get it almost for free. CCSprite just represents a lonely sprite. Both are CCNodes and they implement a draw method, and in terms of Cocos2D this is what we care about for this problem.

So say you start your shoot’em up game implementation, you start modeling a spaceship. You might say: Spaceship is-a CCSprite, or Spaceship has-a CCSprite (and you take care of adding the _sprite the spaceship has to the tree representation so that it’s drawn). For the sake of not going crazy while writing about this, lets just say we picked the is-a representation and I’ll comment briefly the has-a case at the end.

So you’ve got a spaceship, but wait! you’ll probably have a lot of objects in your game, so lets say your hierarchy is: Spaceship is-a GameObject, GameObject is-a CCSprite.

You have the following types of spaceships: main spaceship (the one you command), a boss, and the regular bad guys that appear frequently. So, MainSpaceship (understands user input) is-a Spaceship, RegularBadSpaceship (has some kind of way of killing) is-a Spaceship, BossSpaceship is-a RegularBadSpaceship (but harder to kill).

How frequent are RegularBadSpaceships? Lets say a lot, but BossSpaceship will appear once every level. The point here is that you would want to have the RegularBadSpaceship be represented with a CCSpriteBatchNode and CCSprite for the boss… well hello problem!

The point is that when you are way down your class hierarchy design, you find yourself needing to change a kind of root node because you have a couple of particular leafs in the tree that need a special treatment. Creating a whole branch means replicating a lot of code… but say you go this way, it’s not *that* much code. Then you realize you can use batchNodes for all the sprites, and you need to change a lot of code. Or you realize you need a third way for a sprite (may be a custom drawing node, who knows), and you end up replicate more code. It will work, but this is not the way to go.

A solution

A possible solution, that I found is *THE* solution for this is a component oriented design. Every functionality is a component and a whole game object is a set of components. The component wrapper doesn’t have functionality in itself, it is what its components implement.

So, for our problem, we would have two components for displaying an object. One would use CCSprite, the other one a CCSpriteBatchNode. Both nodes would be of the same kind, so they would have the same API and you can exchange the way an object is displayed by just changing the components. Everything else will be transparent.

While designing a component, you would have dependencies between components, some optional and some not.

When handling objects, you won’t care about the actual class for the object, but wether it implements (or has-a or is-a) a component or not. “Can this thing shoot? Then shoot”, “Is this displayable? Then display”, and so on.

Nice, right? If so much text wasn’t clear enough, lets see some code.

Lets tell it in Python

Component oriented design is really nice to implement in a dynamic language like Python, where you can edit everything at runtime.

You could create a method implements() that returns a bool, and have a set with component ids (or may be a class object) and check for a component belonging to that set to resolve wether it has a component.

It may not be the best Pythonic implementation, but you get the idea.

But what about C++?

A more tricky implementation (but not so much) is in a statically typed language like C++.

Design patterns?

Wait, isn’t this just a Mixin pattern? Yes, it depends on what you want to accomplish, what language you use and how much you want to write.

There are always so many ways of doing the same.

Alternatives

So, an alternative to this inheritance madness (imagine an object with 20 components), is having a couple of helper methods or templates, a nice hash data structure and having the components as a has-a relationship. It will have more overhead, but how much is “more”? Not much in my little experience, in a language like C++, the right data structures and not a zillion components and you’re good to go. The overhead will still lie in the same old bad guys: rendering and reacting to input.

This bits of knowledge I gained throw game design headaches were a really nice lesson on framework or library designing too, or anything that may have complex inheritance trees with a lot objects that are like others but not quite.

Personally, when the time came to put this into use, I went with the alternative approach instead of the inheritance madness and the bigger the game got, the more I loved just having to deal with components and how flexible it all was.

Golang my C++ – Defer

A Hives night

Go!

A month ago I started learning Go, since I can’t learn by doing a ton of simple samples I started implementing a physically based renderer (may be more on this later). While reading about the language features there were a couple that sounded really great. The main one being defer calls.

Defer

The idea behind defer is simple: “execute X at the end of this method”. So wherever the method “ends” you can be sure X will be executed. This sounds simple, but if you have a complex method with multiple return statements, things can get tricky.

Lets illustrate this with an example from [0]:

“Awesome” is the word you are looking for.

In Python’s words

Python has something somewhat similar, which is the with statement. You can read about it in many places, for example [1].

Yeah, it’s far from a defer, but you have a really nice way of creating controlled execution in terms of keeping the state as you want it without much trouble. Either you use the Python implemented classes, like file (with open(blah) as f:), or you can create your own object with __enter__ and __exit__ implemented and you can with’ it all you want.

Show me that C++

Actually, these tricks can be done in Python too, but given the handling of scopes, it might get tricky. So I’ll just keep this C++ for now.

So, I read a couple of days ago about a trick like defer with a std::shared_ptr with a lambda deallocator which acts as a defer call when the object gets deleted at the end of its scope. It might sound nice, and under certain circumstances you might even say it’s reasonable… may be. But I don’t like it very much. You know, this may be one of those “tabs vs spaces for indentation” discussions.

Later on, I read about another implementation that did basically the same as the shared_ptr one but in a more correct way, if I may call it that way. Basically you create an object with a lambda as a parameter, and that gets called in the object’s destructor at the end of its scope. We are almost there.

There is at least one issue here, one of the nice things about Go’s defer is that you can do this:

and it’ll print 3 2 1 0, because it’ll execute it in LIFO order. But with C++ you cannot do that, because if you declare an object inside a block {}, it’ll get deleted at the end of it, which would be at the end of each loop. So we need some more C++ sugar to try to emulate Go’s defer to the last detail.

Implementation

Why don’t we skip all these words and see some code? Ok.

Something worth mentioning regarding this implementation is that you need to run allow_deferred() after all the objects you want to defer a call, otherwise you might get into a problem. A solution would be to use std::shared_ptr, but that depends on the method you are working on.

Other than that, it would be nice to avoid macros but given the approach I’ve taken, it is not possible. A solution would be to have a Deferrer singleton that handles “temporal marks” so you mark it when you run allow_deferred() with the current time and the deferrer will execute the methods until a mark is found. This has two pitfalls, the first is that you’ll need to also call a end_deferrer() at the end of the method, which would defeat the purpose of handling multiple return methods (and obviously forgetting about calling something at the end). And the other problem is that you will need one deferrer per thread (or something equivalent) if you are running in a multithreaded app. So, all in all, I think this approach is a good compromise.

Oh yeah, here’s the output:
3
2
1
0
Inside A::f
Finishing A…
i = 42

Is this a complete solution?

According to [0], there a couple of things that defer does:

A deferred function’s arguments are evaluated when the defer statement is evaluated

This is not true if you bind a parameter, but it is if you use a lambda. So, we have more flexibility but we have to be carefull, as usual.

Deferred function calls are executed in Last In First Out order after the surrounding function

This is true, as we’ve seen from the example. Not much more to say.

Deferred functions may read and assign to the returning function’s named return values

Kind of true, C++ doesn’t have named return values, but anything passed as reference (or a pointer for that matter) can be modifyable if treated properly in the deferred call (lambda with capture by reference, or copy if handling std::shared_ptr).

References

[0] http://blog.golang.org/2010/08/defer-panic-and-recover.html
[1] http://effbot.org/zone/python-with-statement.htm

Objective-C my C++ – Blocks

Some view

The Objective-C side

A really handy Objective-C feature that came to my attention not so long ago are blocks. Now I’m not going to write much about blocks themselves, for that there are much better docs. But here’s the basic idea:
Blocks are anonymous functions that retain the environment where they are created and abstract functionality. Depending on how that environment is presented, the block can modify it or not. Simple, right? Not exactly, but lets see an example:

A key word from before is “retain”. Objective-C uses retain count (or use count, if you are more C++) and that concept is embeded in blocks. That’s something we should keep in mind.

C++0x lambdas

C++’s latest additions gives us lambdas, which are the closest concept to blocks so far. The main difference resides in how it handles its environment.  As with everything in C++, you don’t have too many restrictions and you control a lot, and in this case you control how your lambda handles its environment. Lets see a simple example:

As a quick reference, the extract [=, &modifiable] means “treat modifiable as a reference and capture as a copy everything else”. Which is equivalent in this tiny example to [nonModifiable, &modifiable].
Everything is beautiful, but with a lot of flexibility comes a lot of things to think about so the building doesn’t fall to pieces after a few loops.

Limitations of lambdas

Lambdas don’t have limitations per-sé, but depending on what you want to use them for, there may be a couple of problems with them. So, where can we use lambdas? Completion handlers, error handlers, event handlers. If you want to do really useful things with these, you will probably want to call some method of an object for example, but who handles the life of that object? Is it just a pointer? A reference? Probably not a copy. There’s an easy way of handling this though.

Final thoughts

I’m trying to embrace the new standard as much as possible lately, and the new features come to the rescue, as usual. This time it’s called std::shared_ptr.
The basic idea is to capture by copy everything that you need. Use mostly shared_ptrs, and everything will be ok.
So why am I so worried about the lifetime of objects? With every type of deferred call, it’s something you always have to have in mind, but with game programming particularly, when one of your objects might literally die because it’s may be a bomb that has just went off, or an enemy spaceship that you just eliminated, you really need to be careful. The point in this kind of scenarios is probably check that the object the shared_ptr is wrapping is in use by something else, otherwise what’s the point of modifying something only the lambda has ownership for?
And here’s some code with the help of Cocos2d-X:

Once the lambda is created, the use_count() for the gameObject is increased in 1, and as long as the lambda exists the gameObject will be retained. So this toy scenario might have a bug, where gameObject is never released completely. So this should be handled carefully, but we can just remove the menu after the gameObject explodes. In general, Cocos2d-X is not prepared for this kind of uses yet, especially because they do their own reference counting in an explicit way (not like shared_ptr).

References

[0] Cocos2d-X http://cocos2d-x.org/
[1] Mike Ash on blocks vs lambdas http://www.mikeash.com/pyblog/friday-qa-2011-06-03-objective-c-blocks-vs-c0x-lambdas-fight.html
[2] Apple documentation for blocks http://developer.apple.com/library/ios/#featuredarticles/Short_Practical_Guide_Blocks/_index.html
[3] std::shared_ptr http://en.cppreference.com/w/cpp/memory/shared_ptr