Python yield paradigm

Table of Contents

• generator or iterator
• generator pipe
• coroutine
• concurrency

The yield keyword in python language is its landmark and invention, it is not just a language syntactic and feature, it lets python programming rocks in the functional programming way.

Here in this article, I try to collect all the yield magic here to surprise the python newbie.

1 generator & iterator

"yield" is the keyword that to make a generator in python, so what the generator is? Before that, I just try to explain what the Iterator is first?

An iterator is an easier and more common concept in objective oriented languages, take java, for example, basically, every container can be converted to an iterator.

List<String> namesList = new ArrayList<String>();

// add 3 names
namesList.add("moses");
namesList.add("jacobs");
namesList.add("davide");

// iterate via Iterator
Iterator<String> namesIterator = namesList.iterator();
while(namesIterator.hasNext()) {
    System.out.println(namesIterator.next());
}

// iterate via "for"
for (String temp: namesList) {
    System.out.println(temp);
}

In python, Iterator is syntactically the same thing,

class iter_obj(object):
    def __init__(self):
        pass

    def __iter__(self):
        return self

    def next(self):
        // do some thing
        if something:
            return something
        else:
            raise StopIteration()

Any object implements the magic __iter__ method and next method, is an iterator, those two methods are what the named protocol.

And by contrast, to implement a generator in python is much simpler, just use yield in a function.

The difference between Generator and Iterator in python is that the Iterator is a protocol in class level while the Generator is a functional level concept. They did the same thing to iterate though a set.

But python is much more a functional programming language, so the generator is much more powerful than Iterator, then the python's way to did iterate is to use yield function in the program instead of the Iterator class.

2 generator pipe

The most powerful apply of the generator is to combine multi generators together as a pipe in nest way.

def gen_int(n):
    for i in range(n):
        yield i

def gen_2(gen):
    for n in gen:
        if not n % 2:
            yield n

def gen_3(gen):
    for n in gen:
        if not n % 3:
            yield n

if __name__ == "__main__":
    for i in gen_3(gen_2(gen_int(10))):
        print i

3 coroutine

The coroutine is another programming pattern by the side of the generator, both of them are implemented by yield syntactic.

def consume_num():
    while True:
        n = (yield)
        print(n)


c = consume_num()
c.next()
c.send(2)
c.send(3)

Compared with a generator, the coroutine is a consumer, by contrast, the generator is a producer.

4 concurrency

While, this's the real magic here, use yield to do concurrency programming in python, you can read David Beazley's tutorial. He uses the yield coroutine combined with unblocking features like select and poll, of course by employ multi process execution poll to do unblocking concurrency programming in a magic way.

It is a little complicated, may take you a while to understand that.

Before that, you should understand coroutine and generator first, again refer David Beazley's coroutine and generator. All of his tutorials are enlightenment and impressive. And that's where I learned this python magic.