Lecture#4: Advanced Python

7/29/2019 Lecture#4: Advanced Python

1/35

High performance computing

for non-programmers

Lecture #4:Python programminglanguage: Advanced

Glib Ivashkevych

junior researcher, NSC KIPT


2/35

List comprehensions

rationale

newlist = []

for elem in oldlist:

newlist.append(somefunc(elem))

Very common pattern

Special syntax for this?


3/35

List comprehensions

syntax

newlist = []

for elem in oldlist:

newlist.append(somefunc(elem))

is equivalent to:

newlist = [somefunc(elem) for elem in oldlist]


4/35

List comprehensions

syntax

zlist = []

for x in xlist:

for y in ylist: zlist.append(func(x, y))

is equivalent to:

zlist = [func(x,y) for x in xlist for y in ylist]


5/35

List comprehensions

syntax

zlist = []

for x in xlist:

for y in ylist:if cond(x,y):

zlist.append(f(x, y))

is equivalent to:

zlist = [f(x,y) for x in xlist for y in ylist

if cond(x,y)]


6/35

Generators

rationale

for elem in somelist:

do_something(elem)

What ifsomelist is huge?

Large memory footprint for storing somelist


7/35

Generators

yield instruction

def huge_generator(num):

for i in xrange(num):

yield someexpr(i)

some_generator = huge_generator(huge_number)

for elem in some_generator:

do_something(elem)

A lot better now. Generators know how to get

next element (some_generator.next())


8/35

Generators

expression syntax

def huge_generator(num):

for i in xrange(num):

yield f(i)

sgen = huge_generator(huge_number)

is equivalent to:

sgen = (f(i) for i in xrange(huge_number))


9/35

Iterators

behind the scenes

for elem in somelist:

...for elem in sometuple:

...

for elem in someiterable:

...

Some pattern exists here, yeap?


10/35

Iterators

behind the scenes

for calls __iter()__ on container object

__iter()__ returns iterator object

iterator knows how to get next element

and where to stop(raises StopIteration exception, when there are no more elements)


11/35

Iterators

how to createclass CustomIterable(object):

def __init__(self, *args, **kwargs):

#initialize your object heredef __iter__(self):

return self

def next(self):

#return some value or raise StopIteration

CustomIterablecan now be used in loops


12/35

Magic methods

what about this __ methods?Reserved methods with special meaning

for el in lst:

uses

LstClass.__iter__(lst)

obj = SomeClass(args)

uses

SomeClass.__new__(SomeClass, args) and

SomeClass.__init__(obj, args)


13/35

Magic methods

dirty secrets of constructors__init__is not a constructor (its initializer)

although its usually to treat it as suchexcept when subclassing immutables

__new__is an actual constructor

you need it when subclassing immutables


14/35

Magic methods

be polite__str__is used to emulate str() built-in

function

__unicode__is used to emulate unicode()

built-in function

__repr__is used to get a string representation

of object (also `obj` and repr() built-in)


15/35

Magic methods

be a container__len__is used to emulate len()

len(cont) -> cont.__len__()

or ContClass.__len__(cont)

__getitem__is used to emulate obj[key]

obj[key] -> obj.__getitem__(key)

or ObjClass.__getitem__(obj, key)


16/35

Magic methods

be a container__setitem__is used to emulate obj[key]

=value

obj[key] = value ->

obj.__setitem__(key, value)

or ObjClass.__getitem__(obj, key, value)


17/35

Magic methods

be a container

__delitem__(self, key): del obj[key]__iter__(self): iter(obj) or in loops

__reversed__(self): reversed(obj)

__contains__(self, item): item in obj


18/35

Magic methods

be smart__setattr__(self, name, value)

Intercept assignment to attribute nameBe aware of infinite recursion!

__getattribute__(self, name)

__getattr__(self, name)

Intercept access to attribute name


19/35

Magic methods

be callable__call__(self, ...)

Intercept call on instanceclass SomeClass(object):

def__call__(self, *args, **kwargs):

do_something

s = SomeClass()

s(1,2,3)


20/35

Magic methods

all together

Magic methods in Python allow you to create classes,

which look and behave like built-in types

See this blog post for general overview

http://www.rafekettler.com/magicmethods.html
http://www.rafekettler.com/magicmethods.htmlhttp://www.rafekettler.com/magicmethods.html


21/35

Decorators

rationale

Add functionality without changing the code

Examples:

logging, timing, tests


22/35

Decorators

rationaledef wrapper(f):

def inner(*args):print Gonna call %s % f.func_name

f(*args)

return inner

def func(*args):

do_something

func = wrapper(func) #additional functionality


23/35

Decorators

rationaledef wrapper(f):

def inner(*args):print Gonna call %s % f.func_name

f(*args)

return inner

@wrapper

def func(*args):

do_something


24/35

Decorators

rationaleAnd now magic happens: wrapper can be used to decorateany function. Code reuse, good design - PROFIT!

...

@wrapper

def f1(*args):

do_something

@wrapper

def f2(*args):

do_something_else

...


25/35

Decorators

parametrizedWhat if your decorator depends on some value? Thats ok.

def dec(a):

def wrapper(f): This is actual decorator

def inner(*args):

print Func is %s, param is %s % (f.func_name,str(a))

f(*args)

return inner

return wrapper

@dec(Some string parameter)

def func(*args):

...


26/35

Decorators

leveraging __call__class MyDecorator(object):

def __init__(self, f):

self.f = f

def __call__(self, *args, **kwargs):

print Starting %s call %s self.f.func_name

self.f(*args, **kwargs)

print Finished %s %s self.f.func_name@MyDecorator

def func(*args, **kwargs):

...

Cool. Really.


27/35

Metaprogramming

code that creates codeclass MyClass(object):

def __init__(self, data):

self.data = data Something interesting

...

...

Classes and instances can beextended at runtime


28/35

Metaprogramming

code that creates code>>> MyClass.new_method = lambda self: self.data * 2

>>> m = MyClass(5.)

>>> m.new_method()

10.

Classes and instances can beextended at runtime


29/35

Metaprogramming

rationaleclass IsothermalIdealGas(object):

pressure = 1.

temp = 1.

volume = 1.

quant = 1.

Not cool. Really. PV = nT? Never mind.(forget about R, this is computer science class)

Developer ofIsothermalIdealGasnever heard

about equation of state. Lets patch it.


30/35

Metaprogramming

templating:jinja2from jinja2 import Template

tpl = Template(def custom_setattr(self, name, value):

if name == 'P':

object.__setattr__(self, name, value)

object.__setattr__(self, 'V', {{V_eq}})

elif name == 'V':

object.__setattr__(self, name, value)

object.__setattr__(self, 'P', {{P_eq}}))v_eq = Template(self.{{T}}*self.{{Q}} / self.{{V}}**{{k}})

p_eq = Template((self.{{T}}*self.{{Q}} / self.{{P}})**(1./

{{k}}))

keys = {T:T, P:P, V:V, Q:Q, k:2}

source = tpl.render(V_eq=v_eq.render(...), P_eq=p_eq.render(...))


31/35

Metaprogramming

templating:jinja2Template variables

{{var}},{{var.someattr}}, {{var[key]}}

for block{% for item in iterable %}

...

{% endfor %}

if block{% if var%}

...

{% endif %}


32/35

Metaprogramming

templating:jinja2from jinja2 import Template

tpl = Template({{foo}} - {{bar}})rtpl = tpl.render(foo=First,bar=Second)

or

rtpl = tpl.render({foo:First,bar:Second})

Ok, now we can generate code at runtime.

Usually, for C/CUDA code.


33/35

Metaprogramming

warningNever,

ever-trust other peoples data when using

it to generate executable code.Always assume the worst. They all

want to crash your code. Always.

(google it, yeah. something like python eval dangerous)


34/35

Python

With great power comes greatresponsibility.- Uncle Ben from SpiderMan

or, for those who prefer classic:

Un grand pouvoir implique degrandes responsabilits.

- Franois-Marie Arouet aka Voltaire
http://conjugaison.lemonde.fr/conjugaison/troisieme-groupe/pouvoirhttp://conjugaison.lemonde.fr/conjugaison/troisieme-groupe/pouvoir


35/35

Questions?

Documents

Lecture#4: Advanced Python