unjoinify

8/4/2019 unjoinify

1/36

unjoinify: a module to tame the SQLbeast

Matt WestcottDjangoCon Europe, 7th June 2011

8/4/2019 unjoinify

2/36

The n+1 query problem

8/4/2019 unjoinify

3/36


Eager loading

8/4/2019 unjoinify

4/36


Eager loading

"that thing that really bugs meabout select_related"

8/4/2019 unjoinify

5/36

8/4/2019 unjoinify

6/36

Festival

8/4/2019 unjoinify

7/36

Festival Award

8/4/2019 unjoinify

8/36

Festival Award Nomination

8/4/2019 unjoinify

9/36

Festival Award Nomination Movie

8/4/2019 unjoinify

10/36

Festival Award Nomination Movie Person (director)

8/4/2019 unjoinify

11/36


Company

8/4/2019 unjoinify

12/36

def show(request, festival_id):festival = get_object_or_404(Festival, id = festival_id)return render(request, 'festivals/show.html', {

'festival': festival} )

{{ festival.name }}

{% for award in festival.awards %}{{ award }}

{% for nomination in award.nominations %}

{{ nomination.movie.name }}(...)

{% endfor %}

{% endfor %}

8/4/2019 unjoinify

13/36

def show(request, festival_id):festival = get_object_or_404(Festival, id = festival_id,

select_related('award__nomination__movie'))return render(request, 'festivals/show.html', {

'festival': festival} )

{{ festival.name }}

{% for award in festival.awards %}{{ award }}


{{ nomination.movie.name }}(...)

{% endfor %}

{% endfor %}

8/4/2019 unjoinify

14/36

8/4/2019 unjoinify

15/36

8/4/2019 unjoinify

16/36

people = Person.objects.filter(movies_directed__nominations__award__festival_id =

festival_id)

{% regroup people by award as award_list %}{% for award in award_list %}{{ award.grouper }}

{% regroup award.list by movie as movie_list %}{% for movie in movie_list %}

{{ movie.grouper }}

{% endfor %}{% endfor %}

8/4/2019 unjoinify

17/36

SELECTimdb_award.id,imdb_award.name,imdb_nomination.id AS nomination__id,imdb_nomination.ranking AS nomination__ranking,imdb_movie.id AS nomination__movie__id,imdb_movie.title AS nomination__movie__title,imdb_person.id AS nomination__movie__directors__id,imdb_person.first_name AS nomination__movie__directors__first_nameimdb_person.surname AS nomination__movie__directors__surname

FROM

imdb_awardLEFT JOIN imdb_nomination ON (imdb_award.id = imdb_nomination.award_id)LEFT JOIN imdb_movie ON (imdb_nomination.movie_id = imdb_movie.id)LEFT JOIN imdb_movie_directors ON (imdb_movie.id =

imdb_movie_directors.movie_id)LEFT JOIN imdb_person ON (imdb_movie_directors.person_id =

imdb_person.id)

WHEREimdb_award.festival_id = ?

ORDER BYimdb_award.name,imdb_nomination.ranking

8/4/2019 unjoinify

18/36

id | name | nomination_id | rank | movie_id | movie_title | director_id | director_name---+---------------+---------------+------+----------+-------------------+-------------+---------------2 | Best Director | 4 | 1 | 1 | The King's Speech | 1 | Iain Canning2 | Best Director | 4 | 1 | 1 | The King's Speech | 2 | Emile Sherman2 | Best Director | 4 | 1 | 1 | The King's Speech | 3 | Gareth Unwin2 | Best Director | 5 | 2 | 2 | 127 Hours | 4 | Danny Boyle2 | Best Director | 5 | 2 | 2 | 127 Hours | 5 | Christian Colson2 | Best Director | 6 | 3 | 4 | True Grit | 10 | Joel Coen2 | Best Director | 6 | 3 | 4 | True Grit | 9 | Ethan Coen2 | Best Director | 6 | 3 | 4 | True Grit | 11 | Scott Rudin1 | Best Picture | 1 | 1 | 1 | The King's Speech | 1 | Iain Canning1 | Best Picture | 1 | 1 | 1 | The King's Speech | 2 | Emile Sherman1 | Best Picture | 1 | 1 | 1 | The King's Speech | 3 | Gareth Unwin1 | Best Picture | 2 | 2 | 2 | 127 Hours | 4 | Danny Boyle1 | Best Picture | 2 | 2 | 2 | 127 Hours | 5 | Christian Colson1 | Best Picture | 3 | 3 | 3 | Black Swan | 6 | Scott Franklin

1 | Best Picture | 3 | 3 | 3 | Black Swan | 7 | Mike Medavoy1 | Best Picture | 3 | 3 | 3 | Black Swan | 8 | Brian Oliver

8/4/2019 unjoinify

19/36

id | name | nomination_id | rank | movie_id | movie_title | director_id | director_name---+---------------+---------------+------+----------+-------------------+-------------+---------------2 | Best Director | 4 | 1 | 1 | The King's Speech | 1 | Iain Canning2 | Best Director | 4 | 1 | 1 | The King's Speech | 2 | Emile Sherman2 | Best Director | 4 | 1 | 1 | The King's Speech | 3 | Gareth Unwin2 | Best Director | 5 | 2 | 2 | 127 Hours | 4 | Danny Boyle2 | Best Director | 5 | 2 | 2 | 127 Hours | 5 | Christian Colson

2 | Best Director | 6 | 3 | 4 | True Grit | 10 | Joel Coen2 | Best Director | 6 | 3 | 4 | True Grit | 9 | Ethan Coen2 | Best Director | 6 | 3 | 4 | True Grit | 11 | Scott Rudin1 | Best Picture | 1 | 1 | 1 | The King's Speech | 1 | Iain Canning1 | Best Picture | 1 | 1 | 1 | The King's Speech | 2 | Emile Sherman1 | Best Picture | 1 | 1 | 1 | The King's Speech | 3 | Gareth Unwin1 | Best Picture | 2 | 2 | 2 | 127 Hours | 4 | Danny Boyle1 | Best Picture | 2 | 2 | 2 | 127 Hours | 5 | Christian Colson1 | Best Picture | 3 | 3 | 3 | Black Swan | 6 | Scott Franklin

1 | Best Picture | 3 | 3 | 3 | Black Swan | 7 | Mike Medavoy1 | Best Picture | 3 | 3 | 3 | Black Swan | 8 | Brian Oliver

{{ movie }}

8/4/2019 unjoinify

20/36

my_results = cursor.execute('''SELECT id, name, nomination.id AS nomination__id,FROM ...

WHERE ...''')

awards = do_some_magic(my_results);

return render(request, 'festivals/show.html', {

'festival': festival,'awards': awards,

})

8/4/2019 unjoinify

21/36

my_results = cursor.execute('''SELECT id, name, nomination.id AS nomination__id,FROM ...

WHERE ...''')

awards = do_some_magic(my_results);

return render(request, 'festivals/show.html', {

'festival': festival,'awards': awards,

})

django-unjoinify:

it isn't magic.

8/4/2019 unjoinify

22/36

from unjoinify import unjoinifyquery = '''

SELECT

imdb_award.id,imdb_award.name,imdb_nomination.id AS nomination__id,imdb_nomination.ranking AS nomination__ranking,imdb_movie.id AS nomination__movie__id,imdb_movie.title AS nomination__movie__title,

FROMimdb_awardLEFT JOIN imdb_nomination ON (imdb_award.id =

imdb_nomination.award_id)LEFT JOIN imdb_movie ON (imdb_nomination.movie_id =

imdb_movie.id)WHERE imdb_award.festival_id = %s'''

awards = unjoinify(Award, query, (festival.id,))

8/4/2019 unjoinify

23/36

idnamenomination__id

nomination__rankingnomination__movie__idnomination__movie__titlenomination__movie__directors__idnomination__movie__directors__first_name

nomination__movie__directors__surnamenomination__movie__production_companies__idnomination__movie__production_companies__name

8/4/2019 unjoinify

24/36

idnamenomination__id

nomination__rankingnomination__movie__idnomination__movie__titlenomination__movie__directors__idnomination__movie__directors__first_name

nomination__movie__directors__surnamenomination__movie__production_companies__idnomination__movie__production_companies__name

awards = (award, [nominations])where:

nominations = (nomination, movie, directors, companies)

{% for award in awards %}

8/4/2019 unjoinify

25/36

{% for award in awards %}{{ award.name }}


{{ nomination.movie.name }}

{% for director in nomination.movie.directors %}{{ director.first_name }}{{ director.surname }}

{% endfor %}

{% for company in nomination.movie.companies %}{{ company.name }}

{% endfor %}

{% endfor %}

{% endfor %}

{% for award nominations in awards %}

8/4/2019 unjoinify

26/36

{% for award, nominations in awards %}{{ award.name }}

{% for nomination, movie, directors, companies in

nominations %}{{ movie.name }}

{% for director in directors %}{{ director.first_name }}

{{ director.surname }}{% endfor %}

{% for company in companies %}

{{ company.name }}{% endfor %}

{% endfor %}

{% endfor %}

8/4/2019 unjoinify

27/36

make_unpack_plan(Movie, columns, prefix = 'nomination__movie__')

id

namenomination__idnomination__rankingnomination__movie__idnomination__movie__title

nomination__movie__directors__idnomination__movie__directors__first_namenomination__movie__directors__surnamenomination__movie__production_companies__idnomination__movie__production_companies__name

8/4/2019 unjoinify

28/36

make_unpack_plan(Movie, columns, prefix = 'nomination__movie__')

id

namenomination__idnomination__rankingnomination__movie__idnomination__movie__title

nomination__movie__directors__idnomination__movie__directors__first_namenomination__movie__directors__surnamenomination__movie__production_companies__idnomination__movie__production_companies__name

=> ({model: Movie, fields: ('id','title')},[{model: Person, fields: ('id','first_name','surname')}],[{model: Company, fields: ('id','name')}],

)

8/4/2019 unjoinify

29/36

id | name | nomination_id | rank | movie_id | movie_title | director_id | director_name

---+---------------+---------------+------+----------+-------------------+-------------+---------------2 | Best Director | 4 | 1 | 1 | The King's Speech | 1 | Iain Canning2 | Best Director | 4 | 1 | 1 | The King's Speech | 2 | Emile Sherman2 | Best Director | 4 | 1 | 1 | The King's Speech | 3 | Gareth Unwin2 | Best Director | 5 | 2 | 2 | 127 Hours | 4 | Danny Boyle2 | Best Director | 5 | 2 | 2 | 127 Hours | 5 | Christian Colson2 | Best Director | 6 | 3 | 4 | True Grit | 10 | Joel Coen2 | Best Director | 6 | 3 | 4 | True Grit | 9 | Ethan Coen2 | Best Director | 6 | 3 | 4 | True Grit | 11 | Scott Rudin

1 | Best Picture | 1 | 1 | 1 | The King's Speech | 1 | Iain Canning1 | Best Picture | 1 | 1 | 1 | The King's Speech | 2 | Emile Sherman1 | Best Picture | 1 | 1 | 1 | The King's Speech | 3 | Gareth Unwin1 | Best Picture | 2 | 2 | 2 | 127 Hours | 4 | Danny Boyle1 | Best Picture | 2 | 2 | 2 | 127 Hours | 5 | Christian Colson1 | Best Picture | 3 | 3 | 3 | Black Swan | 6 | Scott Franklin1 | Best Picture | 3 | 3 | 3 | Black Swan | 7 | Mike Medavoy1 | Best Picture | 3 | 3 | 3 | Black Swan | 8 | Brian Oliver

unpack_with_plan(plan, results)

8/4/2019 unjoinify

30/36

id | name | nomination_id | rank | movie_id | movie_title | director_id | director_name

---+---------------+---------------+------+----------+-------------------+-------------+---------------2 | Best Director | 4 | 1 | 1 | The King's Speech | 1 | Iain Canning2 | Best Director | 4 | 1 | 1 | The King's Speech | 2 | Emile Sherman2 | Best Director | 4 | 1 | 1 | The King's Speech | 3 | Gareth Unwin

2 | Best Director | 5 | 2 | 2 | 127 Hours | 4 | Danny Boyle2 | Best Director | 5 | 2 | 2 | 127 Hours | 5 | Christian Colson

2 | Best Director | 6 | 3 | 4 | True Grit | 10 | Joel Coen

2 | Best Director | 6 | 3 | 4 | True Grit | 9 | Ethan Coen2 | Best Director | 6 | 3 | 4 | True Grit | 11 | Scott Rudin

1 | Best Picture | 1 | 1 | 1 | The King's Speech | 1 | Iain Canning1 | Best Picture | 1 | 1 | 1 | The King's Speech | 2 | Emile Sherman1 | Best Picture | 1 | 1 | 1 | The King's Speech | 3 | Gareth Unwin

1 | Best Picture | 2 | 2 | 2 | 127 Hours | 4 | Danny Boyle1 | Best Picture | 2 | 2 | 2 | 127 Hours | 5 | Christian Colson

1 | Best Picture | 3 | 3 | 3 | Black Swan | 6 | Scott Franklin1 | Best Picture | 3 | 3 | 3 | Black Swan | 7 | Mike Medavoy1 | Best Picture | 3 | 3 | 3 | Black Swan | 8 | Brian Oliver

unpack_with_plan(plan, results)

8/4/2019 unjoinify

31/36


Company

beware the cartesian join

8/4/2019 unjoinify

32/36

a_very_long_column_alias_name_that_exceeds_the_limi

8/4/2019 unjoinify

33/36

a_very_long_column_alias_name_that_exceeds_the_limi

unjoinify(model, query, params, column_names)

8/4/2019 unjoinify

34/36

the masterplan

1. Get select_related to auto-generate the SQL

2. Reconstruct models from the resultset

3. 'Push' child objects into their respectiveRelatedManagers

4. Profit!

8/4/2019 unjoinify

35/36

github.com/simonw/django-queryset-transform

8/4/2019 unjoinify

36/36

https://github.com/gasman/django-unjoinify

pip install django-unjoinify

Thank you!Matt Westcott

http://matt.west.co.tt/@westdotcodottt

Documents

unjoinify