Easy contributable internationalization process with Sphinx @ pyconmy2015

Easy contributable internationalization process with Sphinx

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp

1

こんにちはKon-Nichi-Wa

2

Who am I

@shimizukawa

1. Sphinx co-maintainer

2. Sphinx-users.jp

3. PyCon JP committee

BePROUD co, ltd.3

http://pycon.jp/

4

Agenda

1. Sphinx introduction & Setup for i18n

2. A easy way and Non easy ways to translate

3. Sphinx i18n feature

4. Automated translation process with several services

5. Tips, tricks, traps 5

Question

6

Question1

How many people do you use Open Source

Software?

7

Question2

How many people may have contributed to the

OSS?

8

Question3

Does the translation into familiar language

contribute to other developers?

9

10

Translator

Product Author

Translated Docs

Readers

English Docs

Translators

Tools & Services

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

11

1. Sphinx introduction& Setup for i18n

12

What is Sphinx?

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

13

1. Sphinx introduction & Setup for i18n

Sphinx

reSTreSTreStructuredText

(reST)reST

Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

The history of Sphinx (short ver.)

14

1. Sphinx introduction & Setup for i18n

The father of Sphinx

Too hard to maintenance

~2007

Easy to writeEasy to

maintenance2007~

Sphinx Before and After

Before

There was no standard ways to write documentsSometime, we need converting markups into other

formats

After

Generate multiple output format from single sourceIntegrated html themes make read docs easierAPI references can be integrated into narrative docsAutomated doc building and hosting by ReadTheDocs

15

1. Sphinx introduction & Setup for i18n

Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

16

1. Sphinx introduction & Setup for i18n

Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

17

1. Sphinx introduction & Setup for i18n

SPHINXdocutils

HTML Builder HTML theme(Jinja2)

gettext Builder

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx i18n feature (built-in)

18

1. Sphinx introduction & Setup for i18n

reST Parser(directive / role)

doctree(Intermediat

e)

reSTreSTreStructuredText(reST)

$ pip install sphinx

Support tools for translation

How to install Sphinx with i18n tools

To build a sphinx document

$ pip install sphinx-intl$ pip install transifex-client=0.8

19

1. Sphinx introduction & Setup for i18n

$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

20

1. Sphinx introduction & Setup for i18n

Files & setup conf.py$ tree /path/to/deepthought+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.py| +- make.bat| +- Makefile+- setup.py

21

1. Sphinx introduction & Setup for i18n

Document source

Document build output

Target library for doc1. ...2. 3. language = 'ma'4. locale_dirs = ['locale']5.

doc/conf.py

2. Easy contributable internationalization process with Sphinx

22

What is internationalization?

23

2. Easy contributable internationalization process with Sphinx

I18N

What is easy contributable?

24

2. Easy contributable internationalization process with Sphinx

Not a easy way (example 1/3)

The manual are provided only in the HTML files

Rewriting reST files

25

2. Easy contributable internationalization process with Sphinx

Not a easy way (example 2/3)

The manual are generated fromreST files and

docstrings in the source files

Rewriting reST filesRewriting docstrins in the source files.

26

2. Easy contributable internationalization process with Sphinx

Not a easy way (example 3/3)

OK, we are engineers. We can use git!

Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to

do!)git commitgit pullSometimes resolve conflictgit push

27

2. Easy contributable internationalization process with Sphinx

Not easy vs easy

Not a easy way Easy way1. Learn git2. Learn GitHub3. git clone to get the

code4. Translation5. git commit, pull,

push6. Resolve conflict7. Build html your self

1. No git2. No Github3. No file4. Translation5. Update

Automatically6. No conflict7. You can get a HTML

output w/o hand-build.

28

2. Easy contributable internationalization process with Sphinx

3. Sphinx i18n feature

29

Two i18n features of SphinxOutput pot files:

from reST

Input po files:to generate translated HTML

30

3. Sphinx i18n feature

reST pot

reST

htmlpo

Translation flowGenerate pot

Translate pot into po

Generate Translated HTML

31

3. Sphinx i18n feature

reST pot

reST

htmlpo

pot poTranslator

Pleasetranslate

Translate!Thanks for yourContribution!

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""

msgid "For example:"msgstr ""

32

Output pot files 3. Sphinx i18n feature

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ ls _build/gettextapi.pot examples.pot generated.pot index.pot

generated.pot

reST pot

Preparing po files to translate

doc+- _build/| +- gettext/| +- api.pot| +- examples.pot| +- generated.pot| +- index.pot+- locale/

doc+- _build/+- locale/ +- zh_cn/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/

copy and renamefor each langs

Translate them

Translator

pot po

33

3. Sphinx i18n feature

34

Preparing po files to translate

$ sphinx-intl update -p _build/gettext -l zh_cnCreate: locale/zh_cn/LC_MESSAGES/api.poCreate: locale/zh_cn/LC_MESSAGES/examples.poCreate: locale/zh_cn/LC_MESSAGES/generated.poCreate: locale/zh_cn/LC_MESSAGES/index.po

At first, sphinx-intl copy pot files and rename them

pot po

sphinx-intl

$ make gettext$ sphinx-intl update -p _build/gettext -l zh_cnNot Changed: locale/zh_cn/LC_MESSAGES/api.poUpdated: locale/zh_cn/LC_MESSAGES/examples.po +3, -1Not Changed: locale/ma/LC_MESSAGES/generated.po

After change the document, sphinx-intl update differences

3. Sphinx i18n feature

Translate po files

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr ""

generated.po

pot po

sphinx-intl

Translator

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize ``obj`` to a JSON formatted :class:`str`."msgstr " 序列 ``obj`` 以 JSON 格式 :class:`str` 。 "

generated.po

Translate by using Vim, Emacs, OmegaT, ...

35

3. Sphinx i18n feature

Input po files

36

3. Sphinx i18n feature

reST

htmlpo

Translated

$ make html...Build finished. The HTML pages are in _build/html.

Big picture

37

3. Sphinx i18n feature

reST pot

html

po

make gettext

sphinx-intl

Translator

make html

Doc Author

TranslationCatalog

Translatedcatalog

4. Automated translation process with several services

38

Entire process to translate sphinx docs

39

4. Automated translation process with several services

reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

Translator

Translator

Translator

Autor / Translator

upload

Translator

clone

Translator

To Be

40

4. Automated translation process with several services

reST pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

upload

Translators

To BeAutomated

To BeParallel

clone

Translation tool typesVim / Emacs (Editors)

Edit local filesTranslation support plugins are available.

OmegaT (Translation Tools)Edit local files.Translation support features.

Transifex (Services)Edit onlineTranslation support

features.

41

4. Automated translation process with several services

po

Translators

To BeParallel

Be Parallel by using Transifex

Transifex provides...As API:Upload potDownload po

As Web console:GlossaryTranslation memoryRecommendationAuto-translation

42

4. Automated translation process with several services

po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-clientDownload

Translation on Transifex web console

43

4. Automated translation process with several services

Original Text

Translated Text(you should keep reST syntax)

Suggestions fromTranslation Memory (TM)

Original(pot)

Translation(po)

Copy orig to translationMachine translation

SaveReview (if needed)

Translators

Parallel1

2

4

3

56

To Be Automated

44

4. Automated translation process with several services

po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-clientDownload

reST

html

make gettext

make html

Doc Author

upload

To BeAutomated

clone

To Be Automated

45

4. Automated translation process with several services

pot

Upload sphinx-intltransifex-client

po transifex-clientDownload

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

The procedure for automation

1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html

46

4. Automated translation process with several services

pot

Upload sphinx-intltransifex-client

po transifex-clientDownload

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomate

d

1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources

# update .tx/config8. tx push -s # push pot files to

transifex9. tx pull -l zh_cn # pull po files from

transifex10. make html SPHINXOPTS="-D language=ma"

Shell commands for automationrun.sh

$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh

47

4. Automated translation process with several services

The drone.io

48

WebHook

Deploy

Clone repositoryRun shell script

The drone.io is a continuous integration service.

4. Automated translation process with several services

GitHub + drone.io + S3

49

GitHub

Amazon S3

Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html

4. Automated translation process with several services

2

1

3

Be Automated

50

pot

Upload sphinx-intltransifex-client

po transifex-clientDownload

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated

4. Automated translation process with several services

Automated by drone.io

51

UploadpotDownload

poupload

WebHookclone

1

56

make html

make gettext2

3

4

1WebHook

Automated

4. Automated translation process with several services

View from doc author

Doc Author doesn't require annoying procedure.

UpdateTranslation Source

Doc Author

Notify

See

Commit

52

4. Automated translation process with several services

View from doc translatorsNo gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-

build.

Translators

ParallelSee

TranslateTranslated Pages

53

4. Automated translation process with several services

The entire automated process

54

UpdateTranslation Source

Doc Author

Translators

Parallel

Notify

See

See

Publish

Translate

Commit

DownloadTranslations

Notify

Automated

4. Automated translation process with several services

Summary

Tools1. sphinx - documentation generator2. docutils - base of sphinx3. sphinx-intl - support utility for sphinx i18n4. transifex-client - file transfer tool for

Transifex

Services5. Transifex - translation support service6. Drone.io - continuous integration service

55

4. Automated translation process with several services

5. Tips, tricks, traps

56

TIP: Drone.io limits 15mins for a build

Drone.io limits 15mins for a build.Install from wheel package may help you

57

5. Tips, tricks, traps

1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -

2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh

ex. https://bitbucket.org/sphinxjp/docutils-translation/

TRAP: Version of transifex-client

transifex-client 0.11b3 is not stable (for me)Especially for Windows users

If you have encountered trouble with using newer version, please try:

58

5. Tips, tricks, traps

$ pip install "transifex-client=0.8"

Drone.io only list-up repositories which you have admin permission.

Prepare a empty repository to create a drone.io project.

59

TRICK: Preparing Drone.io project5. Tips, tricks, traps

Examples

Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin

Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin

Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin

60

5. Tips, tricks, traps

Questions?@shimizukawa

Grab me after the session :)

61

Thanks :)

62