Live Deployment.Continuous Integration.Drupal.

Andriy Podanenko, @podarok, ProPeople.

Technologies used1. sh/bash scripting2. drush/drupal/php3. Jenkins server master/slave CI server.4. mysql advanced configuration5. vagrant virtualization, puppet scripting (ruby)6. java (if You need to extend basic Jenkins’es plugins) (optional)7. apache/nginx/php5-fpm/mysql (Full LAMP/MAMP stack)8. phantomjs/javascript scripting9. git/GitHub

10. code sniffer stack (php codesniffer, jshint, scss lint, twig lint etcetera)

The problems● Code merged to *master without real testing● Configs from dev site do not pushed to stage● Different dev/stage/prod server environments● Code review is not comprehensive.● Visual regression needs a lot of love.● Deploy is a horror.● Client make changes that brake upgrade path.● Update/upgrade path not tested.

Demo/Live content.● drush si build_profile_name...● rebuild.sh script with all steps for getting dev build●After first release for content management● liverebuild.sh script for getting live DB from hosting

server to meet latest content/config changes.● helps with visual regression testing when there is

frontend’s pull request for review and content/config changed.

rebuild.sh script example(in repo!!!)#!/bin/sh

service memcached restart

chmod +w sites/default/settings.php && rm -rf sites/default/settings.php

drush -vy si sitename --db-url=mysql://drupal:drupal@127.0.0.1:/drupal --account-name=admin --account-pass=pass

chmod +w sites/default/settings.php && rm -rf sites/default/settings.php && cp sites/default/settings_devel.php

sites/default/settings.php

cd sites/all/modules/contrib/guzzle

composer update

cd ../../../../../

pwd

cd sites/all/modules/custom/salespush

composer update

cd ../../../../../

drush some_custom_command

drush en devel -y

drush cc all

liverebuild.sh script example#!/bin/sh

rm -rf livedb.sql*

wget http://URL_TO/livedb.sql.gz

gunzip -f livedb.sql.gz

mysql --force -e "drop database IF EXISTS livedrupal;create database IF NOT EXISTS livedrupal;use livedrupal;source

livedb.sql;"

rm -rf sites/default/settings.php

cp sites/default/settings_live.php sites/default/settings.php

drush -y updatedb

drush upwd admin --password=dEvPasS

create-solr-instance inst_x 7

create-solr-instance inst_i 7

drush en -y live_solr_settings

drush cc all

drush -dvy en migrate_master

drush -dvy updatedb

MySQL fast restoring from dumpSET foreign_key_checks = 0;SET UNIQUE_CHECKS = 0; SET AUTOCOMMIT = 0;source dbdump.sql;SET foreign_key_checks = 1;SET UNIQUE_CHECKS = 1; COMMIT;------------------------------------------------------------------------------------innodb_file_per_table = 1tmp_table_size = 160M

max_heap_table_size = 160M

innodb_file_format = Barracuda

innodb_file_format_max = Barracuda

innodb_flush_log_at_trx_commit = 2

query_cache_size = 160M

table_cache = 800

innodb_buffer_pool_size = 900M

GitHub ads (PR matters)

- Jenkins- GitHub PR builder plugin (triggers a build job in Jenkins for a PR’s hash)- create a build from scratch using cloned repo from PR, demo content or even live db if any.- Create a comment at PR’s thread with links to build/job results

How it looks like in PR’s comment

Sniffers hell (Code Review matters)...

● PHP CodeSniffer● JSHint● SCSS lint● yslow.js + phantomjs...● Drupal SimpleTest● PHPUnit tests● Performance tests (custom made)

PHP CodeSniffer log exampleFILE: .../smartling.admin.inc

--------------------------------------------------------------------------------

FOUND 3 ERRORS AND 7 WARNINGS AFFECTING 10 LINES

--------------------------------------------------------------------------------

20 | ERROR | global variables should start with a single underscore

| | followed by the module and another underscore

346 | WARNING | Unused variable $s_locale.

451 | ERROR | global variables should start with a single underscore

| | followed by the module and another underscore

464 | WARNING | Unused variable $need_fix.

618 | WARNING | Do not use the raw $form_state['input'], use

| | $form_state['values'] instead where possible

740 | ERROR | global variables should start with a single underscore

| | followed by the module and another underscore

762 | WARNING | Unused variable $need_fix.

--------------------------------------------------------------------------------

JSHint log exampledemo.js: line 5, col 5, Missing "use strict" statement.

demo.js: line 5, col 26, 'hideUnusedFn' is defined but never used.

demo.js: line 26, col 25, 'hideUnused3' is defined but never used.

demo.js: line 10, col 27, 'unusedVariable' is defined but never used.

demo.js: line 12, col 23, 'hideUnused' is defined but never used.

demo.js: line 9, col 43, 'unusedArg' is defined but never used.

demo.js: line 17, col 25, 'anotherUnusedArg' is defined but never used.

demo.js: line 16, col 27, 'unusedArgAfterUsed' is defined but never used.

demo.js: line 23, col 26, 'anotherUnusedArg2' is defined but never used.

demo.js: line 1, col 34, 'unusedGlobalVar' is defined but never used.

10 errors

SCSS lint log example_base.scss:568 [W] Class `footer-article-icon-Case-study` in selector should be written in all lowercase as `footer-article-icon-case-study`

_base.scss:725 [W] Merge rule `.pane-block` with rule on line 612

_base.scss:971 [W] Merge rule `.hp-marquee` with rule on line 966

_base.scss:982 [W] Selector should have depth of applicability no greater than 3, but was 4

_colors.scss:14 [W] Color `#666666` should be written as `#666`

_comments.scss:10 [W] `border: 0;` is preferred over `border: none;`

_comments.scss:171 [W] URLs should be enclosed in quotes

_reset.scss:25 [W] Use `//` comments everywhere

_reset.scss:26 [W] Each selector in a comma sequence should be on its own line

_slider.scss:17 [W] Properties should be sorted in order, with vendor-prefixed extensions before the standardized CSS property

yslow.js graph example$ phantomjs yslow.js --info grade --format tap --threshold '{"yminify": 90}' example.comTAP version 131..24ok 1 B (88) overall scorenot ok 2 C (72) ynumreq: Make fewer HTTP requestsok 3 C (70) ycdn: Use a Content Delivery Network (CDN)ok 4 A (100) yemptysrc: Avoid empty src or hrefnot ok 5 F (12) yexpires: Add Expires headersok 6 A (100) ycompress: Compress components with gzipok 7 A (100) ycsstop: Put CSS at topok 8 A (100) yjsbottom: Put JavaScript at bottomok 9 A (100) yexpressions: Avoid CSS expressionsok 10 N/A (-1) yexternal: Make JavaScript and CSS external # SKIP score N/Anot ok 11 C (70) ydns: Reduce DNS lookupsok 12 A (90) yminify: Minify JavaScript and CSSok 13 A (100) yredirects: Avoid URL redirectsok 14 A (100) ydupes: Remove duplicate JavaScript and CSSok 15 A (100) yetags: Configure entity tags (ETags)ok 16 A (100) yxhr: Make AJAX cacheableok 17 A (100) yxhrmethod: Use GET for AJAX requestsok 18 A (100) ymindom: Reduce the number of DOM element

PHPUnit/SimpleTest tests example./runtests.shConfiguration read from /var/www/pr/*****/build717/app/phpunit.xml.distF.....Time: 219 ms, Memory: 13.00MbThere was 1 failure:1) HP\Bundle\AuthBundle\Tests\Controller\DefaultControllerTest::testIndexFailed asserting that two strings are equal.--- Expected+++ Actual@@ @@-'Authentication required.'+'"Argument 2 passed to Symfony\\Bundle\\TwigBundle\\Extension\\AssetsExtension::__construct() must be an instance of Symfony\\Component\\Routing\\RequestContext, none given, called in \/var\/www\/pr\/*****\/build717\/app\/cache\/test\/appTestDebugProjectContainer.php on line 3496 and defined"'/var/www/pr/***/build717/src/**/Bundle/AuthBundle/Tests/Controller/DefaultControllerTest.php:21 FAILURES! Tests: 6, Assertions: 10, Failures: 1.

Live Deployment map

Full Demo

Useful links● https://www.drupal.org/coding-standards● http://www.squizlabs.com/php-codesniffer● http://jshint.org/● https://github.com/causes/scss-lint● https://github.com/asm89/twig-lint● https://github.com/jenkinsci/ghprb-plugin● https://github.com/podarok/ghprb-plugin (Extended version)● http://puphpet.drupal.ua/ (Custom version for Drupal devs)● http://www.vagrantup.com/● http://drush.ws/

Future plans

Add twig templates lintingVisual Regression toolPerformance tests

Thank You

Questions?

Andriy Podanenko, Software Architect, DevOp.http://dgo.to/@podarokhttp://druler.com/contacthttp://wearepropeople.com