View
420
Download
0
Category
Preview:
DESCRIPTION
A presentation on how changes in Daylight Saving Time were handled at Yahoo!. Originally given at BayLISA in May 2007. Slides are available here:netmeister.org/misc/dst_yahoo.pdf
Citation preview
DST ’07 @ Yahoo! Slide 1
DST ’07 @ Yahoo!
Jan Schaumannjschauma@netmeister.org
PGP: 136D 027F DC29 8402 7B42 47D6 7C5B 64AF AF22 6A4C
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 2
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 3
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 4
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 5
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 6
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 7
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 8
whoami
$ ME=$(id -un)
$ awk -F: -v ME=${ME} ’$0 ~ ME { print $5 }’ /etc/passwd
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 9
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 10
whoami
$ ME=$(id -un)
$ grep ${ME} /etc/passwd | cut -d: -f5
Jan Schaumann
$ groups ${ME}
netbsd sa yahoo
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 11
DST ’07 @ Yahoo!
– Saving the living daylight out of you!
or How the YSA team stopped worryingand learned to love UTC!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 12
In the beginning...
...the universe was created.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 13
In the beginning...
...the universe was created.
“This has made a lot of people very angry and been widely regarded as a bad move.”
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 14
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 15
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 16
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 17
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 18
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 19
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 20
Daylight Saving Time for Dummies
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 21
Fast forward to 2005...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 22
Fast forward to 2005...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 23
Fast forward to 2005...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 24
March 11th, 2007
In 2007,Daylight Saving Time will start on March 11th.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 25
March 11th
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 26
March 11th
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 27
August 8th, 2005
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 28
One day later...
Yahoo! decides to take immediate action!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 29
August, 29th, 2005
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 30
December, 22nd, 2005
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 31
August 8th, 2005
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 32
One day later...
Yahoo! decides to take immediate action!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 33
Alas...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 34
“Persistence of Memory”
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 35
Fast forward to January 2007...
“Hey, we need to know which hostscan’t deal with the new DST.”
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 36
Script 0
See remote0.sh (and remote0.5.sh).
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 37
January 2007
need to know hosts that can’t deal with new DST
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 38
January 2007
need to know hosts that can’t deal with new DST
need to run a short command on each host
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 39
January 2007
need to know hosts that can’t deal with new DST
need to run a short command on each host
So: write a script that runs a single command on hosts given as input.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 40
January 2007
need to know hosts that can’t deal with new DST
need to run a short command on each host
So: write a script that runs a single command on hosts given as input.
$ cat >checkhosts.sh <<EOS
#!/bin/sh
while read host; do
ssh $host "single-command"
done < $1
EOS
$
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 41
“single command”
TIMTOWTDI:
TZ=PST8PDT date -j 200703110200
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 42
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 43
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 44
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"
– triggers a bug in FreeBSD/amd64 and zdump spins out of control
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 45
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"
– triggers a bug in FreeBSD/amd64 and zdump spins out of control
env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 46
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"
– triggers a bug in FreeBSD/amd64 and zdump spins out of control
env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
– Hooray!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 47
“single command”
TIMTOWTDI (or is there?):
TZ=PST8PDT date -j 200703110200
– conflicting flags and return values on different platforms
zdump -v PST8PDT | grep "2007 UTC = Sun Mar 11"
– triggers a bug in FreeBSD/amd64 and zdump spins out of control
env TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
– Hooray!
Yes, we happen to have perl installed on every machine.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 48
Script I
See remote1.sh.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 49
So far, so good!
We now have a list of hostnames, sorted by:
operating system
need to patch
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 50
So far, so good!
We now have a list of hostnames, sorted by:
operating system
need to patch
Surely, our work here is done.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 51
So far, so good!
We now have a list of hostnames, sorted by:
operating system
need to patch
Surely, our work here is done.
Well, not entirely...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 52
Forward to middle of February 2007...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 53
Forward to middle of February 2007...
detailed communication with properties about impact
gather information from vendors
select patching
full-time focus on data collection
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 54
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 55
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 56
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 57
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 58
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
So: revamp tool to not run hard-coded command, but any given script.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 59
Script II
See remote2.sh and remote3.sh.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 60
Data Collection: Input
input from central database
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 61
Data Collection: Input
input from central database
input from DNS zone files
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 62
Data Collection: Input
input from central database
input from DNS zone files
prepare exclusion lists
exclude windows hosts
exclude routers, switches etc.
exclude NetApp filers
exclude hosts in corp
exclude DNS rotations, decomissioned hosts, placeholder DNS entries etc.
...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 63
Data Collection: Output
a lot of information is collected → output per host fairly large
sort output by patch-needed
sort output by OS
sort output by property
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 64
Patch actions
FreeBSD
Linux
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 65
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
Linux
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 66
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
Linux
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 67
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
Linux
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 68
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
patch installation innocuous → no reboot required
Linux
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 69
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
patch installation innocuous → no reboot required
Linux
no change necessary for versions >= RHEL3u7, RHEL4u3
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 70
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
patch installation innocuous → no reboot required
Linux
no change necessary for versions >= RHEL3u7, RHEL4u3
installation of rpm tzdata for other versions RHEL3 and RHEL 4
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 71
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
patch installation innocuous → no reboot required
Linux
no change necessary for versions >= RHEL3u7, RHEL4u3
installation of rpm tzdata for other versions RHEL3 and RHEL 4
upgrade to latest RHEL release for all other rpm based Linux versionsrecommend
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 72
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patch
some versions higher than that (including some 6.x based images) mayrequire a patch
upgrade to latest 6.x based version recommended
patch installation innocuous → no reboot required
Linux
no change necessary for versions >= RHEL3u7, RHEL4u3
installation of rpm tzdata for other versions RHEL3 and RHEL 4
upgrade to latest RHEL release for all other rpm based Linux versionsrecommend
forced (ugh!) installation of tzdata rpm possible
Solaris
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 73
Patch actions
FreeBSD
all versions prior to a certain 4.11 based image require patchsome versions higher than that (including some 6.x based images) mayrequire a patchupgrade to latest 6.x based version recommendedpatch installation innocuous → no reboot required
Linux
no change necessary for versions >= RHEL3u7, RHEL4u3installation of rpm tzdata for other versions RHEL3 and RHEL 4upgrade to latest RHEL release for all other rpm based Linux versionsrecommendforced (ugh!) installation of tzdata rpm possible
Solaris
Oy vey!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 74
TZDATA everywhere!
ICU (Internationalization Components for Unicode)
JVM
MySQL
Oracle
Perl DateTime
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 75
TZDATA everywhere!
ICU (Internationalization Components for Unicode)
need to update ICU to >= 3.6
need to recompile apps linked against libicu∗
JVM
MySQL
Oracle
Perl DateTime
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 76
TZDATA everywhere!
ICU (Internationalization Components for Unicode)
need to update ICU to >= 3.6
need to recompile apps linked against libicu∗
JVM
need to update to recent version
March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); novendor patch but workaround solution
MySQL
Oracle
Perl DateTime
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 77
TZDATA everywhere!
ICU (Internationalization Components for Unicode)
need to update ICU to >= 3.6
need to recompile apps linked against libicu∗
JVM
need to update to recent version
March 8th (!): new bug in JDK’s found (for EST, MST, and HST zones); novendor patch but workaround solution
MySQL
restart only for MySQL 4.0
upgrade “mysql tzdata” package for MySQL 4.1
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 78
TZDATA everywhere!
Oracle
patch following instructions provided by vendor
March 3rd: new patch → repatch!
Perl DateTime
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 79
TZDATA everywhere!
Oracle
patch following instructions provided by vendor
March 3rd: new patch → repatch!
Perl DateTime
update to >= 0.38
total of < 100 hosts using this package
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 80
Post-patch actions
if possible, reboot
verify that host displays correct date
adjust /etc/localtime if not a symlink
patch virtual hosts / jails on given host
ensure all services are restarted (mysql, apache, etc.)
per-application tests
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 81
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
check /etc/localtime
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 82
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
check /etc/localtime
check for virtual hosts / jails
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 83
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
check /etc/localtime
check for virtual hosts / jails
gather output of ps auwwx
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 84
Data Collection
need to know hosts that can’t deal with new DST
need to run a short command on each hostenv TZ=PST8PDT perl -e ’@t=localtime 1173682800; exit 1 - pop @t’
need to know what packages are installed
need to know timestamp of a certain file
need to know if client is running oracle
check /etc/localtime
check for virtual hosts / jails
gather output of ps auwwx
oh, and also gather firmware information, why not
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 85
Script N
See remote.final.sh.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 86
How to run the script remotely?
while read host; do
scp remote.sh $host:/tmp/
ssh $host "/tmp/remote.sh"
done < $1
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 87
How to run the script remotely?
while read host; do
scp remote.sh $host:/tmp/
ssh $host "/tmp/remote.sh"
done < $1
But: scp(1) followed by ssh(1) is expensive
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 88
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh; /tmp/remote.sh"
done < $1
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 89
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh; /tmp/remote.sh"
done < $1
But: /bin/sh vs. Solaris anybody?
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 90
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh; \
/usr/local/bin/bash /tmp/remote.sh"
done < $1
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 91
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh; \
/usr/local/bin/bash /tmp/remote.sh"
done < $1
But: contrary to common assumption, /usr/local/bin/bash does not exist on allhosts
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 92
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh && \
bash /tmp/remote.sh"
done < $1
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 93
How to run the script remotely?
while read host; do
cat remote.sh | ssh $host \
"cat >/tmp/remote.sh && \
bash /tmp/remote.sh"
done < $1
appears to work, even if it relies on $PATH
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 94
Performing data collection
originally done on a single host, sequentially
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 95
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 96
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 97
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host(try to get work done on a machine with a load >100)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 98
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host(try to get work done on a machine with a load >100)
then done in parallel on multiple hosts
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 99
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host(try to get work done on a machine with a load >100)
then done in parallel on multiple hosts(managed to fill up all local diskspace when uploading results)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 100
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host(try to get work done on a machine with a load >100)
then done in parallel on multiple hosts(managed to fill up all local diskspace when uploading results)
finally done massively parallel and unattended on many dedicated hosts withplenty of diskspace
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 101
Performing data collection
originally done on a single host, sequentially(full scan took almost a week)
then done in parallel on single host(try to get work done on a machine with a load >100)
then done in parallel on multiple hosts(managed to fill up all local diskspace when uploading results)
finally done massively parallel and unattended on many dedicated hosts withplenty of diskspace(full scan took about 50 minutes)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 102
“Parallelization” in shell - *cough*
split input into N files
farm out to N hosts (slaves)
on each slave
split input into M files
run M jobs in parallel
run all jobs in screen(1) session, then detach
on master, wait until N slaves report done
perform post-processing on master, dump data into http dir and send mail
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 103
Actual patching
mostly done by property
“leftovers” done by YSA team
FreeBSD patches trivial (thanks to internal kick-ass package manager)
RHEL patching less trivial
Solaris patching least trivial
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 104
March 11th cometh!
SAs everywhere frantically exchange batteries in fire alarms.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 105
March 11th goes!
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 106
March 12th comes!
SAs return to their daily routine.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 107
DST ’07 - Bottom line
a non-event
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 108
DST ’07 - Bottom line
a non-event (excellent!)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 109
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 110
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 111
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 112
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 113
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 114
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 115
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82
we continue to do periodical scans and gather the same data
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 116
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82
we continue to do periodical scans and gather the same data
we still will be on alert on October 28th (old switchback date)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 117
DST ’07 - Bottom line
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 118
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82
up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
we continue to do periodical scans and gather the same data
we still will be on alert on October 28th (old switchback date) and November 4th(new switchback date)
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 119
DST ’07 - Bottom line
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 120
DST ’07 - Bottom line
a non-event (excellent!)
we now have a bunch of useful tools to do massive scans(though they should be rewritten in perl, python, C, anything!)
not all of our machines are entirely idle:up 1885 days, 12 mins, 0 users, load averages: 33.36, 31.59, 29.82
up 72 days, 23:01, 0 users, load averages: 1021.89, 974.45, 881.28
up 235 days, 11:39, 0 users, load averages: 305.07, 344.82, 338.55
we continue to do periodical scans and gather the same data
we still will be on alert on October 28th (old switchback date) and November 4th(new switchback date)
we’ll be rather busy in the Fall of 2037...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 121
Y2K38
time t stored as a signed 32bit int
latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038
Suggested solution: using a signed 64bit value.
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 122
Y2K38
time t stored as a signed 32bit int
latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038
Suggested solution: using a signed 64bit value.
New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 123
Y2K38
time t stored as a signed 32bit int
latest time possible: 03:14:07 UTC on Tuesday, January 19, 2038
Suggested solution: using a signed 64bit value.
New wraparound date on Sunday, December 4, 292,277,026,596 (ie in about 290billion years).
“This is not widely regarded as a pressing issue.”
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 124
To summarize:
∆t′ = γ(∆t −v∆x
c2 )
∆t = γ(∆t′ +v∆x′
c2 )
∆t′ = γ∆t
Time is relative...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 125
To summarize:
∆t′ = γ(∆t −v∆x
c2 )
∆t = γ(∆t′ +v∆x′
c2 )
∆t′ = γ∆t
Time is relative (and an illusion)...
Jan Schaumann BayLISA, May 17, 2007
DST ’07 @ Yahoo! Slide 126
To summarize:
...and Daylight Saving Time doubly so.
Jan Schaumann BayLISA, May 17, 2007
Recommended