Disaster Recovery in distant DC; people, hardware, latency - PLNOG 200909
73
Disaster Recovery in distant DC; people, hardware, latency.
-
Upload
marcin-mazurek -
Category
Technology
-
view
226 -
download
3
description
Transcript of Disaster Recovery in distant DC; people, hardware, latency - PLNOG 200909
Disaster Recovery in distant DC;people, hardware, latency.
• What is really the Allegro.pl?
• Distaster Recovery - true story
• Latency - why 20 ms matters
• Data Centers topology
agenda
Allegro ... QXL ?
Serwisy Aukcyjne Serwisy Ogłoszeniowe Płatności
.pl.cz
.ro
.ua.bg
.hu.sk .lt
.pl
.pl
Sklepy Internetowe
Allegro... beginings
o Allegro... today
o Allegro... today
about 1000 servers
o Allegro... today
about 1000 serversabout 300 IT staff
o Allegro... today
about 1000 serversabout 300 IT staff
about 40 in infrastructure
why bother?
Qvo vadis?
Qvo vadis?
Qvo vadis?
Qvo vadis?
Qvo vadis?
Qvo vadis?
2 hours by plane...
~800 km
Go west...
2 hours by plane...
~800 km
Go west...
WWCKD ?
WWCKD ?
Hope is the mother of lost battles
No guts, no glory.
No pain, no game.
No risk, no fun.
WWCKD ?
Hope is the mother of lost battles
No guts, no glory.
No pain, no game.
tiaaaa...
No risk, no fun.
Building DC3
Building DC3
DC net topology
DC net topology
ISP 1
ISP 2
DC2 , Poznań
DC net topology
ISP 1
ISP 2
ISP 1
ISP 2
DC2 , Poznań DC3 , Frankfurt
DC net topology
ISP 1
ISP 2
ISP 1
ISP 2
DC2 , Poznań DC3 , Frankfurt
2G, EoMPLS
DC net topology
ISP 1
ISP 2
ISP 1
ISP 2
DC2 , Poznań DC3 , Frankfurt
2G, EoMPLS
10G, DWDM
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNSDNS
$ host -t ns allegro.pl allegro.pl name server gtmdc3.allegro.pl. allegro.pl name server gtmdc2.allegro.pl.
Global Traffic Manager
DC2 , Poznań DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS request
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS response
Global Traffic Manager
DC2 , Poznań DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS req: www.allegro.pl
HTTP traffic
Global Traffic Manager
DC2 , Poznań DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań
DNS req: www.allegro.pl
DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
Global Traffic Manager
DC2 , Poznań DC3 , Frankfurt
www.allegro.pl
F5 LTMF5 LTM
F5 LTMF5 LTM
F5 GTM F5 GTM F5 GTM F5 GTM
www.allegro.pl
DNS req: www.allegro.pl
HTTP traffic
Latency vs tennis
SYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
ACK
DATA
DATA
ACK
Control packets
Data packets
bandwidth
RTT / latency
window size = 64 kBytes
Bandwidth-delay Product
BDP (bytes) = bandwidth (KBytes/sec) x RTT (ms)
0,0005secSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,0005sec
0,0005sec
ACK
DATA
DATA
ACK
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
Control packets
Data packets
0,0005secSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,0005sec
0,0005sec
ACK
DATA
DATA
ACK
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
Control packets
Data packets
Scenario #1
• bandwidth = 10Gbit/sec• RTT / latency = 1 ms (1/1000 sec)• window size = 64 kBytes
• data to transfer = 640 000 kBytes (1CD)
What is the time to deliver CD from point A to B?
0,0005secSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,0005sec
0,0005sec
ACK
DATA
DATA
ACK
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
0,0005sec
Control packets
Data packets
Scenario #1
• bandwidth = 10Gbit/sec• RTT / latency = 1 ms (1/1000 sec)• window size = 64 kBytes
• data to transfer = 640 000 kBytes (1CD)
What is the time to deliver CD from point A to B?
1 sec !!!
(640 000 kBytes) / (64 kBytes/ 1ms) = 1000 ms
0,5sekSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,5sek
0,5sek
ACK
DATA
DATA
ACK
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
Control packets
Data packets
0,5sekSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,5sek
0,5sek
ACK
DATA
DATA
ACK
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
Control packets
Data packets
Scenario #2
• bandwidth = 10Gbit/sec• RTT / latency = 1 sek• window size = 64 kBytes
• data to transfer = 640 000 kBytes (1CD)
What is the time to deliver CD from point A to B?
0,5sekSYN
SYN/ACK
ACK
TCP/IP explained
FIN
ACK
0,5sek
0,5sek
ACK
DATA
DATA
ACK
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
0,5sek
Control packets
Data packets
Scenario #2
• bandwidth = 10Gbit/sec• RTT / latency = 1 sek• window size = 64 kBytes
• data to transfer = 640 000 kBytes (1CD)
What is the time to deliver CD from point A to B?
1000 sec !!!
(640 000 kBytes) / (64 kBytes/1000ms) = 1000 sec
800km distance...
800km distance...
storage data base
application / transactionsAIX Oracle
MySQLbackend daemonsfile transfers
server deployment
Linux
800km distance...
just 20ms...
co-operation
co-operation
developers
co-operation
developers infrastructure
co-operation
developers infrastructure
co-operation
developers infrastructure
Recipe for success
Disaster recovery ingredients
Recipe for success
Recipe for success
Recipe for success
Recipe for success
Thank You, Q&A
