View
346
Download
0
Category
Preview:
DESCRIPTION
GitLab Infrastructure
Citation preview
GitLab InfrastructureStatus Report
We have HTTP queue time in monitoring, so we ran an experiment
What if we add more memory to workers?
This had a good impact across the board - less load in general
Specially on API timings (authorized_keys lookup timings)
This is why git-ssh is going faster, but there is still a long way to go.
Some things did not go well with the change
Redis leaves connections behind - GitLab max connections open -> outage
Deploys - RC3 blowed in production
On a Friday 1AM my time.
So we built staging the next Monday staging.gitlab.com is way smaller and less powerful than GitLab.com, but it has all the data.
Thanks @Jeroen
Done is better than perfect
Deploys - RC4 blew in staging that very same Monday
<
Staging Production
Postgres is still dying on us, or was it?
Query counts monitoring allowed us to corner the
problem and get it fixed In RC5
Thanks @marat!
Monitoring - improvements on how methods are measured
We are actually
showing where the time
is going now.
Thanks @Yorick!
Performance - no progress besides the API
Storage● Cephfs - dev.gitlab.org has been running on cephfs for the last month
○ Did you noticed? No? That’s good! :)○ Pushing the linux kernel to it takes 27 minutes ~1.5Gb○ Pushing the linux kernel to GitLab.com takes between 1:30hs to forever
● Our measurements were wrong, Cephfs gives 500/150 IOPS● But it scaled without a hiccup up to 98 workers nodes (clients).● We are testing behaviour when we add more nodes/ODS, etc.● We have a plan to move to Cephfs without downtime
Storage - capacity
Git data - 28TB out of 49TB
Shared data - 3TB out of 4TBwe can grow this one easily-ish
Other news● What’s coming soon
○ Multiple mount points/shards - Thanks @Alejandro!○ 2 new hires
■ Alex as a Production Engineer■ Ahmad as a Performance Specialist
● We are talking with CI to transfer knowledge into Infrastructure.● We are going to take over GitHost.io● We are starting to build infrastructure monitoring that can be shipped with
GitLab● We are hiring!
That’s all folks!
Recommended