With Google since August 2005

Systems administrator in Hardware Operations

Developed several internal tools to assist with information flow

Created release engineering practices for Hardware Operations

Software Categories

Release Prerequisites

Release Notification

Separation of Responsibilities

Inter-team Coordination

Deployment instructions

Backout instructions

Pre-deploy and post-deploy testing (unit tests, service tests..)

End-user readable release notes

Critical software:
Important software:
All the rest:

How many groups are involved?

Release schedules agreed to by all

Change Control Boards

Organisation distribution dictates notification

Critical software gets more notification (24 hours -> 7 days)

Important software gets less notification (1min -> 24 hours)

No pre-notification for the rest

Notification on success is important!

Are postmortems needed?

Developers develop

Sysadmins deploy

Define days that are off-limits

Friday and weekend deploys are often bad

Holiday moratoriums

Developers will be developers! Don't expect them to like release engineering.

Obsess on reducing red tape.

Flexibility is more important than rigid procedure.

q
                                         Q  &  A