S3 Hot Topics in System Administration
Ned McClain, Applied Trust Engineering
9:00 a.m.–5:00 p.m.

Who should attend: System and network administrators who are interested in picking up several new technologies quickly.

Topics include:

• BIND9 Tips and Tricks: A Better DNS
  Most sites have migrated to BIND9, but are you really getting the most out of this major rewrite of the Internet's most popular nameserver? Learn about powerful new functionality such as split views, remote management, and even DNSSEC. This topic is a must for every modern administrator.
• Rapid Linux Disaster Recovery
  Tape backups are essential, but they are not an efficient way to restore a server in an emergency. We evaluate the ins and outs of Mondo, an open source disaster recovery tool that can create bootable recovery CDs from any Linux server. When used in tandem with a solid tape backup system, Mondo recovery CDs can reduce "bare metal" recovery time from hours to minutes.
• Linux Kernel Tuning
  As Linux's popularity in production environments increases, the need for Linux kernel tuning knowledge is more important than ever! Whether it's performance, security, or functionality you're looking to cajole your system into, we'll give you the "what to's" and the "how to's," and even the "what you can'ts" of this rare art.
• Practical Integration of UNIX and Active Directory
  With Active Directory, Microsoft introduced an open LDAP directory that has become the de facto authentication store at many organizations. UNIX/Linux administrators are often tasked with the unthinkable: to integrate UNIX authentication with Active Directory. We'll not only explore the standard integration tools, such as OpenLDAP, PAM, and NSS, but will show you how to create custom scripts to manage Active Directory from UNIX.
• Performance Crises Case Studies #4
  Don't miss the latest episode of this incredibly popular segment! We've taken a new set of real-life system administration performance crises and dissected them, providing insight on how to diagnose and remedy situations that you may someday face. This is a great way to gain practical knowledge in the performance arena.
• Custom Open Source Performance Monitoring
  Most organizations have monitoring systems that provide real-time problem alerts, but few can produce graphs of resource utilization over time. We provide practical examples of extending a monitoring system to collect historical performance trends. We'll use examples specific to Nagios and RRDtool, but the lessons and gotchas discussed here will prove useful to anyone looking to implement any new monitoring system.

Ned McClain (S3), co-founder and CTO of Applied Trust Engineering, lectures around the globe on applying cutting-edge technology in production computing environments. Ned holds a B.S. in Computer Science from Cornell University and is a contributing author of both the UNIX Systems Administration Handbook and the Linux Administration Handbook.

S4 Regular Expression Mastery
Mark-Jason Dominus, Consultant and Author
9:00 a.m.–12:30 p.m.

Who should attend: System administrators and users who use Perl, grep, sed, awk, procmail, vi, or emacs. Attendees should have prior experience using regexes in UNIX utilities such as grep, sed, Perl, Python, vi, or emacs.

Almost everyone has written a regex that produced unexpected results. Sometimes regexes appear to hang forever, and it's not clear what has gone wrong. Sometimes they behave differently in different utilities, and you can't tell why. This class will fix all these problems.

The first section of the class will explore the matching algorithms used internally by common utilities such as grep and Perl. Understanding these algorithms will allow us to predict whether a regex will match, which of several matches will be found, and which regexes are likely to be faster than others, and to understand why all of these behaviors occur. We'll learn why commonly used regex symbols such as ".," "$." and "\1" may not mean what you thought they did.

In the second section, we'll look at common matching disasters, a few practical parsing applications, and some advanced Perl features. We'll finish with a discussion of optimizations that were added to Perl 5.6, and why you should avoid using "/i."

Topics include:

• Inside the regex engine
  • Regular expressions are programs
  • Backtracking
  • NFA vs. DFA
  • POSIX and Perl
  • Quantifiers
  • Greed and anti-greed
  • Anchors and assertions
  • Backreferences
• Disasters and optimizations
  • Where machines come from
  • Disaster examples
  • Tokenizing
  • New optimizations
  • Matching strings with balanced parentheses

Mark-Jason Dominus (S4, S9) has been programming in Perl since 1992. He is a moderator of the comp.lang.perl.moderated newsgroup, the author of the Text::Template, Tie::File, and Memoize modules, a contributor to the Perl core, and author of the perlreftut man page. His work on the Rx regular expression debugger won the 2001 Larry Wall Award for Practical Utility. He lives in Philadelphia with his wife, daughter, and several plush octopuses.

S5 Eliminating Backup System Bottlenecks Using Disk-to-Disk and Other Methods
Jacob Farmer, Cambridge Computer Corp.
9:00 a.m.–12:30 p.m.

Who should attend: System administrators involved in the design and management of backup systems and policymakers responsible for protecting their organization's data. A general familiarity with server and storage hardware is assumed. The class focuses on architectures and core technologies and is relevant regardless of what backup hardware and software you currently use. Students will leave this lecture with immediate ideas for effective, inexpensive improvements to their backup systems.

The end may finally be in sight for the pains of backup and restore. The cost of disk storage has crossed the line: it has finally become practical to use disk to enhance or replace tape-based backup systems. In turn, software applications have come to market to facilitate the use of disk in backup systems. Now the problem is sorting out all of the options and fitting them into your existing infrastructure. This lecture identifies the major bottlenecks in conventional backup systems and explains how to address them. The emphasis is placed on the various roles inexpensive disk can play in your data protection strategy; however, attention is given to SAN-enabled backup, the current state and future of tape drives, iSCSI, and virtual tape.

Topics include:

• Identifying and eliminating backup system bottlenecks
• Conventional disk staging
• Virtual tape libraries
• Incremental forever and synthetic full backup strategies
• Information life cycle management and nearline archiving
• Data replication
• Continuous backup
• Snapshots
• The current and future tape drives
• Zero duplication file systems
• iSCSI

Jacob Farmer (S5, S10) is the CTO of Cambridge Computer Services, a specialized integrator of backup systems and storage networks. He has over 15 years of experience with storage technologies and writes an expert advice column for InfoStor magazine. He is currently writing a book on storage networking.
 

S6 Kerberos 5—Revenge of the Three-Headed Dog
Gerald Carter, Samba Team/Hewlett-Packard
9:00 a.m.–12:30 p.m.

Who should attend: Administrators who want to understand Kerberos 5 implementations on both UNIX/Linux and Windows clients and servers.

For many organizations, Kerberos is an an old technology that has been driven to the forefront by deployments of Microsoft Active Directory domains. The introduction of a standard authentication protocol into Windows domains has caused many network administrators to reexamine ways to integrate UNIX/Linux and Windows clients in a single authentication model.

Topics include:

• Key concepts of the Kerberos 5 protocol
• Specific related authentication interfaces such as SASL and GSSAPI
• The specifics of implementing of Krb5 realms
• Implementations of Krb5 cross-realm trusts
• Integration of Windows and UNIX/Linux clients into Krb5 realms
• Possible pitfalls of using popular Krb5 implementations such as MIT, Heimdal, and Windows 200x

Gerald Carter (S6, T6, W3) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing.

S7 Advanced Shell Programming
Mike Ciavarella, University of Melbourne
9:00 a.m.–12:30 p.m.

Who should attend: Junior or intermediate system administrators or anyone with a basic knowledge of programming, preferably with some experience in Bourne/Korn shells (or their derivatives).

The humble shell script is still a mainstay of UNIX/Linux system administration, despite the wide availability of other scripting languages. This tutorial details techniques that move beyond the quick-and-dirty shell script.

Topics include:

• Common mistakes and unsafe practices
• Modular shell script programming
• Building blocks: awk, sed, etc.
• Writing secure shell scripts
• Performance tuning
• Choosing the right utilities for the job
• Addressing portability at the design stage
• When not to use shell scripts

Mike Ciavarella (S7, S12, M7) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching Software Engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer.

S8 Next-Generation Security Tools
Peter Baer Galvin, Corporate Technologies
9:00 a.m.–12:30 p.m.

Who should attend: Systems managers and security managers interested in current security problems and the new generation of tools designed to solve those problems.

This course covers a variety of topics of importance to those designing or implementing security solutions for their installations. It starts with the nasty world of current security threats and the problems sites have to solve. It then talks about what is solvable and what still has no solution. Finally, it covers each of the possible solutions in detail. (Note: Most of these solutions are commercial products.)

Topics include:

• A security methodology
• Determining the state of your world
• Determining the problems to solve
• Policy and procedure
• Risk assessment, security audit, and penetration testing
• Firewalls: Why don't they work?
• Protecting Web servers
• Reducing spam
• Patch management and avoiding patching
• Network snooping
• Gaining status knowledge of your facility
• Content filtering and antivirus software
• Weak and strong authentication
• Spyware and peer-to-peer networks

Peter Baer Galvin (S8, M3, T3) is the Chief Technologist for Corporate Technologies, Inc., a systems integrator and VAR, and was the Systems Manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines. He wrote the "Pete's Wicked World" and "Pete's Super Systems" columns at SunWorld. He is currently contributing editor for Sys Admin, where he manages the Solaris Corner. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials on security and system administration and has given talks at many conferences and institutions on such topics as Web services, performance tuning, and high availability.

S9 Perl Program Repair Shop and Red Flags
Mark-Jason Dominus, Consultant and Author
1:30 p.m.–5:00 p.m.

Who should attend: Anyone who writes Perl programs regularly. Participants should have at least three months' experience programming in Perl.

You've probably been working too hard when you program, writing twenty lines of code when you only needed ten. But there is a better way, and I will show it to you. You'll learn how to improve your own code and the code of others, making it cleaner, more readable, more reusable, and more efficient, while at the same time making it 30-50% smaller. Smaller code contains fewer bugs and takes less time to maintain.

We will examine several real code examples in detail and see how to improve them. We'll focus on red flags—warning signs in your code that are plainly visible once you know what to look for—and on techniques that require little complex thought or ingenuity. All the bad code in this class is guaranteed 100% genuine and typical.

Participants are encouraged to submit their own code for anonymous review in the class. (Send it to mjd-usenix-2005@plover.com by March 1.) Class content varies depending on submissions, but is sure to include some of the topics listed below.

Topics include:

• Families of variables
• Making relationships explicit
• Refactoring
• Programming by convention
• The Flesh Blanket
• Conciseness
• Why you should avoid the "." operator
• Elimination of global variables
• Superstition
• The "use strict" zombies
• Repressed subconscious urges
• The cardinal rule of computer programming
• The psychology of repeated code
• Techniques for eliminating repeated code
• What can go wrong with "if" and "else"
• The Condition That Ate Michigan
• Resisting "Holy Doctrine"
• Trying it both ways
• Structural vs. functional code
• Elimination of structure
• Boolean values
• Programs that take two steps forward and one step back
• Programs that are 10% backslashes
• 'print print print print print '
• C-style "for" loops
• Loop counter variables
• Array length variables
• Unnecessary shell calls
• How (and why) to let "undef" be the special value
• Confusion of internal and external representations of data
• Tool use
• Elimination of repeated code with higher-order functions
• Learning to use a hammer
• The "swswsw" problem
• Avoiding special cases
• Using uniform data representations

Mark-Jason Dominus (S4, S9) has been programming in Perl since 1992. He is a moderator of the comp.lang.perl.moderated newsgroup, the author of the Text::Template, Tie::File, and Memoize modules, a contributor to the Perl core, and author of the perlreftut man page. His work on the Rx regular expression debugger won the 2001 Larry Wall Award for Practical Utility. He lives in Philadelphia with his wife, daughter, and several plush octopuses.

S10 Next-Generation Storage Networking and Data Protection
Jacob Farmer, Cambridge Computer Services
1:30 p.m.–5:00 p.m.

Who should attend: Sysadmins running day-to-day operations and those who set or enforce budgets. This lecture is technical in nature, but it does not address command-line syntax or the operation of specific products or technologies. Rather, the focus is on general architectures and various approaches to scaling in both performance and capacity. Since storage technologies tend to be expensive, there is some discussion of the relative cost of different technologies and of strategies for managing cost and achieving results on a limited budget.

There has been tremendous innovation in the data storage industry in the past few years, and this year the pace has quickened. Proprietary monolithic SAN and NAS subsystems are giving way to open-system and distributed architectures. Data-transfer protocols such as SCSI, NFS, and CIFS are facing competition from VI and DAFS. Fibre-channel and parallel SCSI interfaces are challenged by Gigabit Ethernet, iSCSI, and serial ATA. Bottlenecks imposed by I/O buses and stacks stand to be eliminated by Infiniband and RDMA.

This tutorial describes the latest technologies to hit the market for storage networking: SAN and NAS architectures, virtual storage, parallel file systems, storage interfaces, etc.

Topics include:

• Fundamentals of storage networking
• Shortcomings of conventional SAN and NAS architectures
• Comparison of storage interfaces: fibre channel, SCSI, serial ATA, Infiniband, Ethernet
• Comparison of storage protocols: CIFS, NFS, SCSI, VI, DAFS
• Open systems storage virtualization
• The convergence of SAN and NAS
• High-performance file sharing (NAS on steroids)
• SAN-enabled file systems
• Wide-area file systems
• Parallel file systems
• Content-addressable storage

Jacob Farmer (S5, S10) is the CTO of Cambridge Computer Services, a specialized integrator of backup systems and storage networks. He has over 15 years of experience with storage technologies and writes an expert advice column for InfoStor magazine. He is currently writing a book on storage networking.
 

S11 Over the Edge System Administration, Volume 1
David N. Blank-Edelman, Northeastern University
1:30 p.m.–5:00 p.m.

Who should attend: Old-timers who think they've already seen it all, and those who want to develop inventive thinking early in their career. Join us and be prepared to be delighted, disgusted, and amazed. Most of all, be ready to enrich your network and system adminstration by learning to be different.

Can you think "out of the box" about system administration? One of the things that distinguishes the really great sysadmins from the good ones is their ability to be creative in their approach to problems and solutions. It's time to learn how to break the rules, abuse the tools, and generally turn your system administration knowledge inside out. This class is a cornucopia of ideas for creative ways to take the standard (and sometimes not-so-standard) system administration tools and techniques and use them in ways no one would expect. We'll also cover some tools you may have missed.

Topics include:

• How to (ab)use perfectly good network transports by using them for purposes never dreamed of by their authors
• How to increase user satisfaction during downtimes with 6 lines of Perl
• How to improve your network services by intentionally throwing away data
• How to drive annoying Web-only applications that don't have a command line interface—without lifting a finger
• How to use ordinary objects you have lying around the house, such as Silly Putty, to make your life easier (seriously!)

David N. Blank-Edelman (S11, M4) is the Director of Technology at the Northeastern University College of Computer and Information Science and the author of the O'Reilly book Perl for System Administration. He has spent the last 19 years as a system/network administrator in large multi-platform environments, including Brandeis University, Cambridge Technology Group, and the MIT Media Laboratory. He has given several successful invited talks off the beaten path at LISA and is the LISA '05 Program Chair.

S12 Documentation Techniques for SysAdmins
Mike Ciavarella, University of Melbourne
1:30 p.m.–5:00 p.m.

Who should attend: System administrators who need to produce documention for the systems they manage or who want to improve their documentation skills.

Attendees should be able to make immediate, practical use of the techniques presented in this tutorial in their day-to-day tasks. Particular emphasis is placed on documentation as a time-saving tool rather than a workload imposition.

Topics include:

• Why system administrators need to document
• The document life cycle
• Targeting your audience
• An adaptable document framework
• Common mistakes
• Tools to assist the documentation process

Mike Ciavarella (S7, S12, M7) has been producing and editing technical documentation since he naively agreed to write application manuals for his first employer in the early 1980s. He has been a technical editor for MacMillan Press and has been teaching system administrators about documentation for the past eight years. Mike has an Honours Degree in Science from the University of Melbourne. After a number of years working as Senior Partner and head of the Security Practice for Cybersource Pty Ltd, Mike returned to his alma mater, the University of Melbourne. He now divides his time between teaching software engineering, providing expert testimony in computer security matters, and trying to complete a Doctorate. In his ever-diminishing spare time, Mike is a caffeine addict and photographer.

S13 Troubleshooting: A Basic Skill
Geoff Halprin, The SysAdmin Group
1:30 p.m.–5:00 p.m.

Who should attend: System administrators wishing to hone their ability to troubleshoot a problem under pressure, on a system of which their knowledge may be limited.

One of the most basic skills a system administrator must be able to call upon is that of problem diagnosis and resolution, that is, troubleshooting. It doesn't matter what else you do; if the system is broken, your priority is to fix it.

Topics include:

• A general process for troubleshooting
• Specific techniques that will help you get to the root of the problem
• Ways to identify candidate solutions with confidence

Geoff Halprin (S13, T8) has spent over 25 years as a software developer, system administrator, consultant, and troubleshooter. He has written software from system management tools to mission-critical billing systems, has built and run networks for enterprises of all sizes, and has been called upon to diagnose problems in every aspect of computing infrastructure and software. He has spent more years troubleshooting other people's systems and programs than he cares to remember. Geoff was on the board of the System Administrators Guild (SAGE) and is now a member of the USENIX board of directors.

Who should attend: System and database administrators, software architects, developers and programmers, performance and systems analysts, and IT architects wanting to obtain a deeper understanding of the key Solaris subsystems, as well as the tools and facilities that can be used to:

• Observe, trace, and debug to optimize performance
• Observe, trace, and debug to root-cause aberrent behavior
• Observe and trace to understand how the application workload interacts with the operating system
• Better understand the system as a whole

Applications are becoming more complex every day, and many of the new Solaris features significantly reduce the effort required to administer and anazlyze performance of the entire application and operating system stack.

You may take this class as either a one-day experts class or a two-day complete class. On Day One, we provide an architectual overview of the major Solaris subsystems and an introduction to Solaris performance analysis. On Day Two, we cover advanced topics and spend significant time with hands-on case studies, using the latest tools, including dtrace, mdb, memtool, mdb, trapstat and the Solaris process "ptools."

Topics include:

DAY ONE:

• Solaris overview with performance monitoring and tuning
  • Introduction to the Solaris kernel
  • Using DTrace for performance optimization
  • Overview of Solaris perf tools
• Memory
  • Overview of Solaris virtual memory
  • Observing and managing memory
  • Understanding memory utilization, optimizing, and monitoring
• Process managment & scheduling
  • Introduction to the Solaris process and thread model
  • Developing and tuning multi-threaded processes
  • Observing debugging processes with the ptools
  • Controlling processes with ptools
  • Introduction to scheduling
• Filesystems
  • Overview of Solaris file system architecture
  • Understanding caching
  • Measurement and tuning
  • Filesystems in Solaris: UFS, NFS, and the new S10 ZFS
• Networking and I/O
  • Solaris I/O architecture
  • TCP/IP recap
  • Solaris network performance and tuning

• Solaris observability and debugging tools
  • Mastering Solaris DTrace
  • How to debug/monitor with "mdb"
  • Kernel profiling and lock statistics with lockstat
  • Application lock statistics with plockstat
• Advanced memory architecture and tuning
  • TLB analysis using trapstat
  • Using large pages with the MPSS features
  • NUMA memory allocation and techniques
• Filesystem performance
  • Tools for measuring and characterizing
  • Analysing file system performance using dtrace
• Advanced thread scheduling and tools
  • Thread scheduling, parking lots and queues
  • Tracking thread priorities and sleep events
  • Using CPU binding and processor sets
• Advanced dtrace
  • Attributing network, file I/O to applications
  • Investigating complex inter-process performance problems
  • Tracing unmodified customer applications
• Workload consolidation and resource management
  • Introduction to tools for workload and resource management
  • Workload measurement
  • Using Solaris resource manager to isolate and control workloads
  • Using Solaris Zones to create Application Containers

James Mauro (T2, W2) is a Senior Staff Engineer in the Performance and Availability Engineering group at Sun Microsystems. Jim's current projects are focused on quantifying and improving enterprise platform availability, including minimizing recovery times for data services and Solaris. Jim co-developed a framework for system availability measurement and benchmarking and is working on implementing this framework within Sun.

Richard McDougall (T2, W2) is a Sun Microsystems Distinguished Engineer who specializes in operating systems technology and system performance. He is based at the Menlo Park Performance and Availability Engineering group, where he drives development of performance and behavior enhancements to the Solaris operating system and Sun's hardware architectures. He has led the development of resource management principles, has contributed to the development of virtual memory and file systems within the Solaris operating system, and has architected many tools for analysis, monitoring, and capacity planning. He is the lead author of Resource Management (Prentice Hall). He has written numerous articles and papers on measurement, monitoring, and capacity planning of Solaris systems and frequently speaks at industry and customer technical conferences on the topics of system performance and resource management.

Richard and Jim authored Solaris Internals: Architecture Tips and Techniques (Sun Microsystems Press/Prentice Hall, Feb 2000, ISBN 0-13-022496-0) and are currently collaborating on an update of the book for Solaris 8, as well as volume II.

T3 Solaris 10 Security Features Workshop
Peter Baer Galvin, Corporate Technologies
9:00 a.m.–5:00 p.m.

Who should attend: Solaris systems managers and administrators interested in the new security features in Solaris 10 (and features in previous Solaris releases that they may not be using).

This course covers a variety of topics surrounding Solaris 10 and security. Solaris 10 includes many new features, and there are new issues to consider when deploying, implementing, and managing Solaris 10. This will be a workshop featuring instruction and practice/exploration. Each student should have a laptop with wireless access for remote access into a Solaris 10 machine.

Topics include:

• Solaris cryptographic framework
• NFSv4
• Solaris privileges
• Solaris Flash archives and live upgrade
• Moving from NIS to LDAP
• Dtrace
• WBEM
• Smartcard interfaces and APIs
• Kerberos enhancements
• Zones
• FTP client and server enhancements
• PAM enhancements
• Auditing enhancements
• Password history checking
• ipfilters

Peter Baer Galvin (S8, T3) is the Chief Technologist for Corporate Technologies, Inc., a systems integrator and VAR, and was the Systems Manager for Brown University's Computer Science Department. He has written articles for Byte and other magazines. He wrote the "Pete's Wicked World" and "Pete's Super Systems" columns at SunWorld. He is currently contributing editor for Sys Admin, where he manages the Solaris Corner. Peter is co-author of the Operating Systems Concepts and Applied Operating Systems Concepts textbooks. As a consultant and trainer, Peter has taught tutorials on security and system administration and has given talks at many conferences and institutions on such topics as Web services, performance tuning, and high availability.

T4 Advanced Perl Programming
Tom Christiansen, Consultant
9:00 a.m.–5:00 p.m.

Who should attend: Anyone with a journeyman-level knowledge of Perl programming who wants to hone Perl skills. This class will cover a wide variety of advanced topics in Perl, including many insights and tricks for using these features effectively. After completing this class, attendees will have a much richer understanding of Perl and will be better able to make it part of their daily routine.

Topics include:

• Symbol tables and typeglobs
• Symbolic references
• Useful typeglob tricks (aliasing)
• Modules
• Autoloading
• Overriding built-ins
• Mechanics of exporting
• Function prototypes
• References
• Implications of reference counting
• Using weak references for self-referential data structures
• Autovivification
• Data structure management, including serialization and persistence
• Closures
• Fancy object-oriented programming
• Using closures and other peculiar referents as objects
• Overloading of operators, literals, and more
• Tied objects
• Managing exceptions and warnings
• When die and eval are too primitive for your taste
• The use warnings pragma
• Creating your own warnings classes for modules and objects
• Regular expressions
• Debugging regexes
• qr// operator
• Backtracking avoidance
• Interpolation subtleties
• Embedding code in regexes
• Programming with multiple processes or threads
• The thread model
• The fork model
• Shared memory controls
• Unicode and I/O layers
• Named Unicode characters
• Accessing Unicode properties
• Unicode combined characters
• I/O layers for encoding translation
• Upgrading legacy text files to Unicode
• Unicode display tips
• What's new in Perl lately
• Switch statement
• Defined-or operators
• Pre-compiled modules
• Dynamic handles
• Virtual I/O through strings

Tom Christiansen (T4) has been involved with Perl since day zero of its initial public release in 1987. Author of several books on Perl, including The Perl Cookbook and Programming Perl from O'Reilly, Tom is also a major contributor to Perl's online documentation. He holds undergraduate degrees in computer science and Spanish and a Master's in computer science. He now lives in Boulder, Colorado.

T5 RPM Package Management
Joshua Jensen, IBM
9:00 a.m.–5:00 p.m.

Who should attend: System administrators deploying, or interested in deploying, RPM-based Linux systems in a production environment. Attendees should be familiar with the basics of system administration in a UNIX/Linux environment, user-level commands and TCP/IP networking. Novice administrators and gurus alike should leave the tutorial having learned something.

Whether your environment is a single server or a network with thousands of desktops, workstations, and servers, Linux application deployment, upgrades, and errata policy can be effectively managed with RPM packages. From simple command-line queries to source build environments, from networked package management solutions such as Novell's Zenworks for Linux and Red Hat's RHN to the simple but effective Yum, this course emphasizes real-world solutions, covering everything you need to know to use, create, and manage RPM packages and systems.

Topics include:

• Introduction to RPM: What's a package and what isn't
• Working with RPMs: Basic functionality explored
• Advanced RPM use: Auto dependency aid, rollback, and more
• Building RPMs: Source RPMs, spec files, RPM macros
• Special considerations for dual architecture systems
• Advanced functionality: Triggers and pre/post scripting
• RPM security: Build signing and pre-installation verification
• Package management systems: APT, RHN, RCE/Zenworks, Yum

Joshua Jensen (T5) has worked for IBM and Cisco Systems, and was Red Hat's first instructor, examiner, and RHCE. He worked with Red Hat for 4 1/2 years during which he wrote and maintained large parts of the Red Hat curriculum: Networking Services and Security, System Administration, Apache and Secure Web Server Administration, and the Red Hat Certified Engineer course and exam. Joshua has been working with Linux since 1996, and finds himself full circle having recently left IBM to work with Red Hat Linux for Cisco Systems. In his spare time he dabbles in cats, fish, boats, and frequent flyer miles.

T6 Managing Samba 3.0
Gerald Carter, Samba Team/Hewlett-Packard
9:00 a.m.–5:00 p.m.

Who should attend: System administrators who are currently managing Samba servers or are planning to deploy new servers this year. This course will outline the new features of Samba 3.0, including working demonstrations throughout the course session.

Topics include:

• Providing basic file and print services
• Centrally managing printer drivers for Windows clients
• Configure Samba's support for Access Control Lists and the Microsoft Distributed File System
• Making use of Samba VFS modules for features such as virus scanning and a network recycle bin
• Integrating with Windows NT 4.0 and Active Directory authentication services
• Implementing a Samba primary domain controller along with Samba backup domain controllers
• Migrating from a Windows NT 4.0 domain to a Samba domain
• Utilizing account storage alternatives to smbpasswd such as LDAP

Gerald Carter (S6, T6, W3) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing.

T7 Practical System and Network Monitoring
John Sellens, SYONEX
9:00 a.m.–5:00 p.m.

Who should attend: Network and system administrators interested in real-life, practical, host- and network-based monitoring of their systems and networks. Participants should have an understanding of the fundamentals of networking, basic familiarity with computing and network components, and some familiarity with UNIX and scripting languages.

Participants will leave this tutorial able to immediately start using a number of monitoring systems and techniques that will improve their ability to manage and maintain their systems and networks.

Topics include:

• Monitoring: goals, techniques, reporting
• SNMP: the protocol, reference materials, relevant RFCs
• Introduction to SNMP MIBs (Management Information Bases)
• SNMP tools and libraries
• Other (non-SNMP) tools
• Security concerns when using SNMP and other tools on the network
• Monitoring applications: introductions, use, benefits and complications, installation and configuration (Big Brother, Nagios, SNIPS, MRTG, Cricket, etc.)
• Special situations: remote locations, firewalls, etc.
• Monitoring implementation roadmap: policies, practices, notifications, escalations, reporting

John Sellens (T7, W4) has been involved in system and network administration since 1986 and is the author of several related USENIX papers, a number of ;login: articles, and the SAGE Short Topics in System Administration booklet #7, System and Network Administration for Higher Reliability. He holds an M.Math. in computer science from the University of Waterloo and is a chartered accountant. He is the proprietor of SYONEX, a systems and networks consultancy. From 1999 to 2004, he was the General Manager for Certainty Solutions in Toronto. Prior to joining Certainty, John was the Director of Network Engineering at UUNET Canada and was a staff member in computing and information technology at the University of Waterloo for 11 years.

T8 Release Engineering and Project Lifecycle
Geoff Halprin, The Sysadmin Group, and Lee Damon, University of Washington
9:00 a.m.–5:00 p.m.

Who should attend: Software developers, system administrators, and managers who deal with internal or external project/product lifecycles and quality.

We will look at projects varying in scope from "Here's a new product we just bought; roll it out," through "We need an internal _foo_ server; make it happen," to "Here's this new thing we are developing; let's do it right so we can ship it." We will cover matters from quick projects to "This will take a year and 20 people to deploy."

The focus of this class will be on internally developed projects for internal use with some extrapolation to sold or shipped products. Large and small projects alike can benefit from proper planning and roll-outs. We will prove that the old adage, "There is never time to do it right, but there is always time to do it over," is never a good answer or a good philosophy. We will also examine the phenomenon of "Shoot the engineer and ship the product" in light of the reality most of us face: "The first 90% of the project takes 10% of the time; the remaining 10% takes 90% of the time."

Students should have a project or large task-set in mind when coming to the class. There will be break-out sessions where the student's real world tasks will be used as examples.

This is not intended to replace formal training for project planners. It is intended to make life better for people who deal with projects on a day-to-day basis. We will provide you with the ideas and some of the tools to take your projects from "We need this quick" to a supportable and released project—or even product.

Topics include:

• Test planning and implementation
• Project release planning
• Project lifecycles
• Replacement planning and updating

Geoff Halprin (S13, T8) has spent over 25 years as a software developer, system administrator, consultant, and troubleshooter. He has written software from system management tools to mission-critical billing systems, has built and run networks for enterprises of all sizes, and has been called upon to diagnose problems in every aspect of computing infrastructure and software. He has spent more years troubleshooting other people's systems and programs than he cares to remember. Geoff was on the board of the System Administrators Guild (SAGE) and is now a member of the USENIX board of directors.

Lee Damon (M7, T8) has a B.S. in Speech Communication from Oregon State University. He has been a UNIX system administrator since 1985 and has been active in SAGE since its inception. He assisted in developing a mixed AIX/SunOS environment at IBM Watson Research and has developed mixed environments for Gulfstream Aerospace and QUALCOMM. He is currently leading the development effort for the Nikola project at the University of Washington Electrical Engineering Department. He is past chair of the SAGE Ethics and Policies Working Groups.

See Part 1, T2, for the description of the first day of this tutorial.

Who should attend: System and database administrators, software architects, developers and programmers, performance and systems analysts, and IT architects wanting to obtain a deeper understanding of the key Solaris subsystems, as well as the tools and facilities that can be used to:

• Observe, trace, and debug to optimize performance
• Observe, trace, and debug to root-cause aberrent behavior
• Observe and trace to understand how the application workload interacts with the operating system
• Better understand the system as a whole

Applications are becoming more complex every day, and many of the new Solaris features significantly reduce the effort required to administer and anazlyze performance of the entire application and operating system stack.

You may take this class as either a one-day experts class or a two-day complete class. On Day One, we provide an architectual overview of the major Solaris subsystems and an introduction to Solaris performance analysis. On Day Two, we cover advanced topics and spend significant time with hands-on case studies, using the latest tools, including dtrace, mdb, memtool, mdb, trapstat and the Solaris process "ptools."

Topics include:

• Solaris observability and debugging tools
  • Mastering Solaris DTrace
  • How to debug/monitor with "mdb"
  • Kernel profiling and lock statistics with lockstat
  • Application lock statistics with plockstat
• Advanced memory architecture and tuning
  • TLB analysis using trapstat
  • Using large pages with the MPSS features
  • NUMA memory allocation and techniques
• File system performance
  • Tools for measuring and characterizing
  • Analysing file system performance using dtrace
• Advanced thread scheduling and tools
  • Thread scheduling, parking lots and queues
  • Tracking thread priorities and sleep events
  • Using CPU binding and processor sets
• Advanced dtrace
  • Attributing network, file I/O to applications
  • Investigating complex inter-process performance problems
  • Tracing unmodified customer applications
• Workload consolidation and resource management
  • Introduction to tools for workload and resource management
  • Workload measurement
  • Using Solaris resource manager to isolate and control workloads
  • Using Solaris Zones to create Application Containers

James Mauro (T2, W2) is a Senior Staff Engineer in the Performance and Availability Engineering group at Sun Microsystems. Jim's current projects are focused on quantifying and improving enterprise platform availability, including minimizing recovery times for data services and Solaris. Jim co-developed a framework for system availability measurement and benchmarking and is working on implementing this framework within Sun.

Richard McDougall (T2, W2) is a Sun Microsystems Distinguished Engineer who specializes in operating systems technology and system performance. He is based at the Menlo Park Performance and Availability Engineering group, where he drives development of performance and behavior enhancements to the Solaris operating system and Sun's hardware architectures. He has led the development of resource management principles, has contributed to the development of virtual memory and file systems within the Solaris operating system, and has architected many tools for analysis, monitoring, and capacity planning. He is the lead author of Resource Management (Prentice Hall). He has written numerous articles and papers on measurement, monitoring, and capacity planning of Solaris systems and frequently speaks at industry and customer technical conferences on the topics of system performance and resource management.

Richard and Jim authored Solaris Internals: Architecture Tips and Techniques (Sun Microsystems Press/Prentice Hall, Feb 2000, ISBN 0-13-022496-0) and are currently collaborating on an update of the book for Solaris 8, as well as volume II.

W3 Implementing LDAP Directories
Gerald Carter, Samba Team/Hewlett-Packard
9:00 a.m.–5:00 p.m.

Who should attend: Both LDAP directory administrators and architects. The focus is on integrating standard network services with LDAP directories. The examples are based on UNIX hosts and the OpenLDAP directory server and will include actual working demonstrations throughout the course.

System administrators today run a variety of directory services, although these are referred to by names such as DNS and NIS. The Lightweight Directory Access Protocol (LDAP) is the up-and-coming successor to the X500 directory and has the promise of allowing administrators to consolidate multiple existing directories into one.

Topics include:

• Replacing NIS domains
• Integrating Samba user accounts
• Integrating MTAs such as Sendmail, Qmail, or Postfix
• Creating address books for mail clients
• Managing user access to HTTP and FTP services
• Integrating with DHCP & DNS servers
• Scripting with the Net::LDAP Perl module
• Defining custom attributes and object classes

Gerald Carter (S6, T6, W3) has been a member of the Samba Development Team since 1998. He has published articles with various Web-based magazines and teaches courses as a consultant for several companies. Currently employed by Hewlett-Packard as a Samba developer, Gerald has written books for SAMS Publishing and is the author of the recent LDAP System Administration for O'Reilly Publishing.

W4 System and Network Monitoring: Tools in Depth
John Sellens, SYONEX
9:00 a.m.–5:00 p.m.

Who should attend: Network and system administrators ready to implement comprehensive monitoring of their systems and networks using the best of the freely available tools. Participants should have an understanding of the fundamentals of networking, familiarity with computing and network components, UNIX system administration experience, and some understanding of UNIX programming and scripting languages.

This tutorial will provide in-depth instruction in the installation and configuration of some of the most popular and effective system and network monitoring tools, including Nagios, Cricket, MRTG, and Orca.

Participants should expect to leave the tutorial with the information needed to immediately implement, extend, and manage popular monitoring tools on their systems and networks.

Topics include, for each of Nagios, Cricket, MRTG, and Orca:

• Installation—Basic steps, prerequisites, common problems, and solutions
• Configuration, setup options, and how to manage larger and non-trivial configurations
• Reporting and notifications—proactive and reactive
• Special cases—how to deal with interesting problems
• Extending the tools—how to write scripts or programs to extend the functionality of the basic package
• Dealing effectively with network boundaries and remote sites
• Security concerns and access control
• Ongoing operation

John Sellens (T7, W4) has been involved in system and network administration since 1986 and is the author of several related USENIX papers, a number of ;login: articles, and the SAGE Short Topics in System Administration booklet #7, System and Network Administration for Higher Reliability. He holds an M.Math. in computer science from the University of Waterloo and is a chartered accountant. He is the proprietor of SYONEX, a systems and networks consultancy. From 1999 to 2004, he was the General Manager for Certainty Solutions in Toronto. Prior to joining Certainty, John was the Director of Network Engineering at UUNET Canada and was a staff member in computing and information technology at the University of Waterloo for 11 years.

W5 Administering Linux in Production Environments
Æleen Frisch, Exponential Consulting
9:00 a.m.–5:00 p.m.

Who should attend: Both current Linux system administrators and administrators from sites considering converting to Linux or adding Linux systems to their current computing resources. We will be focusing on the administrative issues that arise when Linux systems are deployed to address a variety of real-world tasks and problems arising from both commercial and research-and-development contexts.

Topics include:

• Recent kernel developments
• High-performance I/O
  • Advanced filesystems and logical volumes
  • Disk striping
  • Optimizing I/O performance
• Advanced compute-server environments
• Beowulf
• Clustering
• Parallelization environments/facilities
• CPU performance optimization
• High availability Linux: fault tolerance options
• Enterprise-wide authentication
• Fixing the security problems you didn't know you had (or, what's good enough for the researcher/hobbyist won't do for you)
• Automating installations and other mass operations
• Linux in the office environment

Æleen Frisch (W5) has been a system administrator for over 20 years. She currently looks after a pathologically heterogeneous network of UNIX and Windows systems. She is the author of several books, including Essential System Administration (now in its 3rd edition).

 

Richard Bejtlich (R2) is technical director for specialized security monitoring in ManTech International Corporation's Computer Forensics and Intrusion Analysis division. He was previously a principal consultant at Foundstone, performing incident response, emergency network security monitoring, and security research. Prior to joining Foundstone in 2002, Richard served as senior engineer for managed network security operations at Ball Aerospace & Technologies Corporation. From 1998 to 2001 Richard defended global American information assets as a captain in the Air Force Computer Emergency Response Team (AFCERT). He led the AFCERT's real time intrusion detection mission, supervising 60 civilian and military analysts. He is the author of The Tao of Network Security Monitoring: Beyond Intrusion Detection and the co-author of the forthcoming Real Digital Forensics, both published by Addison-Wesley. He also wrote original material for Hacking Exposed, 4th Edition, and Incident Response, 2nd Edition, both published by McGraw-Hill/Osborne. He acquired his CISSP certification in 2001 and CIFI credentials in 2004. His home page is https://www.taosecurity.com and his popular Web log resides at https://taosecurity.blogspot.com.

R3 Configuration Management with Cfengine
Mark Burgess, Oslo University College
9:00 a.m.–5:00 p.m.

Who should attend: System administrators with a basic knowledge of scripting who wish to get to grips with cfengine to automate the maintenance and security of their systems. UNIX administrators will be most at home in this tutorial, but cfengine can also be used on Windows 2000 and above. This tutorial works as a guide to the extensive documentation, focusing pragmatically on the key issues and filtering out details.

Cfengine is a tool for setting up and maintaining a configuration across a network of hosts. It is sometimes called a tool for "Computer Immunology"—your computer's own immune system. You can think of cfengine as a very high-level language, much higher-level than Perl or shell, together with a smart agent. The idea behind cfengine is to create a single "policy" or set of configuration files that describes the setup of every host on your network, without sacrificing their autonomy.

Cfengine runs on every host and makes sure that it is in a policy-conformant state; if necessary, any deviations from policy rules are fixed automatically. Unlike tools such as rdist, cfengine does not require hosts to open themselves to any central authority nor to subscribe to a fixed image of files. It is a modern tool, supporting state-of-the-art encryption and IPv6 transport, that can handle distribution and customization of system resources in huge networks (tens of thousands of hosts). Cfengine runs on hundreds of thousands of computers all over the world.

Topics include:

• The components of cfengine and how they are used
• How to get the system running
• How to develop a suitable policy, step by step
• Security
• Organizing configuration files (updating and configuring)
• Ordering issues in configuration management
• Cfservd security and key deployment
• Searching for data with filters
• Special functions and arrays
• Alerts and persistent classes
• Multi-homed host issues
• IPv6 issues
• Methods and modules and when to use them
• Host monitoring with FriendStatus
• Anomaly detection and response with cfenvd
• What is coming in cfengine?

Mark Burgess (R3) is a professor at Oslo University College and is the author of cfengine. He has been researching the principles of network and system administration for over ten years and is the author of Principles of Network and System Administration (John Wiley & Sons). He is frequently invited to speak at conferences.