The Observation Deck

Close this search box.

Month: June 2005

I spent the day today at JavaOne in the City, where DTrace was featured prominently in the morning keynote: JohnnyL described DTrace in the abstract, and drew a (surprisingly large!) round of applause when he announced that DTrace now supports Java. He then called Adam on stage to do a demo,
the most remarkable aspect of which is that it was actually DTrace and not the canned, gussied-up gimcrack that normally passes for a keynote demo. In fact, this demo was so tangibly real, Adam has blogged the scripts from the demo. With these scripts, you can now give Adam’s keynote in the comfort of your own home!

In other JavaOne news, Adam, Stephen and I are going to be leading the charge at an OpenSolaris BOF tonight at 7:30pm at the San Francisco Marriot in Golden Gate B1. With the IBM announcement today supporting Solaris, with the new super-fast, super-cheap Solaris Opteron workstation, and with new Solaris technologies like DTrace and Zones, Solaris has never been more attractive to the Java developer; swing by our BOF to see what all the fuss is about!

Technorati tags:

Even though the launch of
OpenSolaris was well over a week ago,
and even though the
Opening Day entries
have now been sifted through in
five different blog entries
here and
there are still
some great uncategorized entries. Without further ado…

Phew! I think that about does it. When we first tried to encourage
Solaris engineers to blog on Opening Day, I thought we were going to have
a hard time convincing engineers to blog — I knew that providing in-depth,
content takes a lot of time, and I knew that everyone had other priorities.
So when we were planning the launch and talking
about the possibility of dealing with a massive amount of Opening Day content,
my response was “hurt me
with that problem.” Well, as it turns out, most engineers didn’t need
much convincing — many provided rich, deep content — and I was
indeed hurt with that problem! While it was time consuming to sift
through them, hopefully
you’ve enjoyed reading these entries as much as I have. And let it be
said once more: welcome to OpenSolaris!

Technorati tags:

Despite there being now four blog entries to sift through the
Opening Day entries
(one from me,
one from Liane,
one from Claire
and another from me),
there are still some great entries that have gone uncategorized. Many
of these entries fall into the category of “debugging war stories” —
describing a particularly tough or interesting bug that they nailed.
proliferation of these kinds of stories among the Opening Day entries
is revealing: in
some organizations,
you’re a hero among engineers if you come up with a whizzy new app
(even if the implementation is a cesspool),
while in
others you might gain respect among engineers by getting a
big performance win
(even if it makes things a bit flakey),
but in Solaris
development, nothing gains the respect of peers like debugging some
diabolical bug that has plagued the operating system for years.
So if you’re new to
and you’re looking to make a name for yourself, a good place to start is
in the never-ending search for elusive bugs.
To get a flavor for the thrill of the hunt, check out:

And here are a few more on the simple (but thorough) satisfaction from
fixing old bugs:

That about does it for today. There are still many more entries to
categorize, but fortunately, I think I can see the light at the end of the
tunnel — or is that just the approach of death from the exhaustion of
sifting through
all of
this content

Technorati tags:

If you didn’t see it,
Liane Praza
picked up where
my sifting
left off, adding
a blog entry pointing to
more Opening Day entries
— this
in the categories of
devices and device configuration, security, networking,
and standards. But there are still a ton of entries to
categorize, so picking up again in no particular order…

  • System calls.
    System calls are the among most fundamental mechanisms in operating systems:
    they are the mechanism by which untrusted, unprivileged software requests
    a service of trusted, privileged software. We are lucky to have two
    great entries describing the architecture-specific mechanisms of
    system calls in Solaris:
    check out
    Russ Blaine’s entry
    system calls on x86, and
    Gavin Maltby’s
    entry on
    calls on SPARC
    . Then, to understand the architectural-neutral aspects of
    system calls, head over to
    Eric Schrock‘s
    entry on
    how to add a system call.

    As a quick aside, that
    last entry is a great example of how we in Solaris Kernel Development
    are using blogs to write
    down information that (believe it or not) has just been an unspoked part
    of the craft before now. As
    Tim Bray observed,
    blogs have become a critical conduit of information for us — we believe
    that they are the most scalable way to get information from the
    people who have it to the people who need it. If (when?) you become
    an OpenSolaris developer,
    you can expect some friendly peer pressure to create a blog and
    join the party.

  • Build process and workspace management.
    We pride ourselves on a seamless build process,
    and a couple of entries have gone into various aspects of this in depth.
    To give you an idea of how seriously we take the build process — and
    why — check out
    Scott Rotondo’s
    entry on using lint to find security vulnerabilities.
    In particular, note what Scott says when he added a new lint option that
    500 new warnings: “I needed to fix all of these before integrating
    my change to
    Makefile.master because we require the Solaris source to be
    lint-clean.” To which I add only, “dammit.”
    Next, head over to
    Jim Carlson’s
    entry describing the work he did to support
    non-root builds. Jim’s entry demonstrates how difficult it is to
    radically change the build process — and how he managed to pull it off.
    Finally, if you want to really let your makefile flag fly,
    check out
    Mark Nelson’s
    entry describing the build support for localized messages.

    In terms of workspace management, you’ll want to check out
    Will Fiveash’s
    entry describing our workspace management tool, wx. For a long
    time, wx
    was a shell script in
    Bonwick’s home directory.
    It was incredibly useful, but it was also easy to accidentally blow your
    brains out.
    Bart is fond of saying, it
    was “all blade and no handle.”) Will’s rewrite made for a much more
    safer, much more sophisticated wx — and it was a huge help to
    us in automating the final approach of the
    DTrace integration.

  • Debuggability. If you read just a couple of the
    Opening Day entries,
    you probably noticed a trend: many of the entries were about finding
    some nasty bug in the system.
    This is an accurate reflection of our ethos in developing Solaris:
    the operating system must be reliable above all else, and we view
    debugging the operating system as our primary responsibility.
    This responsibility runs deeper than just the act of debugging, because
    our needs so outstripped existing tools that

    we designed and built
    our own
    — most notably
    and DTrace.
    Fortunately, we ship these tools to you, so you can use them on your
    own system and on your own applications.

    There are many entries describing these tools and how they were used
    to tackle a problem.
    Fittingly, a good place to start is
    Mike Shapiro’s
    entry describing using mdb to debug a sendmail bug. This bug is
    described in
    which has one of the
    greatest bug synopses of all time: “sendmail died in a two SIGALRM fire.”
    For more on the power of mdb,
    take a look at
    Eric Saxe’s
    entry on
    using mdb to debug a scheduling problem,
    Ashish Mehta’s entry
    mdb to debug
    a race condition
    , and
    Eric Kustarz’s entry demonstrating an mdb debugger command (“dcmd”) that he wrote to
    retrieve NFSv4 recovery messages postmortem.
    This last example is a particularly good one
    because this is exactly the kind of custom debugging
    infrastructure that mdb’s modular architecture makes easy to build.
    For a comprehensive example of how we have developed subsystem-specific
    debugging infrastructure, read
    Sasha Kolbasov’s
    entry on the
    dcmds related to STREAMS
    As Sasha mentions, the place to start for learning to write your
    own modules is the
    but you can get a flavor for it by reading
    Yu Xiangning’s
    entry on writing a
    a module for kmdb
    kmdb is the in-situ kernel debugger that implements mdb, and when you
    need it, nothing else will do — as
    Dan Mick describes
    in his entry on debugging with kmdb and moddebug.
    For more details on kmdb itself,
    check out
    Matt Simmons’
    entry on
    kmdb’s design and implementation.
    To see how mdb can help debug your application, take a look at
    Will Fiveash’s notes
    on using debugging application memory problems. Will
    mentions ::findleaks, a debugger command that I originally
    implemented for kernel crash dumps, and that
    Jonathan Adams
    ported to work on application core files and — as he mentions in
    his entry —
    reworking it substantially in the process.

    While mdb is the acme of postmortem debugging,
    if the manifestation of a bug is non-fatal, it’s often more
    to use DTrace to debug it.
    For an exanple of this,
    look at
    Bart Smaalders’
    entry on using DTrace to debug jitter.
    It was gratifying to see Bart debug this problem using DTrace, because
    latency bubbles were actually one of the motivating pathologies behind
    And finally, debuggability doesn’t end with tools; subsystems must be
    designed with
    debuggability in mind, as
    Stephen Hahn
    describes in his entry on
    designing libuutil for debuggability.

I think that about does it for today. As someone pointed out on Liane’s
blog, we need a Wiki for this; we agree — it’s on the list of planned
enhancements for Until then,
stay tuned for more sifting…

Technorati tags:

For those of you in the Bay Area, I will be giving a talk and demo on DTrace at tonight’s
BayLISA meeting in Cupertino.
Starts at 7:30p, here are the directions. I will be demo’ing DTrace tonight on my laptop running OpenSolaris bits that I downloaded and built myself from outside of Sun’s internal network. So if you’re interested in DTrace, OpenSolaris or both (and you’re in the Bay Area), you might want to check it out…

Technorati tags:

Yesterday was Opening Day for
we welcomed OpenSolaris with
of blog entries

various aspects of
the implementation.
The breadth and depth of our blogging
will hopefully
put to rest any notion that open sourcing Solaris isn’t a grass-roots
effort: if nothing else, it should be clear that we in the trenches
are very excited to finally be able to talk about the system
that we have poured so much of our lives into — and to welcome
new would-be contributors into the fold.

In our excitement, we may have overwhelmed a tad:
there was so much content yesterday, that it would have been impossible
for anyone to keep up — we blogged over 200,000 words (over 800 pages!)
yesterday alone.
So over the next few days, I want to highlight some entries that you
might have missed, broken down by subject area. In no particular order…

  • Fault management. Fault management in Solaris 10 has been completely
    revolutionized by the new predictive self-healing feature pioneered
    by my longtime co-conspirator
    Mike Shapiro. There are
    two must-read entries in this area:
    Andy Rudoff’s entry
    providing a
    predictive self-healing overview, and
    Dilpreet Bindra‘s
    entry going into more depth on PCI error handling. (If for nothing
    read Dilpreet’s entry for his Reading of the Vows between OpenSolaris and the

  • Virtual memory.
    The virtual memory system is core to any modern operating system, and
    there are several interesting entries here.
    Start with
    Eric Lowe‘s
    extensive entry
    describing page fault handling. As Eric rightly points out,
    page fault handling is the epicenter of the VM system; one can learn a
    tremendous amount about the system just by following page fault processing —
    and Eric is a great guide on this journey.
    Once you’ve read Eric’s entry,
    check out Michael Corcoran‘s
    entry on page coalescing,
    a technique to assure availability of
    large-sized pages — which are in turn necessary to increase TLB reach.
    And discussion of page_t‘s leads naturally
    brings you to
    Rick Mesta
    entry describing a
    big performance win by
    these structures during boot.

    A less-discussed aspect of virtual memory is the virtual memory layout
    of the kernel itself. To learn about some of the complexities of this,
    check out
    Kit Chow’s entry
    on address space limitations on 32-bit kernels.
    The limitation that Kit describes is one of the nasty gotchas of running
    32-bit x86 in flat mode. As Kit mentions, the best workaround is to run
    a 64-bit kernel — but if you’re stuck with a 32-bit x86 chip, you’ll want
    to read Kit’s suggestions carefully. Kit’s entry is a good segue to
    Prakash Sangappa’s
    entry describing his work on
    dynamic segkp for 32-bit x86 systems. Prakash’s work was critical
    for getting some more breathing space on 32-bit x86 systems — saving hundreds
    of megabytes of precious VA. Of course, the ultimate breathing space is
    that afforded by 64 bits of VA — and in this vein check out
    Nils Nieuwejaar‘s
    entry on the kernel address space layout on x64. Both
    Prakash and Nils
    quote one of those comments in the kernel source code that you really need to
    know about if you’re going to do serious kernel development: the comment
    describing the address space layout in
    i86pc/os/startup.c and
    This comment is one of the canonical ASCII-art comments (more on these
    eventually), and I usually find these comments in startup.c by
    searching forward for “----“.

  • Linking and Loading. One of the most polished subsystems in Solaris
    is the linker and loader — the craftsmanship of the engineers that have
    built it has been an ongoing inspiration for many of us in Solaris
    development. To learn more about the linker,
    start with
    Rod Evans’ entry
    taking you on
    a source tour of the link-editors, and then head over to
    Mike Walker’s
    entry describing library bindings.
    As long as you’re checking out
    the linker, be sure to look at past entries like
    Rod’s entry
    tracing of a
    As you can imagine, because the
    dynamic linker is invoked whenever a dynamically-linked binary is executed,
    it’s a natural place to improve performance — especially with
    complicated programs like Mozilla or StarOffice that are linked
    to hundreds (!) of shared objects. We’ve certainly found some big wins
    in the linker over the years, but we’ve also discovered that it’s difficult
    to help megaprograms without hurting nanoprograms — and vice versa.
    For an interesting description of this tradeoff, check out
    David Safford’s
    entry on dynamic
    linker performance
    . If nothing else, you’ll see from David’s work
    the research element of operating system development: we often aren’t
    assured of success when we endeavor to improve the system.

  • Scheduling. CPU scheduling is one of the most basic properties
    of a multitasking operating system. Despite being an old problem,
    we find ourselves constantly improving and extending this subsystem.
    To learn about CPU scheduling, start with
    Bill Kucharski’s
    entry describing
    architecture-specific elements of context switching
    . Then head
    over to
    Gavin Maltby’s
    entry describing
    short-term prevention of thread migration
    . (Before Gavin introduced
    this facility, the only way to prevent migration was to prevent kernel
    preemption — an overly blunt mechanism that led to
    really nasty latency bubble
    that I debugged many years ago.)

    If you’re going to understand thread dispatching, you’ll need to understand
    the way thread state is manipulated — and for that you’ll want to look at
    Saurabh Mishra’s
    entry describing
    . Thread locks are different from normal synchronization primitives,
    as you can infer from
    my own entry describing
    bug in user-level
    priority inheritance

    which is a good segue to a more general problem when dealing with
    thread control: how does one change the scheduling properties of a
    running thread?
    For an idea of how tricky this can be,
    check out
    Andrei Dorofeev’s
    entry describing
    processes to resource pools
    Andrei’s problem was even more challenging than traditional thread
    manipulation, as he needed to
    change the scheduling properties of a group of threads atomically.
    If for no other reason,
    you should read Andrei’s entry to learn of the
    “curse of disp.c.”
    Speaking of the cursed, wrap up your tour of scheduling
    Eric Saxe’s entry describing
    a wedged kernel
    — you’ll see from Eric’s odyssey
    that scheduling problems can require a lot of brain-bending (and patience) to debug!

Okay, I think that’s enough for today — and yet it
barely scratches the surface! I didn’t even touch on gigantic
topics with many Opening Day entries
like security, networking, I/O, filesystems, performance, scheduling,
service management, observability, etc. etc. Stay tuned — or check out
Opening Day entries
for yourself…

Technorati tags:

Recent Posts

November 26, 2023
November 18, 2023
November 27, 2022
October 11, 2020
July 31, 2019
December 16, 2018
September 18, 2018
December 21, 2016
September 30, 2016
September 26, 2016
September 13, 2016
July 29, 2016
December 17, 2015
September 16, 2015
January 6, 2015
November 10, 2013
September 3, 2013
June 7, 2012
September 15, 2011
August 15, 2011
March 9, 2011
September 24, 2010
August 11, 2010
July 30, 2010
July 25, 2010
March 10, 2010
November 26, 2009
February 19, 2009
February 2, 2009
November 10, 2008
November 3, 2008
September 3, 2008
July 18, 2008
June 30, 2008
May 31, 2008
March 16, 2008
December 18, 2007
December 5, 2007
November 11, 2007
November 8, 2007
September 6, 2007
August 21, 2007
August 2, 2007
July 11, 2007
May 20, 2007
March 19, 2007
October 12, 2006
August 17, 2006
August 7, 2006
May 1, 2006
December 13, 2005
November 16, 2005
September 13, 2005
September 9, 2005
August 21, 2005
August 16, 2005