Improving Performance on Solaris, Linux and SmartOS

For many commercial companies the center of the Solaris world is at Oracle. However, for many of us that have lived and breathed Solaris, that center has shifted quite a while back to Joyent.  Oracle’s high licensing costs around a Solaris x86 license makes it uninteresting to many. Meanwhile, Joyent offers an open-source version of Solaris that actually offers a number of significant improvements. Joyent’s SmartOS has become one the most innovative operating systems available on the scene today.  A derivative of OpenSolaris – it has vastly improved on it, creating an excellent cloud-centric operating system with a wealth of features.  Improved are all the compelling Solaris technologies – zones, ZFS, dtrace, SMF, etc – but more important is how they have added new technologies (KVM) and offered a wealth of new features. One of the big wins with SmartOS is improved performance.  If you are interested you can find more information on SmartOS here :


One of the areas where Solaris and SmartOS excel is dtrace – a framework that allows  observation of the running operating system and it’s applications and processes at a very deep level. On more than one occasion I was able to observe the details of applications running in production by using dtrace.  It is typical to see huge performance wins by using dtrace to diagnose performance issues or simply examine running applications.  Twenty examples of performance wins – with the bulk coming from use of dtrace :



Brendan Gregg is a performance guru and he is really well worth listening to. One of the more interesting articles he has written compared Solaris Zones, KVM and Xen and looked at in detail at them.


He recently presented SCaLE 12x and his presentation is, as usual, informative. Discussing what Linux can learn from Solaris-based systems and vice-versa when it comes to performance.
Brendan Gregg has written a number of books on the topic of performance.  He co-authored a book on DTrace :

bgreggdtraceand also authored a comprehensive book on performance :



Recommended Reading : Systems Performance, SmartOS and the Cloud

There is a new book out, Systems Performance : Enterprise and the Cloud by Brenden Gregg, and if you have to put up with slow software, book_brendenthen on a Monday, you need this book.  If you don’t know who Brenden Gregg is, then you haven’t been paying attention to a revolution in some of he  software that came from his team at Sun just a few years back. And this book constitutes an excellent way at  looking at system performance.  As described on Amazon : “Large-scale enterprise, cloud, and virtualized computing systems have introduced serious performance challenges. Now, internationally renowned performance expert Brendan Gregg has brought together proven methodologies, tools, and metrics for analyzing and tuning even the most complex environments. Systems Performance: Enterprise and the Cloud focuses on Linux® and Unix® performance, while illuminating performance issues that are relevant to all operating systems. You’ll gain deep insight into how systems work and perform, and learn methodologies for analyzing and improving system and application performance. Gregg presents examples from bare-metal systems and virtualized cloud tenants running Linux-based Ubuntu®, Fedora®, CentOS, and the illumos-based Joyent® SmartOS™ and OmniTI OmniOS®. He systematically covers modern systems performance, including the “traditional” analysis of CPUs, memory, disks, and networks, and new areas including cloud computing and dynamic tracing. This book also helps you identify and fix the “unknown unknowns” of complex performance: bottlenecks that emerge from elements and interactions you were not aware of. The text concludes with a detailed case study, showing how a real cloud customer issue was analyzed from start to finish.”   You can get a quick overview of the book by looking at his presentation and for the first time being introduced to the DTrace pony (page 6, but we will come back to the DTrace pony later) you can experience modern performance tools.


Any so begins your magical mystery tour if you don’t know DTrace. If you haven’t experienced it you have missed out on a tool capable of literally

DTracegiving you deep observability into your system.  Though the best versions of DTrace run on open-source Solaris-based operating systems like SmartOS and illumos-based variants, their are also versions that run on Linux.  Couple with DTrace are utilization and latency heat maps and flame graphs which offer a visual view of what DTrace is reporting. As you get more interested in this tool you may want to examine what people like Brendan Gregg and Joyent are doing with DTrace in the Cloud –


Joyent is using a forked version of OpenSolaris based on the illumos (open source) version.  Joyent’s open source version, SmartOS,  which they have customized around cloud deployments provides a large number of features unavailable on Oracle’s version of Solaris. And because SmartOS leverages OpenSolaris, one more thing becomes useful, ZFS, one of the most sophisticated file systems available.


Another virtualization aspect of SmartOS is that it leverages Solaris zones (extremely lightweight) and Linux KVM (ported, debugged and tuned via DTrace).  And this all has given Joyent very special insight into dense virtualization and the architectures that virtualized clouds are built on.


All of this is very interesting and offers an alternative to the usual VMware cloud, it also offers very strong open source enterprise-tested features of virtualization, file systems, tracing software and a suite of great features both server, storage and software-centric.  You can learn more about Joyent’s cloud version of Solaris at

Comparing Linux and Solaris

Some Linux advocates suggest that Oracle Solaris 11 is struggling against Linux distributions and they point to benchmarks in Phoronix that show Linux doing reasonably well against Solaris. One key issue is that the benchmarks use the slower GCC 4.5.2 compiler on Solaris, instead of the Solaris Studio C compiler which is a high performance compiler. Another aspect is that Solaris-based distributions offer a number of features that are lacking in Linux  or in early releases on Linux (ZFS, Zones, DTrace, Fault Management Framework, Management Framework, etc) and it can easily be argued that not everything is about performance. Linux emerged as an alternative to commercial versions of Unix by creating open-source copies of Unix technologies. Now, Oracle itself is trying to add Solaris features to their Linux distribution – but it is a difficult process.  Oracle itself makes a good case for Solaris in a comparison between Solaris and Red Hat Enterprise Linux.

Comparing SolarisLinux

Attempts at Moving Solaris Technologies to Linux.  Oracle is trying to move Solaris technologies to Linux – for example,  Zones on Linux and DTrace on Linux.  Work is also underway on porting ZFS to Linux.  Interestingly, Parallels, IBM and Google have all aimed at delivering something akin to Solaris Zones to Linux. Despite the release of Solaris 11, a good case can be made that Solaris has suffered at the hands of Oracle’s stewardship of the platform.  Oracle’s non-support of OpenSolaris created a vacuum that has made possible a wholly independent version of OpenSolaris called illumos.  The illumos distribution is offering a plethora of updates to OpenSolaris technologies and these have been contributed by key software and cloud companies.  Companies such as Joyent, Delphix and Nexenta have contributed to this distribution.  Joyent, for example, has released a highly innovative cloud-oriented version based on illumos called SmartOS which has added Linux’s KVM, ZFS IO throttling and a number of other features.  Joyent’s moving of KVM to Solaris is an interesting move of a major Linux technology to Solaris. An interesting conversation about SmartOS   :

In another video – you can see that the Linux community is discovering lightweight zone virtualization :

Joyent, meanwhile, has improved their SmartMachine experience in their cloud (effectively zones) by being able to deeply introspect each zone.  You can see more about that on their Cloud Analytics page. While Linux has become the more popular flavor of Unix-oriented operating systems, both Solaris-based distributions and Linux offer strong Unix-based variants.  The emergence of illumos-based distributions offers an independent, open source alternative to Linux that offers a wealth of Solaris features. One thing is clear – Linux and illumos-based Solaris distributions are benefiting from each other’s existence.


gotostorageGo to more posts on storage and flash storage at


Three Videos : DTrace and Monitoring and Debugging

If you are interested in sophisticated diagnostic software – DTrace should be of interest to you. Recently, Three talks on DTrace.  First, Dan Kimmel of Delphix, gave a talk about their efforts to use DTrace for monitoring.

At the same event, Adam Levanthal talked about DTrace (his slides are here). Here is the video :

And finally, Max Bruning provides a nice video showing how to use DTrace for debugging :