Archive

Archive for March, 2009

A Short Review of OpenSolaris 2008.11

March 28th, 2009 16 comments

I decided to finally check out OpenSolaris 2008.11. While this release came out back in November of 2008 (hence the 2008.11 naming convention), it has taken me this long to finally give it a chance. Maybe it is because I am still somewhat skeptical with the whole OpenSolaris Project and still do not know what to make of it…yet. And because I am virtualizing it through Sun’s VirtualBox, I am not going to get into massive details about hardware compatibility. Although I have read reviews that there is still a lot of catching up that needs to be done for this project. I must also admit that even through Sun’s very own VirtualBox, the performance with OpenSolaris was horrible. With any other OS I installed, everything ran smooth. OpenSolaris was the only one to run this bad and slow. Initially I thought it may have had something to do with running from the Live CD but even after installation, the performance was just as bad.

Installation

From the very begin I had confused feelings about the operating system. Booting from the CD image you are loaded into a Live CD environment. There is no option to go directly to the installation. When the Live CD loads into memory, you have to click on the desktop icon to download the operating system. I can understand giving the user a chance to play around with the operating system prior to installing it locally but what if I do not want to mess with this? What if I just want to go directly to the installation? This requires a couple extra steps and if I am deploying this OS across multiple nodes, it may get a bit annoying.

I can understand the project’s attempt to introduce a bit of simplicity but this too gave me a mixed opinion of the approach taken. For example, just before the Live CD loads, you are set in an almost command like shell where you are asked a couple of questions. For those of us familiar with a Solaris installation, this is nothing out of the ordinary. In traditional Solaris installations, you worked in a terminal window set inside a pseudo CDE shell of a desktop environment. Now after the Live CD loads and I move towards installing the operating system locally, I select the disk device to install OS to, maybe modify my partitions if necessary and input the root and a user name/password. That is it! There is no opportunity for me to select the packages I want to install, for the possibility of working with a light weight and customized installation and not have so much clutter to worry about wasting space on my local disk drive.

These are just minor details that can be resolved after installation. Sure it can take up a little extra of your time to install Sun’s Open Office, which is not installed by default. Or even Sun’s packaged version of GCC if you wish to do C development for the platform. I must admit that their package manager is somewhat simple to navigate and use. Just like Synaptic on top of aptitude or anything else on top of yum, all dependencies are resolved when selecting a package and it seems to look and act just as clean.

Working with OpenSolaris

On the surface this operating system looks and feels like any other UNIX/Linux operating system running with a GNOME desktop environment. So to the average user, very little difference may be seen. That is why I am not about to waste time on installing and using some of the same applications that can be found on just about any other UNIX/Linux operating system.

Outside of the obvious kernel, the most interesting difference was using the ZFS file system. And with that, I enjoyed playing with ZFS’s snapshot capabilities alongside the Time Slider feature implemented into GNOME. By default ZFS is installed as your primary volume manager and file system. Unless you specifically specify partitions during installation, all zpools will be allocated appropriately for your default installation.

Default ZPool Setup

I have worked with ZFS before so it was not all that new to me. As mentioned above, what was new to me was the Time Slider feature built into the GNOME desktop environment and working from the built-in ZFS snapshot features. You can schedule incremental snapshots for specific paths where you can always (in real time) revert back to an older snapshot. So if I modified a set of files or deleted them, I can always revert back to a previous snapshot of that enabled directory. You can configure Time Slider from the System->Administration menu located at the top of the screen. When you open up a Time Slider enabled directory, you simply click on the Time Slider icon and you can view the number of snapshots taken. From this you can go back in time. Below is an image of the directory in the present:

Present

Here is a past snapshot of the same directory path:

Past

I thought this was an extremely cool feature to implement. I just hope we may see something similar in a future and stable release of Btrfs for GNU/Linux.

One last thing worth mentioning is the fact that I enjoyed how you can view and manage your devices and device drivers from a GUI interface as opposed to the command line. This is sort of like your Microsoft Windows Device Manager. This is also where I feel GNU/Linux falls short from. I know that a few distributions such as Canonical’s Ubuntu have their own implementation of it and the last time I played around with it was on my wife’s computer running Ubuntu 8.04. I still felt that is was a very weak implementation and with regards to device management I was still more efficient on the command line. I personally use Fedora/Red Hat and would like to see something at least somewhat similar for it. In OpenSolaris, you can access the device driver management window from the Applications->System Tools menu at the top.

Device Driver Manager

Not only does it list your devices but it also lists its associated driver (if it exists and is supported). If no, you can always Submit your hardware profile to the project which will hopefully bring support to that device in a near to future release. In Linux, I cannot tell you how many times I have been stuck trying to figure out which kernel module(s) a specific device is utilizing. This management utility just makes it all the more simple.

Overall Impression

My overall impression and opinion of the OpenSolaris 2008.11 release operating system was that it still needed a lot more time to be where it needs to be if it wishes to stand alongside the other open source giants such as Linux and BSD. These open source operating systems had time to mature and while a lot of the work has been taken care of to help speed up OpenSolaris development (i.e. GNU environment and applications, Mozilla applications, etc.), the project still needs more time to add more software (and according to the review, hardware) support while still polishing up the installation process.

It is worth noting that during the installation process, you are provided with a few Sun advertisements focusing on features such as ZFS to even DTrace. While I understand the advantages to using such tools, I personally feel that Sun tends to focus too much on these features rather than focusing on something new. For example, Linux has caught up with DTrace by developing System Tap. The Linux community is still currently working on Btrfs to compete with ZFS. Maybe it is time for Sun to start focusing on their next generation feature to help it stay on the cutting edge of computing.

Categories: OpenSolaris, Solaris, UNIX Tags:

device-mapper (dm): working with multipath-tools. Part 1

March 21st, 2009 3 comments

Device-mapper (hereafter, dm) is one of the best collection of device drivers that I have ever worked with. It brings high availability, flexibility and more to the Linux 2.6 kernel. Device-mapper is a Linux 2.6 kernel infrastructure that provides a generic way to create virtual layers of a block device while supporting stripping, mirroring, snapshots, concatenation, multipathing, etc. While many modules are built on top of device-mapper, the focus of this article is on multipath-tools. Note that I will be using the terms multipath, multipath-tools and dm-multipath interchangeably to signify the same package. Also note that dm-multipath is the name of the repackaged multipath-tools redistributed under Red Hat in their Advanced Server Linux distribution.

Device-mapper multipath provides the following features (taken from HP dm-multipath reference guide):

  • Allows the multivendor Storage RAID systems and host servers equipped with multivendor Host Bus Adapters (HBAs) redundant physical connectivity along the independent Fibre Channel fabric paths available
  • Monitors each path and automatically reroutes (failover) I/O to an available functioning alternate path if an existing connection fails
  • Provides an option to perform fail-back of the LUN to the repaired paths
  • Implements failover or failback actions transparently without disrupting applications
  • Monitors each path and notifies if there is a change in the path status
  • Facilitates the load balancing among the multiple paths
  • Provides CLI with display options to configure and manage Multipath features
  • Provides all Device Mapper Multipath features support for any LUN newly added to the host
  • Provides an option to have customized names for the Device Mapper Multipath devices
  • Provides persistency to the Device Mapper Multipath devices across reboots if there are any change in the Storage Area Network
  • Provides policy based path grouping for the user to customize the I/O flow through specific set of paths

Installing multipath-tools

Installing multipath-tools is usually as simple as going to your distributions repository, finding the package and select it to be installed. You can always download it and build it from source; but most likely your distribution should have it in its repository. Again, note that multipath-tools runs on top of device-mapper, so you will need device-mapper installed in order to utilize multipath-tools.

Configuring multipath-tools

The two main or key components to manage and monitor in the multipath-tools package are the multipath.conf file and also the multipathd daemon. Both serve vital functions to help load a configuration and monitor it. Sometimes after the multipath-tools package has been installed, the multipath.conf file could be found in /etc. If not you can always run a search for an existing template, which in some distributions can exist in the following directories:

Redhat –

$ cd /usr/share/doc/device-mapper-multipath-<version no.>/multipath.conf.defaults
$ cp multipath.conf.defaults /etc/multipath.conf

SuSE –

$ cd /usr/share/doc/packages/multipath-tools/multipath.conf.synthetic
$ cp multipath.conf.synthetic /etc/multipath.conf

To edit the multipath.conf file simply open it up in a text editor:

$ vim /etc/multipath.conf

This is just an example. Your multipath.conf file may be configured differently to accommodate certain features and limitations with the external data storage that you are working with

# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#devnode_blacklist {
#       devnode "*"
#}
##
## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
##
defaults {
multipath_tool  "/sbin/multipath -v0"
udev_dir        /dev
polling_interval 10
default_selector        "round-robin 0"
#       default_path_grouping_policy    multibus
default_path_grouping_policy    failover
default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
default_prio_callout    "/bin/true"
#       default_features        "0"
rr_min_io               100
path_checker            tur
failback                3
no_path_retry      2
}
devnode_blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][0-9]*"
devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}

In older versions of this multipath.conf file there is a known typo. In the blacklist section make sure that you correct the known error:
devnode “^hd[a-z][[0-9]*]” should read devnode “^hd[a-z][0-9]*”

Please understand what you set before activating the dm multipathed disk devices. For example the default_path_grouping_policy is set to failover and not multibus. That means despite the number of LUN path I have accessing the same logical volume, only one remains active at a single time. If the active path were to fail, then there would be a failover to a secondary defined path. Multibus simply send I/O requests across all paths which are marked as active unless failed. My path_checker is a Test Unit Ready (TUR), which is a low level SCSI command (opcode 0×00) to validate that the SCSI unit is ready to accept I/O requests. Also supported as path checkers are readsector0 and directio. Here is a guide to some of these field definitions.

Another extremely important field to this multipath.conf file is the blacklist. This tells the multipath-tools module to omit any device with the following characteristics, when scanning and grouping devices into device-mapper for multipathing.

There is so much more to this multipath.conf and I know I am only touching the surface of it but there is a wealth of information out there to help understand the vast amount of details buried within. I must admit though, that the coolest feature is that you can define specific settings for device specific environments. If you are working with a specific model of Compaq, Mylex or even Xyratex storage devices, these can be defined separately without interfering with any other connected storage device. Here is an example taken from the default multipath.conf file:

#       device {
#               vendor                  "COMPAQ  "
#               product                 "HSV110 (C)COMPAQ"
#               path_grouping_policy    multibus
#               getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
#               path_checker            readsector0
#               path_selector           "round-robin 0"
#               hardware_handler        "0"
#               failback                15
#               rr_weight               priorities
#               no_path_retry           queue
#       }

Obviously your definition would not be commented out.

Earlier I had mentioned the multipathd daemon. You can start or stop the daemon in the following ways:

Redhat -

$ service multipathd start
$ service multipathd stop

SuSE -

$ /etc/init.d/boot.multipath start
$ /etc/init.d/multipathd start
$ /etc/init.d/boot.multipath stop
$ /etc/init.d/multipathd stop

Note that multipathd will not function appropriately until you have all the appropriate modules loaded. In my case it is dm_round_robin, dm_mirror, dm_multipath and dm_mod.

Scanning the SCSI bus for multipath devices

To have the utility scan or update the nodes on the scsi bus/channel(s) you must type the following command:

$ multipath -v2
create: 32000000bb55555cd
[size=27 GB][features="0"][hwhandler="0"]
_ round-robin 0
_ 30:0:0:0 sda 8:0   [ready]
_ round-robin 0
_ 31:0:0:0 sde 8:64  [ready]
create: 32001000bb55555cd
[size=27 GB][features="0"][hwhandler="0"]
_ round-robin 0
_ 30:0:0:1 sdb 8:16  [ready]
_ round-robin 0
_ 31:0:0:1 sdf 8:80  [ready]
create: 32002000bb55555cd
[size=92 GB][features="0"][hwhandler="0"]
_ round-robin 0
_ 30:0:0:2 sdc 8:32  [ready]
_ round-robin 0
_ 31:0:0:2 sdg 8:96  [ready]
create: 32003000bb55555cd
[size=92 GB][features="0"][hwhandler="0"]
_ round-robin 0
_ 30:0:0:3 sdd 8:48  [ready]
_ round-robin 0
_ 31:0:0:3 sdh 8:112 [ready]

Everything gets grouped according to WWID into a dm device.  Multiple LUN mappings of the same LD will be given an alias (read below for these aliases).
To kill this mapping table you can simply run:

$ dmsetup remove_all

or by removing each individual entry:

$ dmsetup remove /dev/mapper/32003000bb55555cd

Once the mapping table is created you should be able to look in the following 3 paths to find a list of all dm devices either written as a dm-x device or under its WWID (World Wide Identifier).

$ ls /dev/dm-
dm-0  dm-1  dm-2  dm-3
$ ls /dev/mapper/
32000000bb55555cd  32001000bb55555cd  32002000bb55555cd  32003000bb55555cd  control
$ ls /dev/mpath/
32000000bb55555cd  32001000bb55555cd  32002000bb55555cd 32003000bb55555cd

Pick one of the paths and format the device as you normally would with any other raw Linux device to be mounted.

$ mke2fs -F /dev/dm-0 ......

Mount the device and verify that it is mounted by typing df at the command line.  Now to see all your active paths and monitor them during the test procedure you can type either multipath –ll or multipath –l at the command line.

$ multipath -ll
32002000bb55555cd
[size=92 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [active]
_ 30:0:0:2 sdc 8:32  [active][ready]
_ round-robin 0 [enabled]
_ 31:0:0:2 sdg 8:96  [active][ready]
32001000bb55555cd
[size=27 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [active]
_ 30:0:0:1 sdb 8:16  [active][ready]
_ round-robin 0 [enabled]
_ 31:0:0:1 sdf 8:80  [active][ready]
32000000bb55555cd
[size=27 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [active]
_ 30:0:0:0 sda 8:0   [active][ready]
_ round-robin 0 [enabled]
_ 31:0:0:0 sde 8:64  [active][ready]
32003000bb55555cd
[size=92 GB][features="1 queue_if_no_path"][hwhandler="0"]
_ round-robin 0 [active]
_ 30:0:0:3 sdd 8:48  [active][ready]
_ round-robin 0 [enabled]
_ 31:0:0:3 sdh 8:112 [active][ready]

The results show that there are two LUN paths representing a single logical volume. One is set to active while the other is enabled and ready until the path fails over.

Note that a lot of individuals make the mistake of formatting and mounting the sd devices.  This is not allowed when using device-mapper.  sdc and sdg present dm device dm-0 or WWID 32002000bb55555cd.  These are virtual devices labeled to represent multiple LUN mappings to the same LD.  So you must use the dm labels as opposed to the sd ones.

Stay tuned for Part 2. Whenever that is going to be.

Categories: Linux, SCSI, Storage Tags:

Linux File Systems: Ext4. Btrfs. Do we understand what we need?

March 17th, 2009 17 comments

Lately I have been reading a lot about Ext4, Btrfs and the next generation of Linux file systems in general. What frustrates me about all these articles is the lack of understanding in what the file system is attempting to accomplish/address. You have file systems that are catered to Flash devices, to desktop usage, enterprise class computing while others resort to compression methods as a space saving technique. As an end-user or administrator you will most likely utilize the file system that will be the most appropriate in your environment. What are you trying to solve? What is the I/O profile that you are catering to? How is this storage medium going to be accessed? All of these are important questions to ask when deploying a file system to manage a specific type of data computing.

Let us first start with a brief history of GNU/Linux. Linux began as a hobby/project intended as a desktop alternative. Almost immediately ports became available to the GNU/Linux environment in which now it was able to serve web, print and other services. It did not take much longer for it to be adopted into business environments in which it was starting to run the behind the scenes.

Since its beginning, most file systems utilized were intended for desktop use. That includes ext2/3-fs, ReiserFS (Reiser4), etc. Companies such as IBM and SGI had developed JFS and XFS (and CXFS) as a higher performing and easily scalable solution. This addressed more of the needs for enterprise class computing requirements. The needs being performance. Redundancy was still important but judged on a different scale and usually managed by external methods or through other applications.

Flash forward to the present and let us take a look at Ext4-fs and Btrfs. A lot of journalists/bloggers  have been doing a lot of comparisons between the two and unless I am mistaken, both attempt to resolve two different problems. Ext4-fs (an upgrade to the Ext3-fs desktop file system) will still be a “desktop file system” while Btrfs is a solution to compete with the Sun Microsystems enterprise capable ZFS. Both are to be utilized in completely different arenas. On the one hand, the one will run on a local disk while the other can be deployed and run efficiently with external storage in a data center, some of which are under the management of external RAID controllers with their own features and accessible via Fibre Channel, Serial Attached SCSI (SAS), iSCSI, etc. protocols; as is common with deployments of ZFS. I can attest to this personally.

Again, we are talking about a totally different playing field. And if that is the case, then why are we treating both Linux file systems as competitors with each other? Why are we treating Ext4-fs as a short term file system solution until Btrfs can replace (after it has become stable)?

I am excited to see what the future holds for both file system but am more intrigued to see the path that Btrfs follows, at least once it has become stable and engineered to truly rival ZFS. ZFS is an extremely well designed file system that offers features and functionality that Btrfs is just catching up to. But that is a topic for another time.

Categories: File Systems, Linux, Storage Tags:

The Linux Kernel: A trip down memory lane.

March 13th, 2009 2 comments

Truth be told. I started using GNU/Linux as early as 2002 with Red Hat 7.3. I was coming off of FreeBSD at the time, so I had some experience with a UNIX-like operating system. So when I say “a trip down memory lane”, from experience I can only go as far back sd the Linux kernel 2.4.18-3, which was the kernel in the release candidate of Red Hat 7.3.

And just because that was the earliest I had Linux experience does not mean I cannot go to kernel.org and download and view older versions of the kernel. What spawned the idea for this post was an article I read earlier about Linux kernel version 1.0 being 15 years old, today! That is when I went to kernel.org and downloaded the archived versions of 0.11 and 1.0.

Anybody, please correct me if I am incorrect in any of this, but while I was browsing through the kernel source I had noticed some interesting things. First of all, total line count. The kernel has grown significantly over the years. For example, version 0.11 pulled up 13,839 total lines of code while the first official and stable release, 1.0 pulled up a total of 176,250 lines of code. As of this writing, the current stable kernel (2.6.28.7) has a total of 10,117,178 lines of code. At the end of 2008, I recall reading this article marking the count of 10 million lines. The counting of total lines was accomplished by using the following command line sequence of commands from the kernel’s base directory: for file in `find . -type f`; do cat $file; done | wc -l

Amazing, isn’t it? And by amazing, I mean the fact that a global community built this thing. Other points of interest were, when I was browsing through 0.11, the only supported file system seemed to be the Minix file system (which makes sense, primarily because of Linus Torvalds writing Linux as a Minix replacement as early as 1991). In the 1.0 release, file system support was extended to ext, ext2, hpfs, isofs, msdos (the early label to the FAT-fs), nfs, and a couple more (including procfs). Also by 1.0, additional support was created for varying types of block devices. That is, from your generic ATA hard disk drives to floppy drives, ramdisks and your scsi media.

It is just amazing to observe how far we have come along. Happy birthday Linux kernel 1.0!

Categories: Linux Tags:

This week in the world of virtualization: Sun’s VirtualBox

March 11th, 2009 6 comments

This week I had the pleasure of taking some personal time to play around with Sun Microsystem’s VirtualBox. What else can I say but, “Wow.” I am impressed with this excellent application. I had downloaded (v2.1.4) and configured it for Fedora Linux on my laptop. It was extremely simple. It came as an rpm and the installation took care of everything, including adding the shortcut launchers in my GNOME menu.

I had wanted to install a virtualization application for quite some time now to help make development life simpler. I do a lot of low-level device driver development and for that reason I had invested (years back) in multiple dummy nodes to do my testing on. Well, not only do those nodes take up space and consume too much power but when I hit kernel panics, I do not want to have to wait for the system to restart and re-initialize for me to continue with my work. I want something simplified and that is where virtualization comes into the scene.

This may sound a bit redundant but some of you may understand why I am doing this. I wanted to virtualize and install Red Hat Advanced Server 5.1, so I can continue developing my drivers locked into that kernel tree. This will also aid in my being an ISV with Red Hat when completed, tested and released. That way I do not have to mess with any external hardware.

Setting Up for a Guest Operating System

When the application first loads and you are starting from scratch you must first create a guest PC profile. In this profile you allocate the amount of total physical memory the virtual guest has when running to even the hard disk space. The best part of this virtualization stuff is that you can create virtual disks as image files saved on the local host. You can set the size limits on this.

Once the profile is set you are ready to start the guest profile and begin installing the operating system. This can be done through a number of methods, I chose a CD/DVD method and had my RHAS5 ISO files mount as a CD-ROM drive. In VirtualBox you are given the option of reading the CD/DVD from the local drive or to load the ISO files instead. No other work necessary until you have to change the media device (i.e. select a different ISO image). The installation could not have gone any smoother. It installed like it was on its own server.

The process reminded me of how great UNIX/Linux can be without needing to write the ISO files to CD-Rs.

List of Guest Profile(s)

Loading the Guest Operating System

Everything loaded and functioned like a dream. I was able to access the attached network and it recognized all of my enabled hardware. VirtualBox allows you to enable/disable hardware components from being seen by the Guest Operating System. Overall performance seems great (so far). I have yet to do any real application processing.

There was only one thing that set me back and that was trying to figure out a way for me to access the tty console outside of X (via CTRL+ALT+<F key>). After a quick Google search, I had seen how you must use this <Host Key>+<F Key> to simulate such functionality. The host key that I speak of is a key to allow the virtualization application to understand whether you want to use the keyboard/mouse on the Host or Guest operating systems. By default it is set to Right CTRL and can be changed to anything else. It does not take the user long to get used to it. Technically it only becomes truly necessary when using the mouse. With regards to the keyboard, as long as the window that is running the Guest operating system is set to active, it will take control of the keyboard (for the most part).

Running RHAS 5

Conclusion

While the concept of virtualization has been around for more than a couple of decades, I am just impressed as to how popular it has become in the last 6 years or so. It makes sense too. Companies invest thousands of dollars in these high performing servers which once configured with an operating system only utilize 10-40% (I just made up those numbers) of its resources. You have all this unused real estate that can be used to run other things. Also, with this whole push of going green, it gives system administrators all the more reason to consolidate on to fewer systems and save money on energy costs that would be wasted in running all those operating systems on seperate machines.

Last year in October, I had attended a Red Hat “Road Tour” Seminar and the primary focus of the company was very clear. Focus was on middleware and virtualization. This was also during the acquisition of Qumranet for their IP in KVM. I have tinkered with other virtualization tools here and there but so far, I am most impressed with this. Great work Sun Microsystems! To date, I don’t think I have ever come across a Sun product that I was disappointed with.

Categories: Linux, UNIX Tags:

Understanding what it is to be open source.

March 4th, 2009 Comments off

Yesterday afternoon I was speaking with a local administrator of one of the companies that I work for and in the past we had discussed topics such as GNU/Linux among other technical things but certain “concerns” always seem to come back up, not only with him but also other Microsoft using technical individuals that assume open source is bad, against Microsoft and anything good, and/or insecure (“a result of the source being available to everyone”). Where have these false ideas sprouted from? Was it Microsoft’s initial and blatant attack on open source with their misinforming “Get the Facts” campaign? These are intelligent individuals that I speak of and this post is not intended to attack them in any way but instead help as an educational guide of what it means to be open source. Also, the posting is not meant to convert individuals to using open source application. That rests solely on that individual’s comfort with the environment(s) that they are accustomed to.

First and foremost, I think it is best to understand that open source does not mean anti-Microsoft. It also does not mean Linux. Although the latter is licensed under an open source license (GPLv2). If you pay careful attention there are numerous open source projects such as Mozilla Firefox, MySQL, Apache Server, GIMP, etc. that are available on a wide range of operating platforms including Microsoft Windows. In fact Microsoft hosts a site for open source projects called CodePlex. I have not read too much about it but I also believe that the developer(s) have the right to adopt Microsoft’s open source licenses. Microsoft understands the advantages of open source, even if it still results in them not adopting it as their main business model. Just recently Microsoft had donated $100K to the open source Apache project, admitting it to be superior than their IIS server.

Second, open source does not mean free. Free software is another category and depending on the licenses used can determine how free an open source application is. When it comes to free software there is a saying: “Free as in speech, not as in free beer.” I would delve more deeply into this topic but it is one meant for another posting.

Going back to open source, open source development can be introduced into an environment with design or strategic goals in mind. What I mean by this is that open source can involve a community which can grow and aid in the development of that project. With this same community, the project’s development can also accelerate at a rapid rate. Now there are different approaches to open source development and instead of taking all the words out of his mouth, I will refer you to Eric S. Raymond’s The Cathedral and the Bazaar. Raymond details the advantages and disadvantages to two different styles of open source development, i.e. the Cathedral method and the Bazaar method. To highlight some of his comments on the Bazaar method which is the most traditional and widely used method, Eric states:

Or, less formally, “Given enough eyeballs, all bugs are shallow.” I dub this: “Linus’s Law”. {…}

It’s one thing to observe in the large that the bazaar style greatly accelerates debugging and code evolution. It’s another to understand exactly how and why it does so at the micro-level of day-to-day developer and tester behavior. {…}

One key to understanding is to realize exactly why it is that the kind of bug report non–source-aware users normally turn in tends not to be very useful. Non–source-aware users tend to report only surface symptoms; they take their environment for granted, so they (a) omit critical background data, and (b) seldom include a reliable recipe for reproducing the bug.

The underlying problem here is a mismatch between the tester’s and the developer’s mental models of the program; the tester, on the outside looking in, and the developer on the inside looking out. In closed-source development they’re both stuck in these roles, and tend to talk past each other and find each other deeply frustrating. {…}

What does some of this mean: “given enough eyeballs, all bugs are shallow“? What better strategy than to have not only testers and end-users reporting the bugs but also developers who traverse through the code and understand where a problem may surface even as far as the core of the application itself. This is feedback that is returned from a global basis as opposed to an isolated and closed environment. As Eric had mentioned in his paper, in closed environments most bugs are reported from what is visually seen and not physically coded. Therefore, it is not a fair comparison when stating that an open source version of an application had xxxxxx amount of bugs at release or currently while a closed version had xxx. Most likely xxxxx amount is being overlooked and never known until something in the near future forces it to be seen. This will in turn increase cost and time. Instead of tackling the problem earlier on, resources must be pulled from other location to put out the current fire.

Does this saying mean that because so many eyes view the code that so many individuals can then write malware and or attack the systems that these applications run on? Not even close! All project owners/facilitators control what gets officially placed into the stable builds of a project. So when the project is tested and compiled and/or built into binary form the appropriate facilitators manage what features and functionality get built right in. In the end, chances are that if an individual(s) with negative intentions spots a bug, another good percentage of individuals with the same access to the source code would have spotted at around the same time or possibly sooner. The more popular projects also have frequent release cycles or updates. Through the open source model, a lot of times when a bug is exposed it is almost immediately resolved.

Also, where is the logic that because open source is available to the public, it is insecure? Look at the closed source Microsoft Windows operating system. It is plagued with attacks and nobody except for Microsoft has the source. We see how secure it is. The evidence speaks for itself. It is a result of open source methodology that projects like the Linux kernel can develop new features, have them tested and released as stable long before NT kernel developers finish their planning process.

A third attack toward open source is quality. Most attacks seem to focus on a lack of quality when the case is just the opposite. I guess this would also depend on the reader’s definition of quality. Is this quality measured by stability, usability and/or graphical appeal? While it is true that early on in a good number of open source projects more focus and emphasis had been placed on stability (which includes security) rather than usability or graphical appeal but as of the past 5-8 years or so I find that the other categories have caught up significantly. But I guess I will let you be the judge. Use Open Office instead of Microsoft Office. Use GIMP instead of Photoshop. Mozilla Firefox instead of Internet Explorer. Or run under an entirely different operating environment such as a GNU/Linux or BSD-based distribution instead of Microsoft Windows. Give it a trial run and then return with your feedback on quality. A great portion of these distributions have gotten great in terms of usability and graphical appeal. In fact, some of these projects have gotten far advanced from their closed competitors. This could be a result of involvement from quality filled contributors with rich development and test backgrounds.

A fourth concern is support. The general assumption is that by utilizing open source applications you will either be offered poor to no support for those applications. Again, the fact is the exact opposite. Sun Microsystems offers support subscriptions for MySQL, Red Hat, Novell and Canonical offer support subscriptions for their operating platforms and the supported applications that they offer with them (which range in the thousands). If you do not wish to spend money, then free support is available around every corner from general forums to even the project’s forum (if one exists).

Additional advantages to open source include early exposure to proper coding etiquette. This is extremely important to a developer such as myself. This is something that is rarely taught and maintained in closed environments. During my history of employment and consultation development I have had the opportunity to see what closed and sloppy coding can do. This in fact results in buggier builds which can take a long time to troubleshoot and resolve. Most resolutions in these environments result in hacking quick fixes on top of other quick fixes which in turn produce additional problems and bugs. I can honestly say that I am grateful for the open source community in more ways than one on this issue.

So, if open source was so evil, then why are large and small companies which include IBM, Sun Microsystems and Novell pooring millions of dollars into it a year? Companies such as Red Hat, Canonical, and Mozilla would have never built their business model around it. If open source was so terrible, would Microsoft donate money to the Apache project? If open source was so bad, would the usage of web browsers such as Mozilla Firefox gain such popularity and over time be a contributor to the decline in usage of Microsoft’s Internet Explorer? According to recent Network Application number IE usage has dropped to 67.44% while Firefox has climbed to 21.77% (as of February 2009). Can 21.77% users be wrong in utilizing open source applications?

Believe it or not but open source is a culture and not a virus. It attempts to offer everything good that is possible and runs entire industries to even countries. Its communities have grown and they will not disappear. At least anytime soon.

BTW, in response to a comment made by an Aronzak from a previous post, I made sure to answer most if not all of my questions in my post. ;-)

Categories: Linux, Misc Tags:

Interactive Kernel Map…

March 2nd, 2009 Comments off

Posted on this blog site, here is an interesting interactive map of the Linux kernel. I find it extremely interesting. Yet another thing for me to play around with. ;-)

Categories: Linux Tags: