Good command line tools are more important than ever and not just a relict of ancient times in comparison to rich Internet (RIA) or GUI applications. Experienced system administrators appreciate their power in sophisticated shell scripts and could probably not manage their environments without them. In the world of free software which made tremendous progress during the last 25 years the CLI (Command Line Interface) never lost it’s importance and is still a standard component of well designed applications which expose their APIs and functionality through programming libraries (e.g. in C/C++/C#, Java,…), scripting wrappers (Perl, Python. Ruby, PHP, JavaScript, Tcl, Groovy, Boo, Lua, Lisp, Guile,…), command line utilities, REST/SOAP/XML-RPC/JSON-RPC/D-BUS interfaces and interactive GUI/Web applications.
Command line and terminal programs are also tightly coupled with the history of programming (especially scripting) languages and system design since for a long time they where the major tools for human-computer interaction. Remember the Digital Command Language (DCL) or IBM’s Job Control Language (JCL) for mainframes? Hundreds of shell implementations (Bourne shell, Korn shell, bash, zsh, (t)csh, fish, cmd.exe, iSeries QShell, Windows PowerShell, Pash, Hotwire, etc.) for all kind of operating systems and the long tradition of command line utilities (e.g. AIX smitty) especially in UNIX-like environments clearly show the importance of the CLI. For embedded and networking devices (e.g. routers, switches. etc.) mastering the command line is still a necessary requirement for professional certifications (e.g. CCNP).
From a more modern perspective command line tools and the interrelated mini or little languages (e.g. check Eric Raymond’s book about The Art of Unix Programming) are special cases of Domain-specific languages (DSL) with the additional focus on interactivity and do not look that old fashioned any more! The syntactic resemblance of command line expressions with functional language constructs e.g. in Haskell or Scala shows further paths of investigation and potential innovations.
The past and future of command line tools
So, how can we build smarter command line tools for systems and especially configuration management? Before I try to answer this question and also delve into some implementation details, let’s look at potential features of a more sophisticated CLI. You will easily recognize that they are not new and have been implemented to some extent in a lot of environments but I am not aware of any shell (framework) which realizes all of them.
1. Everything can be done with a command
This looks very similar to the TCL (Tool Command Language) principle everything is a command but has a slightly different meaning here, i.e. you do not have to edit configuration files (e.g. located in /etc) anymore but just do everything interactively or in a shell script with commands. Most router operating systems follow this approach since they want to control the interaction with the user in a very precise way. As an example take the following DHCP configuration using the Cisco IOS shell:
Router(config)# interface ethernet0/0
Router(config-if)# ip address 192.168.1.1 255.255.255.0
Router(config-if)# no shutdown
Router(config)# ip dhcp pool 1
Router(dhcp-config)# network 192.168.1.0 /24
Router(dhcp-config)# domain-name foobar.com
Router(dhcp-config)# dns-server 192.168.1.254
Router(dhcp-config)# default-router 192.168.1.254
Router(dhcp-config)# lease 30
Router(dhcp-config)# exit
This looks very much like editing /etc/dhcpd.conf interactively similar to the omshell (i.e. OMAPI command shell) which allows real-time updates for the ISC DHCP server without restarting it. The Cobbler install server (a Red Hat emerging technologies project written in Python) makes quite extensive use of the omshell borrowing some ideas from the Kusu HPC framework.
Wrapping all system configuration actions in commands is theoretically not a big deal since you can write scripts that manipulate configurations files or database/registry/LDAP/Active Directory entries. But in practice this is a very hard problem because of the many different configuration file formats and all project which tried to solve it basically failed. A novel approach is the Augeas project which is a command line tool to manipulate configuration files from the shell. It also includes a C-library, language bindings and a domain-specific language to describe configuration file formats. Here’s an example with /etc/hosts. For details and explanations check the Augeas website.
# augtool augtool> set /files/etc/hosts/10000/ipaddr 192.168.0.1 augtool> set /files/etc/hosts/10000/canonical pigiron.example.com augtool> set /files/etc/hosts/10000/alias[1] pigiron saugtool> et /files/etc/hosts/10000/alias[2] piggy augtool> save
Augeas is carefully designed to keep the structure of your configuration files intact but also shows the complexity of this problem. Standardizing on XML would not be a solution either and just adds another very interaction unfriendly format to the zoo of formats especially if you expose it directly to the user. Navigating and manipulating an XML tree with commands and completely hiding the XML behind it is a good approach for legacy free projects though. Very instructive examples of configuration file free commands are zpool and zfs from the OpenSolaris project. You can just start using them as shown in the following example:
# zpool create tank c1t0d0 # zfs create tank/home # zfs set mountpoint=/export/home tank/home # zfs set sharenfs=rw tank/home
It should be clear by now what is meant by everything can be done with a command and how powerful this approach is but we will push this principle much further in the following paragraphs. If Unix had takes this approach from the beginning we would probably not have this chaos of configuration files today.
2. Eliminate flat namespaces with object/command hierarchies
Thousands of commands available at your fingertips is nothing unusual on a Unix/Linux system but the global and flat command namespace is getting bigger and bigger every day with many projects introducing new commands. This can be quite challenging and intimidating for beginners.
Technically speaking this is not necessary if you introduce a hierarchy of configuration or system objects similar to the /proc or /sys filesystem and navigate through them with commands like cd, ls, set and create (or mk) instead of using the many ls* (lspci, lshal, lsusb,…), *adm (mdadm, iscsiadm, fsadm,…) or *config* (nss-config, sdl-config, system-config-*,…) commands.
Let’s start this virtual hierarchy e.g. at /system but feel free to choose a different name of your choice. Staying outside this directory you get your normal shell behavior which is important since another principle we will follow is to enable users with new possibilities and not enforce anything if they do not like it. As soon as you enter the /system directory you will get a completely different environment with dynamically created object/filesystem hierarchies, a restricted $PATH variable with commands depending on the context and much more. Let’s take a look at a simple example:
# cd /system (system)# cd users (users)# cd t<TAB> tanita therese tom (users)# cd tom (users/tom)#<TAB> cd ls set ... (users/tom)# ls name = Tom Waits home = /home/tom shell = /bin/bash groups = tom,users ... (users/tom)# cd .. (users)# create user tim (users)# cd tim (users/tim)# set <TAB> name home shell groups ... (users/tim)# set name Tim Brian (users/tim)# set shell /bin/zsh (users/tim)# set groups tim,users (users/tim)# cd .. (users)# ls tanita therese tim tom
So there is no need for specialized commands like adduser, useradd, etc. anymore. You can use standard commands for object creation (in the example above I used create user), set their properties with set (using command line completion) and navigate around with cd, ls, etc. It’s not my intention to standardize commands but this could be a side effect in the long run. Each configuration domain (storage, servers, networking, InfiniBand, FCoE, iSCSI, backup, archiving, security, authentication, file systems, clusters, virtualization, cloud computing, groupware, etc.) should be discussed by subject matter experts and different implementations can compete against each other until a favorite CLI syntax or environment shows up.
It’s also important to start with commands at a higher abstraction level than system commands to get an immediate benefit for such an approach. Very good candidates are Linux cluster (Rocks, xCAT, Oscar), virtual machine or cloud computing frameworks (like virsh, Enomalism, Ganeti, OpenQRM, Eucalyptus, OpenNebula, Nimbus) and Linux appliance construction toolkits.
Take for example virsh to manage virtual machines. Why implement a special shell for this purpose? Just define a new namespace for virtual machines and appropriate commands like in the following example executed on host1:
# cd /system/vms (vms)# ls fedora1 fedora2 debian1 (vms)# ls -l ...more detailed info about the vms including status (vms)# cd fedora1 (vms/fedora1)# ls <TAB> capabilites status vcpus mem .. (vms/fedora1)# reboot (vms/fedora1)# cd .. (vms)# migrate fedora1 to host2 (vms) ls fedora2 debian1
A cd command in this context could also mean that we log into the virtual machine and get a restricted shell (using the same framework) running inside the virtual machine. This leads us to our next requirement.
3. Network transparency
The virtual machine example above shows that it would be very convenient to handle local and remote virtual machines (VMs) in the same way with network transparent commands (cd /systems/host2/vms; ls). The implementation could use Expect–like mechanisms, a simple ssh, XMPP (or even Google Wave), Telepathy DTubes, the Plan9 resource sharing protocol 9P (take a look at the xcpu project using this idea), special tools and libraries (like libvirt) for certain commands, a FUSE based file system or some RPC-mechanism for non-UNIX operating systems.
We can also integrate routers, switches, storage devices, terminal servers, intelligent PDUs and other network enabled equipment into our object hierarchy and access it exactly the same way (e.g. cd /systems/switch1). The user would either get the real shell prompt of the device (if there is one available) or work in a restriced/simulated environment with features like command line completion or object hierarchies even if the real hardware does not support them. Accessing the
- management module of a Blade chassis (cd /systems/chassis1/mm)
- service processor of a rack server (cd /systems/host1/ipmi)
with ssh, SMASH or IPMI are some examples. There is no need to implement a special IPMI shell if you can integrate IPMI commands with ipmicmd, ipmitool, etc. into a more general and powerful framework! SMASH is interesting because it gives you exactly the proposed network transparent hierarchical access to your hardware in an administrator-friendly way hiding all the complexity of CIM-XML over HTTP, WS-MAN and other XML standards. Since SMASH is concentrating more on the hardware itself it is a nice and orthogonal supplement to our proposal.
4. SQL-like and functional command syntax
How are we going to manipulate groups of objects in our framework? Assume you want to migrate your oldest not accessed files bigger than 10 MB to a tape system when your expensive storage gold pool is more than 90% full. IBM’s General Parallel File System (GPGS) uses a very interesting SQL-like syntax (and a parallel implementation) for this kind of problems:
MIGRATE FROM POOL ‘gold' THRESHOLD(90,85) WEIGHT(CURRENT_TIMESTAMP - ACCESS_TIME)
TO POOL ‘hsm’ WHERE FileSize > 10MB
The general syntax is much more sophisticated but you should get the idea from this example. GPFS is using a lot of interesting commands starting with mm (likemmlsfs, mmlsnsd, mmlscluster, mmlsquota, mmlsdisk, mmlsmount, etc.) for its configuration and management. This flat namespace is of course yet another consolidation candidate (cd /system/gpfs/nsd; ls), but let’s come back to the idea of using a SQL-like syntax in our smarter CLI. Assume you could access all objects or commands in your file system hierarchy like in the following example:
# set myadmins select users from group admins where lastlogin greater 30days # cd myadmins (myadmins)# set shell bin/false
The strength of this approach is quite obvious. Depending on your programming language skills and background you will probably suggest different solutions but keep in mind that the little (almost natural) language design has to be command line completion friendly and basically use one-liners! Here is another example:
# cd /system/clusters # migrate hosts from cluster1 where cpuload greater 200% to cluster2
A simplified TCL version with its everything is a command approach could be a very nice test bed for these kind of ideas and I hope the TCL community will accept the challenge and come up with better proposals! Let’s assume for a moment that you are a TCL fan trying to implement the idea above. Instead of using the generic name set we take e.g. select as our initial command. Additionally we put myadmins at the end of the line since it’s the only syntactic element which can not be auto-completed.
# select users from group admins where lastlogin greater 30days as myadmins # cd myadmins
Going one step forward and executing the command in the groupscontext we can simplify the syntax even more by eliminating the word group since it is clear now that we are selecting a group of users from the admins group:
# cd /system/groups # select users from admins where lastlogin greater 30days as myadmins # cd myadmins
We can even construct a command without from admins if we limit the context even further.
# cd /system/groups/admins # select users where lastlogin greater 30days as myadmins # cd myadmins
We can also select a group of users in the users directory
# cd /system/users # select users where lastlogin greater 30days as myadmins # cd myadmins
but this is probably not a good idea and only shows the flexibility of our approach. One question remains though! Is a group of objects created directly after execution of the select command or the first (each) time you use the group? It depends!
One could extend the selectsyntax with more keywords like static, dynamic, first-use or set properties of the generated group (cd myadmins; set generation dynamic) assuming that the default behavior is static. Using different commands like dynselcet, staticselect, etc. for each case is another solution:
# dynselect hosts from hostgroup cluster where cpuload greater 200% as overloaded # dynselect hosts from hostgroup cluster where os equals RHEL-5.3 as rhel5.3 # dynselect hosts from hostgroup cluster where package is-installed OpenIPMI as ipmihosts # dsh ipmihosts chkconfig ipmi on # cd ipmihosts (ipmihosts)# service ipmi start
The last example shows the use of a global parallel shell command (dsh) applied to a group of hosts and also the approach we have been promoting so far (i.e. command execution in a hierarchical context).
We can also take a more functional and less verbose approach for object selection using map, filter and other well known methods from languages like Haskell and Scala. The only difference is that that we should try to avoid brackets and nesting since these constructs are not really command line completion friendly.
# cd /systems/hostgroups (hostgroups)# dynamic overloaded = cluster filter cpuload greater 200%
You do not really need the equal sign but it clearly enhances readability. Instead of writing _.cpuload as in Scala we drop the underscore reducing the possibilities and expressiveness in our example above. We can also use the equal sign in combination with SQL-syntax:
# cd /systems/hostgroups (hostgroups)# ls cl* cluster cluster1 cluster2 (hostgroups)# dynamic overloaded = select hosts from cluster where cpuload greater 200% (hostgroups)# dynamic rhel5.3 = select hosts from cluster where os equals RHEL-5.3
5. Logging, auditing and change/configuration management
Our feature wish list is far from complete, so let’s continue. If every system configuration activity is using a command, logging and auditing is more or less trivial. Just write e.g. user@host, time and command (or a message in standard IETF syslog format) to a file, database or write once device and you know exactly what happened on your system (at least from a configuration point of view). No need for revision control systems or special filesystems watching your files anymore and change management can be done with a list of pre-approved commands or scripts. Another interesting possibility is to use the ZeitGeist event engine to track the commands, analyze (or even block) them, generate reports, etc.
A configuration management database (CMDB) or more abstract descriptions of your systems (e.g. using one of the many Open Source configuration management systems) can still be used in the background but the same observation about simplification as above applies here. By using a sequence of (high level) commands describing how a system achieves its state you can replace the descriptive approach by a procedural one and are probably more in sync with the actual work of system administrators.
6.Transactions, rollbacks and state changes
Transactional commands which can be rolled back (e.g. deliberately or in case of failures) would be nice to have but are difficult or even impossible to implement. Why not using a special command (since everything can be done with a command according to our first rule) to define if a command is transactional (e.g. by setting a property of the command) and add a rollback command if this is the case. In so doing one can enter a transactional mode where only commands with rollback functionality are shown.
Implementing a new installation and boot process using transactional commands would result in a system which can be transformed very easily into another configuration state by running a list of commands. Red Hat’s kickstart files which can be viewed as recipes how to fully automate the installation process are using this approach and only need some minor changes to completely fit our model. All the commands of a kickstart-ng file could then be executed in a shell allowing easy debugging using a standard start-stop-step cycle. This is very close to what the Debian installer d-i is doing (d-i netcfg/get_gateway string 192.168.1.1) but in sharp contrast to a descriptive style using reply files written in XML (e.g. Novell/SUSE’S autoyast).
In a similar fashion many of the available Linux, FreeBSD, OpenSolaris, etc. HOWTOS could be transformed into a sequence of commands you could read almost like a book instead of following explanations how to type some commands, edit configuration files and do other system configuration changes with web applications or desktop GUIs.
B.t.w. what’s the reason why system administrators do not really trust many of the available web and GUI tools? Because they do not know what they are doing in the background. If these tools were a more convenient way of entering data on top of a CLI (showing the executed commands in an expert mode) you would get more confidence in them. AIX’s smitty is doing exactly that and this is one of the reasons why people really like it. Another benefit of putting more attention to the command line in this way are more structured certifications tests (LPI, Red Hat, Novell, Ubuntu, etc.) which could be done with the CLI or the GUI delivering comparable results.
7. Integrated help
Another important feature of our CLI is integrated help. Traditionally GNU and other utilities use man/info pages and have help options like -h and –help to display all other command line parameters (the Z shell can even use this output to automatically generate command line completion for them!). To have the help information at your fingertips while typing with examples you can immediately integrate into the command line would be a very useful feature! Defining help information and syntax examples for commands and their options will be done with commands of course, as you can see in the following very simple examples:
# cd /system/commands/vms/suspend # set help "Suspend a running domain. It is kept in memory but won’t be scheduled anymore." # set example 1 "cd /system/vms/fedora; suspend" # set example 2 "cd /system/host2/vms/fedora; suspend"
Help information will be defined in many different formats (text, html, wiki, man, info,…) and displayed in different ways according to you preferences with
- a simple text interface
- a ncurses interface & split screens (like e.g. in GNU Midnight Commander)
- overlay techniques (using Clutter and Gnome-Shell)
- a HTML widget (the uzbl Browser is a good example of CLI/widget interaction)
- other kind of widgets (take the Hotwire shell as an example)
This is definitively not your grandfathers static text-based help help but could incorporate blog entries, videos, e-learning, RSS feeds, instant messages, e-mails, Google waves, wiki pages, HOWTO’s, personal notes, twitter feeds, etc. for a more advanced help experience.
9. ACLs
We all know from Zed Shaw’s that the ACL is dead (I completely agree with that!) but enabling and using them with care should be possible for our CLI. We basically give users and groups access to certain commands or subsets of them. When we define new commands or extend old ones we define who can see and execute which part of it. To enhance security the default behavior of our commands should be no visibility and no rights at all if they are not explicitly defined.
10. CLI syntax
A quick look at some command line utilities shows that there is no universally accepted CLI standard. Many tools use the GNU conventions (with short and long options) but there are many exceptions as the following examples show:
# cd /root
# dd if=/dev/cdrom of=cd.iso
# tar xvfz test.tar.gz
# tar -tf test.tar.gz
# tail -f -n 30 /var/log/messages
# cobbler system add --name=h1 --mac=00:14:5E:EC:10:EE --ip=192.168.1.5 --profile=F11-i386
# qemu linux.img -net nic,macaddr=52:54:00:12:34:57
# qemu linux.img -drive if=ide,index=1,media=cdrom
# chkconfig --list ipmi
# chkconfig --level 2345 ipmi on
# ip route show
Since we are looking for a CLI with command line completion friendly commands fitting on one line (e.g. max. 80 characters including the prompt) our general approach will be much simpler. There should be no restrictions for more sophisticated users though! New commands of our CLI will be defined by special commands and not by using a descriptive approach with XML files and the like (take a look at the interesting clish shell for the second approach). We will also put the commands themselves into a command hierarchy applying our first two principles also to commands.
# cd /system/commands/ # create command select # cd select (select)# set generation static (select)# set (select)# set transactional yes (select)# set.... (select)# set ACL ... (select)# set syntax "..." (select)# set option -v "..."
Finding a good syntax for command definitions will be an interesting challenge and hopefully stimulate the imagination and creativity of many developers and language designers. This leads us to the question how to implement a shell with all the proposed features.
11. CLI implementation details
Looking at many different shell implementations I came to the conclusion that implementing a new shell had almost no chance of universal acceptance. Router, firewall and embedded systems often take this approach, sometimes with rather poor results because of time constraints (i.e. no command line completion!).
A new universal shell written in Java, C#, C++ or a scripting language like Python, Perl, Ruby, TCL, Lisp, etc. would not attract all the different communities so C is almost a must for the core framework. But implementing, debugging and stabilizing a new C-based shell is far from trivial and takes a long time.
Of course you could take a higher level framework like pycopia (written in Python) and make it extensible with other programming languages but language flame wars are quite predictable with this approach. The same is true for CLIs being closely related to one programming language (like IPython for Python which is very popular in the scientific computing community) so they are not an option either.
There are hundreds of Open Source projects facing exactly the problems described here and most of them just implement their own command line tools or shell frameworks. One interesting example is the Vyatta Open Source Router having its root in the Quagga-project. Vyatta used the very powerful xorp shell but for a typical Linux user it felt somewhat unnatural so they switched to a bash based CLI reusing the built-in command line completion of bash.
Although Vyatta is using a flat command hierarchy it shows how our CLI could be implemented by taking a standard shell like bash (e.g. the Vyatta bash version) or zsh and their command line completion frameworks (bash’s compgen and complete commands or zsh’s completion system and completion widget). The bash-completion and the many zsh completion scripts show how powerful these shells are regarding command line completion so there is really no need to reinvent the wheel.
In addition the cd command needs to dynamically manipulate the $PATH variable so the full namespace is only visible when navigating outside e.g. /system. The implementation of the whole feature set of our CLI needs some effort though to make it useable for non scripting experts so they can concentrate on little language designs and not bother with shell scripting issues. Maybe some kind of plug-in, templating or even AOP system could turn out to be useful so contributors can extend commands (with some specialized commands) without manipulating the whole syntax (e.g. adding an option, an example, etc.) or add aspects to enable fancy features.
12. Some last words
The more I was thinking about a smarter CLI the more it became evident that the topic is far from dead. Quite the contrary is the case and I hope some people will pick up the ideas presented here and start producing some code. I am convinced that in the process of implementing this smarter CLI a lot of new ideas, different approaches and hopefully much better command line tools than the ones available today will pop up. I am even convinced that you can use this CLI to describe use cases for new software with carefully chosen new commands instead of just plain text descriptions.
Tom's Readme 