Augedit prototype

Augeas is a great tool for string to tree transformation. I created a GUI to make it visual and more interactive. The prototype app is called Augedit. It displays the Augeas tree on the left side, and on the the string representation on the right side. By selecting a node in the tree, the label and the value matched are highlighted in the text view. Here is an example with a trivial key/value lens, one record per line.

Behind the scene, it uses span information of the selected path to recover the file content and highlight the key and the value. The app is in Vala and uses Gtk toolkit.

The prototype as of now is only a viewer. This could be extended in many ways:

• Edition capabilities, where the tree and the text are synchronized as the user types
• Providing a view to query and display results
• XPath expression builder, so that they can be validated and then used in scripts
• Highlight the whole match for a sub-tree  The information is available from the span, but I haven't been able to set alpha channel for colors in Gtk.TextView. Using pango/cairo/webkit based widget to display the text may be the solution.

Vala is quite nice, I like the idea of having bare metal binaries and the integration with Augeas with the bindings works great, documentation on valadoc.org is excellent. However, GCC complains because of the C code generated and it's a bit annoying  part of it is my fault (using deprecated stuff, like Gtk.Box() for instance). For more code examples, I'm using yorba projects as reference, like geary and shotwell. Finally, debugging is not as convenient, because of the additional layer of code generation.

The code is on github: https://github.com/giraldeau/augedit

Have fun!

Managing Apache config with Augeas

Release 0.8.0 of Augeas includes a special lens for managing Apache web server configuration files. First, we will see how the lens maps sections and directives to the tree. Then, we will review some recepies to modify the configuration and show results.

Quick start

Apache configuration has two main components. The first component is section, which is similar to XML elements. Sections can be nested and contain directives. Directives are composed of a name and positional arguments.

As you may know, Apache is very modular. A module can define its own directives and Apache will run the module's callback to parse arguments. This is reflected in the way the configuration is handled in Augeas. The lens is mostly generic and allows arbitrary sections and directives, with arbitrary arguments. Apache allow case insensitive sections are directive. The actual version enforce that open and close tags to be the same case. I recommend to standardize the casing of the configuration for simplicity. Now, let's look at a small example, with one section and one directive.

<VirtualHost *:80>
  ServerAdmin foo@example.org
</VirtualHost>

This configuration produces the following tree.

{ "VirtualHost"
    { "arg" = "*:80" }
    { "directive" = "ServerAdmin"
      { "arg" = "foo@example.org" }
    }
  }

Let's say we want to modify the email address of the server admin, we can do a simple set:

augtool> set "/VirtualHost/*[self::directive='ServerAdmin']/arg" "bar@example.com"

To select the ServerAdmin directive, we need to search for it as a child of VirtualHost. We then set the first argument of the corresponding directive to a new value. Here is the resulting diff from the original file. If there are more than one VirtualHost entries in the file, then a simple set will fail, a set multiple (setm) is required to change them all at once.

$ diff -u httpd.conf httpd.conf.augnew 
--- httpd.conf 2011-02-22 22:03:34.000000000 -0500
+++ httpd.conf.augnew 2011-02-22 22:04:34.844099001 -0500
@@ -1,3 +1,3 @@
 <VirtualHost *:80>
-  ServerAdmin foo@example.org
+  ServerAdmin bar@example.org
 </VirtualHost>

Tree structure

There are 4 special things to know about the tree structure of apache configuration.

• Section names are labels of the tree, they appear on the path.
• Directives are held in value of a "directive" node.
• Arguments of either sections or directives are held in values of "arg" node.
• Arguments nodes must be before any other children.

To demonstrate this, let's create a configuration from scratch, and see it build at each step in augtool. First, launch augtool. I suggest to use the -n option to save to a new file.

$ sudo augtool -n

The variable $conf holds the path to the target empty file, for example under Ubuntu it would be:

augtool> defvar conf /files/etc/apache2/sites-available/foo

Step 1: create VirtualHost section

The clear command empty the value of a node and it create it if the node doesn't exists yet.
Command:

augtool> clear $conf/VirtualHost

Result:

<VirtualHost>
</VirtualHost>

Step 2: Set the argument of VirtualHost section.

Command:

augtool> set $conf/VirtualHost/arg "172.16.0.1:80"

Result:

<VirtualHost 172.16.0.1:80>
</VirtualHost>

Step 3: Create the ServerAdmin directive under VirtualHost.

Command:

augtool> set $conf/VirtualHost/directive "ServerAdmin"

Result:

<VirtualHost 172.16.0.1:80>
ServerAdmin
</VirtualHost>

Step 4: Set the ServerAdmin first argument to configure the email.

Command:

augtool> set $conf/VirtualHost/*[self::directive='ServerAdmin']/arg "baz@example.org"

Result:

<VirtualHost 172.16.0.1:80>
ServerAdmin baz@example.org
</VirtualHost>

The order matters

What if we perform step 3 before step 2? Here is the result:

augtool> set $conf/VirtualHost/directive "ServerAdmin"
augtool> set $conf/VirtualHost/arg "172.16.0.1:80"
augtool> print $conf
/files/etc/apache2/sites-available/foo
/files/etc/apache2/sites-available/foo/VirtualHost
/files/etc/apache2/sites-available/foo/VirtualHost/directive = "ServerAdmin"
/files/etc/apache2/sites-available/foo/VirtualHost/arg = "172.16.0.1:80"
augtool> save
Saving failed
augtool> print /augeas/files/etc/apache2/sites-available/foo/error/message = 

Failed to match ({ /arg/ = /[...]/ }({ /arg/ = /[...]/ })*)?(<<rec>>
| { /directive/ = /[...]/ } | { /#comment/ = /[...]/ } | { })*  

with tree

{ \"directive\" = \"ServerAdmin\" } { \"arg\" = \"172.16.0.1:80\" }"

I reformated the error message to highlight why it failed. The lens require "arg" nodes to appear before "directive", "#comment" and empty nodes. In this case, the node "directive" is before the "arg" node, and it doesn't match the tree structure. So, how to make sure that we insert nodes before any other? Here is the twist. First, we will remove the arg node and create it at the right place.

augtool> rm $conf/VirtualHost/arg
rm : $conf/VirtualHost/arg 1
augtool> ins arg before  $conf/VirtualHost/*[1]
augtool> set $conf/VirtualHost/arg "baz@example.org"
augtool> save
Saved 1 file(s)

And the corresponding result is the one expected. The predicate [1] will always select the first node in a node set and is equivalent to [position()=1]. There is one gotcha, that is if there are no children under VirtualHost, no node will match and the insert will fail. In this case, a simple set is required.

Also, take care to set a value to "arg" nodes! Otherwise, Augeas use the section name lens to render it. Here is the result of having an "arg" node with a null value.

<VirtualHost>
<arg>
</arg>
</VirtualHost>

Use cases

Modify document root

The first use case I show is to modify document root in the default configuration of Apache under Ubuntu. The default document root is "/var/www", but say we want to change it to "/srv/www". We have to modify the DocumentRoot directive and one Directory section. I'm using the match command to verify that the right node is selected before applying changes. (sorry for long lines...)

augtool> match $conf2/VirtualHost/*[self::directive="DocumentRoot"]
/files/etc/apache2/sites-available/default/VirtualHost/directive[2] = DocumentRoot
augtool> set $conf2/VirtualHost/*[self::directive="DocumentRoot"]/arg /srv/www
augtool> match $conf2/VirtualHost/Directory/arg[. = "/var/www/"]
/files/etc/apache2/sites-available/default/VirtualHost/Directory[2]/arg = /var/www/
augtool> set $conf2/VirtualHost/Directory/arg[. = "/var/www/"] "/srv/www/"
augtoo> save
Saved 1 file(s)

Here is the resulting diff.

--- /etc/apache2/sites-available/default 2011-02-22 23:30:48.000000000 -0500
+++ /etc/apache2/sites-available/default.augnew 2011-02-22 23:48:10.000000000 -0500
@@ -1,12 +1,12 @@
 
  ServerAdmin webmaster@localhost
 
- DocumentRoot /var/www
+ DocumentRoot /srv/www
  <Directory>
   Options FollowSymLinks
   AllowOverride None
  </Directory>
- <Directory /var/www/>
+ <Directory /srv/www/>
   Options Indexes FollowSymLinks MultiViews
   AllowOverride None
   Order allow,deny

Changing permissions

On Ubuntu, Apache is configured to serve documentation on http://localhost/doc/. We will share this documentation with everybody by changing Order and Allow from directive and delete Deny directive of the doc directory. We are using the regexp function to select the node with or without leading slash and with or without quotes. We remove all arguments of "Allow from" to make sure that arguments are sane.

augtool> match $conf2/VirtualHost/Directory[arg =~ regexp(".*/usr/share/doc.*")]/
/files/etc/apache2/sites-available/default/VirtualHost/Directory[4] = (none)
augtool> defvar dir $conf2/VirtualHost/Directory[arg =~ regexp(".*/usr/share/doc.*")]/
augtool> rm $dir/*[self::directive="Allow"]/arg
augtool> set $dir/*[self::directive="Allow"]/arg[1] from
augtool> set $dir/*[self::directive="Allow"]/arg[2] all
augtool> set $dir/*[self::directive="Order"]/arg[1] "allow,deny"
augtool> rm $dir/*[self::directive="Deny"]
rm : $dir/*[self::directive="Deny"] 3
augtool> save
Saved 1 file(s)

In this example, the predicate "*[self::directive='Allow']" can be rewritten "directive[ . = 'Allow']", both are equivalent.

Here is the resulting diff.

--- /etc/apache2/sites-available/default 2011-02-22 23:30:48.000000000 -0500
+++ /etc/apache2/sites-available/default.augnew 2011-02-23 00:10:14.000000000 -0500
@@ -33,9 +33,8 @@
     <Directory "/usr/share/doc">
         Options Indexes MultiViews FollowSymLinks
         AllowOverride None
-        Order deny,allow
-        Deny from all
-        Allow from 127.0.0.0/255.0.0.0 ::1/128
+        Order allow,deny
+        Allow from all
     </directory>

Conclusion

Augeas is a very powerful and precise tool to modify Apache configuration file. We can do about any changes with a very simple tree structure. As always, don't forget to reload Apache configuration for changes to take effect.

Thanks to David Lutterkort, who designed the amazing recursive lens that Apache lens uses and feedback for the square lens, and Raphaël Pinson for the feedback and stress test of the lens on more than 2000 Apache configurations and the help for this blog post.

Happy hacking!

Augedit: regedit for Linux

Few months ago, a prototype of GUI for Augeas was posted on the augeas-devel mailing list. I found it awesome, so I wanted to take it further. I think this is the kind of tool that can easy the configuration management on Linux, like regedit does on Windows. It's still require knowledge of what you're doing, but at least, you don't need to worry about the syntax of a particular config file! Here is a screenshot of the latest version.

On the left pane, there is a GtkTreeView, that shows the tree path and the corresponding value, if any. Then, on the right pane, there is the actual file, where the label and the value are highlighted. At this early stage, the views are read only, but the goal is to make any side editable and make the changes propagate to the other side.

Some of ideas that this kind of tool should have: basic edit operations (add, modify, move, delete, ...) searching, filtering, bookmarks, auto-completion.

If you want to try it, you will need the "myhead" branch of augeas, up to date python-augeas branch "aug_info" that adds the API function for highlighting, and the code of Augedit itself.

http://github.com/giraldeau/augeas
http://github.com/giraldeau/python-augeas
http://github.com/giraldeau/augedit

Leave your ideas about this tool in comments! ;)

Square lens for Augeas

Augeas has the ability to handle XML like tag. Here is an example of how to do it with key and del lens in a simplified version for a paragraph tag of an HTML document.

let para = [ Util.del_str("<") . key "p" . Util.del_str(">") .
                    store /[a-zA-Z0-9 \r\n\t]*/ .
                    Util.del_str("</p>") ]

That's working fine, but there is a gotcha: we have to list all HTML tags as strings, not as a regexp. Let's create a new version of this lens to process arbitrary tag.

let tag = [ Util.del_str("<") . key /[a-z]+/ . Util.del_str(">") .
                 store /[a-zA-Z0-9 \r\n\t]*/ .
                 Util.del_str("</") . del /[a-z]+/ "abc" . Util.del_str(">") ]

Let see what happens with the tree cases get, put and create.

• The get will accept arbitrary tags and set the node label accordingly 
• <p>text</p> ---> {"p" = "text"}
• The put direction, without updating the label, will yield correct result
• {"p" = "text"} ---> <p>text</p>
• In the create operation, in the case of a new label, this will yield the default value for the close tag, and this will produce a syntax error
• <p>text</p> ---> {"p" = "text"} ---> {"b" = "text"} ---> <b>text</abc>

Also, this lens accepts a malformed tags, like "<a>text</p>", and we should throw an error in this case.

We need that the closed tag be linked to the key. The square lens is just about that. Let's rewrite the example with the square lens. The lens takes two arguments. First, a regexp that describe the tag, and behaves as a key. The other is a lens that represent what's inside the tag.

let content = Util.del(">") . store /[a-zA-Z0-9 \r\n\t]*/ . Util.del_str("<")
let tag = [ Util.del_str("<") . square /[a-z]+/ content . Util.del_str(">") ]

The first difference is that, now, the open and close tag are related. In the get direction, we can test that the second tag is the same as the first, and then detect syntax errors in the input document. The other difference is about the create operation in the put direction, because now we can copy the key of the node at the end of the content, and then yield correct behavior.

"<p>text</p>" ---> {"p" = "text"} ---> {"b" = "text"} ---> "<b>text</b>"

The first lens to benefit from it will be httpd.conf lens. Stay tuned, the patch is cooking!

Character feedback for XSugar

In the previous post, we showed that embedding white chars inside the XML is not a viable solution for strict bidirectionality. Automaticaly pre-process the modified XML to fix it and make sure it conforms to the stylesheet seems not possible based on the stylesheet itself, because the XML tree and AST tree are both orthogonal and not related to each other.

How does it works in Augeas? Augeas is based on lenses primitives and combinators. The one we are most interested here is the del lens. In the put direction, the del lens remove the matched content, and stores in a skeleton. In the put direction, the lens restores the original content from the skeleton or provides a default value in case of a new node. The skeleton values are retreived either by sequence or by key. To understand the difference, let's see what happens in case of a subtree move. If skeleton is aligned by sequence, then deleted content by the del lens will be restored in the order in which it appears in the original document. It means that white chars are sticking in place while the content move. In the case of insert, every record following the insert will be modified and get the previous skeleton value, and the last record will get the default value. This behavior doesn't correspond to minimal edit. A better approach is to use the key alignment. Then, skeleton is matched by key, and the correct original chars are selected even in the case of move, insert or delete of records.

The idea behind this principle of operation is to create a feedback of the original document to insert previous values when possible, while preserving modifications.

It's possible to do something similar in XSugar, by working on the AST level. In the rest of this blog, I present the principle of operation, a prototype with basic algorithm, obtained results and conclusion about this solution.

The goal is to to get back original white chars inside the updated document, without embedding them in the XML document. By using the strict version of the stylesheet, where every items are labeled, the resulting AST preserves every parsed chars, and we call strict nodes those that contains chars that would have been lost otherwise. Then, we could use this strict AST to match nodes from the updated XML and the original document, and then copy only strict elements from the original to the updated document.

The matching algorithm used to test this hypothesis works as follow. For each node of the first tree, search in the second tree a node where each child labels and text matches, except for strict nodes. The algorithm works by traversing the AST in level order, and has a complexity of O(n^2).

Here is the expected behavior of this simple algorithm.

• Round trip without updating content should restore all original values.
• Inserted node should get default values.
• Updated nodes should get default values, because they will not match any previous values. They are handled the same way as an insert.
• Moved nodes should get their original values.
• Deleted nodes should not render any previous associated value.

To test and highlight the behavior, I modified slightly the students.xsg example. Here is the format of records that the stylesheet parses :

( [0-9]+ [z]+ [0-9]+ [a]+ )*

Hence, two numbers are separated by space, and multiple records are separated by new line, but "z" represents spaces and "a" represents new line. The default number of z's and a's is one, to show the behavior, we use two and more chars to detect when a default value is inserted.

A parse tree and a match between original document and updated document in which a record has been deleted is shown in the following figure.

Here are the results of tests.

Test 1 : round trip without modifications

input:   1zz1aa11zzz11aaa111zzzz111aaaa

default: 1z1a11z11a111z111a

merged:  1zz1aa11zzz11aaa111zzzz111aaaa

All characters are recovered, and the resulting document is exactly the same as the input.

Test 2: insert

input:   1zz1aa11zzz11aaa111zzzz111aaaa

default: 1z1a11z11a99z88a111z111a

merged:  1zz1aa11zzz11aaa99z88aaaa111zzzz111a

The new record "99 88" gets a default "z", while the last record gets a default "a".

Test 2 : move

input:   1zz1aa11zzz11aaa111zzzz111aaaa

default: 1z1a111z111a11z11a

merged:  1zz1aa111zzzz111aaa11zzz11aaaa

The moved record gets the right number of "z", but "a" chars keeps their order.

Test 4 : delete

input:   1zz1aa11zzz11aaa111zzzz111aaaa

default: 1z1a111z111a

merged:  1zz1aa111zzzz111aaa

The central record is deleted, and the last record gets 3 a's in place of 4.

From this results, we can see that strict elements z's are retrieved by keys, and a's by sequence. This is caused by the fact that branch nodes that have only one strict leaf nodes are match by their level in the tree. We can't determine if this node should be matched to the parents, or are themselves a parents for other nodes, and in consequence, sequence match is the only remaining possible match. As I understand, other tree matching algorithms can't address this issue.

The algorithm can be improved to match nodes by leaf node string similarity, using Levenshtein distance. It may be possible to restore chars from updated nodes. But then, the match will depend on the number of changes made to the document, and a node may match another one more similar, may confuse the user by an apparent non-deterministic behavior and may produce non-minimal edit.

In conclusion, we showed that it's possible to use the original document to merge back otherwise lost chars. But, alignment of original content is mixed between sequence and key. The user can't control which one will apply. It may lead to non-minimal edit of the resulting file. In that respect, lenses primitives and combinators used in Augeas provides a more predictive behavior, by letting explicitly choose between key or sequence alignment by the lens developer.

Limitations of XSugar

In previous posts, I explained how XSugar could be used to transform configuration files to XML and back. Preserving white characters (space, tab, carriage return, etc.) is possible by creating special elements to hold them. In a simple round-trip, when the XML document is not modified, it works well. But, there are issues when the XML document is modified. For example, be this simple XML with strict nodes, from the students example of XSugar.

<students xmlns="http://studentsRus.org/">
  <student sid="19701234">
    <name>John Doe</name>
    <email>john_doe@notmail.org</email>
  </student>
  <space1> </space1>
  <space2> </space2>
  <newline>
</newline>
  <student sid="19785678">
    <name>Jane Dow</name>
    <email>dow@bmail.org</email>
  </student>
  <space1> </space1>
  <space2> </space2>
  <newline>
</newline>
</students>

There are spaces and newline elements that follow the student record. They are used to restitute the exact formating of the original string. We can then modify an existing value, and spaces will be preserved, and this correspond to a minimal edit.

But, say another record is inserted after the first student, but before space1 element, then the first record will see it's spacing reset to default values, and the new inserted element will get the spaces from the first record! This doesn't correspond to minimal edit, and is similar to alignment problem found by Foster et. al. in Boomerang.

If a student element is removed, but spaces elements are left, then it will cause a syntax error according to XSugar stylesheet. If an element is moved, it must inserted to right place in the XML, otherwise syntax error may occur.

It may be possible to fix the XML before transformation to non-XML, by adding, moving or deleting space elements. The idea to use the original document and the modified one to produce the updated output is already what does a lens, and is exactly what Augeas does. And now, with the introduction of the recursive lens, Augeas is about to support context-free languages, and opens the door to handle the Web server Apache configuration, and many other.

That's all for now, stay tuned!

Augeas : state of art configuration management

If you are like me, you probably find that configuration management under Linux is a real mess. Each software has it's own configuration file syntax and semantic. Since every project has it's own needs and is developed by different teams, a global and standard repository for configuration is unlikely to ever work in the open source model. We saw some initiatives that worked, like LSB and Freedesktop, but such a shift for configuration file would require a huge coordination and agreement, and is unlikely to happen, and since configuration values are usually highly coupled in software, it makes it hard to change it.

There are few ways we can address the situation. You can choose to not manage at all your config files, you can do backups, use revision management software like subversion. You can also parametrize your configuration files with variables and generate them. You can also do some grep'ing and sed'ing on your config files to change values. Everything there tends to be cumbersome and error prone. But, there is better. If it was possible to have a parser for every config file out there and get in memory representation, change whatever you want and save it back, then you have the st-graal of file configuration management. This way, you can manage configuration at a high semantic level, not manipulate file lines but configuration elements.

Guess what, Augeas aims is exactly that! The concept is simple. For each file type, you write a lens that describe the format of the file. Then, the lens can be used to parse an instance of that config file and you get a tree in memory. You can access the tree and modify it. When you're done, the lens is used again to write back the file, including any modification you made to the tree. A lens is bidirectional, you don't have to specify how to read and write a file, only one description describe both directions.

Last year, at the Linux Symposium, I listen to the talk of David Lutterkort about the progress of the project. There is one limitation of Augeas today related to the type of files for which you can write a lens. For example, we can't write lens to process config files that have recursive structures. For example, Apache configuration has this kind of nested structure. Actualy, this is not possible, because the lens semantic doesn't allow to define recursive grammar.

So, I choosed to do that for my master. When this will be done, it will be possible to manage more file types with Augeas. I will post my findings here, thoughts and ideas on the project. Stay tuned!