Few months ago, a prototype of GUI for Augeas was posted on the augeas-devel mailing list. I found it awesome, so I wanted to take it further. I think this is the kind of tool that can easy the configuration management on Linux, like regedit does on Windows. It's still require knowledge of what you're doing, but at least, you don't need to worry about the syntax of a particular config file! Here is a screenshot of the latest version.
On the left pane, there is a GtkTreeView, that shows the tree path and the corresponding value, if any. Then, on the right pane, there is the actual file, where the label and the value are highlighted. At this early stage, the views are read only, but the goal is to make any side editable and make the changes propagate to the other side.
Some of ideas that this kind of tool should have: basic edit operations (add, modify, move, delete, ...) searching, filtering, bookmarks, auto-completion.
If you want to try it, you will need the "myhead" branch of augeas, up to date python-augeas branch "aug_info" that adds the API function for highlighting, and the code of Augedit itself.
http://github.com/giraldeau/augeas
http://github.com/giraldeau/python-augeas
http://github.com/giraldeau/augedit
Leave your ideas about this tool in comments! ;)
vendredi 8 octobre 2010
jeudi 2 septembre 2010
Windows Phone 7 adventure
I went to Ottawa Mobile Conference, and I won as a presence prize 2 day training on Windows Phone 7 (WP7) plateform, at the Microsoft office in Ottawa. The training is provided by Colin Melia from DevTeach, which is knows the plateform very well. This was for me the occasion to get an idea of what it will look like. With the iOS, Android and RIM devices, is there a place for WP7? What it stands for? Millions of theses devices are supposed to hit the market before Thanks Giving in the US, made by Samsung, LG and Dell and others, so it has to be good.
WP7 is the Microsoft implementation of the whole mobile concept from Apple. Mobile Market is equivalent of AppStore, Zune have the same role as iTunes to sync the device and basic apps are about the same, except they are based on Microsoft technology, for example Internet Explorer replace Safari, Bing map and search replaces Google. A light version of Office and SharePoint may be free on the phone for the release. XboxLive will be the basis for all games on the device. Interesting fact, IE doesn't ship with Flash plugin. Seems that Microsoft agree with Apple on this point.
Concerning the interface, it uses the design pattern Hub and Spoke, that force the user to focus on one task at a time. A tile view of apps acts as the Hub. There are three mandatory physical button on the phone itself: back, start and search. Back goes to the previous view, start goes to the tile view and the search go to Bing. There can't multi-process running simultaneously, the same as the iPhone before multitasking. Say that you start an app, then go to the tile view and start another app, the first app instance is lost.
You can use two different libraries for an app, Silverlight for straight forward forms or XNA for games that requires more heavy video display. We saw mainly how to use Silverlight to design the UI. The application code can be written in C#, using the .NET framework classes. What is realy interesting is the clear separation between the logic and display of the application. It has been thought in such a way that the designer can work on the view independently of the programmer, that fills the view. Some preview data can be supplied to the view at design time, so the designer doesn't need the whole application to work. The UI is completely defined in XAML file in a declarative manner. A factory then read the file at runtime and create corresponding objects. .NET library provides a high level of abstraction for XML handling, HTTP and SQL queries, so in very few lines of code, a lot of things occur.
For those of you who may want to give it a try, here are some notes about installation on a MacBook Pro 1,1. The requirement and installation instructions are accurate. You need the VMX extension for virtualization to get good performances. It has to be enabled into EFI boot loader. See the Parallels knowledge base to get instructions. Microsoft supplies a small utility to check if the extension is available. But, maybe more critical is this MacBook Pro have an ATI X1600 card, and the latest (and the last) driver from AMD doesn't support WDDM 1.1, which is required for decent display performance with the phone emulator. Any computer with proper drivers for Windows 7 should do, it just means I will need a new computer to run the emulator appropriately.
In conclusion, Microsoft is jumping in the arena seriously with WP7. SDK is quite lean yet, and this is a great piece of integrated technology AFAIK. Their first step is obviously to copy recipies of competitors with their own technology. It's right, because innovation comes a lot from inspiration from others. iPhone is still the blue ocean device, and except for the Office app, there is nothing that a WP7 seems to do better, so the price point must be somewhat attractive to change the market forces. The fate of WP7 is now in the hand of customers.
WP7 is the Microsoft implementation of the whole mobile concept from Apple. Mobile Market is equivalent of AppStore, Zune have the same role as iTunes to sync the device and basic apps are about the same, except they are based on Microsoft technology, for example Internet Explorer replace Safari, Bing map and search replaces Google. A light version of Office and SharePoint may be free on the phone for the release. XboxLive will be the basis for all games on the device. Interesting fact, IE doesn't ship with Flash plugin. Seems that Microsoft agree with Apple on this point.
Concerning the interface, it uses the design pattern Hub and Spoke, that force the user to focus on one task at a time. A tile view of apps acts as the Hub. There are three mandatory physical button on the phone itself: back, start and search. Back goes to the previous view, start goes to the tile view and the search go to Bing. There can't multi-process running simultaneously, the same as the iPhone before multitasking. Say that you start an app, then go to the tile view and start another app, the first app instance is lost.
You can use two different libraries for an app, Silverlight for straight forward forms or XNA for games that requires more heavy video display. We saw mainly how to use Silverlight to design the UI. The application code can be written in C#, using the .NET framework classes. What is realy interesting is the clear separation between the logic and display of the application. It has been thought in such a way that the designer can work on the view independently of the programmer, that fills the view. Some preview data can be supplied to the view at design time, so the designer doesn't need the whole application to work. The UI is completely defined in XAML file in a declarative manner. A factory then read the file at runtime and create corresponding objects. .NET library provides a high level of abstraction for XML handling, HTTP and SQL queries, so in very few lines of code, a lot of things occur.
For those of you who may want to give it a try, here are some notes about installation on a MacBook Pro 1,1. The requirement and installation instructions are accurate. You need the VMX extension for virtualization to get good performances. It has to be enabled into EFI boot loader. See the Parallels knowledge base to get instructions. Microsoft supplies a small utility to check if the extension is available. But, maybe more critical is this MacBook Pro have an ATI X1600 card, and the latest (and the last) driver from AMD doesn't support WDDM 1.1, which is required for decent display performance with the phone emulator. Any computer with proper drivers for Windows 7 should do, it just means I will need a new computer to run the emulator appropriately.
In conclusion, Microsoft is jumping in the arena seriously with WP7. SDK is quite lean yet, and this is a great piece of integrated technology AFAIK. Their first step is obviously to copy recipies of competitors with their own technology. It's right, because innovation comes a lot from inspiration from others. iPhone is still the blue ocean device, and except for the Office app, there is nothing that a WP7 seems to do better, so the price point must be somewhat attractive to change the market forces. The fate of WP7 is now in the hand of customers.
mardi 31 août 2010
No network, no problem!
I got a brand new smart phone, and the first thing I noticed is that it worth nothing without network. Wifi hotspot availability is uncertain, everybody has yet beg someone to get some access. 3G data is still expansive. Roaming above all, at 20$ bucks per megabyte, could ruine somebody looking at few youtube videos. And even 3G network is not always available, for example in subways, and will make your device as useful as a stone.
We already know the solution: data cache. At least, a device with some type of cache will be useful at some level. I want to bring with me my own files on any of my devices. This way, we do not rely on the cloud for every lookup, and we can sync with low cost network connexion when it becomes available. Mail is an obvious example of asynchronous system that is working well. With IMAP and a good mail client, you can get a cache of all your emails, read them when offline, even compose an email and send them later when network is back.
Another drastic example of a device that rely only on a cache is the WikiReader, that I bought recently. It has a cache of all 3 millions Wikipedia articles on a 8 Gb SSD card. It can be updated with a computer or by subscribing to an update service by old fashion mail. Never forget that a truck loaded with SSD cards has a lot of bandwith!
Though, keeping in sync multiple disconnected data sets is harder. Yet, many interesting projects, like the distributed replicated database system CouchDB, used in Ubuntu One for online services, borrow my contacts where ever I need them. At the file system level, I'm using Dropbox that replicates my files on all my devices. Beyong replication, it acts as a continuous backup, so I don't worry anymore with hard drive crashed. Also, HTML5 cache is a great addition, and can be used to lower your precious G3 bandwith, server load and lower latency. Those are example of disconnected aware cloud services, that I think are going to take off.
This said, sync all data does make sens only if the device belong to the user for a fair amount of time. In the case of shared workstation, for example a school lab, roaming profiles are known to be a disaster, because the data has to be synchronized from scratch multiple time, and it makes the cache useless. We may think of better synchonization, based on the content the user uses frequently, and fetch other things on demand. But the "keep it simple and stupid" way would be to make user reuse the same computer, or at least the same set of computers, and not flush the cache. Keeping users files on a shared computer is always tricky for privacy reasons, but providing encryption based on the user's credentials could be a solution. Another solution would be to let the student own and borrow their computer. With all those Linux mobile tablets that are going to hit the market soon, I think that it will create a great opportunity for providing affordable computers to each and every students. And with 16Gb of flash storage on basic models, there is enough room to cache almost everything! This way, the cache hit will be much higher, and users will be happier.
In summary, let's take a GPS as an example. Would you prefer a GPS that downloads all it's bitmap maps just-in-time, that rely heavily on network, or a GPS that is using it's embedded geo data. The later will not work without network, the later is not ofter updated, or even not at all. What I would consider the best design is to get a device with both. The mapping data should be embedded and rendered on the device itself, and this sort of cache should be updated each time you get network. Or even if you don't have all the data, you should be able to select areas to cache.
In conclusion, the cloud is so fantastic that we sometimes forget that without a link to it, we're screwed. I consider that devices that are not designed to work without network is bad design, and those devices will end into the Computer History Museum soon.
We already know the solution: data cache. At least, a device with some type of cache will be useful at some level. I want to bring with me my own files on any of my devices. This way, we do not rely on the cloud for every lookup, and we can sync with low cost network connexion when it becomes available. Mail is an obvious example of asynchronous system that is working well. With IMAP and a good mail client, you can get a cache of all your emails, read them when offline, even compose an email and send them later when network is back.
Another drastic example of a device that rely only on a cache is the WikiReader, that I bought recently. It has a cache of all 3 millions Wikipedia articles on a 8 Gb SSD card. It can be updated with a computer or by subscribing to an update service by old fashion mail. Never forget that a truck loaded with SSD cards has a lot of bandwith!
Though, keeping in sync multiple disconnected data sets is harder. Yet, many interesting projects, like the distributed replicated database system CouchDB, used in Ubuntu One for online services, borrow my contacts where ever I need them. At the file system level, I'm using Dropbox that replicates my files on all my devices. Beyong replication, it acts as a continuous backup, so I don't worry anymore with hard drive crashed. Also, HTML5 cache is a great addition, and can be used to lower your precious G3 bandwith, server load and lower latency. Those are example of disconnected aware cloud services, that I think are going to take off.
This said, sync all data does make sens only if the device belong to the user for a fair amount of time. In the case of shared workstation, for example a school lab, roaming profiles are known to be a disaster, because the data has to be synchronized from scratch multiple time, and it makes the cache useless. We may think of better synchonization, based on the content the user uses frequently, and fetch other things on demand. But the "keep it simple and stupid" way would be to make user reuse the same computer, or at least the same set of computers, and not flush the cache. Keeping users files on a shared computer is always tricky for privacy reasons, but providing encryption based on the user's credentials could be a solution. Another solution would be to let the student own and borrow their computer. With all those Linux mobile tablets that are going to hit the market soon, I think that it will create a great opportunity for providing affordable computers to each and every students. And with 16Gb of flash storage on basic models, there is enough room to cache almost everything! This way, the cache hit will be much higher, and users will be happier.
In summary, let's take a GPS as an example. Would you prefer a GPS that downloads all it's bitmap maps just-in-time, that rely heavily on network, or a GPS that is using it's embedded geo data. The later will not work without network, the later is not ofter updated, or even not at all. What I would consider the best design is to get a device with both. The mapping data should be embedded and rendered on the device itself, and this sort of cache should be updated each time you get network. Or even if you don't have all the data, you should be able to select areas to cache.
In conclusion, the cloud is so fantastic that we sometimes forget that without a link to it, we're screwed. I consider that devices that are not designed to work without network is bad design, and those devices will end into the Computer History Museum soon.
dimanche 8 août 2010
Firefox memory management nightmare
Can you believe that you can crash any linux machine with a web page displaying stamp size images, while an iPod can display it without an glitch? Try this dead page with you browser, that displays large images resized and watch your memory usage.
WARNING: this page requires ~300Mb RAM to display!
https://pages.usherbrooke.ca/fgiraldeau/crashff/
Ok, what's going on? I did a small utility to monitor the memory usage of Firefox, and the memory usage was quite normal. Where did all this memory allocation occur? After few top, it appears that the Xorg process blew up. Here is a graph that show both process on crashff loading.
The test shows the page loading, opening other tabs and coming back to the original page. The full scale image is transfered to the Xserver for rescaling the image. On localhost, it's fast, but when Firefox is started remotely, then all this data has to be transfered on the network! After a while, the image is flushed from Xserver, and display the tab requires doing all this work again. And because the Xserver allocates the memory, if you run out of memory, the Out Of Memory (OOM) killer of the kernel will be triggered, but it's likely to kill the Xorg process and leads to the user's session lost. This is really critical.
There is one optimization that may help this, and wanted to know the impact. When setting MOZ_DISABLE_IMAGE_OPTIMIZE to 1, it was supposed to change the behavior. Let see the same experiment with this environment value set.
Ok, so we see that the Xorg process stays at some low level of memory usage, this time firefox blows up. At least, if the user hits the memory limit, then only the Firefox process gets killed, and the user's session will not crash.
I did also the test with Opera 10.60.6386 and Konqueror 4.4.2, and the result is mostly the same as with Firefox with optimization. Konqueror uses a less memory, and doesn't deallocate the tab, so when displaying it again, it's really fast. Here are the graphs.
So MOZ_DISABLE_IMAGE_OPTIMIZE does the trick, and IMHO, this option should be the default.
Note: this blog is related to this firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=395260
WARNING: this page requires ~300Mb RAM to display!
https://pages.usherbrooke.ca/fgiraldeau/crashff/
Ok, what's going on? I did a small utility to monitor the memory usage of Firefox, and the memory usage was quite normal. Where did all this memory allocation occur? After few top, it appears that the Xorg process blew up. Here is a graph that show both process on crashff loading.
![]() |
| Memory consumption for crashff page loading according to time |
The test shows the page loading, opening other tabs and coming back to the original page. The full scale image is transfered to the Xserver for rescaling the image. On localhost, it's fast, but when Firefox is started remotely, then all this data has to be transfered on the network! After a while, the image is flushed from Xserver, and display the tab requires doing all this work again. And because the Xserver allocates the memory, if you run out of memory, the Out Of Memory (OOM) killer of the kernel will be triggered, but it's likely to kill the Xorg process and leads to the user's session lost. This is really critical.
There is one optimization that may help this, and wanted to know the impact. When setting MOZ_DISABLE_IMAGE_OPTIMIZE to 1, it was supposed to change the behavior. Let see the same experiment with this environment value set.
![]() |
| Memory consumption for crashff page loading, with optimization, according to time |
Ok, so we see that the Xorg process stays at some low level of memory usage, this time firefox blows up. At least, if the user hits the memory limit, then only the Firefox process gets killed, and the user's session will not crash.
I did also the test with Opera 10.60.6386 and Konqueror 4.4.2, and the result is mostly the same as with Firefox with optimization. Konqueror uses a less memory, and doesn't deallocate the tab, so when displaying it again, it's really fast. Here are the graphs.
So MOZ_DISABLE_IMAGE_OPTIMIZE does the trick, and IMHO, this option should be the default.
Note: this blog is related to this firefox bug: https://bugzilla.mozilla.org/show_bug.cgi?id=395260
samedi 3 juillet 2010
Mouse safety
For those of you that are working long hours in front of a computer, you may encounter pain in your wrist at some point. This may be caused by repetitive movement, that creates inflammation of tendon.
The inflammation can be treated with ice, rest and acetaminophen (paracetamol). But the cause, for me, was the abuse of the scroll wheel. Since I'm avoiding to scroll, my hand and wrist pain diminish progressively.
Hope that this trick helps you. Happy hacking!
The inflammation can be treated with ice, rest and acetaminophen (paracetamol). But the cause, for me, was the abuse of the scroll wheel. Since I'm avoiding to scroll, my hand and wrist pain diminish progressively.
Hope that this trick helps you. Happy hacking!
mercredi 9 juin 2010
APLWACA what?
I went to the conference Analysis and Programming Languages for Web Applications and Cloud Applications (APLWACA) held in Toronto, and I wanted to share one cool stuff I learned: Javascript security.
The kenote was about the security of untrusted Javascript that browsers may run. This is actually happenning because trusted website can embedded code from other, like analysis code to gather statistics and display ads. Here are two examples of those threath.
This code will replace every links in a document to somewhere you don't trust.
Another bigger threath is that, if you have an active session with a trusted server, then the actual embedded Javascript code can interact with your session like this:
To prevent this, all code from untrusted Javascript should be wrapped to prevent calls to XMLHttpRequest. Here is an actual snippet of code that verify simple lookup o[s].
Some implementation of this kind of wrapper exists, like Facebook Javascript, Google Caja and Yahoo ADsafe. But, how can we actually prove that these wrappers are safe, and they really don't allow to execute the XMLHttpRequest? For example, our simple lookup, s may be an object, will not be equal to the searched string and will be evaluated. But then, s.toString() function will be evaluated, and can return "XMLHttpRequest"!
The solution proposed is to statically typecheck Javascript. String that may evaluates to "XMLHttpRequest" are carried, and then all code is checked to look at unsafe calls that may be made.
Static typecheck has been performed on ADsafe itself. It showed one possible exploit as far. The result is that it could be possible to prove it's secure.
In the mean time, disabling Javascript may not be a bad idea after all...!
The kenote was about the security of untrusted Javascript that browsers may run. This is actually happenning because trusted website can embedded code from other, like analysis code to gather statistics and display ads. Here are two examples of those threath.
This code will replace every links in a document to somewhere you don't trust.
els = document.getElementsByTafName("a");
for (var el in els) {
el[url] = "http://dangerous-site.com/";
}Another bigger threath is that, if you have an active session with a trusted server, then the actual embedded Javascript code can interact with your session like this:
var x = window.XMLHttpRequest();
x.open("/account");
x.send("some nasty command");To prevent this, all code from untrusted Javascript should be wrapped to prevent calls to XMLHttpRequest. Here is an actual snippet of code that verify simple lookup o[s].
lookup function(o, s){
if (s == "XMLHttpRequest"){
return "Not allowed";
} else {
return o[s];
}
}Some implementation of this kind of wrapper exists, like Facebook Javascript, Google Caja and Yahoo ADsafe. But, how can we actually prove that these wrappers are safe, and they really don't allow to execute the XMLHttpRequest? For example, our simple lookup, s may be an object, will not be equal to the searched string and will be evaluated. But then, s.toString() function will be evaluated, and can return "XMLHttpRequest"!
The solution proposed is to statically typecheck Javascript. String that may evaluates to "XMLHttpRequest" are carried, and then all code is checked to look at unsafe calls that may be made.
Static typecheck has been performed on ADsafe itself. It showed one possible exploit as far. The result is that it could be possible to prove it's secure.
In the mean time, disabling Javascript may not be a bad idea after all...!
mardi 1 juin 2010
Square lens for Augeas
Augeas has the ability to handle XML like tag. Here is an example of how to do it with key and del lens in a simplified version for a paragraph tag of an HTML document.
let para = [ Util.del_str("<") . key "p" . Util.del_str(">") .
store /[a-zA-Z0-9 \r\n\t]*/ .
Util.del_str("</p>") ]
That's working fine, but there is a gotcha: we have to list all HTML tags as strings, not as a regexp. Let's create a new version of this lens to process arbitrary tag.
let tag = [ Util.del_str("<") . key /[a-z]+/ . Util.del_str(">") .
store /[a-zA-Z0-9 \r\n\t]*/ .
Util.del_str("</") . del /[a-z]+/ "abc" . Util.del_str(">") ]
Let see what happens with the tree cases get, put and create.
Also, this lens accepts a malformed tags, like "<a>text</p>", and we should throw an error in this case.
We need that the closed tag be linked to the key. The square lens is just about that. Let's rewrite the example with the square lens. The lens takes two arguments. First, a regexp that describe the tag, and behaves as a key. The other is a lens that represent what's inside the tag.
let content = Util.del(">") . store /[a-zA-Z0-9 \r\n\t]*/ . Util.del_str("<")
let tag = [ Util.del_str("<") . square /[a-z]+/ content . Util.del_str(">") ]
The first difference is that, now, the open and close tag are related. In the get direction, we can test that the second tag is the same as the first, and then detect syntax errors in the input document. The other difference is about the create operation in the put direction, because now we can copy the key of the node at the end of the content, and then yield correct behavior.
"<p>text</p>" ---> {"p" = "text"} ---> {"b" = "text"} ---> "<b>text</b>"
The first lens to benefit from it will be httpd.conf lens. Stay tuned, the patch is cooking!
let para = [ Util.del_str("<") . key "p" . Util.del_str(">") .
store /[a-zA-Z0-9 \r\n\t]*/ .
Util.del_str("</p>") ]
That's working fine, but there is a gotcha: we have to list all HTML tags as strings, not as a regexp. Let's create a new version of this lens to process arbitrary tag.
let tag = [ Util.del_str("<") . key /[a-z]+/ . Util.del_str(">") .
store /[a-zA-Z0-9 \r\n\t]*/ .
Util.del_str("</") . del /[a-z]+/ "abc" . Util.del_str(">") ]
Let see what happens with the tree cases get, put and create.
- The get will accept arbitrary tags and set the node label accordingly
- <p>text</p> ---> {"p" = "text"}
- The put direction, without updating the label, will yield correct result
- {"p" = "text"} ---> <p>text</p>
- In the create operation, in the case of a new label, this will yield the default value for the close tag, and this will produce a syntax error
- <p>text</p> ---> {"p" = "text"} ---> {"b" = "text"} ---> <b>text</abc>
Also, this lens accepts a malformed tags, like "<a>text</p>", and we should throw an error in this case.
We need that the closed tag be linked to the key. The square lens is just about that. Let's rewrite the example with the square lens. The lens takes two arguments. First, a regexp that describe the tag, and behaves as a key. The other is a lens that represent what's inside the tag.
let content = Util.del(">") . store /[a-zA-Z0-9 \r\n\t]*/ . Util.del_str("<")
let tag = [ Util.del_str("<") . square /[a-z]+/ content . Util.del_str(">") ]
The first difference is that, now, the open and close tag are related. In the get direction, we can test that the second tag is the same as the first, and then detect syntax errors in the input document. The other difference is about the create operation in the put direction, because now we can copy the key of the node at the end of the content, and then yield correct behavior.
"<p>text</p>" ---> {"p" = "text"} ---> {"b" = "text"} ---> "<b>text</b>"
The first lens to benefit from it will be httpd.conf lens. Stay tuned, the patch is cooking!
Inscription à :
Articles (Atom)




