Salt, we use it everywhere

I have dedicated my mind for the past little while to somehitng that I have developed very high hopes for, something that I call -Salt.

For a few years now I have enjoyed a system management application called func. Func, the Fedora Unified Network Controller is a remote execution system which lets you execute programs on a swath of systems. Func really acts like an interface to your infrastructure as a whole.

Now, while I adore func, and it has been a huge asset to my work in system development, I kept running into issues when I started to make the infrastructures more dynamic, primarily because func hangs when it cannot find a server, and func is somewhat serial, communicating with a single system at a time.

I have found that these problems need to be addressed in the architecture of the remote execution system. Red Hat and Puppet Labs seem to be addressing this problem in the same way, via the new and fancy amqp protocols. Puppet Labs recently purchased the Marionate Collective, and in the ongoing Red Hat cloud project OVirt, func has been replaced with an application called Matahari. Both of these systems use external AMQP servers, and both systems are very powerful, as well as somewhat complex.

But as an Arch Linux community developer, and an avid believer in the Arch Linux way, I wanted something that is extremely portable, fast, efficient, simple to set up and use, completely self contained and very programmable. So I have started to develop a func replacement which I call “Salt”.

Instead of using an existing AMQP broker (which adds complexity) I have built in the targeted features of a message passing broker via the ZeroMQ api, so the broker is built in and completely flexible. Instead of using slower ssl encryption and public key infrastructue for encryption, I am using pki only for the authentication layer (which is simple, and built in) and then using a revolving aes key for payload encryption.

Next the command execution is decoupled, so commands which are published out can be sent in a continued stream and then the return from the commands can be read in a lazy manner. This means that Salt can be used for more communication processes than just remote execution. I hope to make Salt able to handle everything from replacing remote execution systems, like the aforementioned ones, but also as a replacement for systems like Cacti and Munin which are used for stat monitoring. Finally Salt will hopefully work for managing highly dynamic infrastructures, and manage them in a more efficient way, since commands are sent only to systems which are listening to the Salt publisher, and since the command execution is decoupled, all systems receive the commands in parallel.

Before you get excited about salt development, it is still under heavy development (read – “it no worky yet”), but it is getting close, the message passing framework is complete, and the encryption and authentication are also functional. So it is coming along, and I expect the first alpha release to be available in the next few weeks. If you want to follow salt production (and you should!) check out the repo on github:

https://github.com/thatch45/salt

Advertisements

2 responses to this post.

  1. Cool! I’m going to have to play with this.

    FWIW, as for Func being serial, “–async –forks 10”, but yeah, those can block within a fork and once a given node has a work queue assigned to it, it can’t be served by another fork — so there were limitations to forkbomb.py

    The push based stuff wasn’t super awesome. I think the one thing that would be really nice is being able to queue up commands for online systems and be able to store the results of the last run of a particular module. Puppet’s storeconfigs wanted to be SORT OF, this way, but it is quite slow and I’m sure it could be done much better. Storing data in something like Riak/Couch seems possible, but you’d want some kind of access control around it. In Puppet this was quite a performance bottleneck, but a lot of it had to do with the server being very CPU bound, if it’s just a no-SQL database that the server is dumping data into more or less blind, those bottlenecks would be gone. Basically I thinking if you do something about how func-inventory worked with git, you could basically build a monitoring application off it almost for free. Or any kind of data collection based stuff. You could also have the systems check in to get “orders” from that database.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: