Salt Minion Swarm!

I just committed a new kind of test system to salt, one of the problems we have been running into is that in very large deployments some strange and difficult to reproduce issues were popping up. To overcome this issue I developed a script that spins up massive numbers of minions with different ids on a single system.

While the minion swarm script is young and could still use additions, it has allowed me to test working with 1000 minions with only one physical server (granted, it is a somewhat beefy server).

I have discovered some important things, like the fact that a salt master running on a vm with 256 megs of ram has a hard time when you call 1000 minions to authenticate at the same time (the salt master on this vm was able to handle 500 minions without issue). 

Using the new script is easy, just make sure that the minion is installed on the system and then execute the minion swarm script. The script can be downloaded from here:

https://raw.github.com/saltstack/salt/develop/tests/minionswarm.py

Currently it does not take a master argument, so the master that the swarm will be attaching to needs to be resolvable via the dns name “salt”.

When invoking the script the number of minions to spin up can be set with the -m flag, so only do this if you have a lot of ram:

./minionswarm.py -m 1000

 

Any way about it, it is a fun tool to work with and has already helped me track down some salt bugs!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: