A Taste of Salt: Like Puppet, But Less Frustrating

A taste of salt

Have you gotten frustrated with Puppet? Corey Quinn offers an impassioned argument for his favorite open source alternative.

If you’re responsible for the care and feeding of multiple servers, and you haven’t heard about configuration management yet, you have not been paying attention. CFengine was one of the first configuration management systems that was deployed in anything approaching widespread use, and was followed later by Puppet and Chef. A bit over two years ago, Salt Stack‘s “Salt” entered the market, and took a radically different approach to the problem of “configure all of my servers to do X.”

Salt started life as a remote execution system: a class of software applications written to address concerns of the form, “I have this command I want to run across 1,000 servers. I want the command to run on all of those systems within a five second window. It failed on three of them, and I need to know which three.”

Other systems were designed to do this, of course, but they failed in several ways. MCollective (which Puppet Labs acquired several years ago) was (and remains!) fiendishly complex to set up. Chef works atop ssh, which – while the gold standard for cryptographically secure systems management – is computationally expensive to the point where most master servers fall over under the weight of 700-1500 clients. Salt’s approach was far simpler.

Salt leverages the ZeroMQ message bus, a lightweight library that serves as a concurrency framework. It establishes persistent TCP connections between the Salt master and the various clients, over which communication takes place. Messages are serialized using msgpack, (a more lightweight serialization protocol than JSON or Protocol Buffers), resulting in severe speed and bandwidth gains over traditional transport layers, resulting in in the ability to fit far more data quickly through a given pipe. This translates into a non-technical statement of, “Salt establishes a persistent data pipe between servers in your environment that’s extremely fast and low-bandwidth.”

Getting up and running is almost embarrassingly simple compared to other configuration management systems. Setup consists of three steps:

  • Install the package on the master server (clients are referred to as “minions”), and start the salt-master service. (apt-get install salt-master on Debian-based systems, though there are packages for virtually every operating system in widespread use, including an agent for Windows servers.)
  • Make sure that the host “salt” resolves (via DNS hacks or editing the hosts file) on the minion to the IP of the master server; then install the package and start the salt-minion service (apt-get install salt-minion).
  • Accept the minion’s key on the master. (salt-key -a minion_hostname)

At this point, you’re done; you’ve achieved remote execution.

From the master you can run commands across any or all of your minions in a massively parallel manner:

[root@salt]# salt '*' cmd.run "date"

{'log': 'Sat Mar 9 23:18:27 PST 2013'}

{'irc': 'Sat Mar 9 23:18:27 PST 2013'}

{'git': 'Sat Mar 9 23:18:27 PST 2013'}

{'code': 'Sat Mar 9 23:18:27 PST 2013'}

{'mail': 'Sat Mar 9 23:18:27 PST 2013'}

{'www': 'Sat Mar 9 23:18:27 PST 2013'}


Once this system worked, it didn’t take much for the Salt creators to make the logical leap from “running arbitrary commands across an entire server fleet” to realizing they had the makings of a superior configuration management system on their hands. Its use for a configuration management system follows the same model as above, but instead of “run this command,” Salt extended it. Now you can make sure that Apache is installed, /etc/sudoers has these contents, or postfix is running.

The syntax for this is written simply in YAML, an intuitively obvious “list” format.

For instance, to ensure that your ssh server’s configuration is uniform across all of your systems, you would drop a copy of what that file should look like onto your salt master, then call it from the state tree:



    - managed

>    – source: salt://base/files/sshd_config

The next time the system is run, that file is pushed out to all of the listening minions.

What made Salt compelling for my use was that, following the steps in the tutorial, all of what I just described took less than 20 minutes to get working. Someone new to configuration management can be productive with Salt before lunchtime, and someone who’s familiar with configuration managers can perform complex tasks relatively easily. Simple is good.

The folks at SaltStack seem to be unwilling to settle for merely “doing what other configuration management solutions can do.” Part of the value that Salt is adding is in their ecosystem of other offerings tied into Salt.

For instance, Salt-cloud serves as an orchestration layer that can spin up instances in EC2, Openstack, Linode, or other cloud computing providers. The idea is that Salt lets you not only define configuration within running instances, but also handle the initial provisioning, ongoing maintenance, and deprovisioning of the entire instance swarm.

Salt-virt is a newly released module that serves as a well-architected wrapper around the libvirt, a popular virtualization library for KVM, Xen, OpenVZ, and others. This is similar to salt-cloud, but rather than making API calls to a third-party provider, it makes libvirt calls to hypervisors on your behalf to instantiate new virtualization guests, as well as providing monitoring data around running instances.

Salt-vagrant is a salt extension that lets you use Salt to manage Vagrant instances. Vagrant is an established project that wraps around a number of existing virtualization providers to allow for extremely quick provisioning of disposable, consistent environments. This is targeted specifically at developers, to solve the fairly common “Develop on a Mac to deploy into a Linux environment” library problems that sometimes crop up.

Salt-monitor is also in the works as one of the next major milestones. This is intended to serve as a soup-to-nuts monitoring solution that scales well, by replacing serial checks iterating through an environment with broadcast conversations, such as, “All servers: Tell me if you don’t have enough free disk space.”

Lastly, salt-ui is currently in early alpha stages. While it’s still in early development, it’s designed to be a Web interface to Salt, ideally making Salt administration even easier for users who have a bit of trepidation around using the command line. While not yet ready for prime time, it’s worth keeping an eye on as development progresses.

Perhaps the most compelling aspects to SaltStack is its vibrant community.  Despite Puppet’s nearly six year head start, Salt boasts more contributors to its code base (as per Ohloh.net), a superior comment-to-code ratio, an increase in year-over-year commits, and a lower barrier to entry for new contributors.  Community support is provided both via an active IRC channel (#salt on irc.freenode.net) and on the salt-users mailing list.

Having stood up a number of different configuration management systems across a wide variety of environments, I’ve yet to find a solution that’s as rapid to deploy, simple to scale, or as well architected as Salt.

Ideally I’ve given you a bit of a taste of Salt. I hope it whets your appetite.

UPDATE: Since this post was published, I’ve been in contact with a number of Chef aficionados who have informed me that my understanding of the Chef client <-> server communications model was severely flawed. Chef uses https to communicate, not ssh, and its bottlenecks have previously been cited as being based in implementation details of Ruby / CouchDB.  To my understanding, this has largely been mitigated in recent versions; my apologies for the error.

About the author:

Corey Quinn is a contributor to Salt. When not slagging Puppet, he’s a Puppet Labs trained senior systems engineer and sometimes-consultant based out of Los Angeles.  He volunteers for the freenode IRC network, where he believes that no project is so well run that it can’t be complained about endlessly. Get off his lawn.

See also:


  1. Max Arnold says:

    Have you compared Salt with Ansible?

    • It doesn’t compare.. Ansible is ssh only and a bootstrap bitch.

      • Oops, I think ansible supports pull-mode too: https://www.stavros.io/posts/automated-large-scale-deployments-ansibles-pull-mo/

        Salt is a bootstrap bitch, it needs a master, minion services, the master needs the complete of recipes, so you gpg-encrypt variables, but then the master leaks memory and fills your 32GB RAM …. but ansible just needs ssh access (unless you use pull mode) and doesn’t need an agent.

        • Ive been working with Salt for 2 years and have never seen any memory leaks, and Im pushing out state files constantly.

        • Ive used Salt and Puppet for almost years, recently more Puppet due to my line of work (currently on Salt 2015.5.3 and Puppet 4.3)

          quick pros/cons for each tool as compared to each other

          (If anyone sees something that I missed, please reply as I may not be aware of newer developments with the applications)

          Salt vs Puppet
          SALT PROS:
          – extremely fast execution, fastest of any tool ive used
          – very easy to setup infrastructure (and lightweight diskspace wise)
          – remote execution is easily done as compared to Mcollective(Puppet),

          $> salt testbox123 cmd.run “cat /proc/cpuinfo”
          $> salt -G ‘os:windows’ status.uptime

          – Salt remote exec is built into the framework, Mcollective is an external piece with many moving -parts like ActiveMQ brokers, MCO Server/Agents, RPC scripts, etc
          – active dev community, very helpful

          SALT CONS:
          – jinja/yaml templating is confusing and ugly to look at
          – does not have a dedicated DB like PuppetDB (postgres) that can be used for detailed reporting
          – doesnt have a PuppetForge-like utility to install states/modules
          – as far as Im aware, lacks rspec and lint tools for code cleanup
          – lacks something like R10K to manage environments and git integration (although R10K isnt the simplest method for env management)
          – lacks the size of Puppet’s module list and use, Puppet is simply covers more grounds, from Solaris boxes to Network switches and the # of supported modules is in thousands.
          – lacks the resource and relationship description of Puppet’s DSL. People in these comments are complaining about DSL, but I think this is the greatest strength of Puppet, I can literally write out my entire infrastructure and its complex relationships using classes, virtual resources and defined types.. especially with the new Puppet Orchestrator piece (although Salt also has an orchestration piece which I havent played with)

          To sum up, both are great tools that do a fantastic job at config management, although through different methods

  2. One DDOS coming up! :))

  3. HNDDOS, to be precise.

  4. Alen Komljen says:

    Couple months ago I started to use salt after I “tried” to use puppet. Never liked the pp syntax. For me great tool and pretty simple to use.

  5. Luke Kanies says:

    Why is it that everyone feels the need to dump on things, just because they like something else?

    I’m biased, because I wrote the damn thing, but Puppet clearly doesn’t “suck” – it’s in use by thousands of organizations, and tens of thousands of people, many of whom are passionately fond of it. It might suck for you, or you might suck, or it might just not be appropriate for your use cases and/or experience.

    It’d sure be easier to have a constructive conversation if you didn’t start out this way. Let’s stick to the merits of the tools, rather than just casting aspersions. Hopefully the rest of the Salt community is more focused on solving their users’ problems than you are.

    I’m happy to see different approaches on the market, but Puppet and Salt aren’t that comparable that I can see. YAML is not a configuration language. I look forward to seeing you trying to cram variables and iteration and functions into YAML, for instance; yuck. There’s some great new tech here, and I agree the simplicity is great, but don’t act like you’ll be able to build truly reusable solutions without a bit more functionality than yaml provides.

    Of course, your 3 steps is far more than that. Puppet is just as easy:

    First, install Puppet and Mcollective and Facter and everything else it needs.

    Second, set up all of the SSL keys and make sure it all works.

    Third, write the code that automates your whole infrastructure.

    That’s obviously stupid, like the days when the linux people would say “step 1, recompile your kernel”. It’s easy to say something is simple when you get to combine and skip steps.

    And of course, one of the reasons Puppet Enterprise exists is to make the whole thing super-simple for the majority of our users. If you want a complex, hand-tuned setup, you can, but if you just want it to work, bam, it does.

    • One could just as easily say that Puppet is not a configuration language. Honestly, you had all the power of Ruby at your disposal and you just had to layer something else on top of it. Chef on the other hand does the right thing. Pure Ruby + reusable, idempotent code components. Give me a stack trace and not some mangled error message that is a combination of a parse error, a stack trace, and something about failed constraints.

      • Luke Kanies says:

        Meh. It does the right thing from your perspective. Weirdly, not from the perspective of the tens of thousands of Puppet users. Chef is great as long as you like writing gigantic distributed applications that run as root and are ok with no simulation mode, all clients having read/write access to a central database, and don’t mind idempotency only existing for the parts where you feel like it.

        The truth is that the DSL is a trade-off. You don’t like that trade-off. Sweet. Enjoy your stack traces. Weirdly, I spent an hour with a Chef user last week who complained mercilessly about the stack traces and really just wanted error messages. Like, with line numbers and file names. Huh, so wait, your opinions don’t map to every other person’s opinions? Weird.

        I’m totally fine with you saying X works better for you; it’s a stretch to go from that to saying it works better for everyone.

        I personally love riding my bike to work; if you don’t, I assume that you’re just wrong, rather than having different priorities. Right?

        • Ajith Antony says:

          One feature I liked when I evaluated chef was a very pretty diff of the files it changed or would change (–why-run). With puppet I just get a line that says the hashes don’t match. Probably could go either way in terms of user preferences.

          I haven’t gotten too deep with either tool, but that little detail made me feel a lot better about chef in terms of being new to the tools, and getting confirmation that it is in fact about to do the thing that I actually wanted it to do.

          • Eric Campusano says:

            You can get Puppet to provide you with the exact same diff output and a dry run before making any changes by using the “–verbose” and “–noop” options:

            /usr/sbin/puppetd –no-daemonize –onetime –verbose –noop

            • Ajith Antony says:

              I did try almost exactly that. I used –verbose –noop, but did “puppet apply”. Would that make a difference?

              screenshot attached (i think)

            • Luke Kanies says:

              No, it doesn’t make a difference, but you might need to add –show_diff (which can also be added to your puppet.conf).

        • I would say that Puppet’s DSL provides some kind of trade-off if it actually did provide any. You know what is a trade-off? Static types for better performance optimizations both at compile-time and run-time. What does the Puppet DSL give me in return for taking away the top-down, left-to-right flow of imperative logic? Nothing except for weird errors about unsatisfied constraints. Half the time it steamrolls over failed constraints and the other half of the time it provides a cryptic message about something that I have to chase down to some .pp file which is not even the source mentioned in the error message.

          Ansible and Salt are going to win in the end because they approach the problem from the right perspective. Orchestration and configuration management is a subset of a robust RPC mechanism. Puppet, as somebody said, is Ruby hit with a truck and then kept alive on life support. If I was you I would improve upon RPC capabilities and get rid of the nonsense that is MCollective’s DDL files because as usual there is no trade-off here. All it does is add verbosity and constraints and gives nothing back in return. Not to mention all the nonsense with Timeouts.

          We can continue to argue about this if you want. I have all sorts of opinions backed by actual production use of both Puppet and MCollective.

          • binford2k says:

            > What does the Puppet DSL give me in return for taking away the top-down, left-to-right flow of imperative logic?

            It gives you a state model, which means you don’t have to write imperative logic. You simply describe the state you want the system to be in and trust in your tools instead of micromanaging like a PHB.

            • And what exactly is this state model? Do you even know? It sounds pretty vague and I don’t see a definition in your answer anywhere. Do you know what is a formal state modeling language? I’ll give you a hint. A guy by the name of Alan Turing came up with it and then there was this other dude called Alonzo Church. Smart cats in general. You and Luke Kanies, just regular cats. Oh, and there is this other tool I kinda like and trust. Most people think it’s called RubyOnRails but really it’s called Ruby and it just works.

            • rubykilledme says:

              Chef and Puppet have trade-offs, especially as the number of recipes/modules on a node grows. With Puppet’s DSL, you are insulated to a large degree from the insanity which is Ruby.

              First, I’ll acknowledge the major challenges with Puppet are: circular dependencies or having two complex modules both wanting to manage the same resource. The resolution is often to create your own module which manages both.

              And, yes, with Puppet, you can generate a graph file that SHOWS you everything Puppet manages. All files, packages, users, etc.

              With Chef, the nightmare is dealing with the myriad ways Chef and its recipes interact with Ruby. I asked people on the Chef user group what to do about Recipes and conflict, each installing Ruby – via RVM, via RBENV, via the ancient version embedded in Chef-Omnibus, etc. etc.etc. And the answers are:
              – well, if a recipe uses execute() then one Ruby is used
              – if ruby_block is used, then another ruby is used
              – But, I’ve given up on RBENV and RVM and use CHRUBY !

              Most recipes use “gem install” even on real operating systems with package managers. This introduces wonderful conflicts. Puppet abstracts packages and files in such a way that you can generate multi-platform modules much easier than Chef.

              With Chef, you basically get little more than the ability to replace your Bash installation scripts with Ruby installation scripts. That would be a win if it encouraged multi-platform recipes, but, outside MySQL, almost every Chef script only runs on Ubuntu. That’s because people are basically just wrapping “apt” commands in Ruby.

              THAT is why Ansible is loved among people who realize: Chef is just another way to wrap imperative Bash commands – why deal with the pain which is Ruby?

              Go ahead, try to use Ruby to install gitlab. I dare you. In doing do, I found:
              1- The current Postgres recipe is broken
              2- If you fall for the lie that there is a robust Chef community of working recipes you will quickly discover they have mutually exclusive ways of managing user and/or system ruby installations. THAT is what you gain by not relying on raw ruby to run your commands.

              BTW, Puppet is a bear unless you (properly) use hiera. With hiera, you can finally segregate data and code/modules, which had been Puppet’s short-coming.

              I may end up with Salt or Ansible if I won’t have to deal with the insanity which is Ruby management. Ruby as a language may be ok, but managing it is a nightmare.

            • I don’t really understand how choice of Ruby is actually a problem. Both Puppet and Chef assume there is some installation of Ruby on the system you are managing. Are you talking about bootstrapping a system that doesn’t have Ruby already installed because your argument doesn’t really make much sense otherwise?

              I do agree that Ansible in general is a better designed system but I have yet to see any orchestration system that doesn’t have some kind of shortcoming. This is because orchestration in general is a hard problem and requires expertise and domain knowledge. Orchestrating systems running on AWS is different from orchestrating systems running on Rackspace even though all these tools are trying to abstract all of those things away and provide an interface other than shell scripts for managing them.

              The goals are noble but all these systems fall short of providing actual re-usable units of infrastructure automation.

            • rubykilledme says:

              I’m not talking about bootstrapping Ruby, I’m talking about the fact that the general problem of managing Ruby is solved in multiple different and incompatible ways by various people (rbenv, rvm, chruby, embedded, etc) and this results in: incomatible recipes, and broken recipes.

              My main point is that Puppet’s DSL insulates you from the major problem with Chef, which is Ruby management.

              No, It’s not necessary (or necessarily desirable) to assume Ruby is installed. In fact, Chef’s own installer installs its own embedded Chef.

              Many Chef recipes build their own version of Ruby (thus the popularity of the ruby_build recipe). On top of that, recipes often use RVM or RBENV (or some other way) to install a service/user Ruby or system-wide Ruby.

              You then end up with the following sorts of problems:
              1- vagrant chokes if the system ruby is managed by RVM or RBENV. There are yet more hacks to deal with that, but currently those plugins/patches are broken.
              2- Some Chef recipes install a version of Ruby but then fail to use it properly because they use execute() instead of ruby_block or vice-versa.
              3- Chef recipes run much longer because they tend to prefer gem-managed systems and building ruby from source.

              The problem with Ruby is that if you don’t install a required OS package before building ruby, then you need to rebuild Ruby. Thus, you end up with the forest of ways to manage Ruby, often, each app will want its own version of Ruby. That’s a nightmare, and those problems bleed into the Chef recipes.

            • I still don’t understand what your problem is. Not once have I run into a chef recipe problem that involved some kind of incompatibility between the recipe and the Ruby version. Both Puppet and Chef need a Ruby interpreter so how you go about solving that problem is irrelevant to discussing the merits of either system.

              You are no longer discussing anything related to chef and puppet and instead you are describing a problem with unmaintained recipes. Bit rot is a problem even with ansible playbooks so once again I fail to see the relevance of your argument.

            • Cliff Wells says:

              This works great when it does, terrible when it doesn’t. There are some things that can only be expressed as a procedure, and the Puppet DSL makes these things impossible to express.

              For example, I recently implemented iSCSI mounts in Puppet. What was previously a handful of lines in Bash turned into a hodgepodge of defined types, classes, Ruby functions, facts, etc, and *still* required two sequential Puppet runs to complete, because the *procedure* could not be described in a declarative fashion.

          • the more i read the last paragraph, the more utterly insane it seems.

        • Cliff Wells says:

          The fact that tens of thousands of people use Puppet isn’t really an indicator of how good Puppet is. The market for configuration tools is relatively young and the options few. Some people heard Puppet was good and so chose it. By the time they’d figured out it wasn’t that good, they had already committed significant resources into building an infrastructure around Puppet, so they hang on, hoping that things will suck less in the future.

          I say this because this is the situations we’re in. We converted our infrastructure from cfengine (which sucks worse than Puppet, btw) only to learn that Puppet is full of both design and coding flaws that make every day a frustration. I’ve considered moving to Saltstack, but of course, we have several months of effort (not to mention several thousand dollars in licensing and training costs) into Puppet and there’s no guarantee that Salt won’t be just as bad.

          Puppet gives error messages that include line numbers and file names? That’s cool. Let me know when it does it consistently, because we’re running the latest Puppet Enterprise, and I can assure you that there are plenty of times it doesn’t.

          Let me know when that amazing DSL provides either a unification of the varying types (classes, defined types, resources), or a way to syntactically distinguish them, so that I can say, know when I can pass noop = True without reading the source code. DSLs are great things, but they require someone with a bit of background in language design to make them good. The Puppet DSL looks pretty good at first glance, but it’s a mess of fuzzy design decisions that will frustrate anyone used to a real programming language.

          PuppetDB uses PostgreSQL, but apparently doesn’t ensure referential integrity using foreign keys (we discovered this after a botched upgrade that left invalid references in the DB, something that can’t happen with foreign key constraints).

          Also, your docs suck. Despite having a wealth of pages and tons of text, they seem to convey a minimum amount of information.

          I won’t even discuss the Puppet Enterprise dashboard, since I know you’re aware of how deeply that sucks.

          If I had to do it all over again, I’d certainly choose Chef or Saltstack rather than Puppet. So keep that in mind when you count me in your “tens of thousands of users”.

          • “Tens of thousands of people use Puppet”
            That point is moot as long as
            “Hundreds of millions of people use Microsoft products.”

        • mickrussom says:

          ” about the stack traces and really just wanted error messages”

          Lol, puppet gives some of the most absurd errors I’ve ever seen . Chef is annoying, yes, moreso than puppet, but you need to stop being so cocky. Its rough automating with puppet.

        • John Mind says:

          Luke, stop saying “tens of thousands”…

          I was one of them, I know many others.
          We used Puppet because there was no SaltStack, or Ansible, or back then.
          And now it’s a bit late to replace and stays as legacy tech.

          For new projects / flexible organizations, no way we will pick Puppet.
          Expect the tens of thousands to go down.

    • Corey Quinn says:

      Hi, Luke!

      Firstly, I want to thank you — for taking the time to comment, and, more importantly, for creating Puppet. Without it, the configuration management space would be a *lot* sadder than it is today. “We stand upon the shoulders of giants” and Puppet represents a massive leap forward in this space.

      Secondly, I’d like to apologize for coming across as negative. You’re absolutely right that solving user problems needs to be concern #1, and I’m sincerely sorry if the tone I struck in my article makes it more difficult to evaluate my points from that shared perspective.I’d also like to clarify where I’m coming from.

      I’m not a Ruby guy, a Python guy, or even a developer at all (until very recently). I’m a systems guy who leverages awesome software (and yes, (and yes, pageview-grabbing headline notwithstanding, I do include Puppet in that list) to accomplish interesting things. Historically, on the Puppet projects I encountered, I spent far more time wrestling with Puppet itself than I wanted. What made Saltstack such a shocking revelation to me was that it cut the complexity down stupendously. Compared to what it took to get Puppet installed and running at the time, getting a Salt cluster spun up was so painless the first time thaI was thoroughly convinced I’d done it wrong (I know it doesn’t come across this way when I’m discussing my ultimate conclusion, but my default assumption in situations like this is that the problem is with me rather than with any given tool/program/etc.). As I began discussing my findings with others, I quickly realized I wasn’t alone.

      Finally, with any of these systems, the bulk of the use cases out there are “I want to manage a service, a package, a user, and a file. If I can do those four things, I’m almost entirely done.” From that perspective, anything at all that’s better than ssh in a for-loop is going to be up to the task, and the determining factor really comes down to simplicity. I will say that Puppet Enterprise is *very* simple to get working (close enough to Salt’s bootstrapping process that I’d call it a wash), and it’s much appreciated– what’s less appreciated is having to shell out a licensing fee to reduce the complexity to a point where a junior person can make a go of it without hurting themselves or the environment.

      As far as being able to accurately describe an environment in YAML, I agree with you. That’s why I find the ability to use jinja2 in Salt’s state files to be so valuable.

      I also certainly don’t mean to suggest that there aren’t a lot of use cases where Puppet may very well make a lot more sense, and I likewise can’t speak to Puppet’s future plans in terms of where it’s heading in the space. Like I said, even for the use cases I’m most familiar with, it does some very handy things, and it’s entirely possible there’s a puzzle piece here that I’m missing that would make everything go “click” and I’d realize I’ve been going about this whole comparison all wrong. Next time we’re at the same conference, I’d love to buy you a beer and have a more in-depth discussion about what I might be missing. I mean it — a fancy microbrew, even!

      • Luke Kanies says:

        Heh. I appreciate it, although bourbon might be the order of the day in that case.

        I’m happy to see the state of the art pushed, and I think Salt, Ansible, Chef, and others are doing so, which is great. The biggest reason I founded Puppet Labs was because I despaired of exactly that (the whole “fear and desperation” thing). I just wish everyone was a bit more productive about it – build up, don’t tear down.

        There’s so much room for a technical, functional conversation about what works well, and your post did pretty well on a lot of that; but the subject caused the conversation to go in a direction that didn’t help anyone. And, of course, made me think you might poison my beer. 🙂

        This whole YAML, er rather, YAML + jinja2, is exactly the kind of conversation I’d love to have. Specification is fundamentally hard. You can dodge the whole thing and go turing complete, like Chef did, but that has its own issues. You can act like you’re dodging it, like Salt did, but then you need templating, and it turns out, most templates are also turing-complete (I know Puppet’s are).

        One of the big problems I had with Cfengine was that most sophisticated cfengine implementations were really a sophisticated templating implementation; I’m curious to hear what will keep Salt from becoming that same thing.

    • mickrussom says:

      Puppet is not fun. Its sloppy and yes its used by many but many many dev programmers take care of trivial piles of scripts. Also it is picky about DNS names, it has the worst error messages Ive ever see and it take massive effort to get simple things done. Chef is even more byzantine.

    • mickrussom says:

      “Second, set up all of the SSL keys and make sure it all works.”

      Note even Kanies see these need to constantly mop up after the SSL/KEY/DNS mess puppet chokes on all the time.

      Whole infra? Like BIOS settings? Switches? True SDDC soup to nuts is hardly trivial and puppet is not the swiss army knife you think it is.

    • I have to agree, Ive been learning Salt last few weeks and its a great tool, but talking with developers in several organizations (i.banks, financial shops, etc), they all swear by Puppet with many preferring it to Salt due to its features. Its really a preference thing. I prefer Python over Ruby so I would go w Salt, if your business is full of developers who are Ruby centric and are comfortable w Puppet, Puppet is the clear choice, its a great product.

      As for learning curve, Salt is much much easier to pick up for non-developers, this is a major factor for organizations to consider.

    • Mick Russom says:

      I’ve waste _years_ of my life dealing with puppet and the pile of sucky trash that it is. You are a curse. I’ve been doing automation and devops since before even the name “cloud” was ever widely used. Your framework stinks, the errors produced are horrible, your samples are garbage, and the code stinks – it may look good or claim to do be able to do things, but the reality is certs are a constant problem, most code that has been written, even simple stuff, collapses like the World Trade Center if you add a little dash of complexity. Then the no rollbacks, the half way through a task and it blows up and now its broken forever. Its HORRIBLE.

      To be fair Chef is sucky garbage too. Not sure about Salt yet but the reason we have 50 different automation frameworks rather than 2 clear winners is they are all sucking pretty hard.

      You people writing this crap are mostly script kiddies who know little to nothing about the actual system and how it works, then your trash gets cut and pasted and we have a pile of stuff that utterly collapses if someone ever even thinks about touching any of the ruby gems that do laughably simplistic things and if anyone dates update any of them the whole mountain of trash falls down.

      I wish we had another cold war. We used to have engineers that made things that didnt break because if they did there was the threat of nuclear annihilation for doing substandard non-working shoddy trash. All the GenX and later pukes write trash by default and if anyone relied on this trash for critical stuff it would all be over.

  6. Hugh Brown says:

    In python 2.7.3, error during import:
    from _sre import MAXREPEAT
    Does not occur in 2.7.4 or 3.3. Sounds like there are version dependencies you should mention.

    • Marius Gedminas says:

      That looks like a typical stale virtualenv issue. It goes like this: you create a virtualenv using Python 2.7.x, then upgrade the system Python to 2.7.(x+1) and now your virtualenv emits various strange import errors. I believe I’ve seen the _sre one.

      The fix is to replace the bin/python executable in the virtualenv (you can run virtualenv with no arguments in the right working directory and it’ll do that for you).

  7. Baustin213 says:

    It sounds like that might have to be a future blog post, Max! It seems as though a lot of people commenting on Hacker News would like to see this comparison as well.

  8. iafonov says:

    Chef works over ssh? Are you sure? Maybe it would suck less if you’d spend 5 minutes reading chef wiki?

    • Corey Quinn says:

      You’re absolutely right– the one Chef environment I’ve worked within before was apparently non-standard, and was either using Knife, or some other form of ssh as its transport. My apologies, and thank you for catching my error!

      • If this is an error, then surely it should be corrected in the blogpost instead of way down here in the comments?

  9. Corey Quinn says:

    Not yet. Working on it!

  10. Eric Campusano says:

    I use Puppet and am a fan of it. I am taking a wait and see attitude with Salt. I evaluated it and thought it was pretty neat but I wasn’t ready to stake my job on it when there were so many more people using Puppet that could answer my dumb questions about it. I will say that Salt’s remote execution capabilities were quite easy to set up and I was very impressed with that. I agree that setting up mcollective with the open source version of Puppet is more complex and could use some improvement.

  11. BrianWilkins29 says:

    Someone once said, “A poor craftsman blames his tools…”

    I use an older version of Puppet and I am experimenting with the newer version, which I heard has easier filebucketing tools. That being said, a lot of the heartburn I feel with Puppet is not the tool itself, but the fact that as an Administrator, you sometimes feel the need to manage EVERYTHING with Puppet and that has its drawbacks. Recently, I was working on a system and couldn’t figure out why the software wasn’t working. It turned out that Puppet had an older version of a configuration and it just wiped out what I was working on. Oops! Turns out, Puppet is great and the templates work pretty good.

    With any tool, whether it be Salt, CFEngine, Chef, or Puppet, you have to use it correctly. Otherwise, you’ll end up shooting yourself in the foot.

    • binford2k says:

      > as an Administrator, you sometimes feel the need to manage EVERYTHING

      Why would you not? What do you do when a meteor hits your datacenter and blows up your server? With Puppet, you don’t care. You spin up another instance, classify it like the old machine, and head back to start filling out the insurance paperwork.

      • really?? Is it possible to configure PostgreSQL database cluster along with databases, user accounts along with db privileges (including ownership) using Puppet?

        • Legooolas says:

          Yes it is possible.
          Database content could even be injected, but that’s what backups or clustered databases are.

          I’ve configured mysql databases including user accounts, db privileges etc using Puppet.
          It’s not that difficult to express as the state that you want the system to be in.

          Anything that has an API and allows you to check the status can have a state described and managed in the same way.

  12. GreenLantern says:

    Interesting article. We’ve also just started replacing puppet with saltstack. We still use foreman for deployments and the rest is done using saltstack. Salt is extremely easy.

  13. Chip Schweiss says:

    I just found this article because I’m fed up with long running problems with puppet and investigating alternatives.

    I may be late to the discussion, but here’s why I tend to agree that Puppet sucks.

    I’ve been using Puppet for a bit over 3 years, it’s always been a love hate relationship. I’ve been running version 2.6.x since day one. Every attempt to upgrade has been a grueling task that I eventual ran out of time to work on and gave up. Almost every attempt to upgrade results in catalogs that will not compile because the newer version believes there are circular dependencies. When there are literally 1000s of objects in the catalog, nothing Puppet has to offer helps resolve these types of problems.

    Now I’m at an impasse, upgrade or move away. I have over a year’s total work in my Puppet manifests, templates, etc. so it’s a torn decision. Puppet 2.6 will no longer on CentOS 5.10, and Puppet 2.6 is no longer maintained. I must continue to maintain some CentOS 5.x systems so something has to give.

    On my more extensive Puppet configurations, a single Puppet run takes over 10 minutes to complete. When run once an hour on over 100 VMs, that adds up to a lot of CPU time just for Puppet. Besides, the horrible backwards compatibility problems I’ve seen, this is the number one reason I’m considering moving away from Puppet.

    Sure the Puppet master scales really well, they work hard on that. The client side does not scale well at all. Unfortunately this takes the longest to figure out. Handling a package list and a few config files is no big deal for puppet, but by the time it starts assembling an entire application with lots of dependencies, it starts falling on its face for performance.

  14. Denys Rtveliashvili says:

    Just had a glance at its documentation… I see no fundamental difference to puppet from practical point of view.
    For instance, it has the same ridiculous approach to crontab management where you need to declare that an entry is to be removed explicitly if you need to get rid of it. Now let’s say you had a config which says an entry is to be installed on server1. Then you decided to move it to server2. Instead of just changing the name of the node you have to change it and also declare that server1 must NOT have the entry anymore. What a waste of time…

  15. ask all the server for their stats. “Having stood up a number of different configuration management systems across a wide variety of environments, I’ve yet to find a

    better call saul online

  16. Good answer back in return of this issue with firm
    arguments and describing the whole thing about that.

  17. Lastly, players must enter variety of gems to be added and press the Generate” button within the Clash of Clans Hacker

  18. Silly me, just when I thought I had the Cloud thing all hung up and salted.

    I tend to read web articles with a grain of salt for obvious reasons, but your article here is clearly above the salt and worth the reading. Great writers like you are few and the salt of the earth.

    Entertaining to read Luke Kanies get salty on you. Engaged, I read through it like a dose of salts.
    Now I don’t mean to pour salt on his wounds, but frankly I’d tell him to go pound salt!
    Anyone worth their salt can see your being helpful and It’s just human nature to classify things such as “suck” or not.

    Anyhow, back to the salt mines.
    Signed, An Old salt

  19. Quoting from Salt’s best practices guide:

    “To attain this level of modularity this state will need to be broken into two states.


    {% set apache = salt[‘grains.filter_by’]({
    ‘Debian’: {
    ‘server’: ‘apache2’,
    ‘service’: ‘apache2’,
    ‘conf’: ‘/etc/apache2/apache.conf’,
    ‘RedHat’: {
    ‘server’: ‘httpd’,
    ‘service’: ‘httpd’,
    ‘conf’: ‘/etc/httpd/httpd.conf’,
    }, merge=salt[‘pillar.get’](‘apache:lookup’)) %}

    tmpl: salt://apache/files/httpd.conf

    {% from “apache/map.jinja” import apache with context %}

    – name: {{ apache.server }}
    – name: {{ apache.service }}
    – enable: True

    {% from “apache/map.jinja” import apache with context %}

    – apache

    – name: {{ apache.conf }}
    – source: {{ salt[‘pillar.get’](‘apache:lookup:config:tmpl’) }}
    – template: jinja
    – user: root
    – watch_in:
    – service: apache
    This new structure now allows users to choose whether they only wish to install the default Apache, or if they wish, overwrite the default package, service, configuration file location, or the configuration file itself. In addition to this the data has been broken between multiple files allowing for users to identify where they need to change the associated data.”

    So, Salt may be easy to install, just as Puppet 4 & Chef are, but YAML+Jinja = confusing, hard to read configuration management.

    Puppet & Chef are here to stay guys.

  20. Лучшие проститутки Москвы. Блог.

  21. Wow. That is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious.

  22. The worst thing with all of this is that I’ve been in IT for 25 years, programmed in many different languages from assembly and on up. Have been a Unix admin and developer in that time and NO configuration management tool is even remotely recognizable as an actual programming language, with logic, loops and conditionals, and not a single dev I’ve ever met of the web/Ruby clan (those that take to puppet), has even the slightest most vague idea of what proper correct ITIL based secure production quality systems administration is. So all these tools exist seemingly to alienate actual admins from ease of systems management while redirecting web application devs from their true vocation to managing iaas when they are as disinterested in this as a real admin is in learning a hideously designed declarative language that sports no similarity to how computer programming and logic works.

    The only clear thing to me that has come from devops is that puppets marketing far outshines it’s usefulness. Exactly the way window s has since its invention. When senior execs are mandating the use of a specific configure management tool at the coal face IT has completely and utterly lost its way. Now as a dev or an admin(Op), the market only appears to exist for those who are willing to comply and hate their career.

    Worse still is the loss of trust that corps have come to show their own IT departments. Business units actively fight IT shops in house and to me it’s because of this constant feeding off marketing bs about tools that supposedly take complexity away from the ultimately complex.

    Smarter people is the answer, not dumber tech. But in true MS fashion these tools lower the bar to even worse computing than before… it’s the end user software that needs to be idiot proof. Us intelligent people are just fine running the tech (our way)…

  23. Very informative article, Corey. Thanks!

Speak Your Mind