So, what's going on here, I think, is that you have some microservices which call each other. Each microservice is apparently put in a container to avoid dependency issues. Now you have a network overhead problem if the containers are on different machines, so there's something called a "pod" to make them run on the same machine. This is what the author calls a "sidecar". The author may be tangled in his own jargon. Cloudflare says a "sidecar" is another container inside a "pod", but one that is for some auxiliary function rather than being one of the main microservices doing the work. Hence the name "sidecar". The link in the article for "sidecar" is to the author's book, so you have to pay to find out for sure.
Now you have to connect up the messaging, start, stop, and update things, scale to multiple instances and machines, log, and monitor. So you need an orchestration system of some kind. The "operator" decides what to do, and the applications and the Kubernetes system, via the Kubernetes operator API, do it. The "operator" is shoved into the "pod" as a "sidecar".
Since making an operator that does the right thing is a hard problem, there's quite a collection of buzzword-intensive products to do some part of that job.
A better explanation is in the Kubernetes documentation.[1]
Did I get this right?
(About 15 years ago, I had to write something which did this sort of thing for QNX real time programs for a robot vehicle. Had to deal with things like inserting monitoring into a message passing system. The Robot Operating System (which is not an OS, it's a message management layer on top of Linux) does something similar. Now it's gone mainstream. Does it make things easier or harder?)
Sidecars are extra containers that run alongside the main container in a pod. Both sidecars and pods are k8s jargon terms. Pod is what k8s calls a running image, a container in Docker lingo. Pods have metadata like resource claims that k8s uses for scheduling.
The point of the article is a bit like the distributed system analog of aspect oriented programming.
A typical problem might be that you want all your network communications to be encrypted, but you don't want to configure SSL for every service, because the libraries are all different, configuration files to point at rotating certificates are all different, etc. So instead you run an SSL proxy in a sidecar, and make the traffic go through that instead. You sort of bolt on extra horizontal functionality onto your existing set of services.
Operators are slightly orthogonal IMO. I think the core pattern of k8s, the control loop, is great for operational resilience, but it's a bit alien to many existing bits of software, so operators are the glue which converts an otherwise high-maintenance service into the control loop idiom.
I think the article does have a point, it points to a unifying theme for writing apps for k8s, and it's not necessarily just a book ad.
Sure. And sidecars share a bunch of things with the main container. I am simplifying in an inexact handwavey way. You should still think of a pod as primarily a single container, and design deployment around that.
> Now you have a network overhead problem if the containers are on different machines, so there's something called a "pod" to make them run on the same machine.
This is not correct. A pod is a collection of processes which share the same life cycle. They are expected to scale simultaneously, shut down simultaneously, and restarted simultaneously.
> This is what the author calls a "sidecar". The author may be tangled in his own jargon.
This assertion is also quite wrong, and appears to sprout from your misconception of what a pod is.
The concept of a sidecar is very old and well established. A sidecar is a process that is designed to run alongside a master application to augment or offload is functionalities.
You can define a pod so that one process is the master application and one or more applications are sidecars. There is no overlap or misconception.
> Cloudflare says a "sidecar" is another container inside a "pod", but one that is for some auxiliary function rather than being one of the main microservices doing the work. Hence the name "sidecar". The link in the article for "sidecar" is to the author's book, so you have to pay to find out for sure.
The concept is decades old, probably older than you and me. No need to depend on Cloudflare or the author's books to check that out.
> Robot Operating System ... Does it make things easier or harder?
Some things. Pub/sub over ROS topics makes it easy to introspect and check on what is being published, why which node and at what frequency, bandwidth, who is subscribed etc.
For orchestration there are launch files that can also start ROS node son other machines in the network.
> The "operator" is shoved into the "pod" as a "sidecar"
I didn't really follow this part, since the operator should be running on the master nodes as a k8s extension, not in the pod which it manages.
btw, I don't thing the k8s docs does a good job of explaining why one would do this much less push someone think that way. The article might be just a long ad but at least it pushes you to think about it.
No you didn't get this right, although I'm sure you do know what the right explanation is and are exaggerating for effect.
Operators and sidecars are definitely not the "new model" but a pretty old concept with various names and are effective for what they do. You can simply call it a layer of abstraction, the same way NGINX can serve as a reverse proxy to abstract authentication and load balancing while your app just serves HTTP requests.
> Today’s developers are expected to develop resilient and scalable distributed systems.
The author start from within a silo. Many of today's developers don't have these expectations.
Operators are non-trivial software. Some of them are over 50k lines of code and still not near complete. It takes a fair amount of work to develop and test them.
It is sometimes implicitly shared that you MUST use an operator. There are times they are useful. There are many other times you don't need them. I'm still amazed at how often I can get away with a Kubernetes Deployment or StatefulSet and be just fine.
I'm a bit skeptical of sidecars. It seems like a pattern born of organization separation between ops and dev instead of an optimal solution.
I mean, I think I get the benefits. You have your magical smart pipes that just work deployed along with but separate from the grubby app code. Supposedly the app code is simplified because a lot of the complexity is handled in the side car. They're separate containers so you can "compose" and deploy them separately.
But in practice, why is this not strictly worse than a set of libraries? You're spending overhead and do you gain, really?
The side car is language agnostic...but you need a language specific library and you need to interface with the side car through some indirection (network, file, IPC) instead of in process execution. You DO consider the side car or you're double implementing the complexity in both the app and the side car anyway. Reaping the benefit is the same as depending on it, no? So now you have a dynamically linked dependency, so to speak.
Why not just save a lot of overhead and use a library when its viable and relegate the sidecar to a stopgap instead of promoting it as an optimal pattern?
It's not an either this or that, but a matter of convenience. If it's easy to build that library into your app in terms of testing, go ahead. But it might be easier to validate the library and app separately, and then run them together.
For instance, I have an app that needs to securely access sensitive data. I can build that access into my main app, as a wild violation of "separation of concerns" or build a sidecar which handles the access and exposes the data as a read-only file system to the app container. I test both locally and separately then rely on the container platform to ensure that the app only accesses the data through a read-only volume mount.
Which is easier? If you are green field, monolithic or meet all devs each day, maybe the library is easier. But what if the two parts are built by different teams in different countries? and Ops is a third team in a third country?
If the library has a well-defined API and is built and maintained by a team following good engineering practices, and your app is built using a language/platform that makes accidentally working around their API impossible, the difference between that and a sidecar container being communicated with via a readonly FS or local network seems largely academic.
I don't think there are many languages where you can give certain permissions (e.g. r/w access to a directory, or TCP connection to a certain host) just to a single library in the application, in a way that can be enforced at runtime.
> the assumption is an attacker can break into the main app only?
Yeah, the idea is that the main app is more vulnerable (both due to being more exposed, and also because it's often more complex), hence it should have very limited permissions into the internal infra.
This approach is also used in other situations, like browsers, where the process that processes website content is very locked down, and can only perform actions in the system (network, filesystem, etc) through a limited channel to an outside process.
I think this misses the point. The author of the operator delivers software as the operator + sidecar and a different can deliver a different software as the main app being operated and sidecar'd.
Imagine you only want to write a REST API without worrying about DB, for example. An operator manages you DB cluster and REST API server scaling
And assuming you don't have annoying issues like language incompatibility, or have already accepted CORBA/DCOM into your heart (then nothing stops you from running the separated code in a sidecar :P)
Allowing independent teams to deploy their service into production is a hard problem to manage. One way is to provide them with self-service to abstractions that are managed with guard rails.
Persistent storage, key/value stores, caches, naming, service discovery, networking, access control, authentication, authorization, versioning etc etc.
Having that as an API means you can deploy safely. Having it as a library means that you're missing the indication that these are networked services that you are relying on and must accomodate.
Having it as a networked "sidecar" means that it can be upgraded independently, so is an extension of the control plane beyond the lower level system services of k8s and equivalent. It can express business logic in terms of deployment policies, sign offs, audits, accounting etc.
Lacking Kubernetes specific experience, I have no idea what the article is talking about. Is it just “you can write micro services in different languages and deploy them in separate containers, and they can still talk to each other?” Why is that observation insightful?
Is there some special inter- on intra- container communication mechanism that the author is trying to highlight?
A sidecar is an alternative to an application-level library for something that needs to be present in every instance, e.g. knowledge about how to contact other services, or authorization checks on inbound calls.
It's similar to another microservice in the sense that we're putting functionality across a network API instead of a function call. But unlike another microservice, a sidecar is deployed alongside each instance and usually inside the same network namespace, so there is no discovery/routing problem and latency is minimal.
>A sidecar is an alternative to an application-level library for something that needs to be present in every instance
I see where you're coming from there; but it's worth also being explicit that it may be the only option (rather than an alternative) if, e.g. you do not own the code for which you're running the sidecar.
Good examples here are sidecar proxies for service meshes: you're probably doing TLS termination and authorization in the proxy, and the underlying service might be a closed-source database with a binary protocol you know nothing about.
Advanced, proper SOAP usage allowed for various patterns that acted similarly, but that generally falls under anything with middleware that jacks into messaging system
A library needs to be of the same language, sidecars can be polyglot. You can use a golang based sidecar with a python app, can't do that with a library.
We have exactly that for a sidecar on one of our team’s k8s clusters. There is a legacy python 2.6 twisted service. It will not get priority to be updated (functionality is being slowly eroded into other services). It exposes its metrics in a non-standard way and updating it to emit to Prometheus was going to be non-trivial.
Enter the sidecar. It lives next to each instance and scrapes the metrics and converts them over for Prometheus to consume, and happens to be written in Go. It was developed and released in a day or two.
The talk shows how to use CGO to make Go code available as a C library and then to use Cython to make this library available as a Python extension module.
We actually tried this. The Go runtime really does not like to be embedded in Python. When Python forks, as for example Celery likes to do, Go deadlocks. Also had serious troubles with provenance, versioning, and compatibility of the C artifacts. Somewhere along the line there were magic hex numbers. It was a nightmare. The project was scrapped in favor of a sidecar.
Lots of fancy headlines, but they don't even define what they mean by sidecar. Plus this appears to be mainly about Kubernetes microservices, not about monoliths or regular end user software.
The book that is linked has been translated into 4 languages and does a good job explaining what is a sidecar. No need to buy, you can download it for free from here
https://developers.redhat.com/books/kubernetes-patterns
The article is for Kubernetes savvy readers, and the main takeaways (which I agree I failed to convey clearly) are these:
- You can consume generic capabilities from a sidecar through HTTP/gRPC, rather than include these capabilities into a microservice's runtime as a library.
- There are many sidecar examples today (some in a Pod, some shared) Envor, Cloudstate, Skupper, etc. Sidecar is an implementation detail. It can be with containers in a Pod, or it can be a shared sidecar per namespace or host machine - next to a monolith. The best example today, that provides generic developer primitives as a sidecar is Dapr.
- Kubernetes Operators encapsulate operational knowledge into executable form and shipped to software consumers as an "Ops-included" mentality.
Lots of negativity here, and perhaps a little bit of cynicism.
Kubernetes, and in a broader sense Docker, has a number of paradigms which in my opinion won't make it to super-broad adoption.
Docker has multistage builds, but are very impractical to integrate with todays mainstream CI pipelines. Most just suppose you want to clone the repo in their pipeline container and build the target images there. Lots of those power tools (eg. Feed integration...) do not cater for multistage builds.
Kubernetes:
Sidecars do seem useful, but then again not everything that can be separated should be (or is easy to do). My client really went overboard with React HOCs and basically some apps are just 1 long vertical LoC full of "with..." beyond count.
Pod security
RBAC
Cron jobs
Prolly a few more that I forgot about.
Not to say that they are bad. I just don't see them being used that much...
How have people's experiences been with Consul in general? Any pitfalls? We're rolling it out shortly for service discovery and some limited key/value storage (and a separate cluster for Vault storage)
The only thing that burned my team on Consul was DNS routing. At high request load, it would not load balance fairly leading to hotspots that could lead to a node falling over and lead to a thundering herd of node failures. This was several years ago, so maybe they fixed it. I believe it was something to do with a time component to the routing, likely some ill-though-out optimization that failed at scale (one node would be hot for some number of seconds, then some other node would take its place). We got around it by making a routing library that used the (non-exposed) Watcher API to build the membership info for the cluster in memory for each node (instead of relying on DNS), and then did round-robin on the cluster for routing each subsequent request.
I have very limited experience with Consul and Vault. I do know that the older versions of these services, when they were CLI, were very unpleasant, especially the Vault.
Consul didn't immediately meet our needs since it has no form of an audit trail, so the purposes I'd hoped to use it for weren't really fulfilled. I did like what the service seemed to provide and how it provided it.
Just be warned that the lack of audit trail can be a problem if you want to use it for feature flags, etc. It will still need to be backed by some sort of store, like git.
So, what's going on here, I think, is that you have some microservices which call each other. Each microservice is apparently put in a container to avoid dependency issues. Now you have a network overhead problem if the containers are on different machines, so there's something called a "pod" to make them run on the same machine. This is what the author calls a "sidecar". The author may be tangled in his own jargon. Cloudflare says a "sidecar" is another container inside a "pod", but one that is for some auxiliary function rather than being one of the main microservices doing the work. Hence the name "sidecar". The link in the article for "sidecar" is to the author's book, so you have to pay to find out for sure.
Now you have to connect up the messaging, start, stop, and update things, scale to multiple instances and machines, log, and monitor. So you need an orchestration system of some kind. The "operator" decides what to do, and the applications and the Kubernetes system, via the Kubernetes operator API, do it. The "operator" is shoved into the "pod" as a "sidecar".
Since making an operator that does the right thing is a hard problem, there's quite a collection of buzzword-intensive products to do some part of that job.
A better explanation is in the Kubernetes documentation.[1]
Did I get this right?
(About 15 years ago, I had to write something which did this sort of thing for QNX real time programs for a robot vehicle. Had to deal with things like inserting monitoring into a message passing system. The Robot Operating System (which is not an OS, it's a message management layer on top of Linux) does something similar. Now it's gone mainstream. Does it make things easier or harder?)
[1] https://kubernetes.io/docs/concepts/extend-kubernetes/operat...