At our last hackathon, we developed a remote-controlled camera vehicle that can be controlled from anywhere in the world via an internet connection. But before we now write ourselves how we did it, we wrote the following article as an experiment with ChatGPT.

ChatGPT is a prototype of a chatbot that uses artificial intelligence and is constantly being trained by the feedback of users. Thus, ChatGPT can conjure up complete solutions and texts on the screen within a few seconds using a few keywords. Afterwards, we asked ourselves which articles, codes and texts we had read last were actually written by humans.

But feel free to read the following text for yourself. We have translated the content below into German ourselves, you can find the original version in the English version.

We have provided ChatGPT with the following key points:
Write a longer blog article about a hackathon that the Webrunners team hosted in the company’s CreativeSpace. There we built a remote zoom camera vehicle and programmed an application to control it. This allows remote workers to remotely control the camera van and the position of the camera itself.
While the on-site staff assembled the hardware, set up the RaspberryPi and made it accessible to the outside via a tunnel, the remote staff built the application. In order to involve the remote workers as much as possible in the on-site collaboration, we set up several cameras, including a 360° camera that allowed for good integration. The result of the day was a first proof of concept, where the vehicle base and the camera could be controlled remotely. Now the individual aspects can be optimised and components, such as a robotic arm for finer control of the camera, can be developed with the 3D printer.

We have provided ChatGPT with the following key points

Write a longer blog article about a hackathon that the Webrunners team held in the company’s creative space. There we built a remote zoom camera vehicle and programmed an application to control it. Remote workers can use it to remotely control the camera vehicle and the position of the camera itself.
While the on-site employees assembled the hardware, set up the RaspberryPi and made it available to the outside via a tunnel, the remote employees built the application. To ensure that the remote workers were as involved as possible in the on-site collaboration, we set up several cameras, including a 360° camera that enabled good integration. The day’s result was an initial proof of concept, with the vehicle base and camera remotely controllable. Now the individual aspects can be optimized and components, such as a robotic arm for finer control of the camera can be developed with the 3D printer.

ChatGPT result

Last week, the Webrunners team held a hackathon at the company’s creative space. The goal was to come up with innovative solutions to common problems faced by remote workers, and we were excited to see what we could accomplish.
One of the ideas that we came up with was to build a remote zoom camera vehicle that could be controlled by a remote worker. This would allow remote workers to have more control over their camera position during team meetings, and would also allow them to move the camera around the room if needed.
To get started, we split the team into two groups. One group consisted of on-site employees who were responsible for assembling the hardware and setting up the RaspberryPi. They also made the vehicle available to the outside via a tunnel, so that the remote workers could access it.
The other group consisted of remote workers who were responsible for building the application that would control the vehicle. To ensure that the remote workers were as involved as possible in the on-site collaboration, we set up several cameras, including a 360° camera that enabled good integration.
Over the course of the hackathon, the two teams worked together to build and program the remote zoom camera vehicle. By the end of the day, we had a functional proof of concept, with the vehicle base and camera both being remotely controllable.
While we were pleased with the progress we made during the hackathon, we also realized that there was still more work to be done. There were several aspects that we needed to optimize, and we also wanted to develop components such as a robotic arm for finer control of the camera. We decided to use the 3D printer to create these components, which would allow us to customize the vehicle to meet our specific needs.
Overall, the hackathon was a great success for the Webrunners team. We were able to come up with a creative and innovative solution to a common problem faced by remote workers, and we had a lot of fun doing it. We’re excited to continue working on the remote zoom camera vehicle and see how it can help to improve the experience of remote meetings.

¹

Since end of May, thanks to the great efforts of the GitLab project developers, an open source authorization-layer for the docker-registry(v2) is available.
Within this article I´d like to introduce my collegues into the basic steps to work with docker and our new private docker-registry.

The so called docker-registry is like a git-repository for so called docker-images.

Enable the registry for your project

1. Sign in to code.webrunners.de
2. Go to your project: /namespace/projectname
3. Click on Settings: /namespace/projectname/edit
4. Under “Features” enable “Container registry” and save
5. Go back to project and click on the new available “Container Registry“

This is the page where you will see and manage your pushed images.
There is also a nice little help written.

Now your project-namespace is ready to use for pushing and pulling docker images.

Example of usage

On your command line:

1. Create a docker image
2. Name (tag) it to registry.webrunners.de/namespace/projectname:tag
3. Login docker to our registry
4. Push the image
5. Now, others with project-access are able to pull it

Assuming you already have installed docker for your OS…:

• Windows
• Mac OS X
• Ubuntu
• Other

…you should be able to execute the docker client on your command line.

Let´s start

Docker, as it is one big binary that is written in Golang, has different subcommands.

I will list some in a meaningful order:

docker ps # List all currently running processes (docker container)
docker images # List all images, which are the base for containers
docker run image/name # Start a container from an image

Containers are like linked clones to images. You can think of containers as running state and images as frozen state. They don´t alter the images, but are a new isolated layer upon them to make use of preinstalled software.
You can get images from hub.docker.com, or build them locally e.g. to push later to our private registry.

A simple workflow

1. Get an public available image from docker hub
2. Run it locally for testing
3. Adapt it
4. Commit it and save it to a new image
5. Rename the image for pushing to our registry
6. Deleting locally and rerun/pull to verify all is as expected

I) We get this very small alternative to the official python image:

docker pull elyase/staticpython

II) For Windows you could try %CD% instead of $(pwd). Alternatively you can write the full path to the current directory. On Windows use this notation: /c/Users/

docker run -p 8000:8000 -v $(pwd):/src -w /src --rm -it elyase/staticpython sh -c 'python -m SimpleHTTPServer'

Explanation

Don´t forget to check docker --help, docker subcommand --help

• run – Subcommand to start an image
• -p – Publish a port to the docker-host
• -v – Mount a local directory, or a so called Volume within the container
• -w – When starting the container use this as the working directory
• --rm – After stoping the container, delete it. See docker ps -a for exited containers that might be used again, or just eat disk-space.
• -it – Short for -i -t. Start an interactive terminal session
• elyase/staticpython – The image to use
• sh -c 'python -m SimpleHTTPServer' – All the rest is a command to run. Here we only start a little webserver

If you use docker-machine, get the docker-host-IP with docker-machine ls. Now you should be able to open a browser to see your current directory files:

Go to: http://docker-host-IP:8000
For me it´s: http://192.168.99.100:8000

Quit the Container with Ctrl-C.

III) Now we run the container again in detached mode, and run some commands inside. Note that we don’t use a shared Volume this time, as it could not be committed to an image. Also we name the container this time.

docker run -d --name charly -p 8000:8000 -w /src -it elyase/staticpython sh -c 'python -m SimpleHTTPServer' # We wrap the python command, because it didn´t respond to Ctrl-C in my tests

docker exec charly sh -c 'echo "We run webs" > index.html' # We have to wrap the command inside the sh command because of the redirection.

Now I can see the page on 192.168.99.100:8000 again.

We could do some work directly inside a shell session:

docker exec -it charly sh
/src # echo "Go on spider" >> index.html
/src # exit

IV/V)

docker commit -m "Some test" charly registry.webrunners.de/namespace/project:test

You should see now the commit under docker images.
And could rename it later with docker tag.

Now use your GitLab credentials:

docker login registry.webrunners.de

And push

docker push registry.webrunners.de/namespace/project:test

You can manage the image on:
https://code.webrunners.de/namespace/projectname/container_registry.

VI) As the container is still running locally and the just created image (commit) still exist, we now delete them.

docker kill charly
docker rm charly
docker rmi registry.webrunners.de/namespace/project:test
docker images
docker ps

Verify

docker run --rm -it registry.webrunners.de/namespace/project:test

Unable to find image 'registry.webrunners.de/namespace/project:test' locally
test: Pulling from namespace/project
...
...

Build

Now we put all of this inside a build script. The so called Dockerfile.

Create two files

index.html

We run webs
Go on spider

Dockerfile

FROM elyase/staticpython

COPY index.html /src/

WORKDIR /src

CMD ["sh", "-c", "python -m SimpleHTTPServer"]

docker build -t registry.webrunners.de/namespace/project .
docker tag registry.webrunners.de/namespace/project:latest registry.webrunners.de/namespace/project:spider

Check

docker run -it --rm -p 8000:8000 --name charly registry.webrunners.de/namespace/project:spider

Open http://192.168.99.100:8000

Push

docker push registry.webrunners.de/namespace/project:spider

Cleanup

Locally

docker kill charly
docker rm charly
docker rmi registry.webrunners.de/namespace/project:test
docker rmi registry.webrunners.de/namespace/project:spider
docker rmi registry.webrunners.de/namespace/project:latest
docker ps -a
docker images

GitLab

There are buttons to delete the pushed images.

Bugs

#17875:
Docker push will loop to upload and finally give up on big layers.
This should be fixed in a next GitLab release:

Reference

• Dockerfile Reference
• Docker run Reference
• Docker compose
• Gitlab CI using registry
• Docker und .NET – ein Beispiel

• The Old Sailors’ image by Paul Townsend is licensed under CC BY-ND 2.0. ↩

Another alternative is to use parser combinators that combine small atomic parsers to build a complex rule set.

Let’s see how we can use parser combinators in an (admittedly simple) context. One of our customers runs a fascinating web service: Using a easy-to-understand URL scheme, you can remotely control a rendering engine that can effectively combine hundreds of different car parts to render a photo-realistic image of a specific car configuration. One task in a recent project was to extract car configuration data from a web service call.

A real world example

Our customer’s API is confidential, so I had to alter a few specifications and sadly can’t give any example images.

As mentioned before, the URL scheme is straightforward, but it seems to be more human-friendly than machine-friendly, but more about that later. It consists of the following parts:

1. The country version’s ISO code
2. Car metadata (model is mandatory, body and grade are optional)
3. Optional codes for
   1. option codes
   2. equipment packs
   3. accessories
4. Image metadata like resolution, background colour and car angle

https://example.org
    /de
    /vehicle/trabant/5-doors/0815+universal
    /options/1,4711,815
    /packs/p7
    /accessories/a,b,c
    /width/1024
    /height/768
    /exterior-45.jpg

As you can see, there is a bit of inconsistency that might cause some inconvenience while parsing: the country code doesn’t have a qualifier, the car model, body and grade info are delimited by slashes, the other fields are delimited by commas. Also there’s some information encoded in the filename, but we can ignore that as we’re only parsing the car configuration, not the image metadata.

The right tool for our task

The basic idea of any parser combinator is that it takes an input, reads (“consumes”) it until it is either done or fails, in both cases returning the result and the remaining unparsed string, which might be fed into subsequent parsers. In this example we will be using the Parsec library to manage the grunt work for us. It seems well-suited for this task, it’s quite readable and as straightforward as something that calls itself “industrial strength, monadic parser combinator” can be and there are many clones and implementations in various programming languages.

Parsec is written in the pure functional programming language Haskell, but to understand the examples in this post, you won’t need to understand Haskell. I chose Haskell for the code examples because the parsers contain barely any language-specific syntax, all the handling is nicely kept away from us, making the code almost look like pseudo code.

Writing the parser

As mentioned earlier, a parser combinator is built up from simple building blocks combined to complex parsers. There are quite a lot predefined parsers for common tasks that we can re-use and combine. It’s fascinating that all parsers are independent and can be executed on their own, so everything is easily maintainable and testable.

Parsing characters and letters

We’re dealing with a URL, so even without knowing the specs, we can guess that we will encounter “things delimited by slashes”. According to the web service specification, these “things” can only consist of characters and numbers. We’ll start with a parser that reads these symbols between the slashes and call it value.

value = do
    many1 alphaNum

many1 and alphaNum are parsers already defined in the Parsec library. When run, our combined parser expects one or more (many1) alphanumeric symbols, i.e. letters or numbers (alphaNum). If the input matches these characters it will succeed, if it encounters any other symbol, it will fail. The result of the last line in our do-block is returned automatically.

Using the ‘or’ operator

I lied when I said that values can consist only of alphanumeric characters. Actually the web service specifications also allow the use of “+” and “-“, so we’ll need to add these to our parser. A nice way to achieve this is to use the <|>-operator that basically just means “or”:

value = do
    many1 (alphaNum <|> char '+' <|> char '-')

Another option would be to use ... <|> oneOf "+-".

With this out of the way we have everything we need to combine this parser with another parser that only expects a slash. The resulting parser will be used to read the parts of our URL (you remember: things delimited by slashes). For lack of a better name, let’s call it part.

part = do
    char '/'
    value

This does exactly what you might expect: It reads the character “/” and then a value as defined before.

Failing gracefully

I mentioned earlier that parser functions either return the value or fail. In most imperative programming languages, failing usually means that an error or exception will be thrown. Our parsers work a bit differently: Haskell has a special type called Either that represents values with two possibilities called Left and Right. By convention, Left is used to hold an error value and Right is used to hold a correct value. You’ll see in a minute how that looks.

Let’s confirm in Haskell’s REPL that our parsers work. For this we’ll use the parse helper function that expects the following parameters: the parser to execute, a context name (that we’ll just leave empty) and the input string.

> parse part "" "/foo+bar-42"
Right "foo+bar-42"

> parse part "" "quux"
Left (line 1, column 1):
unexpected "q"
expecting "/"

The results should be self-explanatory: The first parser succeeded, returning a Right with the parsed string. The second call failed, returning a Left with a detailed error message.

Some real work: Parsing strings and optionals

Now let’s start to do some real work and write a parser that reads the first part of the URL including the country code and return that. The protocol can be either “http” or “https”, so we need to take care of both options, for the rest we can reuse our part-parser.

This parser introduces two new predefined parsers: string, which reads any given string (as opposed to char that only reads a single character) and optional that (surprise!) marks a parser as optional.

country = do
    string "http"
    optional (char 's')
    string "://example.org"
    part

In action:

> parse country "" "https://example.org/de/otherStuff"
Right "de"

> parse country "" "http://not-the-example.org/de"
Left (line 1, column 5):
unexpected "n"
expecting "://example.org"

Parsing delimited lists

We’ll skip the vehicle part for now and implement the parsers for the car customization first. Options, accessories and packs are all quite similar: a key followed by a slash and a comma separated list of values. Let’s start with an interesting parser: sepBy splits a string using a separator and returns the results. The output won’t be a string now, but a list of strings instead:

commaDelimited = do
    sepBy value (char ',')

In the REPL this looks like this:

> parse commaDelimited "" "1,2,3,4"
Right ["1","2","3","4"]

The rest is easy:

packs = do
    string "/packs/"
    commaDelimited

options = do
    string "/options/"
    commaDelimited

accessories = do
    string "/accessories/"
    commaDelimited

Looking ahead

With that out-of-the-way we can concentrate on the most interesting part now: The vehicle information that can consist of one or more parts. Let’s start with a quite naive approach:

vehicle = do
    string "/vehicle"
    many1 part

This will obviously not work as it will read beyond the vehicle information and into the customization:

> parse vehicle "" "/vehicle/ford/prefect/packs/pack307"
Right ["ford","prefect","packs","pack307"]

We need a way to tell the parser where to stop. That can be done with the manyTill parser that runs a parser until another parser matches. If we want to check for more than one parser, we can use choice which expects a list of parsers and tries them consecutively. The problem is that parsers works greedily – so any character that matches a parser and is therefor already consumed stays consumed, even if the parser should eventually fail. We can avoid this by using try which makes the parser only consume input when it’s fully executed.

The tricky part is: We don’t want that either. We only want to check, if a parser will succeed, but not actually consume anything. The lookAhead parser does exactly that.

customization = do
    choice [try packs, try accessories, try options]

vehicle = do
    string "/vehicle"
    manyTill part (lookAhead customization)

This will first expect the string “/vehicle” and then run the part parser for as long as possible until our customization parser succeeds, but it will only consume the results of part.

Skipping the rest

That’s it, we don’t need the rest (skipMany), but we should still consume it.

theRest = do
    skipMany anyChar

Returning more than one result

We just created all the building blocks we need to parse the url, but we have to combine them. Up to this point, all parsers did something and returned the last result after running all parsers. What if we want the results of all parsers, i.e. the in-betweens?

Let’s take a step back first and write a parser that reads exactly two alphanumeric characters and then expects the input to end (eof):

twoChars = do
    alphaNum
    alphaNum
    eof

> parse twoChars "" "xy"
Right 'y'

That’s nice, but how do we get to the first x? As each parser returns a result itself, we can just pull it out and store it in a variable using the left arrow <- and return a tuple containing all variables. You don’t have to return a tuple, you can return whatever data type holds your values and matches your domain.

twoChars = do
    x <- alphaNum
    y <- alphaNum
    eof
    return (x, y)

Integrating everything

Wrapping it all up, here’s our complete parser.

parser = do
    c  <- country
    v  <- vehicle
    cs <- many1 customization
    theRest
    eof
    return (c, v, cs)

Taking it for a test run:

> parse parser "" "https://example.org/de/vehicle/trabant/5-doors/0815+universal/options/1,4711,815/packs/p7/accessories/a,b,c/width/1024/height/768/exterior-45.jpg"

Right ("de",["trabant","5-doors","0815+universal"],[["1","4711","815"],["p7"],["a","b","c"]])

Nice.

Conclusion

We just wrote a very maintainable parser in less than 40 lines of very readable code. To achieve this, we wrote small and comprehensible parsing functions that we combined to larger parsers.

Nothing we parsed was very impressive or particularly complex and a few loops combined with regular expressions could have achieved the same with maybe even less code. But as soon as a certain complexity is exceeded, the specification is prone to changes, or you want to create cleaner, more testable code that is actually re-usable, parser combinators can be a great alternative.

You can find the complete source (including type annotations) of the parser here: parser.hs.

What’s next?

This barely touches the surface of what Parsec is able to do. If you’re interesting in learning more, an excellent starting point is chapter 16 of Real World Haskell by Bryan O’Sullivan, Don Stewart, and John Goerzen. It goes into much more detail, but expects quite a bit of Haskell knowledge. Also you can Write yourself a Scheme in 48 Hours using Parsec.

If you want to understand parser combinators in general, not necessarily only the Parsec library, check out Graham Hutton’s great book Programming in Haskell. Erik Meijers functional programming MOOC at edX also covers the topic, closely following Hutton’s book.

As mentioned earlier, there are parser combinator libraries for many languages, here’s a short and by no means complete list:
* JParsec for Java
* FParsec for F#
* PyParsing for Python

If you want to know more about the details of the do-notation and return-stuff or just impress your friends with Zygohistomorphic prepromorphisms, Learn You A Haskell is an excellent start. At the time of writing, Chris Allen and Julie Moronuki are about 90% done with their Haskell Book. I’m sure it will be awesome.

The image Legos by qrevolution is licensed under the Creative Commons Attribution 2.0 Generic (CC BY 2.0) licence.

My scenario

The requirements I have for the permission system in my current project are:

• All CMS pages should be available only for authenticated users
• Editing shall only be allowed for staff
• Some pages should be only visible to a certain group

These requirements are somehow basic, right?

Django CMS’ places to set permissions

Since Django CMS has a lot of different places where you can set permissions I’ll go through them step by step.

“Login required” in page permissions

There is a useful flag in the page permission form which enables to restrict view-access to the page for logged in users. In my opinion, there are two issues with that:

First, it interferes with the group- and user-based settings. I think it would be better to integrate authenticated users into the “View restrictions” section. Currently, it’s hard to tell which to use when.
Secondly, user- and group based settings are applicable also for child pages, while “Login required” is not, which is inconsistent.

Django CMS setting: CMS_PUBLIC_FOR

The CMS setting CMS_PUBLIC_FOR offers the following feature when set to "staff":

If the user accesses a page which does not have a view restriction, deny access, if he’s not staff.

If set to "all" it works like:

If the user accesses a page which does not have a view restriction, show it even if he’s anonymous.

That’s nice, but there is no

If the user accesses a page which does not have a view restriction, force him to login, if he’s not authenticated.

That feature would be basically needed to use of Django CMS as an intranet application out-of-the-box.

Users (page) / User groups (page)

As stated in the documentation, you can “easily add users with permissions over CMS page”. The following paragraph says:

“You would be able to create a user with the same set of permissions using the usual Auth.User model, but using Users (page) is more convenient.”

Actually this seems to be the wrong way to go. Django has perfect tools for creating users – why create another one, and even choose another name for it? This leads to confusion for staff and admins and should be removed.
For “User groups (page)” I can hardly tell, because it’s not fully clear what it does: It seems to be a mix of global page permissions and some special user CRUD functionality. In the documentation you find:

“User groups (page) manages the same feature on the group level.”

But this is either confusing or wrong: The group management form has completely other functionality than the user management form. Since there is no further documentation, this feature is somehow useless for me.

Global page permissions

Setting global page permissions gives a certain user or group permissions for all pages. The documentation is pretty poor, but in general, it’s self explaining how it works. It’s a useful feature if you have requirements like “Editors should be able to create pages, but not delete them.” The only problem here is that the global declaration overwrites permissions on page level – this is something I’d not expect.

Django permission: “Can view page”

Another setting that overwrites page-level permission is the Django permission “Can view page”: If this is set for a user or a user group, the page-level settings are also ignored.

How I solved my requirements

First I wrote a custom middleware that redirects all requests to Django CMS pages to the login. Then, I removed the “can view pages” permission from all groups and all global permissions for non-staff.

Finally I removed all view restrictions on the page root and set them only on the particular pages which should be restricted.

Summary

My two major findings which seem to be worth a discussion are:

If you give a user or group the Django CMS permission to view restricted pages on a global level, this setting overwrites all page-level restrictions.

If you give a user or group the Django permission “can view page”, this setting overwrites all page-level restrictions.

I think the permission capabilities of Django CMS are great in general, but they follow a false principle when the global rule overwrites the special one; it’s just not intuitive.

If you find something I overlooked, I’d be happy to discuss!

Measuring and evaluating software and code quality objectively is a difficult task which is illustrated by the following quote in a funny way:

The only valid measurement of code quality is WTFs/minute
— Thom Holwerda

This implies that the only way to measure software quality is a subjective, individual and broad process. This is certainly true to some extend, but we believe that it is possible to apply concrete practices and methods to ensure a good software quality.

Since software quality is a topic that a whole series of books¹ could be dedicated to, I just want to give a high level overview in this post.

Definition of software quality

If we want to evaluate software quality it basically boils down to the question of how much the software meets the particular requirements that where specified for the software in question.

The requirements can be divided into two groups:

• Functional requirements (FR)
  • FRs define what the system is supposed to do
• Non-functional requirements (NFR)
  • Also called quality requirements
  • NFRs define how the system is supposed to be

Software quality characteristics

Some of the most important characteristics of software quality² are:

• Functional correctness
• Reliability
• Usability
  • Adequacy
  • Learnability
  • Robustness
• Maintainability
  • Readability
  • Evolvability
  • Testability
• Efficiency
• Portability
• Security

Functional correctness is a related to the functional requirements. All other aspects tend to be more related to the non-functional requirements.

Functional correctness

A software is considered correct when it meets all the functional requirement.

Reliability

[latex]Reliability = Correctness + Availability[/latex]

Availability of a software system is the percentage of time where the system is fully operational.

Availability ([latex]A[/latex])is defined as:

[latex]A = frac{MTBF}{MTBF + MTTR}[/latex]

([latex]MTBF[/latex]: mean time between failure, [latex]MTTR[/latex]: mean time to repair)

Usability

The usability describes the ease of use of a software and is defined by adequacy, learnability and robustness.

Adequacy means that

1. the user input
2. the program execution
3. the result of the program execution

are reasonable for the given task.

The learnability of a software is influenced by the documentation and by the user interface which should provide efficient use of functionality.

A software is considered robust if the consequences of defect are inversely proportional to the probability of occurrence of the defect.

Maintainability

A software is considered maintainable if

• defects and their cause can be isolated easily
• defects can be fixed easily
• changes can be made easily
• extensions can be made easily

Evolvability is defined as the capability of a system to be extended with new features.

Therefore maintainability is closely related to readability, evolvability, and testability of the code.

Efficiency

Efficient software emphasizes a careful use of resources like e.g. CPU, RAM or external (secondary) storage. Efficiency is not to be mixed up with product efficiency. The product efficiency of a software system is related to the investment of time and money.

Portability

The portability describes how easy it is to port the program to another platform or device.

A software is considered to be portable if the changes that have to be made to the system to run on another device or operating system are less expensive than reimplementation.

Security

The security of a software is partly related to the requirements as security concerns should be part of the functional requirements for the system in question. Besides that security is also related to defects and design flaws which can be minimized by a good code quality and software architecture. There are tools which can help finding security issues like e.g. penetration testing tools.

Here are some aspects that have to be considered for good security:

• Authorization
• Encryption
• Handling passwords
• Session management
• Validation
• Preventing SQL injection

Clean Code Developer

Clean Code Developer (CCD) is an initiative that promotes software quality and the cultivation of higher professionalism within the software industry.

CCD defines four main values:

• Evolvability
• Correctness
• Product efficiency
• Continuous improvement

From these values the CCD initiative derives a hierarchy of principles, patterns and tools that support the values and that are divided into different skill levels.

Evolvability and correctness have already been defined above.

Product efficiency

Product efficiency is influenced by the time and the costs for developing a software system. A software system is considered to be (product) efficient when the investment of time and money is reasonable low. Product efficiency should not be mixed up with efficiency as defined earlier.

Continuous improvement

Continuous improvement is not a software quality characteristic. However, in order to keep the software quality high, it is important for software development teams to make reviews and retrospectives.

Maintainability is crucial

While all the characteristics described above are important, maintainability is crucial. If the code base is messy, it will significantly slow down the development process. Introducing new features will be expensive because they are hard to weave in. Throwing more staff at the project will only result in a growing mess and productivity yet decreases. This scenario is described in Robert C. Martin book Clean Code. He calls it “the total cost of owning a mess”. The cost is that productivity will tend to drive towards zero over time.

Methods and practices

Software engineering practices and methods have great influence on the level of software quality. Some of the most important are:

• Agile software development
• Domain driven design
• Type driven development
• Iterative software development
• Code reviews
• TDD
• Refactoring
• Automated testing
• Continuous integration
• Issue tracking
• Source code conventions
• Penetration testing

Design principles

Here is a set of design principles that are closely related to high maintainability:

• Don’t repeat yourself (DRY)
• Keep it simple stupid (KISS)
• SOLID principles
  • Single responsibility principle (SRP)
  • Open closed principle (OCP)
  • Liskov substitution principle (LSP)
  • Interface segregation principle (ISP)
  • Dependency inversion principle (DIP)
• Separation of concerns (SoC)
• Favor composition over inheritance (FCoI)
• You ain’t gonna need it (YAGNI)

Metrics

In order to find a starting point for improvement of quality and maintainability of the code, it can be valuable to examine some code metrics.

The most important code metrics are:

• Lack of cohesion of methods (LCOM)
• Afferent and efferent coupling (Ca, Ce)
• Cyclomatic complexity (CC)
• Lines of code (LOC)
• Number of methods / fields per class (NbMethods, NbFields)
• Abstractness (A)
• Instability (I)
• Distance from the main sequence (D)

There are several tools for deriving those metrics like NDepend (.NET) or Sonar (Java).

Software Tests

For good software quality it is essential to have a good testing strategy which should include:

• Unit tests
• Integration tests
• System tests
• Regression tests

There are tools that help with testing like

• Unit testing frameworks
• Mocking frameworks
• Test coverage tools

Additional concerns

Make invalid state unrepresentable

Making invalid state unrepresentable is a very valuable way to make a software maintainable, testable and robust. Here is an example of a poor type design:

public class Contact
{
  public string FirstName { get; set; }
  public string MiddleName { get; set; }
  public string LastName { get; set; }

  public string EmailAddress { get; set; }
  public bool IsEmailVerified { get; set; }
}

If this was a domain model from the business layer, then there are several problems. It is not clear

• which values are optional / required
• what the constraints are
• which fields are linked together
• what the domain logic is

E.g. possible invalid states would be, all fields are NULL, EmailAddress is NULL and IsEmailVerified is true, last name is missing, etc.

These problems can be solved by a good design of the types, which will improve the software quality a great deal.

The problem with NULL

Even though most software developers got very much used to it NULL is a problem. In this post the author describes how NULL

• subverts types
• is sloppy
• is a special case
• makes poor APIs
• exacerbates poor language decisions
• is difficult to debug
• is non-composable

Software quality can be improved by choosing a language that does not know NULL like e.g. F# or Haskell.

The problem with state

Managing state can become complex very easily. When passing a mutable object to a function, it can potentially be changed by that function. The function signature might or might not imply this. Anyway, there is no way to be sure that the object won’t be changed. If the object is passed to many functions from many different modules, it will become very hard to keep track.

If state is even mutated from different threads it becomes even more difficult. Using locks is only a poor workaround that results in more complex and less maintainable code.

Here immutability comes to the rescue. Some languages support immutable data structures by default as e.g. F#, Haskell, Clojure and other functional programming languages.

With immutable data structures it is much easier to reason about the correctness of the code. Because of referential transparency there can’t be implicit side effects³. The order of function calls doesn’t matter and the code becomes much more predictable. Less tests are needed because functions can be tested in isolation. We do not have to test a function in the context of all possible or potential system states which will result in a better product efficiency.

Also concurrency is much simpler because we do not have to worry about concurrent updates.

The programming language

Also the choice of programming language can affect the quality of the software. The following language features can improve code quality and product efficiency:

• A good type system
• Type inference
• Concise syntax
• Pattern matching
• Immutability by default
• Structural equality by default
• Scriptability
• Ability for rapid prototyping

A closer look at dependencies

Visualizations of the dependencies between types in a software system can be a good measure for quality besides code metrics. In this article the author presents some interesting examples.

Here are two graphs of the two libraries SpecFlow and TickSpec which both have a similar sized feature set.

SpecFlow:
⁴

TrickSpec:
⁵

The author argues that the complexity of the code is closely related to the programming language. TrickSpec was implemented in F# and shows a clear and clean dependency graph. On the other hand SpecFlow is a well designed library in C#, but still might be more difficult to maintain because of the more complex dependencies.

Conclusion

Software quality is an important but complex concern.

You can see this post as a broad summary of the topic. It can serve as a starting point, a check list, or a guide line when you make considerations about software quality.

Resources

• Software quality
• Clean Code Developer
• Design with types
• The worst mistake of computer science
• Cycles and modularity in the wild
• Principles of OOD
• Immutability

The header image by Luis Llerena is licensed under a UNSPLASH LICENSE. The picture was modified to fit this article.

¹

Reading Time: 15min

What this is about

• Secure and reliable syslog remote logging
• Free yourself from receiving hundreds of logging emails each day
• Setup email alerts
  

Log server

I assume we all have an understanding what logging is and why logging matters.
What we want is a centralized place where all log messages can be sent to, where they are analysed and can trigger alerts.
There is a big picture you can get over long-term logging, called Data-Mining² and which is out of scope for this article.

The first problem I repeatedly faced was how to collect all logging messages securely in a central place. Of course rsyslog can manage it, but has its own pitfalls and doesn’t allow instant further processing.
Logstash³, written in Java, and logstash-forwarder, written in the Go language, came to the rescue.

Logstash-forwarder, Logstash, Graylog2, Elasticsearch, Kibana.

What a chain of tools. Logstash-forwarder (shipper), Logstash (server/filter), Elasticsearch (database) and Kibana (frontend) seem to be an obvious choice, since “brought together” by Elasticsearch in 2013⁴.
Graylog2 (server/filter/frontend) is a different project from Germany that seems to have a lot in common, but on the other hand has some frontend features that are missing in Kibana.

Logstash-forwarder uses the Lumberjack-protocol that should not be mixed up with the “Project Lumberjack”⁵, based on the work of the CEE: Common Event Expression⁶.

CEE is a sort of comedic tragedy of design

— Jordan Sissel, developer of Logstash and logstash-forwarder⁷.

I can only guess why he named his own protocol Lumberjack, but the name seems to be a great fit⁸/⁹ and also a parody of the Monty Python skit “Lumberjack Song”¹⁰.

The lumberjack protocol used by this project exists to provide a network protocol for transmission that is secure, low latency, low resource usage, and reliable.

— Jordan Sissel about logstash-forwarder¹¹

Logstash-forwarder takes files or stdout as input stream. Then it provides it securely and reliable to Logstash, which does further processing. This is exactly what we need, as all applications already write log-files. With Logstash alone we would have plenty of choices for other input formats. For the task of remotely collecting syslog files, the so called “lumberjack-input” suffices, as it is the service that logstash-forwarder connects to.

Forward and process

Logstash-forwarder is sending the syslog file, tagging it as syslog type. Besides the server and SSL-cert configuration, it’s configuration has only a “files”-block, indicating what it is supposed to send to the Logstash server:

"files":[
 {
 "paths":[
 "/var/log/syslog"
 ],
 "fields": { "type": "syslog" }
 }
]

Now we have a lightweight shipper on each host, sending logfiles to our logserver, where we have Logstash and its more heavy requirements Java and Elasticsearch installed. As a frontend view on Elasticsearch, we also have Kibana¹² in place.

Focusing on Logstash, it collects all messages forwarded by the shipper (logstash-forwarder) and outputs everything to the database (Elasticsearch) for later review (Kibana).

The main configuration for Logstash is also very simple.

Have an input:

input {
 lumberjack {
 port => 5000
 ssl_certificate => "path"
 ssl_key => "path"
 }
}

Have an output:

output {
 elasticsearch { host => localhost }
}

Anything that arrives via logstash-forwarder, will be stored to Elasticsearch.
With Elasticsearch’s query language, we have the possibility to search and filter very quickly, to build exactly the view that helps us identifying/analysing problems.
This leads to the questions what to search for, or in other words: How is our data structured?
The next big problem with logging is that most applications have their own log-message standard. But we want to have it normalized. One simple definition of a log-message is:

“timestamp plus data” – Jordan Sissel¹³

That describes enough to have at least one timestamp field required.
As logstash-forwarder just sends files line by line, we need a parser construct for Logstash to analyse the incoming data. Therefore Logstash has filters. A special one, we can use to parse the incoming syslog messages, is called “the grok filter”.
The grok filter provides named regex-patterns to break up a complex regex for parsing a message line. I know this method a bit from fail2ban and am still wondering, if they have anyhow related roots.
With the filter now in place, we extract a timestamp field and some other fields additionally to the base message and are then able to search within Elasticsearch more specifically.

filter {
 if[type] == "syslog" {
 grok {
   match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
 }
 date {
   match =>[ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
 }
}

Logs via email

As every host is sending emails from applications or cronjobs to hostmaster, my mailbox was a pain to look at every morning. Now I forward everything to a special mailbox and parse it via Logstash, which can also trigger a simple alert.
Later on, I will show how we made our setup using Graylog2, which provides alerts via streams over a web GUI.

IMAP input

As Logstash is already running, I only needed another input

input {
 imap {
 host => "dns, or ip"
 password => "password"
 user => "login"
 type => "email"
 }
}

Now every mail that arrives will be tagged as “email”, and has already been processed with a timestamp by the IMAP input.
I want to have a simple alert, which sends me an email for every matched error. Logstash has a conditional expression:

output {
 if[type] == "email" and[message] ~ /error/ {
 email {
 attachments =>["message"]
 to => "email-address"
 }
 }
}

Wow, that was easy. And there are also Logstash outputs to inform our Icinga, or send a note via “hipchat-output”.
But now, imagine using the email alert for a series of logging messages. That could easily lead to hundreds of email alerts. We need a more sophisticated solution for this.

Email alerts

After having Graylog2¹⁴ installed and configured on the same server Logstash is running, I only changed the output of Logstash to gelf:

output {
 gelf { host => localhost }
}

And now everything Logstash gets via inputs and parses via filters, arrives at Graylog2’s gelf-udp-input, which then has an Elasticsearch output again.

Why the heck this overhead? Because Logstash in combination with logstash-forwarder is great at sending messages securely, reliable and low resource. Also it has a lot pre-processing options.
And Graylog2 is great in terms of alerts. I can now configure “streams” that are basically pattern-matched messages, and can trigger an pattern-matched email-alert.

What is the benefit over the Logstash email output?

• Graylog2 stream alerts can be paused after first trigger, leading to a much lower alerting rate
• Graylog2 can manage users and permissions for streams that let me provide logs for a certain host to anybody using it, without showing everything
• Graylog2 has a lot more to offer. E.g. looking promising with additional external dashboards specialized for deployments¹⁵

Additionally, I will keep using Kibana, because it has a much cleaner view on the whole index.

Résumé

With rsyslog forwarding I always duplicated all logs before processing them. Without logstash-forwarder it was a hassle to use logstash (overhead of Java, or rsyslog odysseys). Now logging is fun for me. Lightweight forwarder, secure transmission. The specs for the logserver are a bit heavy (6GB RAM), but as best practice advices, it could be clustered if required.

Disclaimer

I provide by no means a copy and paste ready configuration. You are in responsibility to understand missing requirements and side-effects.

Resources

Basically the program filters a stream of tweets by user input. There are two kinds of ongoing events involved, user inputs and tweets incoming in real-time from Twitter. Here are some detailed specifications:

1. The user can type a search text into a search field
   When the user has entered at least 3 characters and stops typing for at least 2 second the text in the search field will be processed for a new search.
2. User inputs will be used to filter the incoming tweets
   The tweets must contain an exact (but case-insensitive) match of the user input. The filtered tweets are displayed on the UI ordered from most recent (top of the list) to oldest (bottom of the list). Every time the item count of the list exceeds a maximum of 20 the oldest tweet is supposed to be removed.
3. New tweets should be displayed with a sample rate of one second.
   Only a maximum of one tweet should be displayed per second. All other tweets should be dropped.
4. When the search text is changed according to 1. the tweets should be filtered by the new text according to 2.
5. All filtered tweets are classified and sorted into different lists
   Based on the score provided by a classifier each tweet will be considered either as awesome or boring. A score greater than 1000 is considered to be awesome and a score equal or less than 1000 is considered to be boring. Awesome and boring tweets are displayed in separate lists.

(The classifier function can be seen as a black box. In this sample it is a very stupid algorithm that just serves to demonstrate the capabilities of Rx. In another scenario we could imagine it to be some kind of machine learning algorithm or something like that.)

This is how the application’s simple UI will look like:

The source code can be downloaded here.

Implementation

Now we will go through the requirements one by one and explore how they can be implemented using Rx.

Handling user input

To handle the user input we need to apply the Rx operators Where and Throttle.

Where

Where is a filtering operator that filters an observable sequence based on a predicate function. In the diagram below only the result sequence only emits values that are greater than 2.

This can be used to filter out search texts with less than 3 characters.

Throttle

The filtering operator Throttle only emits an item from a source sequence when a given time span has passed without the source emitting any other items. In other words all items that are rapidly followed by other items are ignored.

In our application we can use this when the user is typing in characters. As long as he is rapidly typing no events will let through. But as soon as he stops for a given time span (in this case 2 seconds) an item from the source stream is emitted.

To convert the property that is bound to the view to an observable we can use this extension method as described here. Also we have to make sure that in the view the control’s data binding has an UpdateSourceTrigger that is set to PropertyChanged.

<TextBox Text="{Binding Input, UpdateSourceTrigger=PropertyChanged}"/>

Then we can use this code to create the observable sequence of search text inputs:

var searchTexts = this
    .ToObservable(() => SearchTextBox)
    .Throttle(TimeSpan.FromSeconds(2))
    .Where(x => x.Length > 2);

If we subscribe to this observable by adding all items to a list the output might be something like this:

Combining the search text with the tweet stream

The function allTweetsAbout for searching the tweets is passed into the constructor of the view model. It takes a search text as an argument and returns an observable stream of tweets which is all we need to know. The internals of this function will not be discussed here because this article should focus on the consumption of observables as opposed to creation.

In order to combine the search text stream with the tweet stream we need to introduce the operators Switch, Sample, Publish / Connect and ObserveOnDispatcher. (The Select operator we have already seen in the previous part.)

Switch

The combining operator Switch can be applied to an observable of nested observables. Switch only emits items from the most recent inner observable.

In our program we combine the observable of search texts with the tweet streams using the Select operator. The result is an observable of observable of tweets. As we are only interested in the most recent search this is a perfect application for Switch.

Sample

The filtering operator Sample emits only the most recent item for every specified periodic interval.

Publish and Connect

Publish allows us to share the actual values of an observable with multiple subscribers.

ObserveOnDispatcher

This extension method makes sure that the subscription is made on the UI thread. If we don’t use this in a WPF application it would cause some sort of threading exception.

Note that in order to be able to properly test the application we would use ObserveOn with an IScheduler instance instead which should be injected into our class from outside. More information on this can be found here.

Composing the application

We have introduced all the necessary Rx operators. But for updating the view model we need two more functions, UpdateViewModelAwesomeTweets and UpdateViewModelBoringTweets whose implementations are trivial. Now we have everything to compose the application as follows:

var searchTexts = this
    .ToObservable(() => SearchTextBox)
    .Throttle(TimeSpan.FromSeconds(2))
    .Where(x => x.Length > 2);

var tweets = searchTexts
    .Select(allTweetsAbout)
    .Switch()
    .Sample(TimeSpan.FromSeconds(1))
    .Select(classifyTweet)
    .Publish();

tweets
    .Where(x => x.Score > 1000)
    .Select(x => x.Tweet)
    .ObserveOnDispatcher()
    .Subscribe(UpdateViewModelAwesomeTweets);

tweets
    .Where(x => x.Score <= 1000)
    .Select(x => x.Tweet)
    .ObserveOnDispatcher()
    .Subscribe(UpdateViewModelBoringTweets);

tweets.Connect();

The complete source code of this application can be downloaded here.

Summary

Easy combination of multiple event streams
Combining and handling multiple event streams is not a simple task. But with the help of Rx this becomes really easy.

Dealing with time
When the relationship with time is involved in the event processing Rx provides a set of powerful operators that deal with time like e.g. Throttle and Sample.

Separation of concerns
In the example application we have separated the concerns pretty well. We use small functions that all have a single responsibility like e.g. searching tweets, classifying tweets or updating the model. The integration of all these functions is expressed comprehensively by the composition with the Rx operators.

Resources

• Source code
• ReactiveX – Operators
• Video: An Event-driven and Reactive Future
• Testing Rx
• Typesafe conversion of viewmodel property to observable stream

The header image “tweet” by mozzercork is licensed under a Creative Commons Attribution 2.0. The picture was modified to fit this article.

In this post we will have a closer look at the operators Select, Zip and Skip and review a sample application where those operators are extremely useful.

The application is a very simple WPF application where you can draw lines in a window. For each line the distance will be displayed. In fact the code for this program is so simple and declarative that it really shows the shining beauty and elegancy of Rx.

The source code of the app can be downloaded here.

Operators

Let’s first introduce the operators that are involved in this example.

Select

Select is a transforming operator that projects the elements of an observable sequence into a new form by applying the function f to each element.

Skip

Skip is a filtering operator that bypasses the first n elements of a sequence. The result is a sequence that emits all elements of the source sequence except the first n elements.

Zip

Zip is a combining operator that merges two sequences by applying a function f to each pair of elements. Pairs are created by combining each element of the first sequence with the element of the second sequence that has the same index as the first element.

Operators in practice

To see the operators in action we will review this sample application that lets us draw lines on the UI. For each line the length is displayed. The length is the distance between two positions on the UI represented by two Point instances. In this case the coordinates of the points are determined relatively to the canvas control of the main window.

Let’s go through the steps that are necessary to implement this.

First let’s take a look at the simple class Line which is the main type used in the application:

public class Line
{
    public Point PointA { get; private set; }
    public Point PointB { get; private set; }

    public double Length
    {
        get
        {
            var d = Point.Subtract(PointA, PointB);
            return Math.Sqrt(Math.Abs(d.X * d.X - d.Y * d.Y));
        }
    }

    private Line(Point a, Point b) { PointA = a; PointB = b; }

    public static Line Create(Point a, Point b)
    {
        return new Line(a, b);
    }
}

The implementation is straightforward. The factory method Create takes two points and returns a new Line instance. It is just for convenience because it simplifies the syntax when used instead of the constructor with Rx operators. Length calculates and returns the distance between the two points. For the sake of simplicity the value of Length is not stored locally which would be an optimization.

Now we have to convert the mouse button click events into an observable stream. This can be done with the Observable.FromEventPattern<TEventArgs> function. As the first parameter we have to provide the object instance that exposes the mouse click event which is the MainWindow instance itself. So we can pass in this here. The second parameter is the name of the event that we are interested in which is "MouseUp". The result is an observable sequence of EventPattern<MouseButtonEventArgs>:

Observable.FromEventPattern<MouseButtonEventArgs>(this, "MouseUp")

Next we have to convert each EventPattern into a Position of type Point. We could do this by just using a lambda expression. But instead we define the little nested helper function getPosition. The benefit of the helper function is that it makes the code more compact later when the function is passed to the Select operator. The position is determined relatively to the Canvas UI control named _box.

Func<EventPattern<MouseButtonEventArgs>, Point> getPosition = 
    args => args.EventArgs.GetPosition(_box);

Finally we need a function that takes a line and updates the UI by drawing the line and displaying its length. Therefore we define a private method UpdateUi. The details here are WPF specific and not really relevant to us right now.

Composing the operators

Now we have defined all necessary functions. The only thing left to do is to transform and combine the observable sequence of click events by composing and applying the Rx operators Select, Skip and Zip.

The idea is to create lines by taking each point of the sequence and combine it with its predecessor.

The solution is simple and elegant. After we transformed the click stream to positions with Select(getPosition) we shift the sequence by one with Skip(1). Then we merge the shifted sequence with the original sequence with Zip and pass in the factory method to create lines. That’s it!

There is no need to keep the state e.g. the last clicked position in a local or instance variable.

Here is marble diagram that visualizes how this works:

Connecting the observer

Finally we have to connect the observer (which is the UI in this case) with the observable by calling the extension method Subscribe and pass in the UpdateUi function. Here is the code that is responsible for composing all the functions to create the program:

var points = Observable
    .FromEventPattern<MouseButtonEventArgs>(this, "MouseUp")
    .Select(getPosition);

points
    .Zip(points.Skip(1), Line.Create)
    .Subscribe(UpdateUi);

Summary

We have seen how a program can be composed by using observable sequences and applying Rx operators to them. The source code of this program is very concise and declarative. Also there is no state involved (except for the UI off course) that we have to be cautious of.

Using Reactive Extensions in event-based applications leads to code that is simple and elegant and therefore easy to understand, maintain and extend.

Resources

• Source code
• ReactiveX – Operators

Textual representation

There is a simple textual representation that uses the following symbols:

A sequence with 3 items looks like this:

--O--O--O--|->

Combining sequences

The following diagram visualizes the Select(f) operator applied to Seq1 that creates a result stream (Result) where each emitted value of Seq1 is replaced according to the function f. In this case f adds one to the input value:

Seq1   --1---2-----3---->
       Select(x => x + 1)
Result --2---3-----4---->

Graphical representation

Here is the graphical representation of the example from above:

Resources

• RxMarblesInteractive diagrams of Rx Observables
• ReactiveX – Operators

Rx can be used in desktop as well as web-based applications on either server or client side. A few common situations / problems where you might benefit from using Rx are:

• Unresponsive UI
• Long running computations that could potentially block other threads
• Asynchronous programming where multiple sources of data and events have to be combined
• Enabling and managing concurrency on backend independent of the consumers implementation
• Handling errors and cancellation of asynchronous tasks

In this post I will introduce some general ideas behind Rx and show how it can help in some of the situations stated above. If you are new to Rx it will give you a basic understanding of what Rx is and where it can be useful to apply.

The examples are all presented in C#. But the concepts apply to all programming languages.

Definition

A definition of Rx could be:

Rx is a library for programming with asynchronous data streams. It is a combination of the observer and the iterator pattern as described by the Gang of Four combined with ideas and concepts from functional programming.

Querying Data

To get an idea when it is useful to use Rx let us look at ways how to retrieve data. Basically a query result can be either a single object or a collection of objects. The data can be queried either synchronous or asynchronous. This produces the following matrix:

Sync Single

A function that returns a single object and is executed synchronously is probably one of the most common scenarios that we encounter. In the example the function is called GetData() and the return value is of the generic type T as shown in the following table.

Calling the method and processing the result might look something like this:

string s = GetData();

if(s.Equals(x))
    // do something
else
    // do something else

If this code was implemented behind a user interface the process will be blocked for however long it will take to retrieve the value because this is done synchronously.

Sync Multiple

When we examine the synchronous query of multiple values we will see the same behaviour except that the return value is a collection of objects of type T.

The LINQ library provides a rich toolbox of query operators to create, transform, combine and filter collections. In the example however we want to iterate over all the values and do some processing with each of them.

IEnumerable<string> values = GetData();

foreach(string value in values)
{
    if(value.Equals(x))
        // do something
    else
        // do something else
}

Again this is done synchronously and the UI will be blocked.

Async Single

To make the UI more responsive values can be queried asynchronously. In case of a single value the function might return a Task<T>. A Task in C# can be seen as the equivalent of the Future in Java. It is the result of an asynchronous computation.

Processing the result of a Task can be done synchronously which again will block.

Task<T> task = GetData();

if (task.Result.Equals(x))
    // do something
else
    // do something else

So that is not really an improvement. A better workflow when creating a responsive UI would be:

• Respond to some user action
• Do work on a background thread
• Pass the result back to the UI thread
• Update the UI

To achieve this we should consider the async features in .NET 4.5 or apply a callback. A callback is a function that is passed as an argument and will be called at a later time e.g. with the result of a long running computation. In this case we can use the ContinueWith method of Task that takes a function that is executed asynchronously.

task.ContinueWith(t =>
{
    if (t.Result.Equals(x))
        // do something
    else
        // do something else
});

Async Multiple

Asynchronous computations that yield collections of values can also be handled with callbacks. Actually this is what is done in many cases. But we might quite easily run into several problems as the application becomes more and more complex.

E.g. there is no way to start iterating over the result and process the values before the whole computation completes. If loading the whole collection takes a long time it would be very convenient to start working with the first part of the list while the rest is still processing. But when using callbacks this is not possible.

And what if we want to process each element again asynchronously and then each result of that yet again… and so on? The consequence would be a deeply nested composition of callbacks which will become very unwieldy and won’t be easily maintainable any more.

With nested composition callbacks become unwieldy and lead to the need to synchronize.

This is where Rx comes into play with the main type IObservable<T> – the counter part of IEnumerable<T>.

IObservable<T> represents an asynchronous data stream of objects of type T. Consuming the result is quite simple:

IObservable<T> o = GetData();

o.Subscribe(t =>
{
    if (t.Equals(x))
        // do something
    else
        // do something else
});

The Subscribe method has several overloads. In the example above it takes an action as a single argument which will be applied to each value from the Observable.

Composable functions

As well as LINQ Rx also includes a rich library of higher-order functions that allow us to transform, combine, filter etc. observable sequences. Those functions are easily composable. This is one of the really great benefits of Rx because it makes querying, combining and transforming asynchronous data streams as simple as applying LINQ queries.

Roughly the set of functions can be divided into the following categories:

Transform: Select, SelectMany, Aggregate …

Filter: Skip, Take, TakeWhile, Where …

Combine: Concat, Merge, Zip …

Boolean Operators: Any, All, Contains …

Mathematical Operators: Max, Min, Average, Count, Sum …

Concurrency: ObserveOn, SubscribeOn …

Error Handling: OnErrorReturn, OnErrorResume …

Here is a simple example of how some of these functions can be combined:

IObservable<int> observable = GetData();

observable
    .Skip(1)
    .Take(3)
    .Where(x => x < 10)
    .Select(x => x + 1)
    .Subscribe(Console.WriteLine);

IEnumerable vs. IObservable

To get an even better understanding of what an IObservable<T> is we will compare it to the IEnumerable<T>.

IEnumerable

The method IEnumerable.GetEnumerator() returns an object of type IEnumerator which iterates over collections and has the following methods:

• bool MoveNext() – Advances the enumerator to the next element of the collection. Returns true if it succeeds and false if it fails (when the enumerator points to the last element of the collection or empty list).
• T Current{ get; } – Returns the current element in the list.
• throws Exception – This is actually not a method of the interface. But an exception might be thrown when calling Current so an exception is a potential implicit return value of Current. In other words the return value of Current could be described as Either<T, Exception> or Tuple<T, Exception> or similar types.
• void Dispose() – Disposes of all resources used by the enumerator.

The general task of an enumerator is to pull data from a collection.

Below we can see the enumerator in action (Note that this is just for demonstration and is not good code. Usually we would use a foreach statement to do the iteration over a collection.):

var enumerator = new List<int> { 1, 2, 3 }.GetEnumerator();

while (enumerator.MoveNext())
{
    Console.WriteLine(enumerator.Current);
}
enumerator.Dispose();

IObservable

Each method of the IEnumerator interface has its corresponding method in the IObserver interface:

• OnCompleted() – Notifies the observer that the provider has finished sending data.
• OnNext(T) – Pushes the next value to the observer.
• OnError(Exception) – Notifies the observer that an error occurred.

The general task of an observer is to push data into a stream.

Here is an example of how the methods described above can be called in order to create an observable sequence:

public static IObservable<int> GetData()
{
    return Observable.Create<int>(o =>
    {
        o.OnNext(1);
        o.OnNext(2);
        o.OnNext(3);
        o.OnCompleted();

        return Disposable.Empty;
    });
}

For detailed information on how this works and on all the query operators and more check out the amazing online resource about Rx .NET by Lee Campbell.

The methods of IEnumerator and IObservable relate to each other in a special way as shown in the following diagram. Actually they are dual to each other.

On an observable sequence and on an enumerable sequence the same or at least very similar functions can be applied:

// IObservable<string>
// that emits 75 Strings
GetData()
    .Skip(10)
    .Take(5)
    .Select(x => x + "_transformed")
    .Subscribe(x => Console.WriteLine("next => " + x));

// IEnumerablee<string>
// that contains 75 Strings
GetData()
    .Skip(10)
    .Take(5)
    .Select(x => x + "_transformed").ToList()
    .ForEach(x => Console.WriteLine("next => " + x));

Summary

We have seen how Rx can help to make the UI responsive when dealing with computationally expensive tasks.

Also we could get an idea of how easy and comfortable it is to handle asynchronous data from multiple sources with the use of the LINQ-style query operators.

Enabling and managing concurrency on the backend as well as handling exceptions, cancellation etc. are other areas where Rx really shines. But these details go beyond the scope of this post. For more information on that please check out the resources below.

Things to keep in mind:

• Rx is a library for programming with asynchronous data streams
• Rx provides a rich library of composable functions to transform, combine, filter etc.
• The key types are IObserver<T> and IObservable<T>
• The pair of types IObservable<T> and IObserver<T> is dual to the pair IEnumerable<T> and IEnumerator<T>
• Rx provides rich functionalities for a scenario where a producer asynchronously pushes data to a consumer

Resources

• ReactiveX
• Getting started with the Reactive Extensions to .NET
• Java
• JavaScript
• C#
• Scala
• Clojure
• C++
• Ruby
• Python
• Groovy

The header image “Long exposure head lamps & tail lights” by JC+A is licensed under a Creative Commons Attribution 2.0. The picture was modified to fit this article.