Privacy – My next big thing

Earlier this year I started to work on a privacy team – more specific we are working on tools for anonymization. That’s why I recently started to deep dive into anonymization techniques like differential privacy. I also learned about concepts like homomorphic encryption which is extremely exciting.

At the same time – especially as a parent – privacy becomes more and more important. Seeing all these parents equipping their kids with smart watches that will track their location, terms and conditions of apps being ignored by parents because their kids “need” to use WhatsApp and sharing all these photos and videos from your kids on the internet without their consent. This is something which is really important to me and that seems to become more and more important the older the kids get and the more they use the internet.

To learn more about the different areas of privacy, I became a Certified Information Privacy Technologist (CIPT) and I also read a lot of related books, newsletters and listened to interesting podcasts. I will share some recommendations soon.

If you want to learn more feel free to follow my next blog posts and join me on diving deeper into the field of privacy.

Know Your Tools

A few months back I switched companies. For me this means new team, new domain, new programming language, new tools. After starting with my first project I quickly learned that I had a lot to learn.

Waiting

I am working on end2end tests of a web application. If you ever worked on this you know, that end2end tests are very slow.. Because I just started with the programming language (actually this was my first code I wrote in it), I made a lot of newbie errors. Missing imports, missing semicolons, … But the tools I thought were standard didn’t do auto imports or highlighting compiler errors (this does not mean that they did not support this, but I definitely used them wrong). So I always run the build command just to learn 30seconds later that I missed something obvious. When finally everything compiled, the tests were failing. I was also new to the testing framework, so I did not know all features that were supported nor how to use it properly. I need to do some trial and error.

After spending a whole afternoon waiting, I felt very frustrated. I am working in an engineering productivity team. This was definitely not engineering productivity. I want to consult people and support them in being more efficient, but I was not able to be efficient myself. Not even close.

Increase Productivity – The Power of the Debugger

So I decided to spend the next day to work on my developer setup. I read the documentation and guess what, they were suggesting a different IDE. After setting up the IDE I felt relieved. Compiler errors were displayed immediately, as well as missing imports. This alone would save me hours. I learned how I can execute single test cases (I already knew how to execute single test files but running single tests increased the build runtime even more).

I asked the dev team how I can reduce execution time and got a few more hints. And then I realized that I could just debug the test and evaluate expressions and scroll threw the object instances on the fly. Suddenly I was pretty fast compared to the day before. End2End tests are still slow, I’m not able to change this (at least not in short term), but getting to know the tools, their features and how to use them increased my productivity a lot.

In the past I met some developers who just lived with such a situation. “It’s slow, what can I do? That is how it is.” and they waited and waited (and complained about the slow end2end tests and that they do not want to write more tests). Don’t do this. Invest a few hours to make a good setup and learn about the recommended tools. Afterwards you are much faster and much happier.

What is Engineering Productivity?

Every programmer likes different things. I always enjoyed developing tools for the team, automate things and make the developers live easier. A few months ago, I finally stumbled upon a term which summaries all of these things I’m passionate about: Engineering Productivity 💕

Of course it doesn’t make any difference if there’s a term for something or not. But for me it felt like ‘I’m not alone’ and ‘This topic does exist’. And of course it is much more easier to phrase what I really want to do.

But as always, everyone has other things in mind when they hear about Engineering Productivity. There is no one official definition. So let me explain, what Engineering Productivity is for me.

What is part of Engineering Productivity?

Engineering Productivity aims to help developers to deliver high quality code as fast as possible. Most of the people think about Engineering Productivity as setting up environments quickly or have the right test tools and metrics. Of course this is all part of it.

For me it includes especially the following topics (in no specific order):

  • Setup
    Easy setup of a new project with CI/CD and everything you need.
  • Coding
    A useful IDE, easy to setup for new team members and all necessary plugins. (If there is one necessary plugin missing, I’m happy to implement it. 😉 )
  • Process
    Of course the whole process is important. Which meetings do the developer have? Are they useful and are the intervals useful? Where do they loose time because they have to wait? etc. Sometimes things are done because “we always did it this way”, it’s always worth to think about how the process can be improved.
  • CI/CD
    One of my favorite topics. Everything possible should be automated. You should test as much as possible with a minimum build duration.
  • Debugging
    You should have all the tools and knowledge how to debug efficiently. During debugging a lot of time is wasted. Sometimes this is good, because developers learn a lot while debugging. But sometimes it is just a bad setup or a lack of knowledge.
  • Monitoring
    Developers can fix bugs easily if they are discovered quickly. With a good monitoring you will see errors or increasing response times very fast and hopefully can fix it or roll back. Tools for monitoring are sometimes underrated. You need to have a good dashboards and links to further information when something goes wrong. Developers don’t want to waste time by searching through a dozen of different systems.
  • Testing
    The code should be easy to test. For this you need a testing strategy, maybe some testing frameworks, sometimes testing patterns are useful. What is a unit test? When do I want to have an integration test? How to test the front-end, the database, all different devices and which test environments do I need?
  • Documentation
    To be productive you want to find answers and explanations quickly. So documentation is also part of Engineering Productivity. This does include Architectural Design Records, Bug Tracking, Root Cause Analysis, Diagrams, Backlog, etc.

You see, Engineering Productivity is a very wide field and I really enjoy to improve my productivity and the productivity of the team.

Do you miss something? I’m happy to learn about all the areas you think about when you hear Engineering Productivity.

The Story of my Favorite Algorithm

Recently, while surfing threw YouTube I saw a video where someone mentioned his “favorite algorithm”. Whenever I hear the word “algorithm” there is only one in my mind. We have a history. This algorithm is almost a friend.

Back in 2007 (during my diploma studies), we had to implement a software which had a part of searching the “best” operation respecting amongst others input- and return-parameters. Therefor we had to search for all combinations of the parameters. But the number of permutations is factorial so you can very quickly have a massive number of possible permutations.

We decided to implement an algorithm to get the next permutation, only when we needed it. So we had no list of all possible permutations in memory but just the current one. But it wasn’t easy to find an algorithm. We searched a lot (I turns out maybe not really efficient) but didn’t find one in the first place. So I spent hours and hours to get a solution myself. Lots of slips of paper where lying everywhere and my young ambition was aroused. I had to find a solution for this problem. This was really so much fun. I really loved, and still do, to sink my teeth into such kind of technical problems.

I can’t remember, if I found the solution myself (maybe not – if so I would remember), but we finally found an algorithm inside the book “Discrete Mathematics and Its Applications” which described how to generate the next permutation in lexicographic order. Again it was a challenge to implement it in a way that everyone could easy understand it. Our documentation had an appendix with an example and a pseudo code implementation because it seems to be not easy to understand after describing it as text.

After more than 10 years I still remember how much fun I had and how challenging it was for me to solve the problem. That’s the reason why I would call this algorithm my favorite one. But I never found a name for it, as the book refers it as “Generating the Next Permutation in Lexicographic Order”.

I’m happy to share it anyway. Just for completeness. And to practice again how to put it in words. 😉

// For this code snippet I choose an implementation taking the current
// permutation as an input parameter.
public int[] getNextPermutation(int[] currentPermutation) throws NoNextPermutationException {
  if (!hasNextPermutation(currentPermutation)) {
    throw new NoNextPermutationException();
  }

  int[] nextPermutation = Arrays.copyOf(currentPermutation, currentPermutation.length);
  int j = getFirstIndexGreaterThanItsSuccessorStartingBackwardsAtTheSecondLastIndex(currentPermutation);
  int k = getFirstGreaterSuccessorStartingBackwardsAtTheLastIndex(currentPermutation, j);
  swap(nextPermutation, j, k);
  sortingAllValuesAscendingStartingAtJ(nextPermutation, j);
  return nextPermutation;
}

Sometimes Checked Exceptions

The more (Java) code I write, the more I dislike checked exceptions. In the last weeks, I wrote a lot methods which needed to catch an exception which can never occur. But checked exceptions were introduced for a reason, so I wondered how I would improve exceptions in my own ideal Java.

Examples of nasty checked exceptions

I wrote a builder for generating soap request with Javas SOAP API (javax.xml.soap). It seems that this API throws an SOAPException in each method. Of course I implemented the builder in a way that the users could not create invalid requests. Every time a SOAPException occurs it is definitely a bug in the builder. Of course I don’t throw the SOAPException in my API, I don’t want my user’s to handle them. But I need to invest some time to make my builder code readable although I had to catch a lot of SOAPExceptions.

In another example I searched a class annotated with a specific annotation and a specific annotation value. In the code I needed to use Class.forName(classForAnnotationValue). This throws an ClassNotFoundException. But I just found the class in the classpath! Afterwards I call .newInstance() also throwing two exceptions (InstantiationException and IllegalAccessException).

A good reason for checked exceptions

Of course there are good reasons why checked exceptions were introduced. Developers are forced to think about typical exceptional behaviour. It is possible that you try to create an instance of an interface or an abstract class, which is not possible. Or that you try to instantiate an object of a class only having a private constructor. Of course in a lot of cases, searching a class for a string will return no result and therefore it is very good to force the programmer to think about how his program should react if there is no class for the given string. And let’s be honest, who would read the JavaDoc and check which runtime exceptions are thrown and would decide which he should handle and which not? (Maybe we would do, if checked exceptions had never exist..) But if you know that the checked exception you are caching will never occur, it is just annoying to have the boiler plate try-catch-block.

So what to do instead

In my own ideal Java I would have a ‘sometimes checked exceptions’ concept. Sometimes checked exceptions are checked exceptions which you can ignore (similar to PMD rules). If you call a method throwing a checked exception, you would get a compiler warning (a warning, not a compiler error) and of course your IDE would mark the method call. There are two possibilities to handle this warning:

  1. Just use the known try-catch-block
  2. Ignore the exception explicitly (e.g. by an annotation, similar to @SuppressWarnings)

I’m not quite sure which is the most clever way to suppress the warning. Maybe an annotation is not the best way, but obviously a way that will work. If the exception is suppressed and occurs against expectation it is thrown as a runtime exception.

Conclusion

This concept is not finished yet. Maybe there is something I miss, but it feels good and it would reduce a lot of boiler plate code. The code would not only be shorter and cleaner, but also more easy to understand.

I am sure that I am not the first person having such an idea. If you know such a concept or somebody having the same idea, please let me know! I am also very curious what you think about it. Are there any situations where this won’t work?

Is Clean Code less Code?

For me clean code is simple, easy to understand code. No over-engineering, as little boilerplate as possible and of course non-cryptic. But does that lead to less code?

I really don’t think so. Most of the times less code is more cryptic and harder to understand (and therefore harder to maintain).

When working with jBehave and testing the meta filtering, I wrote something similar to the following code:

@Override
public Embedder configuredEmbedder() {
Embedder embedder = super.configuredEmbedder();
ignoreStoriesAndScenariosWithMetaInformationParameter(embedder, "ignore");
return embedder;
}

private void ignoreStoriesAndScenariosWithMetaInformationParameter(Embedder embedder, String ignoreParameter) {
embedder.useMetaFilters(Arrays.asList("-" + ignoreParameter));
}

Later, I had a discussion about that code and a colleague said he just deleted the “unnecessary” private method which leads to the following code:

@Override
public Embedder configuredEmbedder() {
Embedder embedder = super.configuredEmbedder();
embedder.useMetaFilters(Arrays.asList("-ignore"));
return embedder;
}

Obviously shorter and less methods. For us, working on that specific class, it may be obvious what’s happening in this method, but just imagine somebody joining the project and never working with jBehave before. For him the longer code provides additional information and even if he does not understand how it works and what “meta filters” are and what the minus means, he’ll at least understand what we are trying to achieve.

When I try to explain my choice, other developers agreed to my point but said that we can also achieve the same result with a comment. And yes, I totally agree that this is a valid option. I think this is just a question of style. I personally don’t like comments (and Uncle Bob’s “Clean Code” is on my site), but maybe in this case the comment really is the better choice because you can explain the connection between this meta filter code and the jBehave story files.

So the resulting code is the following:

@Override
public Embedder configuredEmbedder() {
Embedder embedder = super.configuredEmbedder();
// ignore stories and scenarios with meta information parameter @ignore.
embedder.useMetaFilters(Arrays.asList("-ignore"));
return embedder;
}

Of course you can argue, that this small example isn’t worth talking about. But I think within a project style it is important. And you can find a common style by discussing specific examples. Maybe the other developer will think about if his code could may be unclear to project joiners and I will think about a comment instead of one-line methods.

Conclusion

So clean code isn’t always less code. You need to trade off the benefits of more small methods against lesser lines of code. I’ll discuss the other examples about coding style in some future blog posts.

Which solution is your favorit one and why?

Clean Code and Java 8

I write more and more Java 8 code. But within the last week I wondered if streams and lambdas may lead to code that nobody understands (like generics back in the days). But I’ll give you a few examples and my opinion about it.

Messy Code because of Streams and Lambdas?

I wrote some code like this:

items.stream()
.filter(item -> mapper.hasMappingFor(item))
.map(item -> mapper.map(item))
.findFirst();

Wouldn’t it be more readable for other developers seeing the code for the first time, if I write it like that?

for(Item item : items) {
if (mapper.hasMappingFor(item)) return mapper.map(item);
}

Or does it just feel that way because I see these for-loops every day? In general, I really like fluent APIs and for me the stream-solution seems to be very elegant. Are developers complaining about Java 8 code because they are just old-fashioned? Or is it really harder to understand and maybe also harder to maintain?

Elegance of Streams and Lambdas

Last week I attend at one of our internal coding events and we worked on the yahtzee kata. I had to leave early but because I had a lot of ideas, I worked on that kata on the train, focusing on Java 8 features and check if the code feels clean. The problem I was solving: get all possible yahtzee categories for a given roll. I solved that with an enumeration of categories, each defining an predicated against a given roll is tested.

public Set<Category> possibleCategories() {
return Stream.of(Category.values())
.filter(category -> category.isPossibleCategoryFor(dice))
.collect(Collectors.toSet());
}

And here the definition of the single categories and the check if a dice matches the category:

enum Category {
ONES(hasAtLeastOne(ONE)),
TWOS(hasAtLeastOne(TWO)),
THREES(hasAtLeastOne(THREE)),
FOURS(hasAtLeastOne(FOUR)),
FIVES(hasAtLeastOne(FIVE)),
SIXES(hasAtLeastOne(SIX)),
THREE_OF_A_KIND(atLeastOneDieOccursAtLeastXTimes(3)),
FOUR_OF_A_KIND(atLeastOneDieOccursAtLeastXTimes(4)),
FULL_HOUSE(atLeastOneDieOccursAtLeastXTimes(3)
.and(anotherDieOccursAtLeastYTimes(2))),
SMALL_STRAIGHT(dicesInARow(4)),
LARGE_STRAIGHT(dicesInARow(5)),
YAHTZEE(allDiceAreEqual()),
CHANCE(matchesForEveryDice());

private Predicate<List<Dice>> rule;

private Category(Predicate<List<Dice>> rule) {
this.rule = rule;
}

public boolean isPossibleCategoryFor(Dice dice) {
return rule.test(dice);
}
...
}

Of course you could implement something like this with earlier Java versions as well. Only the method implementations are different.

Here I struggled again, when I wrote the method creating possible straights:

private static Set<Set<Dice>> possibleStraights(int numberOfDice) {
return IntStream.range(0, Dice.values().length - numberOfDice)
.mapToObj(startDice -> IntStream.range(startDice, startDice + numberOfDice)
.mapToObj(die -> Dice.values()[die]).collect(Collectors.toSet()))
.collect(Collectors.toSet());
}

I fear everyone of you thought “WTF?” at the first sight. Although I chose an self-explanatory name (at least I hope so), it’s not clear, what the code does. You may hope, that it really creates a set of possible straights. But after I extract a method it seems more readable again.

private static Set<Set<Dice>> possibleStraights(int numberOfDice) {
return IntStream.range(0, Dice.values().length - numberOfDice)
.mapToObj(startDice -> createStraightStartingAt(startDice, numberOfDice))
.collect(Collectors.toSet());
}

private static Set<Dice> createStraightStartingAt(int startDice, int numberOfDice) {
return IntStream.range(startDice, startDice + numberOfDice)
.mapToObj(die -> Dice.values()[die]).collect(Collectors.toSet());
}

You can find the code on GitHub.

Conclusion

I really think my feeling that the Java 8 code is less readable and maintainable, is just because these Java syntax is quite new. (And maybe I don’t trust other developers, that they are familiar with these new features.)

I am quite sure, that the code is as easy to read and maintain as the “old-fashioned” java style. And in a few month it will be completely common. But we should always think twice, if streams and lambdas are the best possible solution for our problem. Just because we have a hammer, not everything is a nail.

But what do you think? Do you think the code is less cleaner because of the Java 8 syntax?

Make it a good piece of work

During my diploma studies I had an English lecturer who gave use one simple and important advice (besides that “information” never has an “s” 😉 ):

Make it a good peace of work!

Even now, seven years after I finished my diploma study, these words echos in my head.

The good

Of course this is a very valuable advice. You should always give your best. Because:

Anything worth doing, is worth doing right.

So I took it to heart and try to give my best at everything I do. And that is what I expect from a Software Craftsman. Of course it is hard, you have no time, you need to do ten things at once and so on. But I’m sure the quote, echoing in my head over and over again, makes my results better.

The bad

On the other hand, I am never satisfied. Every time I write an important email, I must to force me to click send, because I always thing about “Can I express it that way? Am I impolite?” and so on. Whenever I write a blog post I think “I can not publish it, because it is not good enough!”. There is always so much to improve. And than I start to doubt, if the whole post is even useful. “What is the message?”, “What is the value for the people reading it?”. Even when I code I always find some parts which I want to hide from everybody else, because to me it seems too messy, and so on.

Conclusion

But luckily most of the time I come to the point where I think “It doesn’t matter. You did your best, if somebody will complain, you can do better next time.”, and I click “send” or “publish” or “commit” anyway. And most of the time I even get positive feedback. And if you think about, it is not that extraordinary, because I spend a lot of time doing it and most of the time I thought “You have to make it a good peace of work!” and so I do. Even if it is not perfect, it wouldn’t be that bad.

So I thank my English lecturer for his wise words and hope you will find them helpful, too.

Know your Tools: IntelliJs Shelve Function

I switched to IntelliJ for half a year now. For backend java development I couldn’t figure out lots of advantages compared to eclipse. It always seems to me like it’s six of one and half a dozen of the other. A few things within moving and renaming classes seems to work better, but therefore I always struggle with the module structure. But yesterday I found my killer feature and I’ll never switch back to eclipse until they have this feature too: Shelving changes.

You may know this feature from git, it’s called git stash or mercurial hg shelve. (If you don’t know it check it out! It’s worth it!) You just put your current uncommitted (maybe not even compiling) changes aside to work on a task that cropped up (maybe an important bug fix). I already loved this feature as I learned it in git. In IntelliJ you just click “Shelve Changes..” in the “Changes” view, select all changed files you want to shelve and IntelliJ creates a patch in your “Shelf” (a tab in the “Changes” view).

After you are done with the other tasks, you can unshelve your previous work, again by right clicking on the files in your shelf and selecting “Unshelve changes”. You don’t even need to unshelve all at once. You can also select “Unshelve…” and select only a few files you want to unshelve. If you do so the patch in your shelf stays untouched. If you want to remove the unshelved files from it you need to do it manually. For more details have a look at the IntelliJ documentation.

So if you didn’t know this feature, try it! And if you know an equivalent feature in eclipse, please let me know!

What is Software Craftsmanship?

Definition

A lot of people don’t know what Software Craftsmanship is. Is it just a synonym for Clean Code? No, it isn’t. I really like the definition of Sandro Mancuso, I already shared on Google+ in 2011:

Software craftsmanship is a long journey to mastery. It’s a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development

Important are the words journey, lifestyle, improving, constantly learning, responsibility, professionalism, pragmatism and pride.

A long journey to mastery

Yes it really is a long journey. You need to get a lot of experience, experiment with technologies and architectures. Learn principles, practices, techniques, tools and so on. Some things you will like, some things will turn out to be not practicable and you will stop doing it.

If you really want to be a software craftsman you need to be totally committet. You can’t be a software craftsman for just a few hours. You need to take responsibility. Either you act professional or not. Either you want to improve your craft by constantly learning or you don’t. Only if you are totally in you can make the long journey.

Project Reality

Unfortunately, I met a lot developers who didn’t seem to care. They worked nonprofessional and they didn’t take responsibility. Those developers demotivate their colleagues. Why should I care, if the others mess it up anyway? Working in such teams can be very frustrating.

Why you should care anyway?

I am sure that Software Craftsmanship is worth it – and I’m quite sure you agree, otherwise you wouldn’t read this, right? Even if none of your teammates share your opinions. You can make the project (and your working environment) better! And I’m sure software craftsmen will become a role model for all other developers and can share their knowledge with their teammates, once they see the improvements. And in the worst case (no one cares and maybe they even laugh at you, although you improve something), you can switch your job and use your experience and achievements when you apply to another company.