Tuesday, May 01, 2018

JaCoCo, Gradle, and exclusions

The setup

My team is working on a Java server, as part of a larger project project, using Gradle to build and JaCoCo to measure testing code coverage. The build fails if coverage drops below fixed limits (branch, instruction, and line)—"verification" in JaCoCo-speak.

We follow the strategy of The Ratchet: as dev pairs push commits into the project, code coverage may not drop without group agreement, and if coverage rises, the verification limits rise to match. This ensures we have ever-rising coverage, and avoid new code which lacks adequate testing.

The problem

At a work project, we're struggling to get JaCoCo to ignore some new, configuration-only Java classes. These classes have no "real" implementation code to test, are used to setup communication with an external resource, yet are high line-count (static configuration via code). So they drag down our code coverage limits, and there is no effective way to unit test them sensibly. (They are best tested as system tests within our CI pipeline using live programs and remote resources.)

JaCoCo has what seems at first blush a sensible way to exclude these configuration classes from testing:

jacocoTestVerificationCoverage {
    violationRules {
        rule {
            excludes ['hm.binkley.labs.saml.SomeConfig']
            limit {
                counter = 'LINE'
                minimum = 0.90
            }
        }
    }
}

Unfortunately, this does nothing. There is no warning or error, and coverage continues to include the whole code base.

A solution

After a lot of experimenting and StackOverflow research, this answer from Juan Vimberg worked exactly as we needed. Following his approach:

final def excludedClasses = ['hm.binkley.labs.saml.SomeConfig']

jacocoTestVerificationCoverage {
    violationRules {
        rule {
            limit {
                counter = 'LINE'
                minimum = 0.90
            }
        }
    }

    afterEvaluate {
        classDirectories = files(classDirectories.files.collect {
            fileTree(dir: it, excludes: excludedClasses.collect {
                it.replace('.', '/') + '.class'
            })
        })
    }
}

The list of excluded classes is extracted so the same trick can be used in the generated reports:

jacocoTestReport {
    executionData test, databaseTest
    reports {
        html.enabled = true
        xml.enabled = true
        csv.enabled = false
    }
    afterEvaluate {
        classDirectories = files(classDirectories.files.collect {
            fileTree(dir: it, excludes: excludedClasses.collect {
                it.replace('.', '/') + '.class'
            })
        })
    }
}

Something to consider: using wildcards (hm.binkley.labs.saml.*) may take additional work.

Why?

Why does this work, and the "obvious" way does not?

JaCoCo has more than one notion of scoping. The clearest one is the counters: branches, classes, instructions, lines, and methods.

Not as well documented is the scope of checks: bundles, classes, methods, packages, and source files. These are not mix-and-match. For example, exclusions apply to classes. Lyudmil Latinov has the best hints I've found on how this works.

Saturday, March 31, 2018

Workaround for jenv on Cygwin

I'd like to use jenv on my Cygwin setup at home. Oracle has moved to a 6-month release pace, and so I find myself dealing with multiple Java major verions. However, my tool of choice, jenv, does not play well with Cygwin.

(Note: There are two jenvs out there. I am talking about jenv.be, not jenv.io. Apologies that neither does HTTPS well.)

As a workaround, I wrote a straight-forward shell function to provide the minimum I need: switch between versions in the current shell:

# Until jenv.be supports Cygwin
function set-java {
    local -a java_v
    local jdk v OPTIND
    for jdk in /cygdrive/c/Program\ Files/Java/jdk*
    do
        jdk="${jdk/\/cygdrive\/c\/Program\ Files\/Java\/jdk/}"
        v=${jdk#-}
        v=${v#1.}
        v=${v%%.*}
        java_v[$v]=$jdk
    done

    local verbose=false
    while getopts :hv opt
    do
        case $opt in
        h ) cat <<EOH
            Usage: $FUNCNAME [-hv] VERSION

            Options:
            -h Print help and exit
            -v Verbose output

            Arguments:
            VERSION One of ${!java_v[@]}
            EOH
            return 0 ;;
            v ) verbose=true ;;
            * ) echo "Usage: $FUNCNAME [-hv] VERSION" >&2 ; return 2 ;;
        esac
    done
    shift $((OPTIND - 1))

    case $# in
    1 ) ;;
    * ) echo "Usage: $FUNCNAME [-hv] VERSION" >&2 ; return 2 ;;
    esac

    if ! [[ ${java_v[$1]+foo} ]]
    then
        echo "$FUNCNAME: No such Java version: $1.  Try $FUNCNAME -h" >&2
        return 2
    fi

    export JAVA_HOME='C:\Program Files\Java\jdk'${java_v[$1]}
    for v in ${!java_v[@]}
    do
        case $v in
        $1 ) ;;
        * ) export PATH="${PATH//${java_v[$v]}/${java_v[$1]}}" ;;
        esac
    done

    if $verbose
    then
        echo "$FUNCNAME: Updated JAVA_HOME and PATH for JDK to $v at $JAVA_HOME"
    fi
}

Try the -h flag (help).

Thursday, January 11, 2018

Automated acceptance criteria

Automated Acceptance Criteria

Automated Acceptance Criteria

I have a dream (Story Card)

What is my dream story card? I don't mean: What's the story I'd most like to work on! I mean: What should a virtual story card look like (as opposed to card stock on a wall)? This may be a trivial question. But for me the user experience working with stories is very important: I work with them daily, write them, discuss them, work on them, accept them, etc. I want the feel of the card to be thoughtful, like that fellow to the right.

And more than that. I am lazy, impatient, hubristic. The acceptance criteria, I want them testable, literally testable in that each has a matching test I can execute. Given my laziness, I don't want to switch systems and run tests; I'd like to execute acceptance criteria directly from the story card.

So is there a system like that today? No. There are bits and pieces though.

Not all story card systems are equal

Some story card systems are particularly awkward to read, understand or use. Special demerits for:

  • A hard-coded workflow that the team cannot be change to fit how they work: the team is expected to fit the tool
  • Workflow state transition buttons are nice, but not so nice is unconfigurable labels, especially when the button labels are misleading
  • A hard-coded or limited hierarchy of stories, so if a team uses epics or features or themes or whatever to organize stories, and there are more than one level to this, the team is out of luck
  • Lack of quality RESTful support, in particular, no simple identifier for story cards, so linking directly to cards is opaque, useless or completely absent

A scenario

Post-development testing on this team is a fairly ordinary role. The developers say: a new web page is ready. Testers then validate the same page features each time for each new page, simple things:

  • Can an account with security role X log in?
  • Can X submit the form? (Or not submit if forbidden?)
  • What form defaults appear for role X? Do they reflect role X?

(Yes, I know — what about developer testing? Bear with me.)

On it goes, the same work each time. Redundant, repetitive, error-prone, fiddly. And worst of all—boring, BORING! This is traditional, manual "user testing" at its worst.

What's to be done? Can we fix this?

After all, the testing is valuable: nobody wants broken web pages; everybody wants to log in. But the tester is valuable, too, more valuable even: is this the most valuable testing a human could do? Surely people are more clever, more insightful than this. And what about all the other page features not tested because of time spent on the basics?

Well, people are more clever than this.

Clearing the path

What guides testing? If you're using nearly any form of modern user story writing, this includes something like "Acceptance Criteria". These are the gates that permit story development to be called successful: the testers are gatekeepers in the manual testing world. In the manual world these criteria might be congregated into a single "Requirements Document" or similar (think: big, upfront design).

We can do better! Gatekeeper-style testing assumes a linear path from requirements to implementation to testing, just as waterfall considers these activities as distinct phases. But we know agile approaches do better than waterfall in most cases. Why should we build our teams to mirror waterfall? Of course the answer is to structure teams to look agile, just as the team itself practices agile values.

So how do we make Acceptance Criteria more agile?

Enter the Three Amigos

In current agile practice, a story card is not a ready to play until approved by the The Three Amigos: BA, Dev, QA. Each plays their part, brings their perspective, contributes to meeting team-agreed "Definition of Ready".

A key component of playable cards are the Acceptance Criteria — answering the question, "What does success look like?" when a story is finished.

The perspectives include:

  • BA: Is the story told right? — What is the way to describe the work?
  • Dev: Is the story the right size? — What is the complexity of the work?
  • QA: Is it the right story to tell? — What is the value of the work?

What are Acceptance Criteria?

But where does this simple testing come from? Any software delivery process beyond "winging it" has some requirements

Well-written agile stories have Acceptance Criteria. What are these? An Acceptance Criteria (AC) is a statement in a story that a tester (QA) can use to validate that all or a portion of the story is complete. The formulaic phrasing for ACs I like best is:

GIVEN some state of the world
WHEN some change happens
THEN some observable outcome results

Generally for web applications this means when I do something with a web page in the application, then the page changes in some particular way or submits a request and the response page has some particular quality or property (or the negative case that is does not have that quality or property).

In some cases it is even simpler: just check that a particular web address fully loads, for example, when testing login access to pages.

Martin Fowler's Test Pyramid

So what's the question?

Wherever possible we want to automate. If something can be done for us by a computing machine, we don't want to spend human time on it. Humans can move on to more interesting, valuable work when existing tasks can be automated.

Consider the Test Pyramid (image right): automating lower-value tests focuses people on higher-value ones. You get more human attention and insight on the kinds of tests which best improve the value of software. You win more.

The Story

This is a sample story card with a simplistic implementation of AACs. Live buttons call back into the story system, to be mapped to calls in the testing system. (Another implementation might have the buttons directly call to the testing system, avoiding an extra call into the story system but showing details in the page source about the testing system.)

Title

Narrative

AS A AAC author
I WANT a mock executable story
SO THAT others can see the value

Details

No actual criteria were validated in the execution of these tests. This is only a mock.

Acceptance criteria

Summary: 1 missing, 1 untested, 1 running, 1 passed, 1 failed, 1 errored, 1 disabled
GIVEN magical thinking
WHEN in Missingland
THEN there's no test
No test (yet) - create one!
GIVEN magical thinking
WHEN in Newland
THEN nothing has happened yet
Test never run - be the first!
GIVEN magical thinking
WHEN in Fastland
THEN tests run quickly
15% done (3s)
GIVEN magical thinking
WHEN in Happyland
THEN Unicorns
GIVEN magical thinking
WHEN in Sadland
THEN there be Dragons
Expected: Dragons, got: Puppies
GIVEN magical thinking
WHEN in Crazyland
THEN nothing works right
Test timed out after 90 seconds
GIVEN magical thinking
WHEN in Slowland
THEN tests are disabled
@dev1 @qa2: BLOCKED on widget spanner

Acceptance Criteria states

Every AC potentially has a message from the testing system giving more detail on state. These are noted below.

Missing

This AC has no matching test in the testing system. Use the Create button to create a new test. This does not run the test.

The message is boilerplate to remind users to create tests.

Untested

The AC has a matching test in the testing system, but the test has never been run. Use the Test button to run the test.

Typically there is no message for this state.

Running

The matching test for the AC is running in the testing system. Use the Cancel button to stop the test, or wait for it to complete.

The message, is supported by the testing system, should give a notion of progress. See REST and long-running jobs for how to do this.

Passed

The matching test for the AC passed last time it ran. Use the Test button to run the test again.

Typically there is no message for this state.

Failed

The matching test for the AC failed last time it ran. Use the Test button to run the test again.

The message must give a notion of why the test failed.

Errored

The matching test for the AC errored last time it ran. Use the Test button to run the test again.

The message must give a notion of why the test errored.

Disabled

The matching test for the AC is disabled in the testing system. Update the testing system to reenable.

The message, if supported, should give a reason the test is disabled when available.

Potential problems

Nothing is free. Potential problems with AACs include:

Integrations

There are no existing integrations along these lines. You need to build your own against JIRA, Mingle, FitNesse, Cucumber, etc. Whether the story system drives the interaction, or the test system does, may depend on the exact combination. Best would be if both systems can call the other.

Scaling

As more AACs run, the complete suite takes longer. For example, adding 1 minute of AAC/story, and 5 stories/iteration, in a 12 iteration project takes 60 minutes to run. This is not specific to AACs but a general problem with acceptance tests. It's still much cheaper than the manual steps for each test, but prohibitive for a developer to run the whole suite locally.

Best practice is for the 3 amigos to run only the tests specific to a story before calling that story Ready to Accept.

Update

Until I published this post on Blogger, I really wasn't certain how it would look on that platform. I manually tested the visuals from Chrome with the local post, and it looked good. After seeing it in Blogger, however, the "aside" sections are laid out poorly, overlapping the text. I won't relay the page: mistakes are the best way to improve, and a subtext of this post is Experiment & Learn Rapidly. Public experiements are the most faithful kind: no opportunity to fudge and pretend all was well on the first try.

Tuesday, January 09, 2018

Sproingk lives!

Sproingk lives!

After months of instability, I again have a fully working pipeline and a live web page for Sproingk, my demo project combining the latest public betas in:

If you're interested in any combination of these, please take a gander.

About some of the items

Kotlin (It's not just for Android!) is really where JVM programming is headed. And it's fun.

Springfox gives you a lovely UI for your REST API. (In the demo, try the "greeting controller".)

Boxfuse make minimal shrink-wrapped Linux images of your software, and handles blue/green AWS deployments.

Wednesday, January 03, 2018

A language stack for 2018

(I talk about myself in the post more than usual. I'm expressing opinions more than usual, rather than observations and advice. Caveat lector. Also, this post is link-rich.)

The stack

After reading Eric S. Raymond (ESR)'s posts on the post-"C" world, I realized that I, too, live in that world. What would my ideal language stack look like for 2018?

For context, here are ESR's posts I have in mind:

  1. The long goodbye to C
  2. The big break in computer languages
  3. Language engineering for great justice
  4. C, Python, Go, and the Generalized Greenspun Law

Read them? Good. So this is the language stack I have in mind:

  • Python — By default
  • Kotlin (JVM) — When you need it
  • Go — When you must

Each of these languages hits a sweet spot, and displaces an earlier language which was itself a sweet spot of its time:

  • Bash and PerlPython
  • Java → Python and Kotlin
  • "C" and C++ → Kotlin and Go

An interesting general trend here: Not just replace a language with a more modern equivalent, but also move programming further away from the hardware. As ESR points out, Moore's law and improving language engineering have raised the bar.

(ESR's thinking has evolved over time, a sign of someone who has given deep and sustained thought to the subject.)

About my experience with these languages

Python

I have moderate experience in Python spread out since the mid-90s. At that time, I was undecided between Python, Ruby and Perl. Over my career I worked heavily in Perl (it paid well, then), some in Ruby (mostly at ThoughtWorks), and gravitated strongly to Python, especially after using it at Macquarie commodities trading where it was central to their business.

Kotlin

I've been a Kotlin fan since it was announced. It scratches itches that Java persistently gives, and JetBrains is far more pleasant a "benevolent dictator" than Oracle: JetBrains continually sought input and feedback from the community in designing the language, for example. Java has been my primary language since the late 90s, bread and butter in most projects. If JetBrains keeps Kotlin in its current directions, I expect it to displace Java, and deservedly so.

Go

This is where I am relying on the advice of others more than personal experience. I started programming with "C" and LISP (Emacs), and quickly became an expert (things being relative) in C++. Among other C++ projects, I implemented for INSO (Microsoft Word multilingual spell checker) a then new specification for UNICODE/CJKV support in C++ (wstring and friends). I still love "C" but could do without C++. Go seems to be the right way to head, especially with garbage collection.

With Ken Thompson and Rob Pike behind it, intelligent luminaries like ESR pitching for it, and colleagues at ThoughtWorks excited to start new Go projects, it's high time I make up this gap.

What makes a "modern" language?

I'm looking for specific things in a "modern" language, chiefly:

Good community

Varying and strong:

You can explore the figures at TIOBE and StackOverflow.

Kotlin is the interesting case. Though low in the rankings, because of 100% interoperability running on the JVM, it's easy to call Java from Kotlin and call Kotlin from Java, so the whole Java ecosystem is available to Kotlin natively.

Further, Google fully supports Kotlin on Android as a first-class language, so there is a wealth of interoperability there. (The story for iOS is more nuanced, with Kotlin/Native including that platform as a target but in progress.)

Lastly, Kotlin/Native is bringing similar interoperability between Kotlin and Go.

(This is related to having a rich ecosystem.)

Garbage collection

(A quick primer (2011) on garbage collection.)

Among Go's advantages over "C" and C++ is solid garbage collection out of the box, though Boehm a valiant effort. It's only been since 1959. No modern programmer—short of special cases—should manually manage memory.

Kotlin gets a head start here. It's built on the JVM (when targeting that environment), which has arguably the world's greatest GC (or at least most tested). It definitely gives you a choice of garbage collectors. There's too much to say about GC on the JVM for this post except that it is first-rate.

Python has garbage collection. Though not as strong as the JVM, it continues to improve. It is unusual for GC to become a limiting factor in a Python program; you will know if it does.

(If you are in "C" or C++, and want some of the benefits of GC, do consider the Boehm-Demers -Weiser garbage collector. No, you won't get the fullest benefits of a language built for GC, but you'll get enough to help a lot, even if just for leak detection.)

Static type inference

Kotlin really demonstrates where Java could improve by leaps rather than baby steps: automatic type inference. At least Java is heading in the right direction. Don't tell the computer how to run your program when it can figure this our for itself! Keep your focus on what the program is for.

Interestingly, Kotlin and Go have strong typing out of the box, but not Python. Python has built-in support for typing, and an excellent optional implementation, mypy, however this is type declaration not type inference, so it loses a bit there. And none of the common alternatives (Ruby, Perl, PHP, etc.) have type inference either. I'll need to check again in a few years, and possibly update this choice.

(Paul Chiusana writes on the value of static type checking.)

Rich ecosystem

Of the languages, Kotlin is a standout here. Because it is a JVM language, it is 100% compatible with Java, and can fully use the rich world of Java libraries and frameworks. The main weakness for Kotlin is lack of tooling: because more advanced tools may make assumptions about bytecode, Kotlin's particular choices of emitted bytecode sometimes confuse them.

(JetBrains has surveyed on the state of ecosystems for programming languages, related to having a good community.)

Close behind is Python, "batteries included" and all, and it has a better organized and documented standard library than Java. For some problem domains, Python has a richer ecosystem, for example, SciPy and NumPy is the best math environment available in any language. (Some specialty languages like MATLAB deserve mention—an early employer of mine.) I may need to reconsider my ranking Kotlin over Python here.

Go is, frankly, too new to have developed an equivalent ecosystem, and full-blown package management is still a work in progress.

Concision and convenience

A common thread in recent language development is lower ceremony: fewer punctuation marks and boilerplate; make the machine do more work, you do less. Kotlin provides the most obvious example compared to Java. Go is know for cleanness and brevity. And Python ranks high here as well.

(Donnie Berkholz writes an interesting post on ranking language expressiveness.)

Code samples

The classic "Hello, World!" showing three things:

  • Writing a "main" callable from the command line
  • Using standard output to print to console
  • String formatting to build the output message

This doesn't, of course, give a sense of how these languages in their full spectrum, but does give a first taste.

Java

Java in a file named MyStuff.java:

package my.stuff;

public final class MyStuff {
    public static final String LANGUAGE = "Java";

    public static void main(final String... args) {
        System.out.println(String.format("Hello, World, from %s", language));
    }
}

Kotlin

Kotlin in a file named my-program.kt:

package my.stuff

const val LANGUAGE = "Kotlin"

fun main(args: Array<String>) = println("Hello, World, from $LANGUAGE")

Go

But also compare Go to C++:

package main

import "fmt"

const Language = "Go"

func main() {
    fmt.Println("Hello, World, from", Language)
}

C++

And C++:

#include <iostream>

int
main()
{
  std::cout << "Hello, World!" << std::endl;

  return 0;
}

Python

And for completeness, Python compared to Perl and BASH:

#!/usr/bin/python

language = 'Python'

print('Hello, World, from {}'.format(language))

Perl

Any Perl:

#!/usr/bin/perl

use strict;
use warnings;

my $language = "Perl";

printf("Hello, World, from %s\n", $language);

BASH

Unfairly simple:

#!/bin/bash

language=BASH

echo "Hello World, from $language"

See also

Update

As usual, I never catch as many problems in my writing as I do after reading it posted publically. Many small edits, and an added, explicit mention of wstring.

Footnotes

  1. A surprising take on Python versus JavaScript from Michael Bolin. And I do not feel Kotlin (JS) is ready yet for front-end work.
  2. In contrast to ESR's thoughtful posts, a nice Steve Yegge rant in favor of Kotlin. (I enjoy both their writing styles.)
  3. YMMV — I use Perl as a example, but it could be Ruby or PHP or similar. And some might prefer Node.js to Python (but don't: see footnote 1. The exact choice is a matter of preference: I prefer Python, and some would keep Ruby (for example).
  4. Mike Vanier wrote a similar list for "scalable computer programming languages" in 2001, at least for the technical elements.
  5. Mike Hearn points out potential pitfalls with Go GC.

Thursday, December 28, 2017

Push early, push often, push on green

(This post follows up on Frequent commits, a post about git command line help for TDD. I assume you are already following good TDD practices. Also, please recall git normally requires pulling before pushing if your repo is behind on commits from the remote.)

Prologue

I'm chatting with a colleague who is new in her role as an Agile Coach (she comes from a PM background). We were talking about ways to organize a team's story board (card wall), and turned to Desk Checks and when they fit into a story's life.

An interesting remark by her: a team had agreed to share work (push) only after the desk check was successful; that is, they did not push code until a story was almost done: work lay fallow on individuals' machines for potentially days at a stretch.

I was surprised. Why would they wait days to push—what did they do about merge conflicts, complex refactorings, integration failures in the pipeline, et al?

Lecture

Entropy

To me this was clearly a smell. Martin Fowler specifically addresses this in Everyone Commits To the Mainline Every Day, and I would go further: Push commits at the earliest responsible moment. This is opposite the advice for refactoring, or especially emergent design, where the "Rule of 3" and last responsible moment cautions waiting for more information before committing to a course of action.

And you can see why early pushes differ from the other two: waiting will not get you more information. On the contrary, waiting will only increase the entropy of the code base! Commits lie fallow in the local repo, increasing the size of potential merge conflicts for others.

  Benefit from more information? Principle Entropy from waiting
Early push None available Earliest responsible moment Rises from fallow commits
Refactoring Get more code examples Rule of 3 Falls after refactoring
Architecture decision Learn more about system Last responsible moment Falls if responsible

(The "information" in the case of pushes are the pulled commits themselves.)

I've definitely experienced this firsthand, when I'd eventually discard my local commits after waiting too long, and letting them grow too much in a different direction from how the rest of the team progressed. Waste!

Complexity

Consider this work cycle:

  1. Local commit
  2. Fetch commits from remote
  3. Merge, if needed
  4. Local commit again, if needed
  5. Push commits to remote

I've grouped these to emphasize what is local to you (the first four) and what is global to your team (the last one).

Considered only locally, you minimize entropy with frequent pulls for yourself, and likewise for your teammates individually, so you can catch merge conflicts early and resolve them when they are small. But considered globally, you need frequent pushes so those local pulls have only small changes in them. The longer you wait to push, the more work for those who pull.

Early push
You Rest of team Work for others
You commit    
You push  
  They pull Less complexity of merge (1 commit)
You commit  
You push  
  They pull Less complexity of merge (1 commit)

Each single push can be treated on it's own. There are two opportunities for merge conflict, but each is a small amount of work.

Late push[1]
You Rest of team Work for others
You commit    
  They pull No changes to merge
You commit  
You push  
  They pull Greater complexity of merge (2 commits)

In each scenario, there are two commits for others to contend with. The larger, combined push has a greater opportunity for merge conflict, and a greater chance for a large amount of work, because of the combined interactions of the two commits.

And as teams work in parallel, there are more opportunities for merge conflicts.

Push early, push often, push on green

From the above discussion, the safest course is to push early rather than wait as commits pile up locally. But when to push—what is the "earliest responsible moment"?

If your codebase is well-tested, and answer presents itself: Push when tests are green and changes alter the complexity.

The goal is to avoid complex commit interactions that lead to merge conflicts. Tests are the safety net. Further, if all else fails and a commit is bad, it is easy to throw away the last commit until things are right again: only a small amount of work is lost, not days worth.

Understanding what kind of changes alter complexity takes skill: skills improve with experience and coaching. The cost of early pushes is low, and the occassional penalty of late pushes high, so this would be a good topic for a "team norms" ("dev practices") discussion.

For example, the team might agree that changes to comments are not in themselves worth a push. At the other end, your refactorings which impact more than one source file almost certainly should be pushed early: discover their impact on others before you add more refactorings.

A good work cycle:

  1. Pull
  2. Build, run tests
  3. Edit sources
  4. Build, run tests
  5. Commit
  6. Pull and push

After a preliminary sanity check (#1 and #2), get in the the cycle of #3 to #6.

Epilogue

I checked with other teams: it is minority practice to wait until a successful desk check to push changes. That's a relief. Hopefully this practice can be made more rare.

One rational reason—itself a smell—is when tests take too long to run frequently. When I design a pipeline, I recommend breaking out "unit" tests from "integration" tests for this exact reason: even when integration tests run long, the initial CI stage with just unit tests should be fast enough to give quick feedback on frequent pushes, and encourage Push early, Push often, Push on (local) green.

Further reading

Footnotes

[1] The simple statement, "a greater chance for a large amount of work", has rather complex reasoning behind it, beyond the scope of this post.

For example, any particular commit can be viewed as applying an exponent to the overall complexity of a program. A neutral change (say, correcting a typo in a comment) has the exponent 1: it does not change the overall complexity; a positive change (say, removing code duplication) has an exponent between 0 and 1: it lowers the overall complexity; a negative change (say, adding a new dependency) has an exponent greater than 1: it raises the overall complexity.

Consider then that these complexity changes are not simple numbers, but distributions ("odds"), and change with time ("bitrot"), and involve more than the code (people or requirments changes).

[2] In Seth's post, do not confuse "publish once" with "wait to push": it means "don't publish the same commit twice" (which does sometimes happen accidentally, even for experts, from merging or rebasing).

Update

Sure enough, right after posting I read an interesting discussion on the value of the statistical mean (average) relevant to the discussion on two commits taken separately or together.

Essentially, even when the merge conflict work averages out over time for pushing two commits separately versus pushing them together, the outliers for pushing them together is significantly worse than for pushing them separately because of interactions and complexity.

Wednesday, November 01, 2017

Old scripting

The landscape

I'm helping someone with a scripting problem on an old system and an old shell. How old? Try IBM AIX 6.1, first released in 2007, and ksh93 "e" released in ... 1993. At least the AIX is a version of 6.1 from 2014! (Kudos to IBM for treating long-term support seriously.)

A second point to ponder. The goal is to improve remote scripting—running scripts on a remote machine. In this environment, ssh exists but is not used. The remote execution tool chosen is rexec, considered one of the most dangerous tools possible. But my remit is not to address the insecurity, just to improve the scripting. (They know this is a bad, and are actively working to eventually resolve.)

So, given these constraints, what problem am I solving?

Example problem

This environment makes extensive use of remotely executed scripts to wire together a distributed, locally-hosted system. Current scripts duplicate the same approach, each implemented as a one-off: Copy a script to a remote machine with rcp; use rexec to invoke the script, capturing the output to a file on the remote host; copy the captured file back to the local host; process the output file; sometimes clean up the remote host afterwards.

Some gotchas to watch out for with ksh93e or rexec:

  • Function tracing - Using the standard xtrace setting to trace script execution in ksh93 has problems with tracing functions, and requires using old-style function syntax
  • Variable scope - To keep variables local to a function in ksh93, you must use the new-style function syntax (note the conflict with tracing)
  • Exit broken with trap - When calling exit to quit a remote script, trap does not get a correct $? variable (it is always 0, as exit succeeded in returning a non-0 exit status). Instead one must "set" $? with the code of a failing command, and then leave with a plain call to exit
  • No pipefail - Release "e" of ksh93 just does not know anything about set -o pipefail, and there is no uninstrusive workaround. This now common feature showed up in release "g"
  • No exit code - Would you believe rexec does not itself exit with the exit code of the remote command, never has, and never will? It always exits 0 if the remote command could be started.
  • Buffered stderr - Empirically, rexec (at least the version with this AIX) buffers the stderr stream of remote commands, and only flushes when rexec exits, so the sense of ordering between stdout, stderr and the command-line prompt is even worse than usual (the actual handling is unspecified)

This problem and environment triggers a memory: The last time I worked on AIX was in 1994, and it was almost the same problem! I really thought I had escaped those days.

A solution

So I refactored. I couldn't change the use of rexec—this environment is not ready for SSH key management—, I couldn't replace KSH93 with BASH or replace AIX with Linux, but I could do something about the imperfect duplication and random detritus files.

The solution

Note the need to call a fail function instead of exit directly because of poor interaction with trap.

Assuming some help, such as a global progname variable (which could simply be $0), and avoiding remote temporary files:

_transfer_exit_code() {
    while read line
    do
        case $line in
            ^[0-9] | ^[1-9][0-9] | ^11[0-9] | ^12[0-7] ) return ${line#^} ;;
            * ) printf '%s\n' "$line" ;;
        esac
    done
    return 1  # ksh93e lacks pipefail; we get here when 'rscript' failed
}

rscript() {
    case $# in
        0 | 1 )
            echo "$progname: BUG: Usage: rexec SCRIPT-NAME HOSTNAME [ARGS]..." >&2 ;;
        * ) script_name=$1 ; shift
            hostname=$1 ; shift ;;
    esac
    # Trace callers script if we ourselves are being traced
    case $- in
        *x* ) _set_x='set -x' ;;
    esac

    rexec $hostname /usr/bin/ksh93 -s "$@" <<EOS | _transfer_exit_code
set - "$@"  # Only reasonable way to pass through function arguments

# Work around AIX ksh93 return code of exit ignored by trap
fail() {
    return \$1
}

# Our hook to capture the exit code for rexec who dumbly swallows it
trap 'rc=\$?; echo ^\$rc; exit \$rc' EXIT

PS4='+$script_name:\$(( LINENO - 14 )) (\$SECONDS) '
$_set_x

# The callers script
$(cat)
EOS
}

Example use

#!/usr/bin/ksh93

progname=${0##*/}

PS4='+$progname:$LINENO ($SECONDS) '

usage() {
    echo "Usage: $0 [-d] HOSTNAME"
}

. rexec.ksh

debug=false
while getopts :d opt
do
    case $opt in
        d ) debug=true ;;
        * ) usage >&2 ; exit 2 ;;
    esac
done
shift $(( OPTIND - 1 ))

case $# in
    1 ) hostname=$1 ;;
    * ) usage >&2 ; exit 2 ;;
esac

$debug && set -x

script_name=My-Remote-Script

tmp=${TMPDIR-/tmp}/$progname.$RANDOM
trap 'rm -f $tmp' EXIT

rscript $script_name $hostname Katy <<'EOS' >$tmp
echo $#: $1
fail 3
EOS

case $? in
    3 ) ;;
    * ) echo "$0: Did not pass through exit code" >&2 ; exit 1 ;;
esac

case "$(<$tmp)" in
    '1: Katy' ) ;;
    * ) echo "$0: Did not pass through arguments" >&2 ; exit 1 ;;
esac

Source

The code is in GitHub.

Tuesday, September 19, 2017

Help for JDBC with Java streams

We wanted to use JDBC with Java Streams, but encountered several difficulties. Fortunately we found solutions with rather small bits of code.

Checked exceptions

The main obstacle was the JDBC API throwing SQLException for all API methods used in our code. SQLException is a checked exception, so must be declared in our method signatures, or caught otherwise. However the Streams API only accepts methods which declare to throw no checked exceptions, so something simple like this will not compile:

stream(results).
        map(row -> row.getString("label")).  // checked exception
        forEach(this::processLabel);

The call to ResultSet.getString(String) throws a checked exception. The usual approach is to wrap the call, and handle the exception in the wrapping method:

method String streamGetLabel(final ResultSet results) {
    try {
        return results.getString("label");
    } catch (final SQLException e) {
        throw new UncheckedIOException(e);
    }
}

(Here UncheckedIOException is an unchecked exception wrapper we wrote for SQLException, similar to UncheckedIOException in the JDK for IOException.)

Then the stream becomes:

stream(results).
        map(this::streamGetLabel).
        forEach(this::processLabel);

This is OK, however needing to write a wrapper method for each time we wanted to use JDBC in a stream became tedious.

Solution

First we wrote a SAM (more on SAM interfaces/classes) interface as a lookalike for the JDK Function interface: this is what Stream.map(Function) wants.The lookalike is different in that it throws SQLException:

@FunctionalInterface
public interface SQLFunction<T, R> {
    R apply(final T t) throws SQLException;

    // Other default methods - no more abstract methods
}

Then we used this in a closed Function implementation to wrap and delegate to the lookalike, and throw UncheckedSQLException if the lookalike throws SQLException:

@RequiredArgsConstructor(staticName = "applyUnchecked")
public final class UncheckedSQLFunction<T, R>
        implements Function<T, R> {
    private final SQLFunction<T, R> wrapped;

    @Override
    public R apply(final T t) {
        try {
            return wrapped.apply(t);
        } catch (final SQLException e) {
            throw new UncheckedSQLException(e);
        }
    }
}

(Here we use the excellent Lombok library to generate our constructor, and give us a static convenience method, "applyUnchecked".)

Finally some static importing, and our example streams use becomes:

stream(results).
        map(applyUnchecked(row -> row.getString("label"))).
        forEach(this::processLabel);

Or with more help:

stream(results).
        map(getString("label")).
        forEach(this::processLabel);

We wrote similar lookalikes and wrappers for Predicate and Consumer. It would be easy enough to write them for other Java functional interfaces, such as BiFunction.

Streaming result sets

The next difficulty we tackled was how to loop over ResultSet, and use them with Streams.

(A side note: ResultSet is not a set but a list: rows are ordered, and they can duplicate each other in their column data. However, they were named after the SQL concept of sets, not the Java one.)

Fundamentally, a ResultSet is not an iterator, but is close:

Iterator ResultSet Returns  
hasNext() next() boolean  
next() this ResultSet (Yes, the ResultSet itself is the equivalent)

Solution

To provide a ResultSet as an iterator:

final List<String> values = new ArrayList<>();
        for (final ResultSet row : iterable(results)) {
            values.add(row.getString("value"));
        }

Moreso, to provide one as a stream:

final List<String> values = stream(results).
                map(getString("value")).
                collect(toList());

Any SQL failures are thrown as unchecked exceptions. The stream has the characteristics: immutable, nonnull, and ordered.

Transactions

We found JDBC transactions to be tricky. Fundamentally they are tied to a connection; there is no proper nesting. (To simulate nesting, use separate connections. Even then, there is no guarantee of ordering from the database engine.) And they have a baroque API, relying on diddling of the "auto-commit" setting with care needed to restore its setting after the transaction concludes. Several bugs ensued before we switched to using small helper interfaces and methods.

Further, some programming languages do not distinguish void from other return types (e.g., Unit type): Java is not one of them. Likewise for user vs primitive types (Boolean vs boolean). Hence, there are separate transaction blocks for consumers, functions, and predicates.

Solution

One example explains them all. Consider functions and the SQLFunction lookalike interface:

@RequiredArgsConstructor(staticName = "applyTransacted")
public final class TransactedFunction<T, R>
        implements SQLFunction<T, R> {
    private final Connection connection;
    private final SQLFunction<T, R> wrapped;

    @Override
    public R apply(final T in)
            throws SQLException {
        connection.setAutoCommit(false);
        try {
            final R out = wrapped.apply(in);
            connection.commit();
            return out;
        } catch (final SQLException e) {
            connection.rollback();
            throw e;
        } finally {
            connection.setAutoCommit(true);
        }
    }
}

(The pattern is the same for other collection operations.)

With a helper and static importing:

final Integer value = applyTransacted(connection, in -> 0).apply("string");

Or when the transaction fails:

applyTransacted(connection, in -> {
        throw new SQLException("Something went wrong");
    }).apply("string");

Some convenience

Many places in these examples are improved with helper functions, or for transactions, with currying (similar to the builder pattern). Hence, the wide use of Lombok static constructors. Transactions are another example as they need a Connection for begin/commit/rollback.

Solution

A simple helper curries connection for transactions:

@RequiredArgsConstructor(staticName = "with")
public final class WithConnection {
    private final Connection connection;

    public <T> Predicate<T> testTransacted(final SQLPredicate<T> wrapped) {
        return UncheckedSQLPredicate.testUnchecked(
                TransactedPredicate.<T>testTransacted(connection, wrapped));
    }

    public <T, R> Function<T, R> applyTransacted(
            final SQLFunction<T, R> wrapped) {
        return UncheckedSQLFunction.applyUnchecked(
                TransactedFunction.<T, R>applyTransacted(connection,
                        wrapped));
    }

    public <T> Consumer<T> acceptTransacted(final SQLConsumer<T> wrapped) {
        return UncheckedSQLConsumer.acceptUnchecked(
                TransactedConsumer.<T>acceptTransacted(connection, wrapped));
    }
}

Example use:

final Optional<Integer> value = Stream.of(0).
        filter(with(connection).testTransacted(in -> true)).
        findFirst();

(Yes, we might also describe the code as partial application. The "object-oriented" implementation confuses matters with the hidden this reference.)

Conclusion

There is nothing we did that was difficult or complex: simple one-liner interfaces, simple wrapper implementations of Java funcional interfaces, some rote JDBC best practices. The main difficulty was conceptual: seeing the duplication of many, small wrapper methods, and pulling out their commonality. This is a good pattern to keep in mind throughout your code.

UPDATE:

And the source: Java helpers for JDBC and Streams.

Friday, August 11, 2017

How to write clean Java

I cannot tell you how to write good Java, but I can help you write clean Java. As usual, automation is key. It's all about the tooling.

The best thing about good tooling is that they work together: each covers a different area, does not impeded another tool, and fixing one complaint a tool reveals often fixes complaints from other tools.

This advice applies to any programming language, not just Java. Java, having the most mature ecosystem, is instructive.

The tools

Use good source control
Git is your best choice. Use either trunk-based development with feature toggles (feature flags), or gitflow patterns, depending on your needs and organization. Require full builds and testing before pushing to a shared repository; hooks can help together with your build tool.
Use good build automation
Maven or Gradle are your best choices; either is excellent. Teach the tool to be strict and fail builds if anything is not just right. Treat your build configuration as code, part of the same repository and held to the same hygiene standards. Keep your local build fast.
Use a good editor
IntelliJ is the best choice. Use inspections, intentions, and reformatting obsessively: code diffs should only show real changes, not whitespace or formatting. Use plugins to ease integrating IntelliJ with other tooling.
Keep your style consistent
Checkstyle is your best choice; other choices are needed for non-Java JVM languages. Fail your build if checkstyle complains. Keep your IntelliJ formatting and Checkstyle rules consistent. Generally, choose either Sun or Google coding standards, and be leary of deviating from them; if unsure, pick Sun.
Fully test your code
JaCoCo works well. Fail your build if coverage drops below a threshhold; rachet up the threshhold as coverage improves. Start low and aim for a 95%+ threshhold. Follow the Test Pyramid; save your end-to-end tests for CI, or run locally only once before pushing commits.
Be zealous in looking for problems
FindBugs, PMD, or Error Prone are your best choices (pick one); other choices may work better for non-Java JVM languages. Fail your build if your choice complains. Be judicious in disabling complaints (for example, FindBugs "experimental" checks, should likely be disabled).
Use code generation
Lombok is your first choice. Add others as needed (or build domain-specific code generators). Generated code does not need test coverage, style checks, or bug detection if the generator is clean and well-tested: trust it.

Update

Dan Wallach reminded me of Error Prone, added above.

Cygwin terminal in IntelliJ

IntelliJ sports an excellent terminal emulator (the "Terminal" tab at bottom of the editor). By default it brings up a terminal native to your Operating System: CMD.EXE on Windows, $SHELL on Linux and Mac.

However I prefer Cygwin when I work on Windows. WSL is incredible, but there are still interoperability issues between its filesystem and Windows-native programs, and IntelliJ (which relies on java.exe, a Windows-native program) is still working on it.

So, how to open a Cygwin terminal in IntelliJ? Setting the program to start in Settings|Tools|Terminal|Shell path, the most obvious thing to do, does not quite work:

C:\cygwin64\bin\bash.exe

This is a non-interactive shell, and does not source your profile. The next try is:

C:\cygwin64\bin\bash.exe --login -i

This produces an error from IntelliJ that it cannot start the program correctly. A little checking says the leading command needs to be quoted, else IntelliJ treats the entire line as the name of the command, not as a command followed by flags. OK:

"C:\cygwin64\bin\bash.exe" --login -i

Hey, I have a shell! Unfortunately, it starts in my home directory, not in my project root. Starting in the project root is one of the nice features of the terminal in IntelliJ. Finally, two changes. First the IntelliJ setting:

"C:\cygwin64\bin\bash" -c "exec /usr/bin/env INTELLIJ=true $SHELL --login -i"

And an addition to my ~/.bashrc:

${INTELLIJ-false} && cd ${OLDPWD-.}

Ipso presto!

Monday, June 12, 2017

Example JVM agent in Kotlin

Oleg Shelajev wrote an excellent tutorial post on writing JVM agents. These are bits of code which run before your main() method. Why do this? It permits some interesting tricks, chiefly modifying classes as they are loaded, but also estimating the actual memory used by Java objects.

I gave this a try myself, but rather than writing my agent in Java, I wrote it in Kotlin. It was straight-forward, with only one gotcha.

AgentX.kt

@file:JvmName("AgentX")

package hm.binkley.labs.skratch.jvmagent

import java.lang.instrument.Instrumentation

fun premain(arguments: String?, instrumentation: Instrumentation) {
    println("Hello from AgentX 'premain'!")
}

fun main(args: Array<String>) {
    println("Hello from AgentX 'main'!")
}

OK, the non-gotcha. You can declare functions at the package level. This acts just like static methods in Java, with simpler syntax (no potentially artificial wrapper class to hold the static method). The two obvious examples in the above code are main() and premain().

But when calling Kotlin from Java, you use a wrapping class name in the the fully-qualified method name. My Kotlin file is named "AgentX.kt", so the default class name for Java is "AgentXKt". I'm lazy, wanted to save some typing, so I used a Kotlin package-level annotation to name the wrapping class just "AgentX".

Output

The JVM requires an absolute path to any agent jar, and I'm running Cygwin, so a little help to get a full path. Similarly, I used the Maven shade plugin to build a single uber-jar holding my own classes, and those of my dependencies (the Kotlin standard library).

$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar) -jar target/skratch-0-SNAPSHOT.jar
Hello from AgentX 'premain'!
Hello from AgentX 'main'!

Project is here: https://github.com/binkley/skratch.

Gotcha

Enough preamble, now the gotcha. Unlike Java, Kotlin helps you protect yourself from nulls without boilerplate code. So in premain() for the "arguments" parameter, you need to use String? rather than String as the parameter type as the JVM may pass you a null. The first time I tried the code, I didn't realize this and it blew up:

$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar) -cp target/skratch-0-SNAPSHOT.jar hm.binkley.labs.skratch.jvmagent.AgentX
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.IllegalArgumentException: Parameter specified as non-null is null: method hm.binkley.labs.skratch.jvmagent.AgentX.premain, parameter arguments
        at hm.binkley.labs.skratch.jvmagent.AgentX.premain(AgentX.kt)
        ... 6 more
FATAL ERROR in native method: processing of -javaagent failed

Interesting! Kotlin found the issue at runtime. It can't find it at compile time as the JVM API for "premain" is pure convention without an interface or class to inspect.

Let's try running the agent a different way. The command-line lets us pass options, and these become the "arguments" parameter:

$ java -javaagent:$(cygpath -m $PWD/target/skratch-0-SNAPSHOT.jar)= -cp target/skratch-0-SNAPSHOT.jar hm.binkley.labs.skratch.jvmagent.AgentX
Hello from AgentX 'premain'!
Hello from AgentX 'main'!

Sneaky. The mere presence of the "=" on the command line turns the "arguments" parameter from null to an empty string.