What to do in case of org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes

I’m currently gathering my first experiences with Apache Spark and in particular Spark SQL.

While I was playing a bit with Spark SQL Joins I suddenly faced an exception like Exception in thread "main" org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: foo.
Followed by the parsed SQL statement etc …

Well, in MySQL the error message would have been
"Unknown column 'foo' in field list"
Aka: You are accessing a column/field foo where this field does not exist.
I was already a bit too close to the problem in order to see it at once – and I only found descriptions dealing with nested structures etc (which wasn’t the case in my situation). So it took me a couple of minutes to realize what Spark want to tell me.

Maybe this helps someone else, too.

How to ignore Maven build erros due to JavaDoc with Java 8

Java 8 is a bit more strict in JavaDoc parsing. This can lead to build failures in Maven when building the repo with warnings like:

Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.7:jar (attach-javadocs) on project [projectname]: MavenReportException: Error while creating archive:
Exit code: 1 - [path-to-file]:[linenumber]: warning: no description for @param

Sure, the good solution would be to fix the JavaDocs. But in cases where you just clone a foreign repo, you probably just want to get it run and not start fixing it.

To ignore the erros, just turn off doclint by adding the following <configuration> tag to your pom.xml:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-javadoc-plugin</artifactId>
    <version>2.10.2</version>
    <executions>
        <execution>
            <id>attach-javadocs</id>
            <goals>
                <goal>jar</goal>
            </goals>
            <configuration> <!-- add this to disable checking -->
                <additionalparam>-Xdoclint:none</additionalparam>
            </configuration>
        </execution>
    </executions>
</plugin>

Some more solutions can be found in this StackOverflow thread.

How to get List of Objects from deeper level in Json via GSON

Sometimes you get a quite nested Json response but the only thing you need is a list of classes in a certain branch of the Json document (like a response of Yahoo’s YQL query).

Assume just the following json document:

{
"fieldA": {
    "fieldB": {
        "fields": [
            { "foo": "test1", "bar": "test2"},
            { "foo": "test11", "bar": "test22"}
         ]
      }
   }
}

And the only thing you need is the fields array.
A Java8 way to get the fields as a list would be:

List<FooBar> quotes2 = Stream.of(gson.fromJson(json, JsonObject.class)
	.getAsJsonObject("foo")
	.getAsJsonObject("bar")
	.getAsJsonArray("foobar"))
	.flatMap(e -> Stream.of(gson.fromJson(e, FooBar[].class)))
	.collect(Collectors.toList());

But that’s quite some code. Okay if you only need it once, but as soon as you need this several times it clearly violates the DRY principle. Gson (which I am using a lot) doesn’t seem to provide a simple way for doing this. Except creating the whole hierarchy as Java Classes, which might just be overkill.

Solving the problem in a more generic way is the way to go – but it als requires creating generic arrays:

class Gsons{
    public static <T> List<T> asList(String json, String path, Class<T> clazz) {
        Gson gson = new Gson();
        String[] paths = path.split("\\.");
        JsonObject o = gson.fromJson(json, JsonObject.class);
        for (int i = 0; i < paths.length - 1; i++) {
            o = o.getAsJsonObject(paths[i]);
        }
        JsonArray jsonArray = o.getAsJsonArray(paths[paths.length - 1]);
        Class<T[]> clazzArray = (Class<T[]>) ((T[]) Array.newInstance(clazz, 0)).getClass();
        T[] objectArray = gson.fromJson(jsonArray, clazzArray);
        return Arrays.asList(objectArray);
    }
}

The only things to do are creating a class for the entities and calling the method:

List<FooBar> fooBars = Gsons.asList(json, "fieldA.fieldB.fields", FooBar.class);

How to (re)schedule an alarm after an App upgrade in Android

In one of my Apps I am using alarms to schedule notifications.
Of course I also want to (re)schedule the alarm when the device is rebooted. Easy: Just set a BOOT_COMPLETED action in the intent-filter of the according schedule reciever:

<receiver android:name=".AlarmScheduleReceiver" android:enabled="true">
<intent-filter>
<action android:name="android.intent.action.BOOT_COMPLETED" />
<category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</receiver>

The problem now just is that when the app is upgraded, your alarm will not be rescheduled! Not too much of a problem – if you know it! Just add another action into the intent-filter:

<action android:name="android.intent.action.PACKAGE_REPLACED" />

I was really lucky that a friend pointed that out when I added that feature to my app! Figuring this out just by getting user complaints that “the alarm sometimes doesn’t work” would not have been very funny!

I would have been pretty glad if the API docs would mention something like “hey, when you listen for BOOT_COMPLETE, you might consider listening for PACKAGE_REPLACED, too”. Well, that’s life.

How to do automatic tagging of articles using Feedly

In this post I will describe a first proof of concept approach about how to implement a supervised learning system to automatically tag RSS posts in Feedly.

Motivation

Everyone using an RSS reader to read daily news surely knows the situation that certain topics keep (re)occuring in the news. Yet most people have topics that they are simply not interested in. Just think about certain sports, political topics or world events. But of course they keep showing up in the daily news stream.

Therefore a system is needed that automatically assigns predefined tags to the corresponding news entities and (maybe) also marks them as read.

A critical point is that the system must integrate into an RSS reader application. A system not being able to attach to an existing system would not be applicable as one still wants to use a mobile / desktop app to read the news and also to (un)tag articles. Implementing the complete value chain comprising fetching RSS, parsing, classifying, providing an aggregated stream AND an application for reading the news is definitely out of scope for a proof of concept.

I wanted to write such a classifier for quite some time but didn’t find a system that provided a convenient API to plug in  a tool for reading, classifying and pushing back the results. Unless I discovered the Feedly API. Unfortunately the Feedly API is not (yet) fully open, so that one has to obtain a time limited API token by hand. Yet, for a proof-of-concept, this is totally acceptable.

The Learning System

So much for the introduction. Let us go in medias res:

The first thought was to start with some clustering using Elasticsearch (for similarity search). But let’s recall the base facts and requirements:

  • Only a hand full of tags are needed
  • start with the simplest approach first
  • it should be able to run either on OpenShift or on my Raspberry Pi

So the choice was to start with a simple Naive Bayes Classifier. Instead of doing an in depth explanation of the Bayes classifier (I recommend Paul Graham’s A Plan for Spam and the page about combined probability), just recall: a Bayes Classifier is just a 0-1 classifier. So a single classifier is required for each tag. This makes it of course unusable for a very large amount of tags! But the big advantage is that the Bayes classifier is just dead easy. Just count how often a word occurs in the desired in class A (the Tag) and class B.

How to train / apply the classifier(s)

The classifier should be trained perdiodically and the user must have the opportunity to correct classification errors. Before dealing with synchronizing & updating entries, the classification workflow for each tag is:

  1. get all entities for the tag and use them as positive samples
  2. get all read and untagged entities and use them as negative samples
  3. get all unread and untagged entries and compute P(tag)
  4. if P(tag) > 0.95, mark the entity with the tag and probably also mark it as read

As input, the all kinds of properties are used that could distinguish between tags. Including the source URL, site keywords, categories etc. Then the content is tokenized / split by all non word characters. Graham writes about some optimizations for spam detection – yet results were pretty convincing without further optimization.

in order to have some positive samples, this of course requires the presence of some entities being tagged already. In this case I started tagging already quite some time ago as I already assumed that I needed some ground truth.

Raspberry PI: Boon and Bane

Raspberry PIs are great as little home servers. The drawback is that the RaspPi has just a single core, 700 MHz ARM CPU and 512 Mb ram which is shared between GPU and system. So, it is a bit slow and is a bit low on resources. Especially if the RasPi is also used for other purposes at the same time that also consume some RAM. In case of very large RSS streams, this could indeed raise a  problem: Running low on CPU is unconvenient (=slow), but running low on RAM is deadly (OOME). Therefore it might be required to replace the HashMap in the Bayes class with a DB layer like MapDB.

Status Quo

The quick test with the Bayes classifier showed already some really fine results! On the RasPi, each Tag is classified within 200 – 230s (14 – 18s on my notebook). The mission “Reduce the amount of information that I am not interested in” can thus be regarded as “successfully tested“!

Also there have hardly been any misclassifications. And the ones I experienced were quite understandable. In contrast to scientific publications I didn’t do extensive accuracy tests – the first attempts were so promising that I simply saved the time and thought about what to try out next that could make my life easier.

If this approach should be followed any further there are of course (as always) some open issues: Besides code cleaning, one could try to filter by TF-IDF, filter certain tokens, adjusting thresholds, etc. But I doupt that the results would get much better.

And of course, the complete code is available at GitHub. Feel free to fork it and play around with it! Beware: The code can change dramatically from one commit to another. For example if I just want to test a new idea.

Java 8 Streams: Collecting items into a Map of (Key, Item)

Once in a while I come across the task where I have a list of Items that I want to filter and afterwards store in a map. Usually the key is a property of the Item: anItem.name -> anItem

In the usual Java way this looked like the following:

Map<String, Stuff> map = new HashMap<>();
for (Stuff s : list) {
    if (!s.name.equals("a")){
        map.put(s.name, s);
    }
}

Nothing really special, but it somehow doesn’t look too nice. Yesterday I thought: in Scala I would emit tuples and call .toMap. Isn’t that also possible with Java 8 Streams? And indeed it is:

Map<String, Item> map = l.stream()
    .filter(s -> !s.name.equals("a"))
    .collect(toMap(s -> s.name, s -> s)); // toMap is a static import of Collectors.toMap(...)

This looks compact and readable!

If you don’t like s -> s, just use identity() function of the Function class. Actually I do not like static imports very much as as they make the code less readable, but in this case I would decide for static imports.

Map<String, Item> map = l.stream()
    .filter(s -> !s.name.equals("a"))
    .collect(toMap(s -> s.name, identity())); // toMap and identity are imported statically

Windows Tomcat start failed command 127.0.0.1 could not be found

I just installed Tomcat 7 on my Windows machine and tried to fire it up through Netbeans. But instead of a running server, I just got an error message that command 127.0.0.1 could not be found (Localized error message: Der Befehl “127.0.0.1” ist entweder falsch geschrieben oder konnte nicht gefunden werden.).

I remember that I read about it in a Tomcat bugtracker (but can’t find it any more). Well the solution is pretty simple:
Just open [tomcat home]\bin\catalina.bat and remove the “-characters from lines 196 and 201 (in the code snippet below it’s line 1 and 6):

set JAVA_OPTS=%JAVA_OPTS% %LOGGING_CONFIG%

if not "%LOGGING_MANAGER%" == "" goto noJuliManager
set LOGGING_MANAGER=-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
:noJuliManager
set JAVA_OPTS=%JAVA_OPTS% %LOGGING_MANAGER%

Enable MySQL Streaming in Cascading / Scalding

Last week I ran into a an ugly problem of Scalding:
I needed to read a really large table from MySQL to process it in a certain job. In generall this is trivial: just use a JDBC Source, select your columns and that’s it.

Usually we do this by using 1-3 parallel connections to the SQL-server. This time I started running out of memory because scalding didn’t (more precicely: couldn’t) swap/spill to disk. The problem here is the default behaviour of the mysql-connector. The api docs says:

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and can not allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.

So, what does this mean: If you query a 10 GB table, you get all the data and the connector tries to buffer it in memory – which is a bad idea if you just want to process tuple by tuple. You can then split this large query into 10 smaller ones: SELECT ... FROM ... LIMIT 0, x, SELECT ... FROM ... LIMIT x+1, y, … etc. This works – but partitioning a large result this way is not very efficient because starting from the second query, MySQL has to iterate over x rows until it can start gathering and returning results. So you partition the big query into 10 smaller results but you put quite a lot of load to the server. And over all you still have to keep a lot of results in RAM.

Continue reading Enable MySQL Streaming in Cascading / Scalding

Compiling Cascading: FAILURE: Build failed with an exception.

Today I ran into a really stupid error message when I tried to recompile cascading-jdbc:

Evaluating root project ‘cascading-jdbc’ using build file ‘/home/…/cascading-jdbc/build.gradle’.

FAILURE: Build failed with an exception.

* Where:
Build file ‘/home/…/cascading-jdbc/build.gradle’ line: 68

* What went wrong:
A problem occurred evaluating root project ‘cascading-jdbc’.
> Could not find method create() for arguments [fatJarPrepareFiles, class eu.appsatori.gradle.fatjar.tasks.PrepareFiles] on task set.

* Try:
Run with –stacktrace option to get the stack trace. Run with –debug option to get more log output.

BUILD FAILED

Total time: 5.355 secs

Solution

Check your gradle version … I ran a brand new Ubuntu with the shipped gradle version 1.4. Well the cascading readme states that gradle 1.8 is required … and it really is.

RaspberryPi Weatherstation – The TV Station (Part III)

The third and last post of this series describes the RasperryPi that is connected to the TV and displays the sensor data in a visually appealing way.


Part 1Part 2Part 3


In the past blog post, the first Raspberry Pi was connected to several sensors. Tiny Python scripts poll the data from the sensors regularly and save them to simple text files that can be copied via SSH/SCP. Current data can be obtained from the sensors by directly connecting to the Brick Daemon which runs on this RasPi.

The main focus of this post is the visualization of the data via a JavaFX application and how to control the FX application by using the regular remote control of the TV. But – before we’re divig into the details, I want to teas you with a screen shot of the final result 😉

Wetterstation blue edition

Nice isn’t it? So let’s get started

Things to discuss

The most important question first: What do I want to see and do?
I had a pretty clear intention already: I want to see the current values of all sensors in a small overview and I want to be able to toggle through time series of the past days.

Toggling should be done by the TV’s remote control as I would have the remote at hand already when I switch to the Weather data display. I’ve seen several guides where an IR-receiver and an additional remote were used to control the RasPi. Yet I simply do not like the idea of having another remote control on the table – right next to the other ones. Also controlling the app via mobile phone wasn’t what I wanted as we usually still simply use the TV’s own remote to control the TV. – Also I know from Raspbmc that it is possible to accomplish this by using libCEC – somehow.

Next question to clarify: How should the data be visualized on the TV. This was easy: JavaFX.
Colleagues asked me why I chose FX instead of HTML5. Well mainly there were three reasons:

  • I simply wanted to do something in JavaFX
  • I’m not a fan of fizzling around with JavaScript and CSS. And the standard way to displaying webapps on the RasPi seems to be using Midori. A browser I never worked with before – and even thinking about possible CSS/JS incompatibilities totally turned me off.
  • I wanted to control the UI with the TV’s remote control. That has to be done somehow using the CEC-commands that are sent through HDMI. I’ve never done that before and anticipated my chances to accomplish this in pure Java ways higher than somehow creating input events that I had to redirect to the browser.

Checking the Hardware

Prior to coding, I did some research if and how the CEC commands are sent through HDMI. And – unfortunately – besides all software problems, a lot of people reported problems to even get the signal to the RasPi! Possible problems mentioned throughout several posts included the TV software having disabled the CEC functionality or HDMI cables that were blocking / not forwarding the CEC signals.

What I found the easiest way to test the CEC compatibility of the setup was Raspbmc. Raspbmc is a mediacenter for Raspberry Pis, which is easy to install and supports CEC. So: download the image, flash it to an SD card, connect the Pi to your TV with the HDMI cable that you want to use later as well and start the Pi. If you see your TV’s remote working: Great! Otherwise: you have my sympathy. Good luck in finding out what is wrong with your technical setup.

Install libCEC

So the CEC signals are technically consumable by the Pi. Let’s istall libCEC.
DO NOT install the packages from the pulse-eight website. And also DO NOT simply recompile and install libCEC following the guide on the GitHub page. It’s important to compile the lib for the Pi! Just follow one of the step by step guides to compile and install:

$ cd /tmp
$ sudo apt-get install build-essential autoconf liblockdev1-dev libudev-dev git  libtool pkg-config
$ git clone git://github.com/Pulse-Eight/libcec.git
$ cd libcec
$ sudo ./bootstrap
$ sudo ./configure --with-rpi-include-path=/opt/vc/include --with-rpi-lib-path=/opt/vc/lib --enable-rpi
$ sudo make
$ sudo make install
$ sudo ldconfig
$ cec-client -l

LibCEC is now installed including cec-client (seen in the last call). Play around a bit with cec-client. Up to now everything was fine on our Samsung TV. But beware, I’ve seen posts where people with TVs from other vendors faced serious problems … If you want to play around with cec-client, CEC-O-MATIC might also be a reference you should have a look at.

Install Java 8

At the time of writing, Java 8 is already released. At the time I was writing the code, it was still an EA release (Early Access). I read several posts that recommended installing the ARM version of Java 8. So I simply followed the OpenJDK guide to install OpenJDK 8:

  1. download Java 8 ARM from Oracle
  2. unpack the file: sudo tar zxvf jdk-8-linux-arm-vfp-hflt.gz -C /opt
  3. and check if Java 8 got installed: /opt/jdk1.8.0/bin/java -version
  4. Set default java and javac to JDK 8:
    $ sudo update-alternatives --install /usr/bin/javac javac /opt/jdk1.8.0/bin/javac 1
    $ sudo update-alternatives --install /usr/bin/java java /opt/jdk1.8.0/bin/java 1
    
    $ sudo update-alternatives --config javac
    $ sudo update-alternatives --config java
    
    $ java -version
    $ javac -version

    java and javac should link to 1.8.0 now.

Next adjust the memory split option as also mentioned on the page to 256mb (gpu_mem=256 in /boot/config.txt). It’s useful to read the page. The page also gives a note which is pretty important to avoid frustration:

Note that the default configuration of JavaFX on the Raspberry Pi does not use X11. Instead JavaFX works directly with the display framebuffer and input devices. So you should not have the X11 desktop running when starting JavaFX.

JDK 8 EA builds for the Raspberry Pi include full support for hardware accelerated graphics, with everything from the base, graphics, controls and FXML modules. Media and Web modules are not included.

So configure the Pi to boot just to the shell (via sudo raspi-config) and do not try to see the JavaFX output via VNC!

Building the GUI

Designing the UI (or: how I started to love JavaFX)

Before I even started to code, I wanted to set up the layout. Coming from a Java-Swing background I was really excited to try this JavaFX thingy that was said to be so much cooler than Swing.

After setting up the project in NetBeans and installing JavaFX Scene Builder I was very positively surprised! By the way, I was only working with Scene Builder 1.1, the current Version 2 is said to be even better. After initializing the project, my focus quickly went into the src/main/resources folder. At this location you can find fxml/FXMLDocument.fxml and styles/base.css (later renamed to metro.fxml/.css). Designing the Gui was about 90% done just in Scene Builder (for the .fxml) and Netbeans (for the.css). The really really nice thing is that it is a true WYSIWYG editor. Also whenever the css file is being edited, the effect is immediately visible in Scene Builder. Simply great compared to what I was used to from my Swing experience.

After the main layouting was done, I started to write some Java code in FXMLDocumentController.java. Accessing UI components is simply done by annotating the according fields. The fields themselves are injected automatically. A nice decoupling of view and controller. Also I was able to compare two very different layouts by just switching one single line (the one referencing the fxml file) without further refactoring.

Time for some user tests! Honestly, I was a bit afraid what others would say. But I wanted to make a NICE UI, so I asked my wife and friends on Facebook and Google+ for feedback. And this decision turned out to be damn right! I got very valuable feedback about colors and layout. I definately recommend asking users – and value their feedback!

Afterwards I implemented the logic for reading the CSV files including filling the model classes. These model classes were then rendered into the graphs. So I also had some test data and continued styling the graph lines via CSS. At this stage I remembered the talks of Gerrit Grunwald. Gerrit is pretty experienced in JavaFX on the Pi and mentioned in some talks that animations on the Pi can be really really slow in the beginning until everything gets hot spot compiled. – I can definitely confirm this! The trick was to simply set animated="false" on the line chart that would only be animated at the beginning (which I didn’t really need or want anyways).

Show the UI on the TV

Time for a first test on the TV! At this point, remember the last part from “Install Java8” above: We do not need an X-server as JavaFX on the Pi paints directly to the framebuffer! So just compile and build the Jar, copy it over to the Pi and start it with java -jar ./ Weatherstation-1.0-SNAPSHOT.jar.

I made the experience that my regular monitor is not the same as the display of the TV. So I spent some time with adjusting css font sizes until I was content with the way everything was displayed on the TV. It feels a bit like webdesign and testing with different browsers – just a bit less painful.

Connecting the sensors

With the gui being done so far, the next step had to be done. When the program is started, archived data is read from files. Current data should be fetched directly from the remote sensors. This was actually plain easy. With the API provided by tinkerforge, everything I had to do was starting a thread off the Event dispatching thread that polled new sensor values periodically and pushed the data to the models. the models then simply updated their UI elements.

Remote control Java

The key strokes should be used to toggle through the values of humidity, temperature, ambient light and air pressure.

Everything that was left now was the remote control part. Unfortunately, libcec is a C-binary only. So I had the choices to try some JNI and talk to the lib directly or to start a Java Thread wrapping a process that just called the cec-client program:
/usr/local/bin/cec-client -d 8 -t prta -o Wetter. The d parameter defines the verbosity of the output, t defines the type of device the cec-client simulates (recorder, …), and o finally defines the string that is shown in the On Screen Display of the TV when brwosing through the HDMI input devices. When a cec-client listens with this command, just switch the TV to the according “device” and press some buttons on the remote control. The key presses are shown on the console.

The console output again is parsed directly from the Java process and calls the appropriate methods in the controller. This works brilliant in my case. Yet PulseEight do not recommend using cec-client in a production environment as it is intended for tests only. But well – I didn’t want to invest even more time into coding a perfect cec-java-bridge. Especially as my C-knowledge is very basic. So chances are high that my implementation would be far from “production ready”, too.

WE ARE DONE

That’s it! And of course, you can fork the complete project on GitHub!


Part 1Part 2Part 3