OpenShift and Drupal

Over the Christmas holidays I had some more time for OpenShift Express experiments. This time I experimented with Drupal based on this template.

After a slightly bumpy start and a quick pull request I got the basic installation up and running. One problem I encountered now, however, was that there is no easy way to manage data created by the application once running.

For example, Drupal users can upload images (per default into sites/default/files). Naturally I would like to add these uploaded resources under version control as well. However, that's not possible at the moment. OpenShift Express had the notion of a OPENSHIFT_DATA_DIR directory whose content is preserved between redeploys, but this directory is outside of the git repository. What I really want is to be able to use the same resource directory for my local development as well as on the remote site and manage the resources via git.

Locally I can add resources created in sites/default/files directly to git, but how to solve this on the sever side? The first problem I had, which I already encountered when installing Gollum on OpenShift, was that the repo directory from which the application is run from is not a git clone, but an git archive which means that there are no git control files. If OPENSHIFT_REPO_DIR were a git clone I could somehow add created files to the git repo, commit, push and finally pull it into my local environment. Unfortunately, the OpenShift gang decided for now against a full git clone mainly (afaik) due to size concerns. The good news is that there is a way to fix this yourself with some lines in the .openshift/action_hooks/build hook:

cd $OPENSHIFT_HOMEDIR/$OPENSHIFT_APP_NAME/runtime
rm -rf repo
git clone $OPENSHIFT_HOMEDIR/git/$OPENSHIFT_APP_NAME.git repo

This will delete the git archive and and replace it with a git clone. This is of course only half the rent. Somehow you need to add and commit new content on the server side. One way is to add some more code to the build hook,but so far I went with another simpler solution. I just ssh into onto the remote server.  As far as I know this was not always possible, but it is now. Just check your git remote url via git remote -v or rhc-user-info.The former will display something like:

ssh://<uuid>@<app-name>-<domain>.rhcloud.com/~/git/<app-name>.git

To ssh to your app use:

ssh <uuid>@<app-name>-<domain>

Once you are on the server you can change into OPENSHIFT_REPO_DIR run your git commands and finally git push. Once you've done that you can do a git pull from your local machine. The disadvatange of this method is that a git push on the remote server is of course running the git hooks meaning you are forcing a redeploy. Whether this is a bad thing depends on the usecase. An alternative could be to use scp to copy new resources to the local machine and then add them there, but of course you would need a way to determine what to copy.

As you can see, I am also still figuring things out. I haven't yet found an approach I am really happy with, but I am getting there :-) Let me know what you think.

--Hardy

P.S. For the Mac developers out there. To develop my Drupal app locally using the default Apache server I added the following to /System/Library/LaunchDaemons/org.apache.httpd.plist:

<!--?xml version="1.0" encoding="UTF-8"?-->

<plist version="1.0">
<dict>
        ...
        <string>system.preferences</string>
        <key>EnvironmentVariables</key>
        <dict>
                <key>OPENSHIFT_APP_NAME</key>
                <string>myapp</string>
                <key>OPENSHIFT_DB_USERNAME</key>
                <string>mydbuser</string>
                <key>OPENSHIFT_DB_PASSWORD</key>
                <string>mydbpass</string>
                <key>OPENSHIFT_DB_HOST</key>
                <string>127.0.0.1</string>
                <key>OPENSHIFT_DB_PORT</key>
                <string>3306</string>
        </dict>
</dict>
</plist>

This way I can develop my Drupal application locally without making any changes to settings.php in php/sites/default which contains the database connection settings.

Programming languages in 2012

A new year is about to be born and it is time to make vows and predictions. Here is my take on programming languages in 2012 and what I will try to do with them.

Java will stay around for much longer just as Cobol has been around for ages. There has been much buzz in the business about Scala being the replacement of Java. I don't think it will happen in 2012. I am not sure Scala will be the next Java either. Java is a language where you can only do things in a few ways. This is a huge benefit for organisations that can not be sure their developers will stick around. A switch to Scala would make it much harder to get into a typical code base with many authors over several years. A nice and clean Scala system with a bit of code conventions would be a bliss to work with. So for enterprise software Java and its microsoft siblings will stay strong for yet some time.

I don't think 2012 is the year when developers start writing tests so I certainly hope that nice dynamic languages like Ruby, Python and Smalltalk will not enter the stage in any big way. But if they do lets pray that it is with developers that test.

Functional programming will continue to come back and this will move Scala (again) and erlang forward. Haskell is a bit too obscure to count.

I also predict that C will come back a little due to arduino and other stuff like that. Try to taste "domain specific languages in C".

One dynamic languages that will continue to shine is of course javascript. As code moves from the backend to the browser javascript will rise and shine. It is not that bad a language but it got several nasty traps for the newcomer. The only hope for a Javascript alternative comes from Coffeescript. Lets pray for native browser support in firefox during 2012. (I guess chrome is busy fiddling with that strange dart.) We will also see much more backend javascript during the year. node.js is here to stay and it will mature and become a serious option even for more conservative shops.

So what will I do with this knowledge in the year to come. This year I have become fluent in Ruby so I will continue to use it for backend stuff. I have also learned more Javascript and will continue down that path. The new language I will try to pick up is Coffescript. It looks much more joyful to code in than Javascript and may also be useful for testing. I guess I will do some Java also. It will feel nostalgic now that I haven't touched it for quite a while.

Do you still need a Christmas present?

Are you still looking for a Christmas present? It might not be too late to get a copy of The Wild Tiger, the debut album of Aravind Adiga. If not the best book I read this year, it was definitely the most entertaining. It gives a great insight into modern India and I really got sucked into the story. I don't want to write a long summary or get too much into reviewing. If you want to know more, just read the wikipedia entry or even better get your own copy. The interesting bit for me was, how reading this book brought back memories of another great - Wild Swans by Jung Chang. Even though both book differ greatly in their form of delivery and even though one is about India and the other about China, both capture the reader completely and deliver great insights. We learn ways to little about the Asian countries in our European upbringing. At least that's my experience :-(
If you know any of these two books and can recommend something similar, let me know. If you don't know them, go and get them. You won't be disappointed. AND they do make a great Christmas present.

Merry Christmas
--hardy

Lessons learnt: business intelligence

I have had the opportunity to work with business intelligence tooling for a customer this fall. It is a new area for me so it has been both frustrating and interesting. The frustration comes from the many dead ends it is possible to walk mindlessly into. The interestingness comes of course from the opportunity to learn new stuff.

My customer does not have "big data" with row counts in the millions rather than the billions. They get by with a relational database with no problem at all. My task has been to extract data from the relational database and put it in a business intelligence tool to enable ease of reports creation. We choose to develop a proof of concept with GoodData a SaaS (Software as a Service) vendor in the business intelligence sector. They are on a mission to redefine the business intelligence business (wow - a bit meta that) making it easy to get up and running with a nice user interface and some queer command line tools to use for integration. This post is a short list of lessons learned while working on this.

Data quality is important. For a system that has been around for a while it is likely that parts of the data is not of the highest quality. This is not visible until someone put it into reports.

GoodData has good support for developers with nice, open, informative forums and fast support. The tools are not outstanding in any way (as you can see to the left). Errors are most often obscured and hard to decipher. A certain amount of trial and error is necessary to get forward. According to various sources - old school business intelligence tools are typically even harder on developers and may cost a fortune to implement. This amazes me.

There are other SaaS players in the business intelligence niche but only some of them has the option to try the solution before buying. GoodData is one of them. Most of them don't have any useful information about how to go about doing an implementation on their home page. This is a bit strange to me but maybe the typical business intelligence tool buyer is a suit and not a developer? Pro tips for SaaS vendors: offer free trial account and put useful information on your site.

Even millions of data rows take a long time to extract.

A flat data structure with lots of redundancies is preferred in the business intelligence end of things. Getting a flat data structure out of a bunch of tables from a highly normalized relational database is not fast at all. My worst case scenario had me at about 13 days to get all the data out. I then abandoned the flatness dream. Export times is not that important though since a full data load should only be necessary once.

When getting data out of a live system and it involves several hours of processing the data will most likely change. If you are to join it outside the database it may then fail because stuff exported later may reference entries that didn't exists in the target dataset. The best way to solve this is probably to just take a copy of the entire database and work with that instead.

Lookup tables is not what it sounds like. If I - for example - has a flight dataset that got one departure airport and one arrival airport I can not connect this to the same lookup table but have to create a departure airports data set and an arrival airports data set. This is very wrong to me. (This may be a GoodData weirdness.)

Anyway. This were scattered thoughts - I have had a good time playing with these things and have as a result more respect for the complexity of BIG DATA.

Global Day of Code Retreat Stockholm 2011 - a retrospect

General ideas

I am on the train back to Göteborg from Global Day of Code Retreat in Stockholm hosted by Valtech. It was a nice day with valuable lessons. The fact that it was for free and even included lunch and free beer did of course not make it worse ;-)

The organization was good and the sessions were nicely moderated by Emily Bache. In the final retrospect of the evening I already mentioned the following ideas to spice up upcoming code retreats 

• randomize pairs
• have multiple problems katas or at least change the problem for the next event
• offer partly implemented projects for the problems (in different languages). This way one can also practice how to evolve existing software. Challenges could be to continue projects with bad or no tests. What do you then? Green field development is great, but is not spotted so often in the wild than the extension/maintenance based development. It is important to also hone your skills of working and improving an existing code base.

Some  thoughts on TDD

The biggest focus during the whole event was on TDD. I am a strong believer in writing tests and very (very) rarely I commit a functional change without a test, however, I am not religious about whether to write tests first or not. It depends on the situation.

It was nice to to some TDD, but some things I saw still got me wondering. We implemented Conway's Game of Life and a quite common approach I've seen in the iterations was to implementing the different (arguably quite simple) game rules. Often one single rule was taken, a test for just this single rule was added and then implemented. The implementation was supposed to just pass, often by initially returning hard coded values. Then a new test was added for the second rule and the implementation changed accordingly. Depending on your level of dummy implemenations you now have to revisit again your first test and implementation, and so on ...

I am wondering whether this is really the intention of TDD. For me there is too much mocking going on. The rules for the Game of Life are quite simple and in my opinion can be implemented in one go. You can still write the tests up front, but why not write all basic tests for the four rules up front and then implement the logic? Does TDD really mean that you have to break down a problem sentence by sentence? Is it really forbidden to implement "more" than required, knowing that you will need it a few minutes down the track anyways? Part of being/becoming an expert is that you are (compared to a novice) able to manage a bigger scoped problems. Breaking down problems is good, but as with everything a balance has to be found here as well.

I also think there is still room and more importantly value in hacking/spiking sessions. Again a balance is needed. For writing production code TDD is a very good method, but it is not so suited for investigating solutions for a problem. A spike to explore the problem space can be helpful and most importantly fun. In one session I was just quickly exploring in ruby on how to actually display the game using Rmagick to generate animated gifs. It was heaps of fun and a very welcome break of all the disciplined work in the other iterations.

Misc

Last but not least, a OO thought. Due to my Java background I naturally tackle most problems in an OO way. In one session though I was the co-pilot and we went for a solution (in ruby) without classes and a very compact data structure for the problem using arrays and maps. I found even thinking about the problem was made much harder, since all was so abstract. There is a certain level of self documentation when you just see names (classes) like Cell and Habitat. It helps your brain to tackle the problem. Advanced data structures are good and maybe sometimes required, but it is definitely a good thing, if you can hide this complexity behind simpler interfaces and classes :-)

Overall a very productive and enjoyable day. Thanks Corey Haines for the idea of code retreats.

--Hardy

sthlm film festival 2011

Some thoughts from Stockholm Film Festival. It is a pity that something that is so near and so good tends to be forgotten again and again. I haven't been to Stockholm Film Festival since 2005 and that was my first time despite living here my entire life. I then said to myself that I should attend every year. It turned out that annually happens every six years for me. In any case I went to some 11 films and here is some notes to remember them.

It started out with me missing The Man from Le Havre by Aki Kaurismäki. Käthe went and said it was perhaps the best movie of the festival. We saw "Mannen utan minne" a couple of years ago an it was a very beautiful and slow film indeed.

Miranda Julys The Future was miserable as expected. She creates bizarre but real personas with such ease. Thinking about it - a theme of the films I saw is definitely "unhappy women".

The most touching film in the "unhappy women" genre was Martha Marcy May Marlene by Sean Durkin about a woman that leaves her life in a sect. It is about two very intense weeks after she left when she stays with her only relative, her sister and her sisters husband. The film clearly depicts the kind of damage one may get in a sect. This touched me very much since I have been brought up in a baptist kinda church that is on the border to sectishness (or whatever it is called....). I can really understand how a badly managed view of the world may create the strangest of minds. On the other hand the mindset of a sect, with its distancing from worldly, capitalist things is really refreshing and should be something to think about for us all. The film contrasted this nicely with Marthas sister being sort of living the capitalist way. Not in an extreme way but still....

The darkest of the "unhappy women" genre films was the russian Twilight Portrait by Angelina Nikonova. It takes place in suburban Russia where evil police and a lonely woman walks the broken streets. All is just very sad in a troubling real way. It also got the most thorough party mood sinker ever seen.

Victorian Britains contribution to the "unhappy women" genre was a "Wuthering Heights" version. In this film everyone was unhappy and it rained a lot and there were lots of mud. Still the camera did some awesome things so it was not a waste.

The japanese contribution to the "unhappy women" genre was called Guilty of Romance and was about a bored housewife that ventures out and eventually finds herself at the bottom of things. Or just finding herself as the films sometimes says. The ending is interesting to say the least. I liked it a lot.

Hysteria generalized the "unhappy woman" theme. Settled in late 19th century England it had lots of messages put into the story about a young doctor that sways between doing the good work and being successful and finally finds himself where one might guess in such a entertaining and high moods film. One of the best.

I saw two documentaries. I only managed half of the VERY long Harrison one by Scorsese. I slipped away in the break being a bit too restless to stay for another 1'45. It was a nice film though. The other one was called Knuckle and is a must see for every feminist wannabe. It was about irish travelers, how different families fight each other for who knows what. The only reasonable persons in the film (except the creator) was women. The men just wanted to fight each other and they passed it on to their sons. Thinking about ridding the planet of males is strongly backed by findings when seeing this nice documentary.

All she can was a nice movie bit it NEVER ENDED. It was about a high school youngster wanting to get to college via winning a qualification competition. It is about her struggle and I guess the nice thing about the film is that it didn't end as expected. There are failures and miseries but in the end it is sort of sorted out  - or not.

The weirdest film was korean Invasion of Alien Bikini. With lots of action, weird sex and awesome characters it was great entertainment.

The last film we saw was Sleeping Beauty about a young student that seems to live a divided life. She works with several things at the same time as studying and communicates a feel of meaninglessness. Then she accepts a job as a sleeping beauty and it is about this experience the film revolves.

So - putting the festival to sleep. I am now a bit film fatigued.... we'll see if there will be another six years until I attend next time. It would be a shame with so many good creations to watch each year.

HTC Sense Ade

I finally had enough and rooted my HTC Desire. After all my problems with this phone I finally decided to root it and install the latest Android version on it. The final straw was that my phone got slower and slower, because it was running out of space on the phone internal storage. Not even App 2 SD would help anymore, since many apps would just refuse to be moved.

There are plenty of instructions out there on how to do this, so I don't want to repeat too much. I basically followed these instructions. There are four simple steps:

• Make a backup (eg MyBackup)
• Root the phone. For Mac users there is a very simple app Unrevoked. I had to reset my phone to factory settings before it worked. Trying to root the phone with the software updates I had was not possible.
• Install the ROM Manager and follow the instructions for ClockworkMod Recovery. I was happy that I did that, because I had to reset my original state several times, before I got things working.
• Download the ROM you want to install follow the instructions to install it. The important thing is say yes, when asked whether to wipe data and cache. Wipe it all! I went through several iterations ending up in endless loops, because I did not do that.  

Once your new ROM is booting and working you can reply your backup and reconfigure your apps.

What did I gain? First of heaps of battery time :-) Now my battery lasts for a day and I can even leave it over night without waking up having a dead phone. So I can definitely confirm that HTC Sense is a useless battery drain. Also some of the new Android Gingerbread feature are quite nice. With the rooted phone I can now also move more apps to the external storage which helped a lot as well.

One could say that I have rediscovered my joy in owning an Android phone :-)

--Hardy