How to process files with spaces using bash

The other day I wanted to rename a whole bunch of images with spaces and brackets in them. I had files of the form filename1 (1).JPG, filename1 (2).JPG, ..., filename2 (1).JPG, ...
In case you are wondering, these file came from a windows box where copies of files get saved in the format filename().xyz. I know that from a OS point of view spaces in file names are no problem, but hell are spaces a pita if you have to process the files in a script. So what I wanted to do was to remove the space with a '_', replace the brackets and while on it lowercase the filenames (I really just don't like JPG, I want jpg). For these kind of problems I would normally try a for loop. Here is my first attempt:

for i in `ls -1 *.JPG`; do j=`echo $i | tr -d "[:blank:]" | tr "[:upper:]" "[:lower:]" | tr "(" "_" | tr -d ")"`; mv "$i" "$j"; done

The idea is to process each file name in a loop, transform the filename with a a set of tr calls and then do a final rename of the files using mv. tr takes its input from standard in and writes to standard out, hence the trick of first echoing the current value of the iteration, passing it through several tr calls and then assigning it to a temporary variable. Note the use of the character classes for the transformation via tr.

This one liner works fine for file names without spaces, but as soon as there are spaces in the file names you are getting into trouble. The default field separator of a for loop is the space character which means file names with spaces are causing two iterations of the loop. One solution to the problem is to set the $IFS variable prior to executing the for loop. The value of this variable determines the field separator. However, in this case you also have to remember to reset the variable afterward.

Another approach is to  use a completely different construct which you don't see so often. Let's use find together with a while loop.

find . -name "*.JPG" -print0 | while read -d $'\0' file; do j=`echo $file | tr -d "[:blank:]" | tr "[:upper:]" "[:lower:]" | tr "(" "_" | tr -d ")"`; mv "$file" $j; done

We use find together with the -print0 option to print all file names into one line separated by a ASCII NUL character. In the loop we use read together with the -d option which determines the delimiter for read. The rest is the same as in the for loop solution.

Hope this helps some people, but maybe its just me who gets frustrated with spaces in file names ;-)

Enjoy,
Hardy

Support at its best

Fredrik and I are working on a little Rails app at the moment (more to this in a later post) and decided to host it on Site 5. The features of the hosting package are what you can expect from any decent hosting company our-days and would not be worth writing about, but there is one thing which really makes Site 5 stand out from the crowd - their support. This guys have excellent support and the best is the support chat functionality. You have a problem? Just click on the Chat Now button, type your question and within seconds you are chatting with someone who tries to help you. Many problems can be resolved on the spot and for others support tickets will be created. After the chat you just provide your email address in the chat window and you get a transcript of the chat sent to you. No waiting in a phone queue for hours the phone pressed to your ear in fear you might miss the moment where it is finally your turn, no unqualified people who don't have a glue what you are saying, no useless problem categorization games before you can talk to someone. No just a simple line of communication to someone who knows what he is talking about.
So listen up you Dells, Telias, Comhems, ... of this world - what works for them can work for you! And for heavens sake, get rid of this ridiculous voice recognition driven help systems. They are just a pita.

Enjoy,
Hardy

buildr overview

Tonight I spent a couple of hours at Jayway listening to Anders Janmyr talking about buildr - the build system that doesn't suck. buildr is a build system for Java and a whole bunch of other languages. It is written in Ruby and installs as a gem. Build scripts are also Ruby with everything that comes with it. It is built with the good things of maven as role model and with the less fortunate aspects of maven as frightening example. Dependencies are downloaded from maven repos and local builds can be installed to a local maven repo. To try it out I cd'd into a directory with a maven pom.xml and ran buildr. It gave me the choice of creating a build file from the maven pom or from the directory structure. When choosing maven pom the generated build file was about half the size of the pom. Now this was used on a very simple pom and according to Anders it may not work at all with more complex maven projects. buildr can do all the typical tasks of a build system and if something extraordinary is needed it is easy to script it with ruby.

Some advantages of buildr: (1) it is Ruby which enables "real" programming in build files, (2) it is Ruby which makes it possible to write build files that looks really nice (3) it is not XML - anyone tired of scripting in XML? (4) it includes the good parts of maven. Some disadvantages: (1) it is Ruby which may result in Perlish code if not careful, (2) it is not Java and XML (thank God) which may make it hard to introduce in larger organisations with static developers. If I was to pick a build system for my own use I would choose buildr but then I perhaps wouldn't choose Java as and the need wouldn't be there.... For larger organisations with competence problems it may be hard to introduce buildr over maven or ant. The threshold into ruby may be a little bit too high for the average java developer.

Anders also went through cucumber - a really nice testing tool that non-dev people can understand - and rake - Ruby make. "rake -T" will be remembered - it will come in handy when forgetting the howtos when coding rails in spare hours.

UPDATE: Peter Lind pointed in two more alternatives to buildr. If you use JRuby you can use Ant and Rake interchangeable as described in this post. Advantages with this approach is that you rely entirely on stable technologies. buildr is a bit new and may not work for all cases yet. And then - of course - there is polyglot maven that promises to: "to leverage the power of Maven through modern JVM language implementations like Groovy, Scala, Clojure and JRuby". Seems like everyone is heading in each others directions.....

PS. Jayway is hiring junior Java developers with 2-5 years of experience. Someone in the audience translated this to "cheap"....

Buquebus vs Seacat

If you are in Buenos Aires and you are planning a trip Colonia or Montevideo everyone will tell you that the best way to get there is with Buquebus. Even Lonley Planet would only lists the web address of Buquebus. We used Buquebus back in December for a short trip to Colonia and these guys charged a killing. It seems they can, because they dominated the market. Some people even go so far as to imply Columbian mafia connections ;-)
But there are alternatives and one of them is called Seacat. Ups, now I spilled the beans! We used them just a couple of weekends ago to get to Montevideo and it only cost us a fraction of what the shorter trip to Colonia has cost us earlier. The best part of it all is that Seacat leaves from the same new fancy terminal building in Puerto Madero which all people believe is exclusively used by Buquebus. In fact our taxi driver who brought us there kept telling us during the whole way that he did not think that there are any other boats leaving from there.

Montevideo itself was great for a weekend, but wouldn't have liked to stay much longer. People say that going to Montevideo is like time travel and you see Buenos Aires in the 1940ties, which is partly true. Unfortunately, there are other sides to it gave my experience of Montevideo a negative touch. A certain decay was noticeable in many neighborhoods and it felt that the broken window syndrome was at play, only that people have given up fixing them :(

Nevertheless, there are a few great spots you should visit in case you get the chance.

• The Mausoleo de Artigas

• Teatro Solis - make sure to get a tour and enjoy a coffee in the new refurbished cafe. Probably one of the best places in Montevideo to do so

• Mercado del Puerto - absolute highlight of the trip. I could have spent hours sitting in this market, eat good food and watch people. I in particular can recommend El Palenque were you can sit at the bar and watch the cooks do their business. I had the most delicious chorizo pan there!

Another thing we discovered was real Spanish style Churros. You find churros a lot in Buenos Aires as well, but there they tend to be of the thick and soggy kind. This guy at a local Carneval celebration made them super thin and crispy. Yummi! Believe me - no point to eat them any other way :)

--Hardy

A MySQL odyssey

I just started a little side project using Ruby on Rails and decided today to start developing against the database I will be using in production - MySQL. So far I have been using SQLite which worked fine, but it is always good to stay as close as possible to the target environment as possible. Anyways, let my odyssey begin...
In theory I just have to edit config/database.yml and change the adapter parameter to mysql. Then restart the rails server script/server -e development. Done!
Well, not so fast. The first restart tells me that mysql is not part of the default rails application anymore. The error message is quite clear and basically recommend to just run gem install mysql. Fair enough, I run the gem install and restart the server. Now I get:

> script/server -e development

=> Rails 2.3.4 application starting on http://0.0.0.0:3000
/Users/hardy/.gem/ruby/1.8/gems/radiant-0.8.1/vendor/rails/activesupport/lib/active_support/dependencies.rb:440:in `load_missing_constant': uninitialized constant MysqlCompat::MysqlRes (NameError)

Ok, not so funny anymore and quite cryptic. Uninitialized constant that cannot be good.
A word to my environment. I run Max OS X (Snow Lepard) on a Mac Book Pro. MySQL is installed via MacPorts and can be found somewhere under /opt/local. The MySQL version is 5.1.40.
What does one do in such a situation? I start googling and end up in some cryptic mail archives. Bugger, worst case scenario. Digging through mail archives is a real pita. Information is often unverified and skimming through through a million >>>>>>>>>> is just no fun. After some searching I suspect that the problem I am experiencing is that the mysql gem needs to be build for 64 bit. I decide to try:

sudo env ARCHFLAGS="-arch x86_64" gem install mysql –with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

After that I start the server again. This time I get:

/Users/hardy/.gem/ruby/1.8/gems/radiant-0.8.1/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:585:in `real_connect': Can't connect to local MySQL server through socket '/opt/local/var/run/mysql5/mysqld.sock' (2) (Mysql::Error)

Argh, another error. At least it is something more familiar. MySQL lets you either connect via TCP/IP or via Unix sockets. But why does it try to via /opt/local/var/run/mysql5/mysqld.sock. I am pretty sure I am using a different socket file. I locate my MySQL config file and find that the socket file is /tmp/mysql.sock. Why did that not get picked up? I add:

socket: /tmp/mysql.sock

to my config/database.yml. One restart later and I get:

/Users/hardy/.gem/ruby/1.8/gems/radiant-0.8.1/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:585:in `real_connect': Access denied for user 'foobar'@'localhost' (using password: YES) (Mysql::Error)

Will this never end? At least it's another classic. MySQL permissions. First step is always the mysql.user table. I probably forgot to add an entry for my user and database. I connect as root and execute:

CREATE USER 'johndoe'@'localhost' IDENTIFIED BY 'foobar';
GRANT ALL PRIVILEGES ON mydb.* TO 'johndoe'@'localhost' WITH GRANT OPTION;

After that I try to start the ruby server again. Same problem. Some quick swearing, but then I remember - flush! Back to the mysql shell:

FLUSH PRIVILEGES;

And now I get an error in the mysql shell:

Table 'mysql.servers' doesn't exist

WTF! After some more swearing (this time in German) I start googling again. I cannot find a reasonable explanation on why this table is needed or why it is not created in my mysql database. Best I find is a table creation script which I decide to run:

CREATE TABLE `servers` (
`Server_name` char(64) NOT NULL,
`Host` char(64) NOT NULL,
`Db` char(64) NOT NULL,
`Username` char(64) NOT NULL,
`Password` char(64) NOT NULL,
`Port` int(4) DEFAULT NULL,
`Socket` char(64) DEFAULT NULL,
`Wrapper` char(64) NOT NULL,
`Owner` char(64) NOT NULL,
PRIMARY KEY (`Server_name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='MySQL Foreign Servers table';

Ahh, now flushing privileges works. One more script/server -e development - success! Finally I am up and running again, but I wonder:"Do things really have to be so painful?"

--Hardy

P.S.: With MySQL permission/user problems I recommend to go through these guides: access- denied and adding-users.

P.S.S: Another problem which had me puzzled for a while one was that I was not able connect to MySQL using:

mysql5 -u foobar -p

Whenever I tried to connect with a password I would get the Access denied for user (using password: YES) error. Connecting without the -p option worked though. I kept looking at the mysql.user table and the entry for foobar. What I did not notice was that I had effectively something like this:

Host       User      Password       
localhost 
%          foobar    

The first line allows any user to connect from localhost using no password. Since localhost is more specific then the % wildcard this rule was always applied before the actual foobar entry.

jfokus 2010 day 2 summary

Day 2 of jfokus started with a content free keynote on the future of Java. Cameron Purdy from Oracle couldn't say much due to the snorcle press conference later today on the acquisition of Sun. Still it was not the worst keynote ever. I think it grabbed a second place far ahead of javafx and robots and far behind first years nice talk on java as open source by Simon Phipps.

Then off to the thought challenging talk on unconscious taylorism by Marcus Ahnve. He traced a lot of our problems with software back to the early days of industrialism and its emphasis on an effective overall system. Every task was specialized and performed in batches leading to skills oriented organisations. You know . system architects in one group and developers in another, usability experts in one group and system analysts in another, maintenance in one place, operations in another and development in a dependency position to the former two. This is the way most organisations do their work. It is like a big engineering practice. Marcus emphasized the need for individuals quoting the agile manifesto: "Individuals and interactions over processes and tools". It was not clear from the speech why this emphasis is good. My answer is that software development is a creative process where the individual must be given a good environment to be creative in. The talk reminded me a lot of Richard Durnalls splendid JAOO talk on "The IT division. Refactored" (review here). They both cited Matsushita on the western failure and while Richard talked more about the alternative (process oriented organisations), Marcus talked more about the cause. Another reference that popped into my head was the programmers stone. Marcus referenced some non technical guy that talked about them and us, evil and good - it sounded a lot like the mapper - packer thing in the programmers stone. I liked this talk a lot and afterwards I've been thinking that there may be similar causes for the use of the "architecture" metaphor - bashed here.

Next I went to the pair programming show where Niclas Nilsson and Hans Brattberg illustrated problems and possibilities with pair programming by actually playing a number of typcial scenarios. Nothing new here but great fun and a good reminder.

After lunch I felt obliged to go enterprise. (Ok - I know I told everyone that I counted out all sessions mentioning enterprise and performance..... I'm sorry - I let you down.) This was the talk about Java EE 6 - a really good talk by Alexis Moussine-Pouchkine about a really boring subject. Seems like the new stuff in Java EE 6 is really nice and useful and will make life better for the developer. I don't particularly like the heavy use of annotations but it sure is powerful. The new Schedule annotation is still very verbose compared to a cron job definition but to be an enterprise solution is looks really smooth.

To outweigh the bad influence of the enterprise I went to the JRuby session with Charles Nutter. This was perhaps the best talk at jfokus this year. Entertaining and technically brilliant. The nicest part was how to transform a Java program to a Ruby one. A great way to show the unnecessary verbosity of Java. I knew most of the stuff in this talk since before but since I like ruby it was great anyway. As for the benefits of Ruby on the JVM I think it is a great selling point to enterprises that already invested in a J2EE infrastructure. No need to change that if you want to use Rails for you applications. (What they don't know is that it would probably be even easier to set up a pure Rails installation using apache....) Arguments for JRuby instead of standard Ruby is mostly around performance. For example - JRuby takes advantage of Javas threading model which makes it easier to scale an application. JRuby also runs faster in general. Another big selling point is access to Java libraries. I am not sure how often you really need something that is only Java but it may make lots of sense in a Java enterprise environment where other in-house apps and libraries are built with Java.

I ended the conference with two fluffy lightning talks about feedback and leadership. The content of the conference was good and I had a lot of networking going on. 5 year celebration next year!

jfokus 2010 day 1 summary

It is hard to top the exotic blog posts from my argentinian coblogger. Nevertheless it may make sense to write up a summary of jfokus - the yearly Java conference in Stockholm. Yet again the venue is Filmstaden Sergel and I won't report anymore on this as compared to last years report.

The day started badly with a 40 minute delay on the commuter trains from the northern suburbs. This made me late to the first tutorial. The first day of two is tutorial day - one before and one after lunch. 4 to choose from. I started with a nice talk on Good Object-Oriented Development by Kevlin Henney. Kevlin went through a lot of basic stuff related to object oriented development. Nothing new really but it is good to get this kind of confirmation know and then. He made some nice examples on encapsulation and showed us some real life code that was just not very good. I see similar stuff often in the systems I work with. One reason for this kind of code is that many developers learn a set of have-tos that they stick to. Some of them should probably be considered harmful. One example: always provide a default constructor. This may be useful if there is such thing as a default state for instances of the class but in many cases it results in objects with undefined state and leads to error prone code. Later on Kevlin removed 'get' from some code with a bunch of getters and I felt like FINALLY. I have never understood the point with the JavaBeans standard (ok - tooling but what else?) getters and setters really suck. (Ruby has a much nicer way to go about it. No big surprise there.....) He also talked for a while about unit testing and a feeling I have had lately that writing tests per method is darn wrong was firmly confirmed. It is much better to name methods after what they test. A method is often tested with several test cases while other simpler methods can be tested indirectly without a dedicated test case. The readability of tests is often low and I know I have failed with this on several occasions but it is not hard to argue that test code should be as readable as the main code. It will all live for a while and it will all be maintained so it should all be in as good shape as possible.

After lunch Anders Hammar delved into the depths of maven. Not the funniest subject but since I have some maven skills and don't work with it currently I felt that it was the best choice of the afternoon. (A kind of attempt to refresh a fading skill set.) Anders mixed slides and live demonstrations which is a very nice way to go about it. He started with lots of examples of the eclipse maven plugin m2eclipse. Looks really nice and I have tried it on a side project also. I didn't realize that m2eclipse runs on maven 3 but apparently it is compatible with maven 2 so you can use m2eclipse and maven 2 on the command line with the same configuration files. The part on dependency management didn't add anything new. More interesting then was the next part on life cycle customization. This is perhaps the area where maven has some serious problems. There are 3 built in lifecycles: default, clean and site. They are built up from phases where each phase has its default plugin that is executed. A problem with this is that this lifecycle view of a system build doesn't always fit all systems. If your plugin doesn't fit nicely into the phases you can run into all sorts of problems. A possiple way out of this problem is to create your own lifecycle. I am not sure whether this is good or not. Maybe Hardy knows more about it? The tutorial was labelled "advanced" but to me it was closer to basics. My needs for repetition of forgotten knowledge were thus satisfactorily filled!

The day ended nicely with dinner at texas longhorn kungsholmen and then some more beverages at bishops arms centralen. Expect a second report after tomorrow!