Orban Botond's Blog

Normalize you DB

2011-10-06T17:04:00.007+01:00

Hi everyone,

In the last couple of days I had the opportunity to normalize a DB. :)

Let me show you a screenshot about a single table:

As you see a lot of data is repeated across the rows. Therefore, I had some bad thoughts about those who let the DB become like this in the beginning. I didn't wanted to do it. As the moments were passing away I let the challenge to excite me. At the end I did it.

Before doing anything the total number of rows in the db was above 100000 and the size of the table was 10,5 MBytes.

My first step was to extract the name of the components into another table and reference them.

This way the size of the table decreased to 8.5 MBytes. And the new table size was 1.5 Mbyte. I earned 0.5 Mbyte, not too much, but the database was much more cleaner.

My second step was to transform the check/change data represented by a string into their numerical representations, this way enabling the database to be internationalized. By doing this the table size decrease to 6.2 MBytes. That means 72% of its original size and the data can be internationalized and we have all the benefits of a normalized DB schema (that means no update problems and a joy to work with).

I also measured the speed of joins operations. Because of the normalization the same data retrieval was increased by 3 times. I put an index onto foreign keys. And the speed was just like before. :)

Design your User Interface

2011-10-06T10:52:00.006+01:00

Hello everybody,

I have just finished the book "The Design Of Everyday Things".
It's a great book written by Donald Norman. The guy who wrote "The invisible computer", who worked for the Apple Computer's Advanced Technology Group.

In this book he talks about many-many interesting things:
-The constraints and strength of our short term & long term memory.
-The slips and mistakes our brain makes, while we do things.
-How can we make use of our cultural knowledge and knowledge in the world while design.
-The importance of natural mapping between controls and our actions.
-Interpretation of a system state, and our mental model about the system.
-The (missing) evolution of product line. The evaluation of the already selled products and the importance of debriefing then evolving the product to a better one.

He also describes how primitive were the computers at time of writing.

And he gives us some hints how he imagines the mobile phones should easy our everyday tasks. I had the impression the he was envisioning the nowadays iPhones. He also describes how a graphical user interface would help us to be more effective with the computer interaction. I often had the feeling that he was describing a vision which is very similar about nowadays iPads, iPhones or Microsoft Surface.

I was also symphatizing with him because I remember times when I needed to setup my Mouse and CD driver in the config.sys in order to use those devices. Nowadays everything is much more easier. Especially that, I am a Macintosh user I can spend my time focusing on relevant tasks, without even realizing that I am working.

I finished the book yesterday. After I put the book back onto my shelf I started to browse the new Apple products... They are just amazing. I was especially amazed by the new Siri feature in the iPhone. It remembers me to the "Computer" from Startreak. It is a good example that if we can imagine something then we can sooner or later achieve it. Another feature I imagined even myself and waited a lot is the iCloud. I tried the beta version by downloading Ethics from Aristotle on my Macbook and the book instantly appeared on my iphone too. :) Definetely it enhances our usability experience.

The next day, other news were expecting me. Sad news. The founder and main dreamer, envisioner and technology pioneer Steve Jobs died. So good journey Steve and rest in peace.

sqlite log filtering

2011-07-18T21:14:00.007+01:00

Hi,

Presumably a lot of you are developing rails applications using sqlite.
In this case the log is spamed by some pesky sql statements every time our app accesses the db.


 SQL (0.6ms)   SELECT name
 FROM sqlite_master
 WHERE type = 'table' AND NOT name = 'sqlite_sequence'
  SQL (0.6ms)   SELECT name
 FROM sqlite_master
 WHERE type = 'table' AND NOT name = 'sqlite_sequence'
... 
 SQL (0.6ms)   SELECT name
 FROM sqlite_master
 WHERE type = 'table' AND NOT name = 'sqlite_sequence'
  SQL (0.6ms)   SELECT name
 FROM sqlite_master
 WHERE type = 'table' AND NOT name = 'sqlite_sequence'
...

These messages bother me because they have nothing to do with my application logic, they are only needed by the sqlite driver to access the db structure.

Therefore I developed a small ruby gem called sqlite_log_filter which filters these messages out. The source can be found on github

All you need to do is to put it in your gemfile


gem 'sqlite_log_filter'

Then bundle it


bundle install

Hopefully it helps :)

Test (Behaviour) Driven Development

2011-06-21T21:53:00.007+01:00

Recently I saw a pretty nice post:

"If you see anyone pushing code to Rails without tests, ping me and I will gladly revert. In 2011, this should be unacceptable."

I remember times when I had a project which was developed test driven at my former place of work. Of course the Test Driven initiator was me, and by time the teammates embraced the methodology and it managed to be a very successful project.

The most exciting thing was that I never had to debug; the tests always showed me the exact the location of the problem -if there was any.
Another almost unbelievable fact was that days before releases we didn't have any unfinished tasks. We were down on the streets having a Budweiser or limonade and we were chatting. What I mean here is that the test batteries saved us so much time that we never had to work overtime. We were just shipping the product confidently days before the planned release. I also remember some sales meetings where this project was showed to the customers as a success story. I also remember one of the questions one customer asked: "Then why don't you write software this way every time?" and the answer was "Because some customers simply dictate to not write any test at all..." Well everyone in the meeting laughed. But we all knew that this decision would cause any customer to throw out hundreds of thousand of dollars and hundreds of hours of overtime for the developers without mentioning the moral downgrade for every involved party. Maybe we were laughing at that unintelligent stereotype.
I also remember one important fact while writing this. In the initial project schedule the testing time wasn't calculated. We simply adopted the test first and then the benefits showed themself over and over again. We couldn't run out of the schedule... :) Almost unbelievable but it was true.

After the project has been finished I have tried to introduce this methodology in as many projects as I could. Unfortunately there were some projects where the mass was so frozen that my attempts were totally ignored. Sometimes the reason for ignorance was simply the lack of infrastructure. Fortunately the infrastructure is not a problem in Ruby on Rails. The community is also mostly test-centered.

I cannot emphasize enough for every developer in any domain to embrace this Test First thinking. There are so many benefits of it that many books have been written about it. And who knows, maybe you who will embrace this approach, will have some similar success story to tell the world

autotest, zentest, autotest-rails, autotest-rails-pure: which gems to choose?

2011-06-20T13:18:00.012+01:00

It was a little bit confusing for me which "autotest like" gems to choose when I want to autotest my rails application.
Well... There are some alternatives depending which testing framework I choose rspec or Test::Unit, also there are alternatives if I want to generate the tests for my controller actions automatically with zentest or I just want to write them in some other way.

Zentest was the first tool in this domain with a lot of features. Then autotest-standalone was factored out from it. There is also an empty gem called Autotest which is dependent on Zentest.

Because the main idea is to simply map the lib and test directories, in order to test rails application we need to tell the rails specific mappings. This is done by the autotest-rails and rspec-rails and autotest-rails-pure gems. So let's walk through each of the alternatives

Test::Unit

In this case the rails specific mappings are done by the autotest-rails-pure or autotest-rails gems.
The code excerpt which makes the mapping for the controllers is:


    add_mapping %r%^app/controllers/application_controller\.rb$% do |_, m|
      files_matching %r%^test/(controllers|views|functional)/.*_test\.rb$%
    end

For the core autotest we can use the autotest-standalone or the feature rich old Zentest.

Rspec

The rails specific mapping is done by the rspec-rails in case we opt for using rspec

Here is an excerpt wich does the mapping for the controllers:

    add_mapping(%r%^app/controllers/(.*)\.rb$%) { |_, m|
      if m[1] == "application"
        files_matching %r%^spec/controllers/.*_spec\.rb$%
      else
        ["spec/controllers/#{m[1]}_spec.rb"]
      end
    }

So there are a plenty of alternatives we can choose from and it is good to know which opportunities are available. We have all we need :)

Making the Factory Girl to support Paperclip

2011-06-11T12:49:00.019+01:00

Hi Rubysts,

Paperclip and Factory Girl Rails both of them are my loved gems.
Unfortunately Factory Girl Rails can't create the model if the model had been Papercliped. Well at least not in Rails 3.0.7. But the good news is that it can be solved by adding a few lines of Ruby code to our factory.

This is my model:


class Club < ActiveRecord::Base
  has_many :dancers
  has_attached_file :photo, :styles => { :medium => "300x300>", :thumb => "100x100>" }
  validates_attachment_presence :photo
  validates_attachment_size :photo, :less_than => 1.megabytes
end

So in order to use the factory with ease we need to tell our factory to upload some pictures into our model.
Therefore let me show you an example how to write the factory which makes the two gems to co-work:


include ActionDispatch::TestProcess

class Club
  attr_accessor :photo_file_name
  attr_accessor :photo_file_size
  has_attached_file :photo, :styles => { :medium => "300x300>", :thumb => "100x100>" },
                    :url  => "/test_can_be_deleted/:attachment/:id/:style/:filename",
                    :path => ":rails_root/public/test_can_be_deleted/:attachment/:id/:style/:filename"
end

Factory.define :club do |c|
  c.title 'Limpex'
  c.sefurl 'Limpasm'
  c.name  'Limpasm'
  c.metadescription 'good etc'
  c.content '<p> a good place</p><p>the icecream is prety damn good</p>'
  c.state 0
  c.city 'Gheorgheni'
  c.email 'some@somplexx.com'
  c.photo   { fixture_file_upload( 'spec/factories/test.png', 'image/png') }
end

As you can observe I have just reopened my model and have redefined the paperclip attachment to point to a different path in order to store the dev and test mode attachments in different places.

That's it. :)

The missing event when the mouse stays in a place

2011-01-26T22:14:00.005Z

Hello everybody,

I just released a jquery plugin which detects if the mouse stays in a place for more than the specified period of millisecs.

I was missing this functionality and I also needed more than twice and :) I needed also in the past so finally I decided to extract as a reusable plugin.

You can download it from here:

https://github.com/orbanbotond/mouseStay

or here:

http://plugins.jquery.com/project/mousestay

It is easy to use as any other jquery thing.

Enjoy :)

Unobtrusive In PlaceEditor for Rails3

2010-10-17T10:10:00.017+01:00

Hello everyone,

It's been one and half month since Rails3 came out and it evolvolved a lot if we compare it to Rails2. This direction of evolution makes me happy.

The topic I want to write about is related to the Rails3 way of unobtrusive Javascript. To be more specific I want to show you a way how to make the scriptaculous' in place editor unobtrusive.

The idea: wouldn't it be nice if by adding some extra attributes to some html tag the displayed entity fields would become spontaneously editable?


<p id="<%= project.id %>_project_content"
  data-in-place-editable="true"
  data-entity="project"
  data-field="content"
  ><%= project.content %></p>

I think it definitely would.

So let's start with it.

Let's suppose that we have a restfull resource called project and we want an in place editor in our view which is bind to the update method of our ProjectsController.

First thing you need to do is to check out the http://github.com/orbanbotond/Fast-In-Place-Editor. into your public/javascript directory of your project.

This file contains the generic repetitive mundane line's of code which would otherwise been repeated field by field in your view.

In your layout file you need to include this javascript:


<%= javascript_include_tag "myextensions/fast_in_place_editor.js" %>

In your view add some extra attributes to the tag which displays your entity's fields. Let's say this is how you were displaying the content of a project:


<p id="<%= project.id %>_project_content"><%= project.content %></p>

In order to make the content in place editable do this:


<p id="<%= project.id %>_project_content"
  data-in-place-editable="true"
  data-entity="project"
  data-field="content"
  data-url="<%= project_path(project, :authenticity_token => form_authenticity_token) %>"
  data-cols="60"
  data-rows="6"
  ><%= project.content %></p>

And that's all you need to do.

Enjoy :)

A Click Heat Map plugin for ruby on rails applications

2009-12-06T21:45:00.022Z

Hi Rubysts,

I've just finished my first contribution to the Ruby on Rails community this evening.
This contribution is actually the repackaging of the click heat map generator included within cannoli into an easy-to-install and deploy ruby on rails plugin.

If you take a look at cannoli you observe that it is based on ruwa web analytic ruby on rails app. I didn't need all of these together.
So... Did I manage to do something better? I believe that it is only rarely the case for something to be in every aspects better than something different. In this particular case I would say that I have made a piece of ruby on rails building block which is less than cannoli but it can be extremely easily incorporated in your Rails ( >2.3.2 ) application.

It has it's own requirements:

you need Rmagick installed
you must have Rails 2.3.2

Maybe you want to improve your sites web usability.
In order to install the plugin:

./script/plugin install git://github.com/orbanbotond/click_heat_map.git

From the click_heat_map plugin directory (not from rails root)

```
rake db:migrate:click_heat_map
```

Then you need to put and adapt this code into your templates:

< src="'/javascripts/overlay.js'" type="'text/javascript'">

< src="'/javascripts/clickmap.js'" type="'text/javascript'">

put this html code into your templates you want to measure:

< id="prova" style="margin:auto;width:800px;height:1px;">

And finally, you reached the point where you want to deploy your rails app using capistrano.
Maybe you ask yourself how capistrano knows how to migrate your plugin to the remote server.
Well...
In my case I put the installed plugin under version control and Capistrano just copied to the remote server.
I also created a file named
20091130000000_create_heat_map_clicks.rb:

require 'vendor/plugins/click_heat_map/lib/db/migrate/20091130000000_create_heat_map_clicks.rb'

This way capistrano creates the tables needed by the plugin on the remote server by running "cap deploy:migrations".

Finally the result makes me happy :)

A quick fix for swfir and prototype incompatibility

2009-11-04T15:37:00.008Z

Hi,

Some of you might know about a small and smart library called swfir. This library intends to extend some of the limitations of classic html images. It can rotate the images and it can draw some fancy borders and fancy shadows around the borders and so on. I like the library despite the fact that it's not perfect. It's not compatible with prototype. :(

Here is a small patch that I posted to the authors and hopefully a new release will come out which will support prototype :)

Untill then you can use the patch from here:

@@ -569,7 +569,11 @@

Flash Vars

var varString = "";

- for(va

r key in this.params){ varString += ("&"+key+'='+ this.params[key]); }

+ for(var key in this.params){

+ if("function" != typeof(this.params[key])){

+ varString += ("&"+key+'='+ this.params[key]);

+ }

Rendering before:

Rendering after:

Subversion 1.5.2

2008-09-10T00:19:00.012+01:00

Hi all,

It's a long time I haven't written about anything.

Yesterday I reinstalled the latest (1.5.2) svn onto my leopard. It was a challenge for me to install from source into the right place: /usr/bin;/usr/lib. This way I have only one svn on my machine the latest (actually the best) one :).

There is a new feature I have wait a lot. It makes software development easier when I want to try something unusual on another branch, and finally want to reintegrate into my trunk.
It is much easier to handle than the old method for merging:

svn merge --reintegrate http://tralalala/branches/3

Here are the steps needed to install it:

Download the sources svn and svn-dep.
/configure CFLAGS='-arch x86_64' APXSLDFLAGS='-arch x86_64' --with-ssl --with-sasl=/usr/lib --prefix=/usr >c.out
make
make check (this is optionally, however I like to see all the tests passed :))
sudo make install

I installed with neon because I wanted to add http repository access. Here are the steps which need to be performed right after download if you want to compile with neon:

Move neon from svn-dep into the unpacked svn directory. This way even neon will be compiled and installed with svn.

Upload progress bar by streaming JSON

2008-02-13T09:54:00.035Z

Have you ever dealt with the problem of making an upload progress bar?

If you did, then it's time to continue reading.

I wanted to make one using javascript.

Basically, there are two alternatives:

The simpler one is to make periodic ajax requests to a url. That url must return the received_size/ total_size pair. All you need to do is to update your progress bar in the browser. This solution works in all those browsers which support AJAX. The disadvantage is that the browser will usually receive the (received_size/totoal_size) data pair delayed. So the progress bar won't show you the real progress.
The second option would be to use streaming. Instead of periodically polling the server, we should make one single request, and the server would stream the receive_size/total_pair data back to the browser.

I like this streaming alternative much more because it gives you a better UI experience.

Let me give you some hints on the streaming alternative.

The simplest and the secure way is to stream sequences of '<script type='text/javascript'>updateProgressBar(total, received)</script>' back to the browser. The returned javascript will be automatically executed in IE 6.0, Firefox, Safari and Opera. You can find a solution for this at Ry Dahl's article. However it works in a lot of browsers it's not the most elegant solution because you need to hardcode updateProgressBar in the server configuration file.
A little bit more elegant solution is to stream JSON expression like {"total_size": 10, "received_size": 0}; {"total_size": 10, "received_size": 1}; {"total_size": 10, "received_size": 2}; ...

If you are streaming JSON expressions you need to write a javascript function which updates the progress bar according to the last JSON slice. You can do this using this javascript utility. This script periodically checks the received content and calls your updateProgressBar function passing the last received JSON slice to it. Unfortunately this script doesn't works with the latest prototype library.

The shortest and the most elegant alternative is to use the "onInteractive" AJAX callback function. By using this callback we don't need to periodically check the response. It is called every time when new data arrives to the browser.

Look at this beautifull code:

new Ajax.Request('/upload/progress', {

pos :0,

onInteractive: function(obj) {

var slice = obj['responseText'].slice(this.pos)

var jsonFormat = slice.substring(0,slice.length - 2);

var json = jsonFormat.evalJSON();

updataProgressBar(json.totalSize, json.receivedSize);

this.pos = obj['responseText'].length + 1;

});

Unfortunately this callback doesn't work in Opera neither in IE.

And I didn't find any w3 recommendation for enforcing the browsers to implement it. :(

Let me summarize the opportunities and how are they supported on different browsers:

	Opera latest	Firefox 2.0.0.2	Webkit	Safari	IE 6	IE 7
Periodic Ajax calls	ok	ok	ok	ok	ok	ok
Streaming <script> tags	ok	ok	ok	ok	ok	ok
Streaming JSON, using Ry's javascript	ok	ok	ok	ok	not working	not working
Streaming JSON, using onInteraction	not working	ok	ok	not working	not working	not working

So I need to keep up hacking till AJAX standard is born and implemented... :))

Social Networking Softwares

2008-02-02T21:36:00.001Z

I received an invitation to a "social network" software from India not long time ago.

I always subcribe to these software at least to check what are they capable of. I also like to observe the Darwinian evolution of software.

This software made me upset right in the first turn. It didn't harvest my relationships organically like FACEBOOK but it wanted to record my relationships using my Gmail contacts by borrowing my Gmail password. I didn't have any other choice if I wanted to see the web-app running, however borrowing my Gmail password was very suspicious to me. In the next step it enlisted all my Gmail contacts. There was a tiny checkbox to send the invitation to all of them or not. I didn't want to invite anybody to this social network, all I wanted to do is to check what it offers. Therefore I've unchecked the checkbox and went to the next page. Well... The service didn't offer me any excepcionally good quality. It was nothing to learn. So it didn't worth borrowing my precious Gmail password. For the sake of security I changed my Gmail password promptly.

I was surprized when many of my friends sent me a message in the next few days that they don't want to subscribe to this at all, nevertheless I unchecked the invitation checkbox. There were some of them who thought that this software is a VIRUS. Well I've unregistered myself from this "social networking" immediately.

Well... This happens when a software harvests its users in an agressive manner rather than adapting the strategy to grow organically. The problem with it is that even sometimes even those people will be harvested who are not even my friends (I send them only one or two email messages in my whole life). So my "social network" will contain some foreign people to.

This precious information is now free and available for everyone, thanks to these softwares. But sometimes they also contain false relationships :)

Ruby on Rails productivity

2008-01-17T22:11:00.002Z

I was always amazed by the productivity of ruby language. Specially by the development infrastructure of rails. I wanted to check it and specially to measure it.

Therefore, I decided to implement something which was already implemented in other languages before using other infrastructure. This happens in our industry very rarely.

I started to implement the payroll system described in the book Agile Software Development. Originally the payroll system was written in C++ and consist of about 3300 LOC. I implemented the first iteration in ruby writing only 383 LOC including even the test code. I was very surprised. :)

And why did I use rails? Ruby would have been enough.

Simply because it has a very effective development environment.

I used migrate files for creating the relational DB, fixtures for setting up the preconditions for testing, zentest as devil's advocate, and rake for a lot.

What about the benefits:

it was fun, easy, and very interactive
I've got fast feedback from zentest
all the O/R mapping was already done
learned a lot along the journey
During the development I was happiest ever

Keep smiling :)

World of Standards

2008-01-12T17:12:00.001Z

I saw an interesting post in my feed reader today about the fact that the IE8 passes the Acid2 test.

I ran the test myself immediately using my installed browsers. I was surprised that firefox didn't pass the test but I was very happy that Safari and Webkit both passed.

The whole test is just about displaying a simple smily face correctly, isn't it?

Taking a closer look what is inside the test I was surprised how many small details are tested.

They also test some seldom used great css features like the generated content, css tables, child selectorsto name just a few one.

The guys there simply made a brilliant job for the web community.

I am anxious to try the next Acid test. :)

Webapplication testing like a Lego story

2007-12-15T11:41:00.002Z

Once I have got the chance to challenge the joys and woes of testing a Webapplication.

I started the whole thing trying out HttpUnit/HtmlUnit and Selenium to experience which one is better for me.

I decided to use Selenium because it uses a real browser for clicking throughout the UI, therefore if the UI works driven by tests it will work in the hands of human users too.

I used the Selenium IDE to create my initial test cases and I saved them as Java test cases. Later as the number of test cases increased the same Selenium IDE generated java code appeared across several test cases. I refactored those repeating code until I had only one helper method for handling the UI elements. From then on writing test cases was like playing Lego in my childhood, except that instead of using the Lego brick I used the methods in my helpers. The number of test cases has grown to a few hundred in one month. Well, playing Lego becomes boring after a certain amount of time.

Fortunately I faced another challenge.

I needed to check if handling the mandatory fields, on forms works or not. Therefore I started to play with my lego set again. After writing the second test case I saw that they are almost identical. Hm... I decided to build a generic machine for testing those forms. In two weeks the machine was up and running and every form was well tested (around 300 cases).

In the meantime the way how the requirements were defined for the mandatory fields had changed from text based sentences into a big matrix. Usually changing letters in matrixes are much more easier than rewriting whole sentences right? So the requirements/matrix changed on every two-three days. All I needed to do was to change one or more letters from "r" to "m" in my test case to support those changes. Yes, my machine was data driven. Not to bad from a Lego system. :)

That was the funny part for a developer like me who was asked to test a web ui.

The whole thing was still under development and fine tuning. Therefore, I wanted to put those tests to check the ui every day automatically. The machine which was given to me for daily regression testing was running linux. It was placed somewhere maybe without a monitor and a keyboard.

Well, my idea was to test it under firefox but I was constrained to use only the text console. Hm... How those two fit together. Fortunately I got a hint from a friend to use a virtual display.

I used the Xvfb for the virtual display.

In this way the cron job could launch the test cases which could open a running fully functional firefox and the webapplication was tested every day. :)

Well, in the beginning I did't want to do all this "testing", but during the journey I gained a lot of experience and I say it worth playing all this Lego like game. :)

Introduction

2007-11-14T23:30:00.002Z

Hello everybody

I am a Software Engineer with over 5 years of experience.

At this moment of writing this blog I am employed by a big software engineering company in Eastern Europe.

I have created this blog with the purpose of sharing ideas about software development, technologies and tools.

Hopefully this will connect me to other professionals around the globe and there will be a place for constructive debates.