So it seems I have the reason isolated. Now what is the solution? Well the applet is part of so-called Xfce-goodies, which is sort of external project. I did not find corresponding bugzilla nor some mailinglist etc. But I think that author is perhaps aware of this behavior, so there is no need to shout at him. But I needed the workaround and I think I have found one. It is simple and it is Linux-like: Just configure all your keyboards in xorg.xonf file in InputClass section and then use the applet as an indicator and switcher.
I am using following snippet of xorg.conf on my ThinkPad X301 which I am connecting and disconnecting to a USB keyboard:
Section "InputClass" Identifier "keyboard defaults" MatchIsKeyboard "on" Option "XkbLayout" "us, cz" Option "XkbVariant" ", qwerty_bksl" Option "XKbOptions" "grp:alt_shift_toggle, terminate:ctrl_alt_bksp" EndSection