Tuesday, September 27, 2011

hpacucli: Error: No controllers detected. with hpsa and SmartArray P410i

I have encountered a fucking weird problem with hpacucli (HP utility for their RAID controllers) on ProLiant DL360 G7 or whatever. The controller was HP SmartArray P410i and the problem was simple:

root@XYZ:~# hpacucli 
HP Array Configuration Utility CLI 8.70-8.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

=> ctrl all show

Error: No controllers detected.

The solution is: Load the sg driver and that's it:

root@XYZ:~# modprobe sg
root@XYZ:~# hpacucli 
HP Array Configuration Utility CLI 8.70-8.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

=> ctrl all show

Smart Array P410i in Slot 0 (Embedded)    (sn: 500143801630C980)

Sunday, September 18, 2011

Moose (a postmodern thing in Perl, or whatever) and circular reference+inheritence

Now speaking of Moose, the postmodern object system in Perl 5, taken from Perl 6, chewed, digested and... Whatever:-) Let's assume I have a class Animal like that:

package Animal;
use Moose;
use Animal::Dog;
use Animal::Cat;

has 'type' => (is=>'ro', isa=>'Str');

sub make_noise { print "Making noise."; }

sub specify {
  my $self = shift;

  return Animal::Dog->new({type=>$self->type()}) if($self->type() eq 'dog');
  return Animal::Cat->new({type=>$self->type()}) if($self->type() eq 'cat');
  return $self;
}

And two subclasses:

package Animal::Dog;
use Moose;
extends 'Animal';

override 'make_noise' => sub { print "Wuf!"; }



package Animal::Cat;
use Moose;
extends 'Animal';

override 'make_noise' => sub { print "Meow!"; }


It simply does not work. You will get some fucking strange message that the first override in the Animal::Cat failed (TODO: Add here the real message...) and I can understand why: In order to compile the Animal class it has to use (=load and compile) the Animal::Dog and Animal::Cat.

Any way around it? To use something like Factory design pattern known from Java. But, does it really mean I can not use any subclass in a superclass with Moose? Is it a feature or a bug?:-)

Thursday, September 8, 2011

Build environment for OpenIndiana

Just a quick post - I wanted to build few things like Nagios NRPE, few Perl modules with C bindings etc. So I needed to install toolchain, compiler (rather Sun Studio than gcc) etc.

The easy way is:

pkg install SUNWcvs SUNWsvn SUNWarc SUNWj6dmo SUNWj6dev SUNWj6dmx SUNWj6dvx \
SUNWj6cfg SUNWj6rtx SUNWj6man SUNWgnu-automake-19 SUNWgnu-automake-110 SUNWaconf \
SUNWmercurial SUNWlibtool ss-dev SUNWsfwhea SUNWhea SUNWxwinc SUNWxorg-headers \
SUNWi2cs SUNWgpch SUNWgnome-common-devel SUNWgmake SUNWbison SUNWflexlex

pkg install SUNWcvs SUNWsvn SUNWarc SUNWj6dmo SUNWj6dev SUNWj6dmx SUNWj6dvx \
SUNWj6cfg SUNWj6rtx SUNWj6man SUNWgnu-automake-19 SUNWgnu-automake-110 SUNWaconf \
SUNWmercurial SUNWlibtool ss-dev SUNWsfwhea SUNWhea SUNWxorg-headers SUNWi2cs \
SUNWgpch SUNWgnome-common-devel SUNWgmake SUNWbison SUNWflexlex

pkg install developer/sunstudio12u1

Saturday, August 27, 2011

Test driven service operation vs. Nagios et al.

I am constantly thinking about network and server outages handling. This was my focus few years ago, I worked as an admin/op in a small hosting company and I was bombarded by SMSes from our home-brewed server/service monitoring system written in Perl. The system has bunch of drawbacks so we decided to replace it by Nagios (3 dot something). It has drawbacks as well, and I would say that even more serious in some cases.

I was responsible for the Nagios migration but I am not an op anymore so I have less experience with that. From my point of view, being aware of my limited insight, I can describe some drawbacks in Nagios 3 (= Nagios Core, simply the OSS version you get when apt-get install Nagios3 on top of Debian...).

The first one and probably the most severe: The configuration is complicated by nature. In addition Debian forces/strongly suggest some ideas about how you should write the config files. And when you try to do so, you have to read manual which does not give answers to all questions. For example: Are  multiple parents in dependency tree of hosts/services in AND, OR or whatever relation? You can find lots of small questions and eventually Google some answers, dig into documentation or whatever, but it takes time. Anyway, writing Nagios configs takes time and one should ask himself: Why? Of course you can use Swiss-made NConf to convert writing config into clicking configs in web interface, but it is not a real improvement. Why can't it be automatic? Let's say the system can auto-discover hosts and test-run all yet-know service tests. If some of tests results to OK, it can suggest that test. It should be able to clone hosts, categorize hosts, make exceptions etc. but on the other hand I prefer text config files over some sophisticated database schema...

The another thing is that sending alarms should be smarter than only triggering scripts like send mail to contacts and send messages to pagers or cell phones in modern days. Well, I like the idea of master alarm. I would like to have a possibility to set some alarms as not crucial for business/system operation and have them listed in web interface/reports and alarmed by less aggressive way to ops. I would like to have a possibility to have some threshold for sounding master alarm and then sending this master alarm (once or N-times but not overflowing ops with hundreds of different and probably correlated errors). And I would like to have a permissive and easy to use system, not a system which does not allow to acknowledge all reported errors and when not acknowledged, it bothers by SMSes over and over. I would like the system to accept my input and respect what I want or want not to save, not like Nagios->Acknowledge->Error: You have to write a comment. Wtf.? I have major network problem, I want to investigate what is going on and not writing stupid comments, especially in situation I do not know what to say, I just want to stop SMSes from bothering me.

I would like to have a monitoring cluster, able to monitor network/servers/services from more locations and give me a overall report. I would like to have a possibility to write own triggers on errors/warnings, to report more complex situations. Let's say that I have a cluster of 10 servers with loadbalancing and I know that 5 servers would be sufficient. I would make sense not to send alarm during nighttime when one of these 10 servers went down. But it make sense to send alarm if only 6 or less servers remains operational. Event more complex situations could be described and it would be nice to set this triggers easily.

And I would like to have a overview on my system. I want to see what is going on, what happened in past and write afterwards how did I solved the problem to have op's log and tip for next time.

I think that technically it should be relatively easy to run few thousands of test each minute on a decent Intel server. Not speaking about parallelization. Then it comes an idea: We have a paradigm/style/philosophy of test driven development. Why not to have a test driven system operation? I think there are two "contras": ComplexNess of configuration and complications with data acquisition and interpretation - i.e. people fears that it would be more complicated to answer a question "what is broken?". But I believe that both "con's" a only drawbacks of current software. Discussion will be appreciated.

Tuesday, July 12, 2011

Nagios NRPE @OpenIndiana

I tried to install $title and I have found excellent howto for (Open?)Solaris 10 here: http://www.utahsysadmin.com/2008/03/14/configuring-nagios-plugins-nrpe-on-solaris-10/

There is as catch which demonstrates itself by a fucking compilation error:
root@spagetka:/usr/src/nrpe-2.12# make
cd ./src/; make ; cd ..
make[1]: Entering directory `/usr/share/src/nrpe-2.12/src'
cc -g -I/usr/include/openssl -I/usr/include -DHAVE_CONFIG_H -o nrpe nrpe.c utils.c -L/usr/lib  -lssl -lcrypto -lnsl -lsocket   
nrpe.c:
"nrpe.c", line 616: invalid source character: <0xffffffe2>
"nrpe.c", line 616: invalid source character: <0xffffff80>
"nrpe.c", line 616: invalid source character: <0xffffff9d>
"nrpe.c", line 616: invalid source character: <0xffffffe2>
"nrpe.c", line 616: invalid source character: <0xffffff80>
"nrpe.c", line 616: invalid source character: <0xffffff9d>
"nrpe.c", line 616: undefined symbol: authpriv
"nrpe.c", line 616: warning: improper pointer/integer combination: arg #2
"nrpe.c", line 618: invalid source character: <0xffffffe2>
"nrpe.c", line 618: invalid source character: <0xffffff80>
"nrpe.c", line 618: invalid source character: <0xffffff9d>
"nrpe.c", line 618: invalid source character: <0xffffffe2>
"nrpe.c", line 618: invalid source character: <0xffffff80>
"nrpe.c", line 618: invalid source character: <0xffffff9d>
"nrpe.c", line 618: undefined symbol: ftp
"nrpe.c", line 618: warning: improper pointer/integer combination: arg #2
"nrpe.c", line 1505: warning: initializer will be sign-extended: -1
"nrpe.c", line 1506: warning: initializer will be sign-extended: -1
"nrpe.c", line 1652: warning: initializer will be sign-extended: -1
"nrpe.c", line 1653: warning: initializer will be sign-extended: -1
cc: acomp failed for nrpe.c
utils.c:
make[1]: *** [nrpe] Error 1
make[1]: Leaving directory `/usr/share/src/nrpe-2.12/src'

*** Compile finished ***

So you have to edit the nrpe.c according to linked howto.

Monday, July 11, 2011

Compiling DBD::mysql @OpenIndiana

I tried to compile DBD::mysql module from CPAN on oi_148 by SunStudio cc using commands:

export PATH="$PATH:/usr/mysql/5.1/bin/amd64"
perl -MCPAN -e shell
cpan[1]> install DBD::mysql

I got:

cpan[1]> install DBD::mysql
(...)
I will use the following settings for compiling and testing:

  cflags        (mysql_config) = -I/usr/mysql/5.1/include/mysql  -xprefetch=auto -xprefetch_level=3 -mt -fns=no -fsimple=1 -xbuiltin=%all -xlibmil -xlibmopt -xnorunpath -m64   -DHAVE_RWLOCK_T -DUNIV_SOLARIS
  embedded      (mysql_config) = 
  libs          (mysql_config) = -lrt -L/usr/mysql/5.1/lib/amd64/mysql -R/usr/mysql/5.1/lib/amd64/mysql -lmysqlclient -lz -lsocket -lnsl -lm
  mysql_config  (guessed     ) = mysql_config
  nocatchstderr (default     ) = 0
  nofoundrows   (default     ) = 0
  ssl           (guessed     ) = 0
  testdb        (default     ) = test
  testhost      (default     ) = 
  testpassword  (default     ) = 
  testsocket    (default     ) = 
  testuser      (guessed     ) = root

To change these settings, see 'perl Makefile.PL --help' and
'perldoc INSTALL'.

Checking if your kit is complete...
Looks good
Using DBI 1.616 (for perl 5.008004 on i86pc-solaris-64int) installed in /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/auto/DBI/
Writing Makefile for DBD::mysql
Writing MYMETA.yml and MYMETA.json
cp lib/DBD/mysql.pm blib/lib/DBD/mysql.pm
cp lib/DBD/mysql/GetInfo.pm blib/lib/DBD/mysql/GetInfo.pm
cp lib/DBD/mysql/INSTALL.pod blib/lib/DBD/mysql/INSTALL.pod
cp lib/Bundle/DBD/mysql.pm blib/lib/Bundle/DBD/mysql.pm
cc -c  -I/usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/auto/DBI -I/usr/mysql/5.1/include/mysql  -xprefetch=auto -xprefetch_level=3 -mt -fns=no -fsimple=1 -xbuiltin=%all -xlibmil -xlibmopt -xnorunpath -m64   -DHAVE_RWLOCK_T -DUNIV_SOLARIS -DDBD_MYSQL_INSERT_ID_IS_GOOD -g  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TS_ERRNO -xO3 -xspace -xildoff   -DVERSION=\"4.019\" -DXS_VERSION=\"4.019\" -KPIC "-I/usr/perl5/5.8.4/lib/i86pc-solaris-64int/CORE"   dbdimp.c
/usr/perl5/5.8.4/bin/perl -p -e "s/~DRIVER~/mysql/g" /usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/auto/DBI/Driver.xst > mysql.xsi
/usr/perl5/5.8.4/bin/perl /usr/perl5/5.8.4/lib/ExtUtils/xsubpp  -typemap /usr/perl5/5.8.4/lib/ExtUtils/typemap  mysql.xs > mysql.xsc && mv mysql.xsc mysql.c
Warning: duplicate function definition 'do' detected in mysql.xs, line 242
Warning: duplicate function definition 'rows' detected in mysql.xs, line 749
cc -c  -I/usr/perl5/site_perl/5.8.4/i86pc-solaris-64int/auto/DBI -I/usr/mysql/5.1/include/mysql  -xprefetch=auto -xprefetch_level=3 -mt -fns=no -fsimple=1 -xbuiltin=%all -xlibmil -xlibmopt -xnorunpath -m64   -DHAVE_RWLOCK_T -DUNIV_SOLARIS -DDBD_MYSQL_INSERT_ID_IS_GOOD -g  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TS_ERRNO -xO3 -xspace -xildoff   -DVERSION=\"4.019\" -DXS_VERSION=\"4.019\" -KPIC "-I/usr/perl5/5.8.4/lib/i86pc-solaris-64int/CORE"   mysql.c
"mysql.xs", line 687: warning: implicit function declaration: mysql_st_next_results
"mysql.xs", line 897: warning: implicit function declaration: is_prefix
Running Mkbootstrap for DBD::mysql ()
chmod 644 mysql.bs
rm -f blib/arch/auto/DBD/mysql/mysql.so
LD_RUN_PATH="/lib:/usr/mysql/5.1/lib/amd64/mysql" /usr/perl5/5.8.4/bin/perl myld cc  -G dbdimp.o mysql.o  -o blib/arch/auto/DBD/mysql/mysql.so  \
           -lrt -L/usr/mysql/5.1/lib/amd64/mysql -R/usr/mysql/5.1/lib/amd64/mysql -lmysqlclient -lz -lsocket -lnsl -lm          \
          
ld: fatal: file dbdimp.o: wrong ELF class: ELFCLASS64
ld: fatal: file processing errors. No output written to blib/arch/auto/DBD/mysql/mysql.so
make: *** [blib/arch/auto/DBD/mysql/mysql.so] Error 1
  CAPTTOFU/DBD-mysql-4.019.tar.gz
  /usr/gnu/bin/make -- NOT OK
'YAML' not installed, will not store persistent state
Running make test
  Can't test without successful make
Running make install
  Make had returned bad status, install seems impossible
Failed during this command:
 CAPTTOFU/DBD-mysql-4.019.tar.gz              : make NO

The problem was that I mixed up 64-bit AMD64 MySQL libraries with stock 32-bit Perl binaries. Solution was to omit amd64 from path:

export PATH="$PATH:/usr/mysql/5.1/bin"
perl -MCPAN -e shell
cpan[1]> install DBD::mysql

Wednesday, July 6, 2011

Telling OpenIndiana firewall to accept my rules

Let's say that I have my own rules:-) For firewall of course. And I want them to be set to kernel instead of automatically generated rules by services, FEA, or whatever.

Then I have to prepare rules into files: /etc/ipf/ipf.conf (rules for IPv4) and /etc/ipf/ipf6.conf (for IPv6).

For example it can be something like this:

/etc/ipf/ipf.conf
#################### top section #####################
block in all
pass in quick on lo0 all
#################### end of top section #####################

# special rules here

########################## default policy ################################

pass out all keep state
pass out proto icmp all

pass in proto tcp from any to any port = ssh keep state

# Munin & Nagios
pass in proto tcp from any to any port = 4949 keep state
pass in proto tcp from any to any port = 5666 keep state

pass in proto icmp all

# Traceroute
pass in proto udp from any to any port 33433 >< 33626 keep state

/etc/ipf/ipf6.conf
#################### top section #####################
block in log all
pass in quick on lo0 all
pass out all keep state
pass out proto ipv6-icmp all
#################### end of top section #####################

pass in proto tcp from any to any port = ssh keep state
pass in proto ipv6-icmp all
pass in proto udp from any to any port 33433 >< 33626 keep state

Then you have to change service parametes to accept the files:
svccfg -s network/ipfilter:default setprop firewall_config_default/policy = astring: custom
svccfg -s network/ipfilter:default setprop firewall_config_default/custom_policy_file = astring: "/etc/ipf/ipf.conf" 
svcadm refresh network/ipfilter
or
svcadm enable network/ipfilter

And finaly verify that rules are present:
ipfstat -nio
ipfstat -nio6

Saturday, June 25, 2011

OpenIndiana Build 148 @ VMWare ESXi 4.1

I have been trying to run an OpenIndiana server in a virtualized envrionment (sort of a development and test machine). And I ran into problems, actually into one huge fucking problem at the very beginning: The OI installer CD image does not boot inside VMWare virtual server when you have more than 4 virtual CPUs set in the virtual machine profile.

Easy solution: Set only 2 CPUs and everything works fine. Well, not everything... The Open Indiana fucking installer freezes in the end instead of restarting the system, but it worked for me after all.

My next article: Why do we virtualize? What the fuck virtualization brings to our lives and what it takes? And why do we suffer so miserably with commodity servers...?:-)