Home

Because of gcc and make package, can led to vulnerability of our system so we need to avoid to in stall these package but anyway some fluentd’s plugins require to gcc and make to compile itself during installation. So today I have some trick to help you install those plugins with out install gcc and make packages.

**Please keep in mind, compiled plugin package must be run on the ruby same version.

We have to 2 steps to do

  1. Build compiled .gem package from compiler machine.
  2. Install compiled .gem package which got from step 1 in the target machine.

Let’s start

OS : Centos 6.8 x86_64 and td-agent version 2.3.1-0.el6.x86_64

Step 1 : Build compiled .gem package from compiler machine

1.1 ) Download fluentd package from official site and install to compiler machine

# mkdir -p /fluentd/fluentd-plugins && cd /fluentd/fluentd-plugins
# wget http://packages.treasuredata.com.s3.amazonaws.com/2/redhat/6/x86_64/td-agent-2.3.1-0.el6.x86_64.rpm
# rpm -ivh td-agent-2.3.1-0.el6.x86_64.rpm

1.2) Install Development packages

# yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
# yum install -y gcc make

1.3) Install gem compiler

# /opt/td-agent/embedded/bin/fluent-gem install gem-compiler -v 0.5.0

OK, We have all we need so in my case, I will pick up some fluentd plugin that I used to faced the problem, it need gcc and make. its name : fluent-plugin-concat

1.4) Download dependency packages for fluent-plugin-concat

# mkdir /fluentd/fluent-plugin-concat && cd /fluentd/fluent-plugin-concat
# /opt/td-agent/embedded/bin/fluent-gem fetch cool.io -v 1.4.5 –platform=ruby
# /opt/td-agent/embedded/bin/fluent-gem fetch msgpack -v 1.0.2 –platform=ruby
# /opt/td-agent/embedded/bin/fluent-gem fetch strptime -v 0.1.8 –platform=ruby
# /opt/td-agent/embedded/bin/fluent-gem fetch serverengine -v 2.0.0 –platform=ruby
# /opt/td-agent/embedded/bin/fluent-gem fetch fluentd -v 0.14.9 –platform=ruby
# /opt/td-agent/embedded/bin/fluent-gem fetch fluent-plugin-concat -v 2.0.0 –platform=ruby

** fluent-gem is the representative of gem command of ruby but it was designed for working with fluentd

1.5) Compile dependency packages for fluent-plugin-concat, in this case there are 3 dependency packages need to be compiled (cool.io,msgpack,strptime)

So what are you waiting for? let’s compile it.

# cd /fluentd/fluent-plugin-concat
# /opt/td-agent/embedded/bin/fluent-gem compile cool.io-1.4.5.gem
# /opt/td-agent/embedded/bin/fluent-gem compile msgpack-1.0.2.gem
# /opt/td-agent/embedded/bin/fluent-gem compile strptime-0.1.8.gem

After compiled, you will get 3 new gem files as listed below

cool.io-1.4.5-x86_64-linux.gem
msgpack-1.0.2-x86_64-linux.gem
strptime-0.1.8-x86_64-linux.gem

** the postfix, -x86_64-linux is depend on your machine architecture.

Tips : the compiled gem file able to use with both centos 6 and centos 7 but the ruby version must be same version.

1.6) Deleted unused files and compress file for easy to transfer

# cd /fluentd/fluent-plugin-concat
# rm -rf cool.io-1.4.5.gem msgpack-1.0.0.gem strptime-0.1.8.gem
# cd /fluentd
# tar -zcvf fluent-plugin-concat.tar.gz fluent-plugin-concat

Step 2 : Install compiled .gem package which got from step 1 in the target machine.

2.1 ) Download fluentd package from official site and install to compiler machine

# mkdir -p /fluentd/fluentd-plugins && cd /fluentd/fluentd-plugins
# wget http://packages.treasuredata.com.s3.amazonaws.com/2/redhat/6/x86_64/td-agent-2.3.1-0.el6.x86_64.rpm
# rpm -ivh td-agent-2.3.1-0.el6.x86_64.rpm

2.2) Make fluentd directory, Copy fluent-plugin-concat.tar.gz and extract to this directory

# mkdir /fluentd
# tar -zxvf fluent-plugin-concat.tar.gz

2.3) Install plugins

# cd /fluentd/fluent-plugin-concat
# /opt/td-agent/embedded/bin/fluent-gem install –local fluent-plugin-concat-2.0.0.gem

Tips : If you locally install and there are require dependecy packages in the same directory of main plugin, fluentd will be detect and install dependency packages automatically.

2.4) Verify installed plugin

# /opt/td-agent/embedded/bin/fluent-gem list

Enjoy!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s