[Hadoop] Flume + Hbase 연동하기

Flume ver : 1.4.0

HBase Ver : 0.94.1

Hadoop ver: : 1.2.1

시나리오 

 - NameNode : 192.168.1.2

 - FlumeAgent : 192.168.1.5

 - Flume Agent에서 Tail로 로그를 받아와 NameNode의 HBase로 저장한다. 

 - NameNode와 Agent와 통신은 Avro 프로토콜을 이용한다.

 - HBase Table 명 : demo (미리 만들어줌)

 - Hbase ColumnFamily 명 : cf

Flume Master Configuration(flume-hbase.conf)

master.sources = node1

master.channels = mem-channel

master.sinks = hbaseSink

# For each one of the sources, the type is defined

master.sources.node1.type = avro

master.sources.node1.bind = 192.168.1.2

master.sources.node1.port = 61616

# The channel can be defined as follows.

master.sources.node1.channels = mem-channel

# Each sink's type must be defined

master.sinks.hbaseSink.type = hbase

master.sinks.hbaseSink.table = demo

master.sinks.hbaseSink.columnFamily = cf

master.sinks.hbaseSink.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializer

master.sinks.hbaseSink.serializer.payloadColumn = col1

#Specify the channel the sink should use

master.sinks.hbaseSink.channel = mem-channel

# Each channel's type is defined.

master.channels.mem-channel.type = memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

master.channels.mem-channel.capacity = 100000

master.channels.mem-channel.transactionCapacity = 10000

Flume Agent Configuration(flume-hbase.conf)

agent.sources = node04

agent.channels = mem-channel

agent.sinks = avroSink

# For each one of the sources, the type is defined

agent.sources.node04.type = exec

agent.sources.node04.command = tail -F /var/log/message

agent.sources.node04.batchSize = 5000

# The channel can be defined as follows.

agent.sources.node04.channels = mem-channel

# Each sink's type must be defined

agent.sinks.avroSink.type = avro

agent.sinks.avroSink.hostname = 192.168.1.2

agent.sinks.avroSink.port = 61616

#Specify the channel the sink should use

agent.sinks.avroSink.channel = mem-channel

# Each channel's type is defined.

agent.channels.mem-channel.type = memory

# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

agent.channels.mem-channel.capacity = 100000

agent.channels.mem-channel.transactionCapacity = 10000

Run

master : #flume-ng agent -n master -c /usr/local/flume/conf/ -f /usr/local/flume/conf/flume-hbase.conf

agent : #flume-ng agent -n agent -c /usr/local/flume/conf/ -f /usr/local/flume/conf/flume-hbase.conf