Hadoop:  challenge  accepted!

Arkadiusz  Osiński arkadiusz.osinski@allegrogroup.com

Robert  Mroczkowski robert.mroczkowski@allegrogroup.com

ToC -­‐‑   Hadoop  basics -­‐‑   Gather  data -­‐‑   Process  your  data -­‐‑   Learn  from  your  data -­‐‑   Visualize  your  data

BigData -­‐‑  Petabytes  of  (un)structured  data

BigData -­‐‑  Petabytes  of  (un)structured  data -­‐‑   12%  of  data  is  analyzed

BigData -­‐‑  Petabytes  of  (un)structured  data -­‐‑   12%  of  data  is  analyzed -­‐‑   a  lot  of  data  is  not  gathered

BigData -­‐‑  Petabytes  of  (un)structured  data -­‐‑   12%  of  data  is  analyzed -­‐‑   a  lot  of  data  is  not  gathered -­‐‑   how  to  gain  knowledge?

Power Big  Data

Data  Lake

Scalability

Petabytes

Mapreduce Commodity

HDFS -­‐‑   Storage  layer

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system -­‐‑   Commodity  hardware

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system -­‐‑   Commodity  hardware -­‐‑   Scalability

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system -­‐‑   Commodity  hardware -­‐‑   Scalability -­‐‑   JBOD

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system -­‐‑   Commodity  hardware -­‐‑   Scalability -­‐‑   JBOD -­‐‑   Access  control

HDFS -­‐‑   Storage  layer -­‐‑   Distributed  file  system -­‐‑   Commodity  hardware -­‐‑   Scalability -­‐‑   JBOD -­‐‑   Access  control -­‐‑   No  SPOF

YARN -­‐‑   Distributed  computing  layer

YARN -­‐‑   Distributed  computing  layer -­‐‑   Operations  in  place  of  data

YARN -­‐‑   Distributed  computing  layer -­‐‑   Operations  in  place  of  data -­‐‑   MapReduce…

YARN -­‐‑   Distributed  computing  layer -­‐‑   Operations  in  place  of  data -­‐‑   MapReduce… -­‐‑   and  others  applications

YARN -­‐‑   Distributed  computing  layer -­‐‑   Operations  in  place  of  data -­‐‑   MapReduce… -­‐‑   and  others  applications -­‐‑   Resource  management

Let’s  squize  our  data  to  get  a  juice!!

Gather  data flume-twitter.sources.Twitter.type = com.cloudera.flume.source.TwitterSource flume-twitter.sources.Twitter.channels = MemChannel flume-twitter.sources.Twitter.consumerKey = (…) flume-twitter.sources.Twitter.consumerSecret = (…) flume-twitter.sources.Twitter.accessToken = (…) flume-twitter.sources.Twitter.accessTokenSecret = (…) flume-twitter.sources.Twitter.keywords = hadoop, big data, nosql

Process  your  data -­‐‑   Hadoop  Streaming!

Process  your  data -­‐‑   Hadoop  Streaming! -­‐‑   No  need  to  write  code  in  Java

Process  your  data -­‐‑   Hadoop  Streaming! -­‐‑   No  need  to  write  code  in  Java -­‐‑   You  can  use  Python,  Perl  or  Awk

Process  your  data #!/usr/bin/python import sys import json import datetime as dt keyword='hadoop' for line in sys.stdin: data = json.loads(line.strip()) if keyword in data['text'].lower(): dt=dt.datetime.strptime(data['created_at'], '%a %b %d %H:%M:%S +0000 %Y').strftime('%Y-%m-%d') print '{0}\t1'.format(str(dt))    

Process  your  data #!/usr/bin/python import sys (counter,datekey=(0,'') for line in sys.stdin: line = line.strip().split("\t") if datekey != line[0]: if datekey: print "{0}\t{1}".format(str(datekey),str(counter)) datekey = line[0] counter = 1 else: counter += 1  print "{0}\t{1}".format(str(datekey),str(counter))    

Process  your  data yarn jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \

-files ./map.py,./reduce.py \

-mapper ./map.py \

-reducer ./reduce.py \

-input /tweets/2014/04/*/*/* \

-input /tweets/2014/05/*/*/* \

-output /tweet_keyword

Process  your  data (….) 2014-04-24 864 2014-04-25 1121 2014-04-26 593 2014-04-27 649 2014-04-28 1084 2014-04-29 1575 2014-04-30 1170 2014-05-01 1164 2014-05-02 1175 2014-05-03 779 2014-05-04 471 (….)

Process  your  data

Recommendations

Which  product  will  be  desired  by  client?

We’ve  got  historical  users  interaction  with  items.

Simple  Example Let’s  just  do  mahout    -­‐‑  it’s  easy!

> apt-get install mahout

> cat simple_example.csv

1,101

1,102

1,103

2,101

> hdfs dfs -put simple_example.csv

> mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -b \

-Dmapred.input.dir=/mahout/input/wikilinks/simple_example.csv \

-Dmapred.output.dir=/mahout/output/wikilinks/simple_example \

-Dmapred.job.queue.name=atmosphere_prod

Simple  Example Tadadam!

> hdfs dfs –text /mahout/output/wikilinks/simple_example/part-r-00000.snappy 1 [105:1.0,104:1.0] 2 [106:1.0,105:1.0] 3 [103:1.0,102:1.0] 4 [105:1.0,102:1.0] 5 [107:1.0,106:1.0]

Wiki  Case

We’ve  got  links  between  wikipedia  articles,  and  want  to  propose  new  links  between  articles.

„Wikipedia   (i/ˌwɪkɨˈpiːdiəә/   or   i/ˌwɪkiˈpiːdiəә/   WIK-­‐‑i-­‐‑PEE-­‐‑dee-­‐‑əә)   is   a   collaboratively   edited,  multilingual,   free   Internet   encyclopedia   that   is   supported   by   the   non-­‐‑profit  Wikimedia   Foundation.   Volunteers   worldwide   collaboratively   write   Wikipedia'ʹs   30   million  articles  in  287  languages,  including  over  4.5  million  in  the  English  Wikipedia.  Anyone  who  can  access”  

Wiki  Case

Wiki  Case

hlp://users.on.net/%7Ehenry/pagerank/links-­‐‑simple-­‐‑sorted.zip

#!/usr/bin/awk -f BEGIN { OFS=",”; } { gsub(":","",$1); for (i=2;i<=NF;i++) { print $1,$i } }  

Wiki  Case

yarn jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \

-Dmapreduce.job.max.split.locations=24 \

-Dmapreduce.job.queuename=hadoop_prod \

-Dmapred.output.key.comparator.class=mapred.lib.KeyFieldBasedComparator \

-Dmapred.text.key.comparator.options=-n \

-Dmapred.output.compress=false \

-files ./mahout/mapper.awk \

-mapper ./mapper.awk \

-input /mahout/input/wikilinks/links-simple-sorted.txt \

-output /mahout/output/wikilinks/fixedinput

Wiki  Case Mahout  lib  count’s  similarity  Matrix  and  gave  recommendations  for  824  articles.

What’s  important,  we  didn’t  gather  any  knowledge  a  priori  and  just  ran  algorithm’s  out  of  box.

Wiki  Case Acadèmia_Valenciana_de_la_Llengua

FIFA Valencia

October_1 Calendar

Prehistoric_Iberia Link  appears  recently

Ceuta Spain  City  at  the  north  coast  of  Africa

Roussillon Part  of  France  by  the  border  with  Spain

Sweden J

Turís municipality  in  the  Valencian  Community

Vulgar_Latin Language  article Western_Italo-­‐‑Western_languages Language  article

Àngel_Guimerà Spanish  wriler

Wiki  Case

Tweets

Let’s  find  group  of: •  tags   • users

Tweets

•  Our  data  is  not  random •  We’ve  picked  specific  keywords •  We’ll  do  analysis  in  two  

orthogonal  directions

Tweets {

"filter_level":"medium",

"contributors":null,

"text":"PROMOCIÓN MES DE MAYO. con ...",

"geo":null,

"retweeted":false,

"lang":"es",

"entities":{

"urls":[

{ "expanded_url":"http://www.agmuriel.com",

"indices":[ 69, 91 ],

"display_url":"agmuriel.com/#!-/c1gz",

"url":"http://t.co/APpPjRRTXn" } ]

}

(…)

 

Tweets #!/usr/bin/python import json, sys for line in sys.stdin: line = line.strip() if '"lang":"en"' in line: tweet = json.loads(line) try: text = tweet['text'].lower().strip() if text: tags = tweet[” entities"][”hashtags”] for tag in tags: print tag[“text”]+"\t"+text except KeyError: continue  

#!/usr/bin/python import sys (lastKey,text) = (None,"") for line in sys.stdin: (key,value) = line.strip().split("\t") if lastKey and lastKey != key: print lastKey+"\t"+text (lastKey,text) = (key,value) else: (lastKey,text) = (key,text+" "+value)  

Tweets

yarn jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \

-Dmapreduce.job.queuename=atmosphere_time \

-Dmapred.output.compress=false \

-Dmapreduce.job.max.split.locations=24 \

-D-Dmapred.reduce.tasks=20 \

-files ~/mahout/twitter_map.py,~/mahout/twitter_reduce.py \

-mapper ./twitter_map.py \

-reducer ./twitter_reduce.py \

-input /project/atmosphere/tweets/2014/04/*/* \

-output /project/atmosphere/tweets/output \

-outputformat org.apache.hadoop.mapred.SequenceFileOutputFormat

Get  SequenceFile  with  proper  mapping

Tweets

mahout seq2sparse \

-i /project/atmosphere/tweets/output \

-o /project/atmosphere/tweets/vectorized -ow \

-chunk 200 -wt tfidf -s 5 -md 5 -x 90 -ng 2 -ml 50 -seq -n 2

Calculate  vector  representation  for  text

{10:0.6292275202550768,14:0.7772211575566166}  {10:0.6292275202550768,14:0.7772211575566166}  {3:0.37796447439954967,14:0.37796447439954967,19:0.654653676423271,22:0.534522474858859}  {17:1.0}  {3:0.37796447439954967,14:0.37796447439954967,19:0.654653676423271,22:0.534522474858859}  

Tweets I’ts  time  to  begin  clusterization

Let’s  find  100  clusters

mahout kmeans \

-i /tweets_5/vectorized/tfidf-vectors \

-c /tweets_5/kmeans/initial-clusters \

-o /tweets_5/kmeans/output-clusters \

-cd 1.0 -k 100 -x 10 -cl –ow \

-dm org.apache.mahout.common.distance.SquaredEuclideanDistanceMeasure

Tweets Glance  at  results

BURN OPEN LEATHER FAT SOFTWARE WALLET WEIGHTLOSS LINUX MAN FITNESS UBUNTU ZUMBA OPENSUSE

PATCHING

Tweets

It  was  easy  because  tags  are  very  dependent  (coocurence).

Tweets Bigger  challenge  –  user  clustering

LINUX UBUNTU WINDOWS OS PATCH MAC HACKED MICROSOFT

FREE CSRRACING WON RACEYOURFRIENDS ANDROID CSRCLASSIC

Tweets Bigger  challenge  –  user  clustering

•  Results  show  that  dataset  is  strongly  curved  by  mobile  and  games

•  Dataset  wasn’t  random  –  we  subscribed    specific  keywords

•  OS  result  is  great!

Tweets HADOOP  WORLD

run  predictive  machine  learning  algorithms  on  hadoop  without  even  knowing  mapreduce.:  data  scientists  are  very...  h:p://t.co/gdmqm5g1ar

rt  @mapr:  google  cloud  storage  connector  for  #hadoop:  quick  start  guide  now  avail  h:p://t.co/17hxtvdlir    #bigdata

Tweets HADOOP  WORLD

Cloudera  wants  to  do  big  data  in  Real  Time.

Hortonworks  wants  to  replace  cloudera  by  research.

Visualize  data add jar hive-serdes-1.0-SNAPSHOT.jar; create table tw_data_201404 ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\012’ STORED AS TEXTFILE LOCATION ‘/tweets/tw_data_201404’ AS SELECT v_date, LOWER(hashtags.text), lang, COUNT(*) AS total_count FROM logs.tweets LATERAL VIEW EXPLODE(entities.hashtags) t1 AS hashtags WHERE v_date like '2014-04-%' GROUP BY v_date,LOWER(hashtags.text),lang    

Visualize  data add jar elasticsearch-hadoop-hive-2.0.0.RC1.jar; CREATE EXTERNAL TABLE es_export ( v_date string, tag string, lang string, total_count int, info string ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler’ TBLPROPERTIES ( 'es.resource' = 'trends/log', 'es.index.auto.create' = 'true') ;    

Visualize  data INSERT overwrite TABLE es_export SELECT distinct may.v_date,may.tag,may.lang,may.total_count,'nt' FROM tw_data_201405 may LEFT outer JOIN tw_data_201404 april ON april.tag = may.tag WHERE april.tag is null AND may.total_count>1;    

Visualize  data

Visualize  data Tag: eurovisiontve

Thank  you!

Questions?