From Fedora Project Wiki

< SIGs‎ | bigdata‎ | packaging

mNo edit summary
 
(28 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Hive =
== Overview ==
From the [http://hive.apache.org/ project site]: "Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL."
From the [http://hive.apache.org/ project site]: "Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL."


The Fedora Big Data SIG is investigating the requirements to adapt the latest version of Hive as a package in Fedora, now that [https://koji.fedoraproject.org/koji/packageinfo?packageID=16841 Hadoop 2.x has been packaged]. Although Hive obviously has a significant dependency on Hadoop, the Java project is not Maven-based and instead is built using Ant and Ivy. The [[:Packaging:Java]] xmvn tooling support in Fedora does not directly apply to the Hive build. In many ways this can be viewed as a simplification instead of a challenge since one can configure a local file-system Ivy resolver relatively easily.
The Fedora Big Data SIG is investigating the requirements to adapt the latest version of Hive as a package in Fedora, now that [https://koji.fedoraproject.org/koji/packageinfo?packageID=16841 Hadoop 2.x has been packaged]. Although Hive obviously has a significant dependency on Hadoop, the Java project is not Maven-based and instead is built using Ant and Ivy. The [[:Packaging:Java|xmvn tooling]] support in Fedora does not directly apply to the Hive build. In many ways this can be viewed as a simplification instead of a challenge since one can configure a local file-system Ivy resolver relatively easily.


Using static build-derived analysis (Ant doesn't really provide something like the Maven dependency plugin), there are a group of dependencies that are currently missing from Fedora which block the build of Hive using Fedora-only installed versions. There are also many dependencies available which are not necessarily version-compatible. However, like the [[:Changes/Hadoop]] outline, those can hopefully be mitigated in the Hive source where possible.
Using static build-derived analysis (Ant doesn't really provide something like the Maven dependency plugin), there are a group of dependencies that are currently missing from Fedora which block the build of Hive using Fedora-only installed versions. There are also many dependencies available which are not necessarily version-compatible. However, like the [[:Changes/Hadoop|hadoop outline]], those can hopefully be mitigated in the Hive source where possible.


Version 0.11 is the latest release and built from source (using the Fedora Hadoop target of 2.0.5a) using:
== Build ==
Version 0.12.0 is the latest release and built from source (using the Fedora Hadoop target of 2.2.0) using:


<pre>ant very-clean package -Dhadoop.version=2.0.5-alpha -Dhadoop-0.23.version=2.0.5-alpha -Dhadoop.mr.rev=23 -DenhanceModel.notRequired=true</pre>
<pre>ant very-clean package -Dhadoop.version=2.2.0 -Dhadoop-0.23.version=2.2.0 -Dhadoop.mr.rev=23 -DenhanceModel.notRequired=true -Dmvn.hadoop.profile=hadoop23 -Dshims.include=0.23 -Dbuild.profile=core -Dthrift.home=/usr</pre>


Note that to do a local build using the [[#SCM|SIG branch]] you must make a directory to store any [[#Dependencies|(currently)]] unpackaged jars:
<pre>mkdir -p ~/hive/lib/missing</pre>
== Dependencies ==
The full Hive dependency list is captured [http://pmackinn.fedorapeople.org/tattletale/hive/hive-0.11.0/dependson/index.html here] but the following table outlines the missing dependencies. The ones in '''bold''' are deemed hard dependencies and must be packaged.
The full Hive dependency list is captured [http://pmackinn.fedorapeople.org/tattletale/hive/hive-0.11.0/dependson/index.html here] but the following table outlines the missing dependencies. The ones in '''bold''' are deemed hard dependencies and must be packaged.


Line 16: Line 24:
  |-
  |-
  | avro-ipc, '''avro-mapred'''
  | avro-ipc, '''avro-mapred'''
  | '''<span style="color:red">Blocked</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|1009170}}
  | <strike>{{bz|1009170}}</strike>
  | [[User:ricardo|ricardo]]
  | [[User:ricardo|ricardo]]
  | Although avro 1.6.2 is packaged, it does not include the ipc and mapred jars. IPC appears to only apply to 0.20 shim. MapRed is used by an Avro output format in QL. To complicate matters, 1.6.2 only supports the old mapred API.
  | Although avro 1.6.2 is packaged, it does not include the ipc and mapred jars. IPC appears to only apply to 0.20 shim. MapRed is used by an Avro reader/input/output feature in QL and is based on the legacy mapred API (i.e.,org.apache.hadoop.mapred).
|-
| commons-httpclient
| '''<span style="color:darkviolet">Available</span>'''
|
|
| Jakarta?
  |-
  |-
  | '''datanucleus-core'''
  | '''datanucleus-core'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  |  
  | <strike>{{bz|1011705}}</strike>
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | Forms backbone of metastore layer for different data sinks. Upstream project at http://www.datanucleus.org/
  | Forms backbone of metastore layer for different data sinks. Upstream project at http://www.datanucleus.org/
  |-
  |-
  | '''datanucleus-api-jdo'''
  | '''datanucleus-api-jdo'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  |  
  | <strike>{{bz|1011962}}</strike>
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | JDO implementation for datanucleus
  | JDO implementation for datanucleus
  |-
  |-
  | '''datanucleus-rdbms'''
  | '''datanucleus-rdbms'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  |  
  | <strike>{{bz|1011960}}</strike>
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | RDBMS plugin adapter for datanucleus
  | RDBMS plugin adapter for datanucleus
|-
| ftplet-api
| '''<span style="color:darkviolet">Available</span>'''
|
|
| Appears to only apply to 0.20 shim
|-
| ftpserver-core, ftpserver-deprecated
| '''<span style="color:darkviolet">Available</span>'''
|
|
| Appears to only apply to 0.20 shim
|-
| geronimo-j2ee-management_1.1_spec
| '''<span style="color:darkviolet">Available</span>'''
|
|
| ''Hopefully'' can substitute jboss-j2eemgmt-1.1-api package
  |-
  |-
  | hbase
  | hbase
  | '''<span style="color:blue">Active</span>'''
  | '''<span style="color:green">Complete</span>'''
  |  
  | <strike>{{bz|1045556}}</strike>
  | [[User:rrati|rrati]]
  | [[User:rrati|rrati]]
  | hbase-handler can be compiled out but seems like a significant omission  
  | hbase-handler can be compiled out but seems like a significant omission  
  |-
  |-
  | high-scale-lib
  | high-scale-lib
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|865893}}
  | <strike>{{bz|865893}}</strike>
  | [[User:gil|gil]]
  | [[User:gil|gil]]
  |
  |
|-
| httpclient, httpcore
| '''<span style="color:darkviolet">Available</span>'''
|
|
| Same as above...Jakarta?
  |-
  |-
  | '''javolution'''
  | '''javolution'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|1009153}}
  | <strike>{{bz|1009153}}</strike>
  | [[User:pmackinn|pmackinn]]
  | [[User:pmackinn|pmackinn]]
  | Used by the QL classes: a '''hard''' dependency
  | Used by the QL classes: a '''hard''' dependency
  |-
  |-
  | '''jdo-api'''
  | '''jdo-api'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|1011696}}
  | <strike>{{bz|1011696}}</strike>
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | [[User:pmackinn|pmackinn]],[[User:gil|gil]]
  | Dependency for datanucleus-api-jdo. CANNOT substitute existing jdo2-api.
  | Dependency for datanucleus-api-jdo. CANNOT substitute existing jdo2-api.
  |-
  |-
  | '''libthrift, libfb303'''
  | '''libthrift, libfb303'''
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|982285}}, {{bz|1000563}}
  | <strike>{{bz|982285}}, {{bz|1000563}}</strike>
  | [[User:willb|willb]]
  | [[User:willb|willb]]
  | Will Benton has some RPM artifacts at http://freevariable.com/thrift/
  | Will Benton has some RPM artifacts at http://freevariable.com/thrift/
  |-
  |-
  | metrics-core
  | metrics-core
  | '''<span style="color:orange">Review</span>'''
  | '''<span style="color:green">Complete</span>'''
  | {{bz|861502}}
  | <strike>{{bz|861502}}</strike>
  | [[User:gil|gil]]
  | [[User:gil|gil]]
  |
  |
  |-
  |-
  | pig
  | pig
  | '''<span style="color:darkviolet">Available</span>'''
  | '''<span style="color:orange">Review</span>'''
  |  
  | {{bz|1060277}}
  |  
  | [[User:pmackinn|pmackinn]]
  | Test and source imports of Pig classes, however they appear to be in the adapter space so may be able to defer.
  | Test and source imports of Pig classes, however they appear to be in the adapter space so may be able to defer.
|-
| stax-api
| '''<span style="color:darkviolet">Available</span>'''
|
|
| Need to sort out which of available stax-api packages could fit (bea, stax2)
  |-
  |-
  | tempus-fugit
  | tempus-fugit
Line 124: Line 96:
  |-
  |-
  |}
  |}
NB: This list is distilled from the overall set of missing dependencies but many of the ones that aren't listed are not required for the latest Fedora version of Hadoop (2.0.5a), assuming the appropriate [[#Build|build properties noted]] are specified.
== SCM ==
The BigData SIG is tracking a set of commits [https://github.com/fedora-bigdata/hive/tree/fedora-0.11 here] to build according to FPG. These will eventually be converted into a patch set for a spec file once all the outstanding missing dependencies are in place. These commits include a set of custom Ivy resolvers that only inspect the local filesystem in typical Fedora Java jar locations. A [https://bugzilla.redhat.com/show_bug.cgi?id=1012612 RFE] was created to make Fedora Ivy map dependencies into the local filesystem implicitly, thus doing away with custom resolvers (as much as feasible).

Latest revision as of 15:02, 19 February 2014

Hive

Overview

From the project site: "Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL."

The Fedora Big Data SIG is investigating the requirements to adapt the latest version of Hive as a package in Fedora, now that Hadoop 2.x has been packaged. Although Hive obviously has a significant dependency on Hadoop, the Java project is not Maven-based and instead is built using Ant and Ivy. The xmvn tooling support in Fedora does not directly apply to the Hive build. In many ways this can be viewed as a simplification instead of a challenge since one can configure a local file-system Ivy resolver relatively easily.

Using static build-derived analysis (Ant doesn't really provide something like the Maven dependency plugin), there are a group of dependencies that are currently missing from Fedora which block the build of Hive using Fedora-only installed versions. There are also many dependencies available which are not necessarily version-compatible. However, like the hadoop outline, those can hopefully be mitigated in the Hive source where possible.

Build

Version 0.12.0 is the latest release and built from source (using the Fedora Hadoop target of 2.2.0) using: 

ant very-clean package -Dhadoop.version=2.2.0 -Dhadoop-0.23.version=2.2.0 -Dhadoop.mr.rev=23 -DenhanceModel.notRequired=true -Dmvn.hadoop.profile=hadoop23 -Dshims.include=0.23 -Dbuild.profile=core -Dthrift.home=/usr

Note that to do a local build using the SIG branch you must make a directory to store any (currently) unpackaged jars: 

mkdir -p ~/hive/lib/missing

Dependencies

The full Hive dependency list is captured here but the following table outlines the missing dependencies. The ones in bold are deemed hard dependencies and must be packaged.

Missing/Questionable Dependencies
Project State Review BZ Packager Notes
avro-ipc, avro-mapred Complete RHBZ #1009170 ricardo Although avro 1.6.2 is packaged, it does not include the ipc and mapred jars. IPC appears to only apply to 0.20 shim. MapRed is used by an Avro reader/input/output feature in QL and is based on the legacy mapred API (i.e.,org.apache.hadoop.mapred).
datanucleus-core Complete RHBZ #1011705 pmackinn,gil Forms backbone of metastore layer for different data sinks. Upstream project at http://www.datanucleus.org/
datanucleus-api-jdo Complete RHBZ #1011962 pmackinn,gil JDO implementation for datanucleus
datanucleus-rdbms Complete RHBZ #1011960 pmackinn,gil RDBMS plugin adapter for datanucleus
hbase Complete RHBZ #1045556 rrati hbase-handler can be compiled out but seems like a significant omission
high-scale-lib Complete RHBZ #865893 gil
javolution Complete RHBZ #1009153 pmackinn Used by the QL classes: a hard dependency
jdo-api Complete RHBZ #1011696 pmackinn,gil Dependency for datanucleus-api-jdo. CANNOT substitute existing jdo2-api.
libthrift, libfb303 Complete RHBZ #982285, RHBZ #1000563 willb Will Benton has some RPM artifacts at http://freevariable.com/thrift/
metrics-core Complete RHBZ #861502 gil
pig Review RHBZ #1060277 pmackinn Test and source imports of Pig classes, however they appear to be in the adapter space so may be able to defer.
tempus-fugit Review RHBZ #1009654 gil Concurrency library. May only be a test dep. Upstream at http://tempusfugitlibrary.org/

NB: This list is distilled from the overall set of missing dependencies but many of the ones that aren't listed are not required for the latest Fedora version of Hadoop (2.0.5a), assuming the appropriate build properties noted are specified.

SCM

The BigData SIG is tracking a set of commits here to build according to FPG. These will eventually be converted into a patch set for a spec file once all the outstanding missing dependencies are in place. These commits include a set of custom Ivy resolvers that only inspect the local filesystem in typical Fedora Java jar locations. A RFE was created to make Fedora Ivy map dependencies into the local filesystem implicitly, thus doing away with custom resolvers (as much as feasible).

Retrieved from "https://fedoraproject.org/w/index.php?title=SIGs/bigdata/packaging/Hive&oldid=370320"