Skip to content

jemygraw/fusion-hadoop

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

hdfs

<property>
    <name>fs.qiniu.access.key</name>
    <value>ACCESS KEY</value>
</property>
<property>
    <name>fs.qiniu.secret.key</name>
    <value>SECRET KEY</value>
</property>

使用方式:

$ hdfs dfs -ls fusion://img.abc.com/2017-05-22/
$ hdfs dfs -ls fusion://img.abc.com/2017-05-22/22/
$ hdfs dfs -ls fusion://img.abc.com/2017-05-22/22/part-00000.gz

spark

spark-shell --jars $HADOOP_HOME/qiniu-lib/fusion-hadoop-1.0.0.jar
sc.setLogLevel("INFO")
sc.hadoopConfiguration.set("fs.qiniu.access.key","ACCESS KEY")
sc.hadoopConfiguration.set("fs.qiniu.secret.key","SECRET KEY")
val textFile=sc.textFile("fusion://if-pbl.qiniudn.com/2017-05-22/16/part-00000.gz")
textFile.count()

About

implements the hdfs interface to access fusion logs

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published