66133

Автор(ы): 

Автор(ов): 

2

Параметры публикации

Тип публикации: 

Доклад

Название: 

Implementing Big Data Processing Workflows using Open Source Technologies

ISBN/ISSN: 

978-3-902734-22-8, 1726-9679

DOI: 

10.2507/30th.daaam.proceedings.054

Наименование конференции: 

  • 30th DAAAM International Symposium on Intelligent Manufacturing and Automation (Vienna, Austria, 2019)

Наименование источника: 

  • Proceedings of the 30th DAAAM International Symposium on Intelligent Manufacturing and Automation (Vienna, Austria, 2019)

Город: 

  • Vienna, Austria

Издательство: 

  • DAAAM International

Год издания: 

2019

Страницы: 

0394-0404
Аннотация
In our implementationresearch,we applyworkflow approachto the modeling and development ofthe Big Data processing pipeline usingopen source technologies. The data processing workflow is a set of interrelatedsteps which launch some particular jobssuch as Spark job, shell job or Postgre SQL command. All workflow steps are chained to form integrated process and imitate the data load from staging storage area to the datamart storage area. The experimental workflow-basedimplementationof a data processing pipeline was performed thatstagesthrough different storage areas and uses actual industrial KPIdatasetof some 30 millions records. Evaluation of implementation results provides proofs ofthe applicability of proposed workflow toother application domains and datasets which should satisfy the data format at input stage of the workflow.

Библиографическая ссылка: 

Сулейкин А.С., Панфилов П. Implementing Big Data Processing Workflows using Open Source Technologies / Proceedings of the 30th DAAAM International Symposium on Intelligent Manufacturing and Automation (Vienna, Austria, 2019). Vienna, Austria: DAAAM International, 2019. С. 0394-0404.