1 August 2018 Palo Alto, California — Squadex, a Silicon Valley-based Cloud Transformation Consulting company, is delighted to announce that we organized and hosted the second local meetup of AWS User Group on July 20, 2018.

In a bid to increase engineers’ knowledge and skills of Amazon Web Services (AWS), a world’s leading cloud computing service, Squadex continued to invite industry heavyweights to participate in the event.

At the second meetup, Giuseppe Angelo Porcelli, Solutions Architect at Amazon Web Services, and Stepan Pushkarev, CTO at Hydrosphere.io, provided their perspective on Amazon Web Services and pondered why its cloud computing capabilities should be used by data scientists and engineers when dealing with Big Data and Artificial Intelligence problems.


“AWS Big Data and AI Services” by Giuseppe Angelo Porcelli

Angelo kicked off his keynote with trends. According to Gartner research, Big Data, Machine Learning, Artificial Intelligence, and Cloud are the top strategic technology trends that will impact most organizations in 2018.

While enterprises start to recognize the value of Big Data, ML, and AI, challenges of their application are many:

  • 80% of time managing data is allocated for analysis
  • 50% of working models go to production
  • 28% of companies generate business value from data

That being said, data scientists should find the ways how to work more efficiently and how to showcase tangible results to the businesses.

In that context, building a database for Big Data and Machine Learning purposes on AWS may be the answer.

The AWS platform is agile, scalable, and secure. It provides engineers with broad capabilities to inject, store, analyze, and consume data at a relatively low cost. Additionally, AWS cloud enables experimentation, since any changes and fixes can be implemented quickly and easily.

Amazon has been utilizing Artificial Intelligence and Machine Learning in business for 20+ years. AI is used for product suggestions at Amazon.com; it manages robots in its warehouses, operates drones that deliver products, and automates Amazon Go stores. Alexa also relies on Artificial Intelligence.

In the meantime, AWS provides developers with services, platforms, and infrastructure. Some of the most notable are:

  • Services: Amazon Rekognition, Amazon Polly, Amazon Lex, Amazon Transcribe, Amazon Translate, Amazon Comprehend
  • Platforms: Amazon SageMaker, Amazon EMR, AWS DeepLens
  • Infrastructure: AWS Greengrass ML Inference, AWS Deep Learning AMI, and Amazon EC2 P3 Instances

To sum it up: Cloud Computing is the future of Big Data and Machine Learning. It allows to facilitate and improve Data Management, which provides enterprises with new dimensions of service quality, interoperability, and business process change.

“Spark Microservices on AWS” by Stepan Pushkarev

Apache Spark is one of the most well-studied batch big data processing engines. It is widely used for machine learning, since it allows data scientists to efficiently solve data problems and graph computations.

Spark can be reinforced with Microservices, with the solution moved to AWS. It can assist data scientists in solving complex data problems because it supports on-demand requesting, multi-tenancy, and event-driven data processing.

However, the solution as such is not easy to implement. The major challenges of Spark Microservices are as follows:

  • Spark Ops can be challenging (Ops heavy)
  • REST and Messaging API to trigger Spark jobs and consume results are not supported
  • No OLS/framework to define the API
  • A single context/session engine

To resolve those issues, Stepan designed a solution (EMR + Mist) and demonstrated its demo.

Finishing the demo, Stepan explained how the EMR + Mist solution could be used to implement:

  • Smart Spark configs managed by Mist
  • Smart EMR configs
  • EMR lifecycle management
  • Cloud formations templates

To sum it up: The proposed solution allows to accelerate machine learning to production, which benefits both data scientists and businesses in the long run.

Squadex will continue to organize meetups of AWS User Group, aiming to increase awareness about Amazon Web Services and share their use cases, solutions, and applications.