Spark’s documentation details the method of running on a cluster managed by Kubernetes. But it mainly involves using spark-submit to directly submit Spark applications to Kubernetes cluster in cluster mode. This article will introduce the REPL environment for running spark-shell on a Kubernetes cluster in client mode.

The configuration shell script is given directly above. When the kubectl run is run, a Spark Client that keeps running will be generated. Finally, kubectl exec is used to start the simplest spark-shell environment. According to our configuration, there are 4 executor instances, and the pods are as follows:

➜ kubectl get…

Jupyter is a very popular web-based interactive computing platform. It can provide great help for the teaching of programming languages ​​and the prototyping of project development. Just as the origin of its name JuPyteR, it was primarily only for Julia, Python, and R, but now it can support as many as dozens of programming languages ​​and environments.

Jupyter is widely used in Python language learning and project development, especially Python computing and machine learning, etc. As a result, many projects have developed Jupyter to support Scala, which in turn supports the kernel for Spark computing. Currently, the most popular Scala…

Jupyter是目前非常流行的基于网页(web-based)的交互式计算(interactive computing)。其能够为编程语言的教学和项目开发的原型设计提供非常大的帮助。正如Jupyter这个名字的由来,最初只是针对Julia,Python,和R的支持,而如今能支持多达几十种的编程语言和环境。

Jupyter广泛地用于Python语言的学习和项目开发,特别是Python的计算和机器学习等等。于是,有好些项目开发了Jupyter支持Scala语言,进而支持Spark计算的核(kernel)。目前,最为流行使用和开发活跃的Scala核项目则是Almond。Brunk在这个文章中有对Almond比较细致的介绍:“Interactive Computing in Scala with Jupy …

Timothy Zhang

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store