Table of contents

Run an Apache Kafka Spark streaming application in Watson Studio Local

Watson Studio Local supports Apache Kafka Spark streaming applications, and can connect to Kafka brokers over SSL.

Whether you submit a Kafka Streaming Application or a Kafka Streaming Application supporting SSL, the steps to run a Spark application in Watson Studio Local are the same as any other application.

Enable SSL for Apache Kafka Spark streaming applications

Restriction: Apache Kafka Spark streaming supports SSL for Kafka version 0.10.0 or later.

If you plan to use SSL for Apache Kafka Spark streaming applications, you need the following security artifacts:

  • Truststore with Certificate Authority (CA) certificate. It is persisted in the global directory of the GlusterFS shared volume at /user-home/_global_/security/customer-truststores/cacerts.
  • Keystore with a key pair and certificate that is signed by CA (used to authenticate the Spark streaming application). It is persisted in the global directory of the GlusterFS shared volume at /user-home/_global_/security/customer-keystores/idp.keystore.jks.

Watson Studio Local provides a idpKeyMan.sh script with the following flags to manage these security artifacts:

  • -certreq: Generates a certificate request.
  • -genkey: Generates a key pair.
  • -import: Imports a certificate or a certificate chain.
  • -list: Lists entries in a keystore.
  • -storepasswd: Changes the store password of a keystore.

To enable SSL for Apache Kafka Spark streaming applications, complete the following steps:

  1. Generate the key and the for the Spark streaming application. For example: /wdp/utils/idpKeyMan.sh -genkey -keystore -storepass keystore_password -alias spark-submit-client-test -validity 365 -dname "CN=ibm.com,OU=Analytics,O=IBM,L=Markham,S=ON,C=CA" -keypass keystore_password where spark-submit-client-test represents an alias that you need to reuse in a later step to associate the private key with the signed certificate.
  2. List the keystore entries to verify that your entry with the specified alias was successfully created. For example: /wdp/utils/idpKeyMan.sh -keystore -storepass keystore_password -list | grep spark-submit-client-test
  3. Export the Spark streaming application client certificate to a file. For example: /wdp/utils/idpKeyMan.sh -keystore -certreq -alias spark-submit-client-test -file client-cert -storepass keystore_password
  4. Create either an intermediary or root CA certificate to sign the Spark streaming application client certificate with.
  5. If you have an intermediary CA certificate that is not signed by a root CA certificate, copy it to the Watson Studio Local cluster and then run the following command to import the CA certificate into the truststore of the Spark submit service: /wdp/utils/idpKeyMan.sh -truststore -import -alias caroot -file ca-cert -storepass keystore_password where ca-cert represents the intermediary CA certificate.
  6. Sign the Spark streaming application client certificate with the CA certificate. This step creates a file artifact, the signed Spark submit client certificate to be imported into the keystore.
  7. Import the CA certificate into the keystore (to establish the chain of trust for the signed Spark streaming application client certificate). For example: /wdp/utils/idpKeyMan.sh -keystore -import -file ca-cert -storepass keystore_password -alias caroot
  8. Import the signed Spark streaming application client certificate into the keystore and associate it the public-private key pair that was generated in the first step. For example: /wdp/utils/idpKeyMan.sh -keystore -storepass keystore_password -import -alias spark-submit-client-test -file cert-signed-client where spark-submit-client-test represents the same value for the -alias parameter that you provided in the first step and cert-signed-client represents the signed Spark streaming application client certificate.

After SSL is enabled, the application developer must update their application with security.protocol and ssl parameters. See Configuring Kafka Clients for details.