Run an Apache Kafka Spark streaming application in DSX Local
DSX Local supports Apache Kafka Spark streaming applications, and can connect to Kafka brokers over SSL. Whether you submit a Kafka Streaming Application or a Kafka Streaming Application supporting SSL, the steps to run a Spark application in DSX Local are the same as any other application.
Enable SSL for Apache Kafka Spark streaming applications
Requirement: Apache Kafka Spark streaming supports SSL for Kafka version 0.10.0 or later.
If you plan to use SSL for Apache Kafka Spark streaming applications, you need the following security artifacts:
- Truststore with Certificate Authority (CA) certificate. It is persisted in the global directory of the GlusterFS shared volume at
- Keystore with a key pair and certificate that is signed by CA (used to authenticate the Spark streaming application). It is persisted in the global directory of the GlusterFS shared volume at
DSX Local provides a
idpKeyMan.sh script with the following flags to manage these security artifacts:
-certreq: Generates a certificate request.
-genkey: Generates a key pair.
-import: Imports a certificate or a certificate chain.
-list: Lists entries in a keystore.
-storepasswd: Changes the store password of a keystore.
To enable SSL for Apache Kafka Spark streaming applications, complete the following steps:
- Generate the key and the for the Spark streaming application. For example:
/wdp/utils/idpKeyMan.sh -genkey -keystore -storepass keystore_password -alias spark-submit-client-test -validity 365 -dname "CN=ibm.com,OU=Analytics,O=IBM,L=Markham,S=ON,C=CA" -keypass keystore_passwordwhere
spark-submit-client-testrepresents an alias that you need to reuse in a later step to associate the private key with the signed certificate.
- List the keystore entries to verify that your entry with the specified alias was successfully created. For example:
/wdp/utils/idpKeyMan.sh -keystore -storepass keystore_password -list | grep spark-submit-client-test
- Export the Spark streaming application client certificate to a file. For example:
/wdp/utils/idpKeyMan.sh -keystore -certreq -alias spark-submit-client-test -file client-cert -storepass keystore_password
- Create either an intermediary or root CA certificate to sign the Spark streaming application client certificate with.
- If you have an intermediary CA certificate that is not signed by a root CA certificate, copy it to the DSX Local cluster and then run the following command to import the CA certificate into the truststore of the Spark submit service:
/wdp/utils/idpKeyMan.sh -truststore -import -alias caroot -file ca-cert -storepass keystore_passwordwhere
ca-certrepresents the intermediary CA certificate.
- Sign the Spark streaming application client certificate with the CA certificate. This step creates a file artifact, the signed Spark submit client certificate to be imported into the keystore.
- Import the CA certificate into the keystore (to establish the chain of trust for the signed Spark streaming application client certificate). For example:
/wdp/utils/idpKeyMan.sh -keystore -import -file ca-cert -storepass keystore_password -alias caroot
- Import the signed Spark streaming application client certificate into the keystore and associate it the public-private key pair that was generated in the first step. For example:
/wdp/utils/idpKeyMan.sh -keystore -storepass keystore_password -import -alias spark-submit-client-test -file cert-signed-clientwhere
spark-submit-client-testrepresents the same value for the
-aliasparameter that you provided in the first step and
cert-signed-clientrepresents the signed Spark streaming application client certificate.
After SSL is enabled, the application developer must update their application with
security.protocol and ssl parameters. See Configuring Kafka Clients for details.