We can also try different values for the weights or how weights are distributed across neurons and test different gradient normalization strategies, applying L2 regularization and dropouts. There is no rule of thumb to choose a constant value for L1/L2 regularization or for dropouts. However, the L2 regularization constant takes a smaller value as it forces the weights to decay toward zero. Neural networks can safely accommodate dropout of 10-20 percent, beyond which it can actually cause underfitting. There is no constant value that will apply in every instance, as it varies from case to case:
A GPU-accelerated environment will help decrease the training time. DL4J supports CUDA, and it can be accelerated further using cuDNN. Most two-dimensional CNN layers (such as ConvolutionLayer and SubsamplingLayer) support cuDNN.
The NVIDIA CUDA Deep Neural Network (cuDNN) library is a GPU-accelerated library of primitives for deep learning networks. You can read more about cuDNN here: https://
developer.nvidia.com/cudnn.
Creating an API endpoint for the image
How to do it...
Persist the model using ModelSerializer: 1.
File file = new File("cnntrainedmodel.zip");
ModelSerializer.writeModel(model,file,true);
ModelSerializer.addNormalizerToModel(file,scaler);
Restore the trained model using ModelSerializer to perform predictions:
2.
MultiLayerNetwork network =
ModelSerializer.restoreMultiLayerNetwork(modelFile);
NormalizerStandardize normalizerStandardize =
ModelSerializer.restoreNormalizerFromFile(modelFile);
Design an API method that accepts inputs from users and returns results. An 3. example API method would look like the following:
public static INDArray generateOutput(File file) throws IOException, InterruptedException {
final File modelFile = new File("cnnmodel.zip");
final MultiLayerNetwork model =
ModelSerializer.restoreMultiLayerNetwork(modelFile);
final RecordReader imageRecordReader = generateReader(file);
final NormalizerStandardize normalizerStandardize = ModelSerializer.restoreNormalizerFromFile(modelFile);
final DataSetIterator dataSetIterator = new
RecordReaderDataSetIterator.Builder(imageRecordReader,1).build();
normalizerStandardize.fit(dataSetIterator);
dataSetIterator.setPreProcessor(normalizerStandardize);
return model.output(dataSetIterator);
}
Create a URI mapping to service client requests, as shown in the following 4. example:
@GetMapping("/")
public String main(final Model model){
model.addAttribute("message", "Welcome to Java deep learning!");
return "welcome";
}
@PostMapping("/")
public String fileUpload(final Model model, final
@RequestParam("uploadFile")MultipartFile multipartFile) throws IOException, InterruptedException {
cookBookService.generateStringOutput(multipartFile);
model.addAttribute("message", "Welcome to Java deep learning!");
model.addAttribute("results",results);
return "welcome";
}
Build a cookbookapp-cnn project and add the API dependency to your Spring 5. project:
<dependency>
<groupId>com.javadeeplearningcookbook.app</groupId>
<artifactId>cookbookapp-cnn</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
Create the generateStringOutput() method in the service layer to serve API 6. content:
@Override
public List<String> generateStringOutput(MultipartFile multipartFile) throws IOException, InterruptedException { //TODO: MultiPartFile to File conversion (multipartFile ->
convFile)
INDArray indArray = ImageClassifierAPI.generateOutput(convFile);
for(int i=0; i<indArray.rows();i++){
for(int j=0;j<indArray.columns();j++){
DecimalFormat df2 = new DecimalFormat("#.####");
results.add(df2.format(indArray.getDouble(i,j)*100)+"%");
//Later add them from list to the model display on UI.
} }
convFile.deleteOnExit();
return results;
}
Download and install the Google Cloud SDK: https://cloud.google.com/sdk/. 7.
Install the Cloud SDK app-engine-java component by running the following 8. command on the Google Cloud console:
gcloud components install app-engine-java
Log in and configure Cloud SDK using the following command:
9.
gcloud init
Add the following dependency for Maven App Engine in pom.xml:
10.
<plugin>
<groupId>com.google.cloud.tools</groupId>
<artifactId>appengine-maven-plugin</artifactId>
<version>2.1.0</version>
</plugin>
Create an app.yaml file in your project as per the Google Cloud documentation:
11.
https://cloud.google.com/appengine/docs/flexible/java/configuring- your-app-with-app-yaml.
Navigate to Google App Engine and click on the Create Application button:
12.
Pick a region and click on Create app:
13.
Select Java and click the Next button:
14.
Now, your app engine has been created at Google Cloud.
Build the spring boot application using Maven:
15.
mvn clean install
Deploy the application using the following command:
16.
mvn appengine:deploy
How it works...
In step 1 and step 2, we have persisted the model to reuse the model capabilities in API.
In step 3, an API method is created to accept user inputs and return the results from the image classifier.
In step 4, the URI mappings will accept client requests (GET/POST). A GET request will serve the home page at the very beginning. A POST request will serve the end user request
In step 5, we added an API dependency to the pom.xml file. For demonstration purposes, we build the API JAR file and the JAR file is stored in the local Maven repository. For production, you need to submit your API (JAR file) in a private repository so that Maven can fetch it from there.
In step 6, we are calling the ImageClassifier API at our Spring Boot application service layer to retrieve the results and return them to the controller class.
In the previous chapter, we deployed the application locally for demonstration purposes. In this chapter, we have deployed the application in Google Cloud. Steps 7 to 16 are dedicated to deployment in Google Cloud.
We have used Google App Engine, although we can set up the same thing in more customized ways using Google Compute Engine or Dataproc. Dataproc is designed to deploy your application in a Spark distributed environment.
Once deployment is successful, you should see something like the following:
When you hit the URL (which starts with https://xx.appspot.com), you should be able to see the web page (the same as in the previous chapter) where end users can upload images for image classification.
Implementing Natural 5
Language Processing
In this chapter, we will discuss word vectors (Word2Vec) and paragraph vectors (Doc2Vec) in DL4J. We will develop a complete running example step by step, covering all the stages, such as ETL, model configuration, training, and evaluation. Word2Vec and Doc2Vec are natural language processing (NLP) implementations in DL4J. It is worth mentioning a little about the bag-of-words algorithm before we talk about Word2Vec.
Bag-of-words is an algorithm that counts the instances of words in documents. This will allow us to perform document classification. Bag of words and Word2Vec are just two different types of text classification. Word2Vec can use a bag of words extracted from a document to create vectors. In addition to these text classification methods, term
frequency–inverse document frequency (TF-IDF) can be used to judge the topic/context of the document. In the case of TF-IDF, a score will be calculated for all the words, and word counts will be replaced with this score. TF-IDF is a simple scoring scheme, but word embeddings may be a better choice, as the semantic similarity can be captured by word embedding. Also, if your dataset is small and the context is domain-specific, then bag of words may be a better choice than Word2Vec.
Word2Vec is a two-layer neural network that processes text. It converts the text corpus to vectors.
Note that Word2Vec is not a deep neural network (DNN). It transforms text data into a numerical format that a DNN can understand, making customization possible.
We can even combine Word2Vec with DNNs to serve this purpose. It doesn't train the input words through reconstruction; instead, it trains words using the neighboring words in the corpus.
Doc2Vec (paragraph vectors) associates documents with labels, and is an extension of Word2Vec. Word2Vec tries to correlate words with words, while Doc2Vec (paragraph vectors) correlates words with labels. Once we represent documents in vector formats, we can then use these formats as an input to a supervised learning algorithm to map these vectors to labels.
In this chapter, we will cover the following recipes:
Reading and loading text data
Tokenizing data and training the model Evaluating the model
Generating plots from the model Saving and reloading the model Importing Google News vectors
Troubleshooting and tuning Word2Vec models
Using Word2Vec for sentence classification using CNNs Using Doc2Vec for document classification
Technical requirements
The examples discussed in this chapter can be found at https://github.com/
PacktPublishing/Java-Deep-Learning-Cookbook/tree/master/05_Implementing_NLP/
sourceCode/cookbookapp/src/main/java/com/javadeeplearningcookbook/examples. After cloning our GitHub repository, navigate to the directory called Java-Deep- Learning-Cookbook/05_Implementing_NLP/sourceCode. Then, import the cookbookapp project as a Maven project by importing pom.xml.
To get started with NLP in DL4J, add the following Maven dependency in pom.xml:
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-nlp</artifactId>
<version>1.0.0-beta3</version>
</dependency>