djaouen
[Book Question] Machine Learning in Elixir: Poor accuracy for Chapter 1's example
Hello,
I am working through the examples in the new book, Machine Learning in Elixir and I am having an issue with poor accuracy in Chapter 1’s example. You can find the Livebook I created here: https://github.com/danieljaouen/machine-learning-in-elixir/blob/main/machine-learning-in-elixir-chapter-1.livemd
And here is the accuracy I am getting on my machine:
Batch: 0, accuracy: 0.0666667
%{
0 => %{
"accuracy" => #Nx.Tensor<
f32
0.06666667014360428
>
}
}
However, the training accuracy seems fine:
Epoch: 0, Batch: 450, accuracy: 0.8331868 loss: 0.5048826
Epoch: 1, Batch: 450, accuracy: 0.8779556 loss: 0.4173653
Epoch: 2, Batch: 450, accuracy: 0.9101056 loss: 0.3732252
Epoch: 3, Batch: 450, accuracy: 0.9288760 loss: 0.3434850
Epoch: 4, Batch: 450, accuracy: 0.9367946 loss: 0.3209158
Epoch: 5, Batch: 450, accuracy: 0.9416718 loss: 0.3026979
Epoch: 6, Batch: 450, accuracy: 0.9494675 loss: 0.2874412
Epoch: 7, Batch: 450, accuracy: 0.9583363 loss: 0.2743504
Epoch: 8, Batch: 450, accuracy: 0.9583363 loss: 0.2629215
Epoch: 9, Batch: 450, accuracy: 0.9626405 loss: 0.2528131
Not sure what I am doing wrong here. Any help? Thanks in advance!
Marked As Solved
grossvogel
I had a few more minutes to play with this, and so far it looks like we can get better results by processing the x and y data into tensors before splitting up test and training sets.
feature_columns = ["sepal_length", "sepal_width", "petal_length", "petal_width"]
label_column = "species"
x_all = Nx.stack(shuffled_normalized_iris[feature_columns], axis: 1)
y_all =
shuffled_normalized_iris[label_column]
|> Explorer.Series.cast(:category)
|> Nx.stack(axis: -1)
|> Nx.equal(Nx.iota({1, 3}, axis: -1))
x_train = x_all[0..119]
x_test = x_all[120..149]
y_train = y_all[0..119]
y_test = y_all[120..149]
Also Liked
grossvogel
You can see what the model is predicting for the test data with Axon.predict/4
Axon.predict(model, trained_model_state, x_test)
grossvogel
I ran into this also, and decided it had to be some kind of typo with how the test set is set up. After a lot of head scratching, I think there’s a more subtle error with the setup of the test data. I believe when the species are assigned their positions in the one-hot encoding vector, that order is determined by the order in which the species are encountered in the test and training data.
For instance, if the species of the first 3 rows of the training set are "Iris-virginica", "Iris-setosa", ""Iris-versicolor", then those entries in the train_y data will look like [1, 0, 0], [0, 1, 0], [0, 0, 1] and the model will learn to predict [1, 0, 0] if the features match what it’s learned about “Iris-virginica.”
If the species are encountered in a different order in the test data, then we may end up with “Iris-virginica” having the 2nd position instead of the first in the test_y data, so the model will predict [1, 0, 0] but the scoring logic will be comparing against [0, 1, 0]
bdarla
Your code is correct (in accordance with the book).
In some runs, I also noticed low accuracy. This is because of the small dataset (150 samples).
If you re-run the steps from the shuffle step and below, then you will receive different results every time. In some cases, it can easily be 96% accuracy. Just, rerun the experiment.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"