dev-notes

發表於 2016-02-26   |   分類於 机器学习   |  

EasyPyPR开发手记

2016.2.26

接下来的主要工作是开发 EasyPyPR的字符识别功能

之前的思路是利用 scikit-neuralnetwork 的神经网络库来完成这个部分。

然而在使用sknn库的过程中遇到了一些问题

sknn开发分类器的例程是这个

1
2
3
4
5
6
7
8
9
from sknn.mlp import Classifier, Layer
nn = Classifier(
    layers=[
        Layer("Maxout", units=100, pieces=2),
        Layer("Softmax")],
    learning_rate=0.001,
    n_iter=25)
nn.fit(X_train, y_train)

然而我使用的是否发现抛出异常了

1
2
3
4
5
6
7
8
Traceback (most recent call last):
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 177, in <module>
    train_model()
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 164, in train_model
    Layer("Maxout", units=128, pieces=2),
  File "/usr/local/lib/python2.7/dist-packages/sknn/nn.py", line 93, in __init__
    raise NotImplementedError("Layer type `%s` is not implemented." % type)
NotImplementedError: Layer type `Maxout` is not implemented.

定位了一下抛出异常的代码
于是将隐含层的”Maxout”修改成”Linear”

1
2
3
4
5
6
7
8
9
from sknn.mlp import Classifier, Layer
nn = Classifier(
    layers=[
        Layer("Linear", units=100, pieces=2),
        Layer("Softmax")],
    learning_rate=0.001,
    n_iter=25)
nn.fit(X_train, y_train)

新发生的情况是

1
2
3
4
5
6
7
8
9
10
11
No handlers could be found for logger "sknn"
Traceback (most recent call last):
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 177, in <module>
    train_model()
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 168, in train_model
    nn.fit(train_data, train_label)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 390, in fit
    return super(Classifier, self)._fit(X, yp, w)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 241, in _fit
    raise e
RuntimeError: Training diverged and returned NaN.

好吧，让我们google一下，
查看StackOverflow
http://stackoverflow.com/questions/26659772/silence-no-handlers-could-be-found-for-logger-paramiko-transport-message

新问题的出现

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
WARNING:sknn:Parameter `pieces` is unused for layer type `Linear`.
ERROR:sknn:
A runtime exception was caught during training. This likely occurred due to
a divergence of the SGD algorithm, and NaN floats were found by the backend.
Try setting the `learning_rate` 10x lower to resolve this, for example:
    learning_rate=0.000100
Traceback (most recent call last):
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 180, in <module>
    train_model()
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 171, in train_model
    nn.fit(train_data, train_label)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 390, in fit
    return super(Classifier, self)._fit(X, yp, w)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 241, in _fit
    raise e
RuntimeError: Training diverged and returned NaN.
[Finished in 35.9s with exit code 1]
[shell_cmd: python -u "/home/ray/dev/EasyPyPR/ANNtrain.py"]
[dir: /home/ray/dev/EasyPyPR]
[path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games]

我觉得 首先把pieces参数给去掉
然后降低学习速率 learning_rate=0.000100

1
2
3
4
5
6
7
8
9
10
11
nn = Classifier(
    layers=[
        Layer("Linear", units=128),
        Layer("Softmax")],
    learning_rate=0.0001,
    n_iter=25)
nn.fit(train_data, train_label)
y_example = nn.predict(train_data[5,:])
print y_example
print train_label[5]

接下来就是不断降低学习速率的尝试

然而当学习速率最低时
learning_rate=0.00000001

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
ERROR:sknn:
A runtime exception was caught during training. This likely occurred due to
a divergence of the SGD algorithm, and NaN floats were found by the backend.
Try setting the `learning_rate` 10x lower to resolve this, for example:
    learning_rate=0.000000
Traceback (most recent call last):
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 180, in <module>
    train_model()
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 171, in train_model
    nn.fit(train_data, train_label)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 390, in fit
    return super(Classifier, self)._fit(X, yp, w)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 241, in _fit
    raise e
RuntimeError: Training diverged and returned NaN.
[Finished in 35.9s with exit code 1]
[shell_cmd: python -u "/home/ray/dev/EasyPyPR/ANNtrain.py"]
[dir: /home/ray/dev/EasyPyPR]
[path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games]

改变一下玩法吧 输出层改线性

1
2
3
4
5
6
7
8
9
10
11
nn = Classifier(
    layers=[
        Layer("Linear", units=128),
        Layer("Linear")],
    learning_rate=0.00001,
    n_iter=25)
nn.fit(train_data, train_label)
y_example = nn.predict(train_data[5,:])
print y_example
print train_label[5]

然并卵啊

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
WARNING:sknn:WARNING: Expecting `Softmax` type for the last layer in classifier.
ERROR:sknn:
A runtime exception was caught during training. This likely occurred due to
a divergence of the SGD algorithm, and NaN floats were found by the backend.
Try setting the `learning_rate` 10x lower to resolve this, for example:
    learning_rate=0.000000
Traceback (most recent call last):
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 180, in <module>
    train_model()
  File "/home/ray/dev/EasyPyPR/ANNtrain.py", line 171, in train_model
    nn.fit(train_data, train_label)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 390, in fit
    return super(Classifier, self)._fit(X, yp, w)
  File "/usr/local/lib/python2.7/dist-packages/sknn/mlp.py", line 241, in _fit
    raise e
RuntimeError: Training diverged and returned NaN.
[Finished in 1.9s with exit code 1]
[shell_cmd: python -u "/home/ray/dev/EasyPyPR/ANNtrain.py"]
[dir: /home/ray/dev/EasyPyPR]
[path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games]

let me google again
关键词: RuntimeError: Training diverged and returned NaN

http://stackoverflow.com/questions/33478998/runtimeerror-nan-in-hidden0-w-no-handlers-could-be-found-for-logger-sknn-sci

看起来是要归一化的问题，那我再归一化一下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
pipeline = Pipeline([
    ('min/max scaler', MinMaxScaler(feature_range=(0.0, 1.0))),
    ('neural network', Classifier(
        layers=[
            Layer("Linear", units=128),
            Layer("Linear")],
        learning_rate=0.0000001,
        n_iter=25))])
pipeline.fit(train_data, train_label)
y_example = pipeline.predict(train_data[5,:])
print y_example
print train_label[5]

归一化以后似乎可用了

之后是极低的准确率

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(10, 120)
predict:
[[10]
 [28]
 [ 1]
 [ 4]
 [11]
 [29]
 [10]
 [19]
 [24]
 [11]]
label:
[1 1 1 1 1 1 1 1 1 1]
[Finished in 1.1s]

感觉是使用问题 准备换成 scikit-learn用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
def train_model():
    train_data = joblib.load('data/ann_train_data.pkl')
    train_label =  joblib.load('data/ann_train_label.pkl')
    # train_label = train_label.transpose()
    # train_label.shape = (2262,1)
    print train_data.shape
    print train_label.shape
    print type(train_label)
    # print 'train data'
    # print train_data[0:10,0:20]
    # print train_label[1000:1005]
    pipeline = Pipeline([
        ('min/max scaler', MinMaxScaler(feature_range=(0.0, 1.0))),
        ('neural network', Classifier(
            layers=[
                Layer("Linear", units=256),
                Layer("Softmax")],
            learning_rate=0.00001,
            n_iter=25))])
    pipeline.fit(train_data, train_label)
    joblib.dump(pipeline, 'model/sknn_ann')
    # nn = Classifier(
    #     layers=[
    #         Layer("Linear", units=512),
    #         Layer("Softmax")],
    #     learning_rate=0.00001,
    #     n_iter=25)
    # nn.fit(train_data, train_label)
    # joblib.dump(nn, 'model/sknn_ann_nn')
def predict_char():
    train_data = joblib.load('data/ann_train_data.pkl')
    train_label =  joblib.load('data/ann_train_label.pkl')
    X = train_data[0:10,:]
    print X.shape
    pipeline = joblib.load('model/sknn_ann')
    y = pipeline.predict(X)
    print "predict:"
    print y
    print "label:"
    print train_label[0:10]

http://scikit-learn.org/dev/auto_examples/classification/plot_classifier_comparison.html
http://scikit-learn.org/dev/modules/generated/sklearn.neural_network.MLPClassifier.html

准备切换工具
不过开发环境似乎仍有问题

ANNtrain.py

找了一个例子
plot_classifier_comparison.html

#神经网络
比特币统计套利-推荐阅读
SKNN(scikit-neuralnetwork)的几个例子