用sklearn做线性回归, 但数据normalization后,出来MSE全部为0。

用sklearn在股票价格数据 做线性回归, 但数据normalization后,出来MSE的结果全部为0。别人说是模型出错了, 但奈何自己是python新手,请求各位帮忙指出其中原因,感谢感谢!!!!

数据是这样子的:

图片说明

这是不加normalization的,

from sklearn.linear_model import LinearRegression
from sklearn import cross_validation
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import Normalizer

LinearRegression=LinearRegression()

scores = cross_validation.cross_val_score(LinearRegression, X_stock1_train, y_stock1_train, scoring='neg_mean_squared_error', cv=10)


print (-scores)
print ('Average score for Linear Regression:', np.mean(scores))

结果看起来还算正常:
[ 0.03666889 0.05985924 0.05718805 0.04757506 0.05605501 0.05602068
0.04308263 0.05089644 0.0489978 0.0384472 ]
Average score for Linear Regression: -0.0494790998005

##分割线##

normalization处理过的:

from sklearn.linear_model import LinearRegression
from sklearn import cross_validation


transformer=Normalizer().fit(X_stock1_train, y_stock1_train)
X_stock1_train=transformer.transform(X_stock1_train)
y_stock1_train=transformer.transform(y_stock1_train)


LinearRegression=LinearRegression()
scores = cross_validation.cross_val_score(LinearRegression, X_stock1_train, y_stock1_train, scoring='neg_mean_squared_error', cv=10)

print (-scores)
print ('Average score for Linear Regression:', np.mean(scores))


结果:
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
Average score for Linear Regression: 0.0

别人说是模型出错了, 但奈何自己是python新手,请求各位帮忙指出其中原因,感谢感谢!!!

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
深度学习做回归问题如何对数据集进行标准化,归一化?
## **问题描述** 使用深度学习RNN模型预测目标轨迹。输入过去一段时间目标的运动轨迹预测未来某一时刻目标的位置。现在已有一些轨迹的数据,在训练之前应该如何对数据进行标准化/或者归一化呢?或者是不需要进行这个处理?(自己试了没有标准化数据时模型难以训练,使用的损失函数是mean_squared_error) ## **找的一些标准化/归一化方法** 1. zero-mean normalization x = (x - μ ) / σ # μ是样本均值,σ是样本方差 很多文献和网上的例子使用这种方法,但是这个方法有个很严重的问题,公式中的μ , σ是更具已有的数据集计算得到的,实际运用中是想利用新测到的数据预测目标未来的位置,显然新测量到的数据无法使用原来的μ ,σ进行处理。 2. **Min-max normalization** x = (x - min)/(max - min) #min是样本最小值,max是样本最大值 由于目标的位置,例如(x, y)未知,min和max无法确定,网友建议使用理论的最大值,最小值计算,但是这可能导致数据发布极不平衡,暂时没尝试过这种方法。 3.**atan函数转换** x = atan(x) * 2 / π 这个方法可以把数据映射到[0,1]。使用这个方法遇到了两个个问题,使用反正切计算后得到的预测结果偏差很大(模型训练时损失在0.02以下),另一个问题是,由于模型有误差,当真实值接近1(标准化以后)时,模型预测可能超过1,这样计算反正切结果为负,所以放弃这种方法。 ## 寻求的帮助 请问各位大佬,这类问题应该怎么对数据集进行预处理,网上找的很多方法,并没有很好的解决。
python 在进行机器学习时,没有键盘输入ctrl C但是出现KeyboardInterrupt错误,这是什么原因呢?
1.在运行中并没有键盘输入ctrl C,运行一段时间后,出现: 2. 错误提示: ``` Traceback (most recent call last): File "F:\new\new world\deep learning\CNN_ Batch_normalization\CNN_Batch_normalization\cat_dog_mian.py", line 101, in <module> trainer.train() File "..\common\trainer.py", line 80, in train self.train_step() File "..\common\trainer.py", line 53, in train_step grads = self.network.gradient(x_batch, t_batch) File "F:\new\new world\deep learning\CNN_ Batch_normalization\CNN_Batch_normalization\cat_dog_deep_convnet.py", line 127, in gradient dout = layer.backward(dout) File "..\common\layers.py", line 176, in backward dx = self.__backward(dout) File "..\common\layers.py", line 185, in __backward dxc = dxn / self.std KeyboardInterrupt ``` 不清楚是什么原因,望大佬解答!!!
tensorflow训练完模型直接测试和导入模型进行测试的结果不同,一个很好,一个略差,这是为什么?
在tensorflow训练完模型,我直接采用同一个session进行测试,得到结果较好,但是采用训练完保存的模型,进行重新载入进行测试,结果较差,不懂是为什么会出现这样的结果。注:测试数据是一样的。以下是模型结果: 训练集:loss:0.384,acc:0.931. 验证集:loss:0.212,acc:0.968. 训练完在同一session内的测试集:acc:0.96。导入保存的模型进行测试:acc:0.29 ``` def create_model(hps): global_step = tf.Variable(tf.zeros([], tf.float64), name = 'global_step', trainable = False) scale = 1.0 / math.sqrt(hps.num_embedding_size + hps.num_lstm_nodes[-1]) / 3.0 print(type(scale)) gru_init = tf.random_normal_initializer(-scale, scale) with tf.variable_scope('Bi_GRU_nn', initializer = gru_init): for i in range(hps.num_lstm_layers): cell_bw = tf.contrib.rnn.GRUCell(hps.num_lstm_nodes[i], activation = tf.nn.relu, name = 'cell-bw') cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, output_keep_prob = dropout_keep_prob) cell_fw = tf.contrib.rnn.GRUCell(hps.num_lstm_nodes[i], activation = tf.nn.relu, name = 'cell-fw') cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, output_keep_prob = dropout_keep_prob) rnn_outputs, _ = tf.nn.bidirectional_dynamic_rnn(cell_bw, cell_fw, inputs, dtype=tf.float32) embeddedWords = tf.concat(rnn_outputs, 2) finalOutput = embeddedWords[:, -1, :] outputSize = hps.num_lstm_nodes[-1] * 2 # 因为是双向LSTM,最终的输出值是fw和bw的拼接,因此要乘以2 last = tf.reshape(finalOutput, [-1, outputSize]) # reshape成全连接层的输入维度 last = tf.layers.batch_normalization(last, training = is_training) fc_init = tf.uniform_unit_scaling_initializer(factor = 1.0) with tf.variable_scope('fc', initializer = fc_init): fc1 = tf.layers.dense(last, hps.num_fc_nodes, name = 'fc1') fc1_batch_normalization = tf.layers.batch_normalization(fc1, training = is_training) fc_activation = tf.nn.relu(fc1_batch_normalization) logits = tf.layers.dense(fc_activation, hps.num_classes, name = 'fc2') with tf.name_scope('metrics'): softmax_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits = logits, labels = tf.argmax(outputs, 1)) loss = tf.reduce_mean(softmax_loss) # [0, 1, 5, 4, 2] ->argmax:2 因为在第二个位置上是最大的 y_pred = tf.argmax(tf.nn.softmax(logits), 1, output_type = tf.int64, name = 'y_pred') # 计算准确率,看看算对多少个 correct_pred = tf.equal(tf.argmax(outputs, 1), y_pred) # tf.cast 将数据转换成 tf.float32 类型 accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32)) with tf.name_scope('train_op'): tvar = tf.trainable_variables() for var in tvar: print('variable name: %s' % (var.name)) grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvar), hps.clip_lstm_grads) optimizer = tf.train.AdamOptimizer(hps.learning_rate) train_op = optimizer.apply_gradients(zip(grads, tvar), global_step) # return((inputs, outputs, is_training), (loss, accuracy, y_pred), (train_op, global_step)) return((inputs, outputs), (loss, accuracy, y_pred), (train_op, global_step)) placeholders, metrics, others = create_model(hps) content, labels = placeholders loss, accuracy, y_pred = metrics train_op, global_step = others def val_steps(sess, x_batch, y_batch, writer = None): loss_val, accuracy_val = sess.run([loss,accuracy], feed_dict = {inputs: x_batch, outputs: y_batch, is_training: hps.val_is_training, dropout_keep_prob: 1.0}) return loss_val, accuracy_val loss_summary = tf.summary.scalar('loss', loss) accuracy_summary = tf.summary.scalar('accuracy', accuracy) # 将所有的变量都集合起来 merged_summary = tf.summary.merge_all() # 用于test测试的summary merged_summary_test = tf.summary.merge([loss_summary, accuracy_summary]) LOG_DIR = '.' run_label = 'run_Bi-GRU_Dropout_tensorboard' run_dir = os.path.join(LOG_DIR, run_label) if not os.path.exists(run_dir): os.makedirs(run_dir) train_log_dir = os.path.join(run_dir, timestamp, 'train') test_los_dir = os.path.join(run_dir, timestamp, 'test') if not os.path.exists(train_log_dir): os.makedirs(train_log_dir) if not os.path.join(test_los_dir): os.makedirs(test_los_dir) # saver得到的文件句柄,可以将文件训练的快照保存到文件夹中去 saver = tf.train.Saver(tf.global_variables(), max_to_keep = 5) # train 代码 init_op = tf.global_variables_initializer() train_keep_prob_value = 0.2 test_keep_prob_value = 1.0 # 由于如果按照每一步都去计算的话,会很慢,所以我们规定每100次存储一次 output_summary_every_steps = 100 num_train_steps = 1000 # 每隔多少次保存一次 output_model_every_steps = 500 # 测试集测试 test_model_all_steps = 4000 i = 0 session_conf = tf.ConfigProto( gpu_options = tf.GPUOptions(allow_growth=True), allow_soft_placement = True, log_device_placement = False) with tf.Session(config = session_conf) as sess: sess.run(init_op) # 将训练过程中,将loss,accuracy写入文件里,后面是目录和计算图,如果想要在tensorboard中显示计算图,就想sess.graph加上 train_writer = tf.summary.FileWriter(train_log_dir, sess.graph) # 同样将测试的结果保存到tensorboard中,没有计算图 test_writer = tf.summary.FileWriter(test_los_dir) batches = batch_iter(list(zip(x_train, y_train)), hps.batch_size, hps.num_epochs) for batch in batches: train_x, train_y = zip(*batch) eval_ops = [loss, accuracy, train_op, global_step] should_out_summary = ((i + 1) % output_summary_every_steps == 0) if should_out_summary: eval_ops.append(merged_summary) # 那三个占位符输进去 # 计算loss, accuracy, train_op, global_step的图 eval_ops.append(merged_summary) outputs_train = sess.run(eval_ops, feed_dict={ inputs: train_x, outputs: train_y, dropout_keep_prob: train_keep_prob_value, is_training: hps.train_is_training }) loss_train, accuracy_train = outputs_train[0:2] if should_out_summary: # 由于我们想在100steps之后计算summary,所以上面 should_out_summary = ((i + 1) % output_summary_every_steps == 0)成立, # 即为真True,那么我们将训练的内容放入eval_ops的最后面了,因此,我们想获得summary的结果得在eval_ops_results的最后一个 train_summary_str = outputs_train[-1] # 将获得的结果写训练tensorboard文件夹中,由于训练从0开始,所以这里加上1,表示第几步的训练 train_writer.add_summary(train_summary_str, i + 1) test_summary_str = sess.run([merged_summary_test], feed_dict = {inputs: x_dev, outputs: y_dev, dropout_keep_prob: 1.0, is_training: hps.val_is_training })[0] test_writer.add_summary(test_summary_str, i + 1) current_step = tf.train.global_step(sess, global_step) if (i + 1) % 100 == 0: print("Step: %5d, loss: %3.3f, accuracy: %3.3f" % (i + 1, loss_train, accuracy_train)) # 500个batch校验一次 if (i + 1) % 500 == 0: loss_eval, accuracy_eval = val_steps(sess, x_dev, y_dev) print("Step: %5d, val_loss: %3.3f, val_accuracy: %3.3f" % (i + 1, loss_eval, accuracy_eval)) if (i + 1) % output_model_every_steps == 0: path = saver.save(sess,os.path.join(out_dir, 'ckp-%05d' % (i + 1))) print("Saved model checkpoint to {}\n".format(path)) print('model saved to ckp-%05d' % (i + 1)) if (i + 1) % test_model_all_steps == 0: # test_loss, test_acc, all_predictions= sess.run([loss, accuracy, y_pred], feed_dict = {inputs: x_test, outputs: y_test, dropout_keep_prob: 1.0}) test_loss, test_acc, all_predictions= sess.run([loss, accuracy, y_pred], feed_dict = {inputs: x_test, outputs: y_test, is_training: hps.val_is_training, dropout_keep_prob: 1.0}) print("test_loss: %3.3f, test_acc: %3.3d" % (test_loss, test_acc)) batches = batch_iter(list(x_test), 128, 1, shuffle=False) # Collect the predictions here all_predictions = [] for x_test_batch in batches: batch_predictions = sess.run(y_pred, {inputs: x_test_batch, is_training: hps.val_is_training, dropout_keep_prob: 1.0}) all_predictions = np.concatenate([all_predictions, batch_predictions]) correct_predictions = float(sum(all_predictions == y.flatten())) print("Total number of test examples: {}".format(len(y_test))) print("Accuracy: {:g}".format(correct_predictions/float(len(y_test)))) test_y = y_test.argmax(axis = 1) #生成混淆矩阵 conf_mat = confusion_matrix(test_y, all_predictions) fig, ax = plt.subplots(figsize = (4,2)) sns.heatmap(conf_mat, annot=True, fmt = 'd', xticklabels = cat_id_df.category_id.values, yticklabels = cat_id_df.category_id.values) font_set = FontProperties(fname = r"/usr/share/fonts/truetype/wqy/wqy-microhei.ttc", size=15) plt.ylabel(u'实际结果',fontsize = 18,fontproperties = font_set) plt.xlabel(u'预测结果',fontsize = 18,fontproperties = font_set) plt.savefig('./test.png') print('accuracy %s' % accuracy_score(all_predictions, test_y)) print(classification_report(test_y, all_predictions,target_names = cat_id_df['category_name'].values)) print(classification_report(test_y, all_predictions)) i += 1 ``` 以上的模型代码,请求各位大神帮我看看,为什么出现这样的结果?
theano 报错 module 'configparser' has no attribute 'ConfigParser' 用的是Anaconda3 python3.6
>theano 报错 module 'configparser' has no attribute 'ConfigParser' 用的是Win10 Anaconda3 python3.6 ``` from sklearn.datasets import load_boston import theano.tensor as T import numpy as np import matplotlib.pyplot as plt import theano class Layer(object): def __init__(self,inputs,in_size,out_size,activation_function=None): self.W = theano.shared(np.random.normal(0,1,(in_size,out_size))) self.b = theano.shared(np.zeros((out_size,)) + 0.1) self.Wx_plus_b = T.dot(inputs, self.W) + self.b self.activation_function = activation_function if activation_function is None: self.outputs = self.Wx_plus_b else: self.outputs = self.activation_function(self.Wx_plus_b) def minmax_normalization(data): xs_max = np.max(data, axis=0) xs_min = np.min(data, axis=0) xs = (1-0)*(data - xs_min)/(xs_max - xs_min) + 0 return xs np.random.seed(100) x_dataset = load_boston() x_data = x_dataset.data # minmax normalization, rescale the inputs x_data = minmax_normalization(x_data) y_data = x_dataset.target[:,np.newaxis] #cross validation, train test data split x_train, y_train = x_data[:400], y_data[:400] x_test, y_test = x_data[400:], y_data[400:] x = T.dmatrix('x') y = T.dmatrix('y') l1 = Layer(x, 13, 50, T.tanh) l2 = Layer(l1.outputs, 50, 1, None) #compute cost cost = T.mean(T.square(l2.outputs - y)) #cost = T.mean(T.square(l2.outputs - y)) + 0.1*((l1.W**2).sum() + (l2.W**2).sum()) #l2 regulization #cost = T.mean(T.square(l2.outputs - y)) + 0.1*(abs(l1.W).sum() + abs(l2.W).sum()) #l1 regulization gW1, gb1, gW2, gb2 = T.grad(cost, [l1.W,l1.b,l2.W,l2.b]) #gradient descend learning_rate = 0.01 train = theano.function(inputs=[x,y], updates=[(l1.W,l1.W-learning_rate*gW1), (l1.b,l1.b-learning_rate*gb1), (l2.W,l2.W-learning_rate*gW2), (l2.b,l2.b-learning_rate*gb2)]) compute_cost = theano.function(inputs=[x,y], outputs=cost) #record cost train_err_list = [] test_err_list = [] learning_time = [] for i in range(1000): if 1%10 == 0: #record cost train_err_list.append(compute_cost(x_train,y_train)) test_err_list.append(compute_cost(x_test,y_test)) learning_time.append(i) #plot cost history plt.plot(learning_time, train_err_list, 'r-') plt.plot(learning_time, test_err_list,'b--') plt.show() #作者 morvan莫凡 https://morvanzhou.github.io ``` 报错了: Traceback (most recent call last): File "C:/Users/Elena/PycharmProjects/theano/regularization.py", line 1, in <module> from sklearn.datasets import load_boston File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\datasets\__init__.py", line 22, in <module> from .twenty_newsgroups import fetch_20newsgroups File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\datasets\twenty_newsgroups.py", line 44, in <module> from ..feature_extraction.text import CountVectorizer File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\feature_extraction\__init__.py", line 10, in <module> from . import text File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\feature_extraction\text.py", line 28, in <module> from ..preprocessing import normalize File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\preprocessing\__init__.py", line 6, in <module> from ._function_transformer import FunctionTransformer File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\preprocessing\_function_transformer.py", line 5, in <module> from ..utils.testing import assert_allclose_dense_sparse File "C:\Users\Elena\Anaconda3\lib\site-packages\sklearn\utils\testing.py", line 61, in <module> from nose.tools import raises as _nose_raises File "C:\Users\Elena\Anaconda3\lib\site-packages\nose\__init__.py", line 1, in <module> from nose.core import collector, main, run, run_exit, runmodule File "C:\Users\Elena\Anaconda3\lib\site-packages\nose\core.py", line 11, in <module> from nose.config import Config, all_config_files File "C:\Users\Elena\Anaconda3\lib\site-packages\nose\config.py", line 6, in <module> import configparser File "C:\Users\Elena\Anaconda3\Lib\site-packages\theano\configparser.py", line 15, in <module> import theano File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\__init__.py", line 88, in <module> from theano.configdefaults import config File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\configdefaults.py", line 17, in <module> from theano.configparser import (AddConfigVar, BoolParam, ConfigParam, EnumStr, File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\configparser.py", line 77, in <module> theano_cfg = (configparser.ConfigParser if PY3 **AttributeError: module 'configparser' has no attribute 'ConfigParser**' 把theano里的configparser.py文件里的ConfigParser改成了configparser还是不行 换了模块import configparsor也不行。。。![图片说明](https://img-ask.csdn.net/upload/201909/30/1569832318_223436.png)
Mnist数据集中通过灰度归一化处理反锯齿问题
The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. the images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 field. 这是官网给的文档,请问The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm是什么操作,通过灰度归一化而达到反锯齿的效果,有没有大神解释解释,这是个什么算法?我想尽可能了解Mnist数据的来源,因为项目需要
Beadarry和limma包分析illumina芯片数据数据报错!
我用beadarry和limma包对GSE49454数据集进行差异基因表达分析时,最后卡在这个group name上面了,尝试了好多方法都不行 代码:library(GEOquery) library(beadarray) library(illuminaHumanv4.db) #下载数据 url<-"ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE49nnn/GSE49454/matrix/" filenm<-"GSE49454_series_matrix.txt.gz" if(!file.exists("GSE49454_series_matrix.txt.gz")) download.file(paste(url, filenm, sep=""), destfile=filenm) gse <- getGEO(filename=filenm) #将下载的数据转换为ExpressionSetIllumina,并注释 summaryData <- as(gse, "ExpressionSetIllumina") rna <- factor(pData(summaryData)[,"characteristics_ch1"]) #去除非匹配 fData(summaryData)$Status <- ifelse(fData(summaryData)$PROBEQUALITY=="No match","negative","regular" ) Detection(summaryData) <- calculateDetection(summaryData, status=fData(summaryData)$Status) #normalization summaryData.norm <- normaliseIllumina(summaryData,method="neqc", status=fData(summaryData)$Status) group <- pData(summaryData.norm)[ ,"characteristics_ch1"] limmaRes <- limmaDE(summaryData, SampleGroup="characteristics_ch1") design <- model.matrix(~0+rna) design colnames(design) <- levels(rna) aw <- arrayWeights(exprs(summaryData.norm), design) aw fit <- lmFit(exprs(summaryData.norm), design, weights=aw) contrasts <- makeContrasts(group: SLE-group: Healthy control of SLE, levels=design) contr.fit <- eBayes(contrasts.fit(fit, contrasts)) topTable(contr.fit, coef=1) 报错: > contrasts <- makeContrasts(SLE-control, levels=design) Error in makeContrasts(SLE - control, levels = design) : The levels must by syntactically valid names in R, see help(make.names). Non-valid names: rnagroup: Healthy control of SLE,rnagroup: SLE > limmaRes <- limmaDE(summaryData, SampleGroup="characteristics_ch1") Calculating array weights Array weights Error in makeContrasts(contrasts = contrast, levels = design) : The levels must by syntactically valid names in R, see help(make.names). Non-valid names: group: Healthy control of SLE,group: SLE
theano 运行报错 安装了MingGW依然不行
想用theano运行regularization 安装了MingGW c++complier依然报错 ``` from sklearn.datasets import load_boston import theano.tensor as T import numpy as np import matplotlib.pyplot as plt import theano class Layer(object): def __init__(self,inputs,in_size,out_size,activation_function=None): self.W = theano.shared(np.random.normal(0,1,(in_size,out_size))) self.b = theano.shared(np.zeros((out_size,)) + 0.1) self.Wx_plus_b = T.dot(inputs, self.W) + self.b self.activation_function = activation_function if activation_function is None: self.outputs = self.Wx_plus_b else: self.outputs = self.activation_function(self.Wx_plus_b) def minmax_normalization(data): xs_max = np.max(data, axis=0) xs_min = np.min(data, axis=0) xs = (1-0)*(data - xs_min)/(xs_max - xs_min) + 0 return xs np.random.seed(100) x_dataset = load_boston() x_data = x_dataset.data # minmax normalization, rescale the inputs x_data = minmax_normalization(x_data) y_data = x_dataset.target[:,np.newaxis] #cross validation, train test data split x_train, y_train = x_data[:400], y_data[:400] x_test, y_test = x_data[400:], y_data[400:] x = T.dmatrix('x') y = T.dmatrix('y') l1 = Layer(x, 13, 50, T.tanh) l2 = Layer(l1.outputs, 50, 1, None) #compute cost cost = T.mean(T.square(l2.outputs - y)) #cost = T.mean(T.square(l2.outputs - y)) + 0.1*((l1.W**2).sum() + (l2.W**2).sum()) #l2 regulization #cost = T.mean(T.square(l2.outputs - y)) + 0.1*(abs(l1.W).sum() + abs(l2.W).sum()) #l1 regulization gW1, gb1, gW2, gb2 = T.grad(cost, [l1.W,l1.b,l2.W,l2.b]) #gradient descend learning_rate = 0.01 train = theano.function(inputs=[x,y], updates=[(l1.W,l1.W-learning_rate*gW1), (l1.b,l1.b-learning_rate*gb1), (l2.W,l2.W-learning_rate*gW2), (l2.b,l2.b-learning_rate*gb2)]) compute_cost = theano.function(inputs=[x,y], outputs=cost) #record cost train_err_list = [] test_err_list = [] learning_time = [] for i in range(1000): if 1%10 == 0: #record cost train_err_list.append(compute_cost(x_train,y_train)) test_err_list.append(compute_cost(x_test,y_test)) learning_time.append(i) #plot cost history plt.plot(learning_time, train_err_list, 'r-') plt.plot(learning_time, test_err_list,'b--') plt.show() #作者:morvan 莫凡 https://morvanzhou.github.io ``` 报错如下: You can find the C code in this temporary file: C:\Users\Elena\AppData\Local\Temp\theano_compilation_error_cns9ecbh Traceback (most recent call last): File "c:\Users\Elena\PycharmProjects\theano\regularization.py", line 2, in <module> import theano.tensor as T File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\__init__.py", line 110, in <module> from theano.compile import ( File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\compile\__init__.py", line 12, in <module> from theano.compile.mode import * File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\compile\mode.py", line 11, in <module> import theano.gof.vm File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\gof\vm.py", line 674, in <module> from . import lazylinker_c File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\gof\lazylinker_c.py", line 140, in <module> preargs=args) File "C:\Users\Elena\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2396, in compile_str (status, compile_stderr.replace('\n', '. '))) Exception: Compilation failed (return status=1): C:\Users\Elena\AppData\Local\Theano\compiledir_Windows-10-10.0.17134-SP0-Intel64_Family_6_Model_142_Stepping_9_GenuineIntel-3.6.5-64\lazylinker_ext\mod.cpp:1:0: sorry, unimplemented: 64-bit mode not compiled in . #include <Python.h> 终端显示gcc有安装 ![图片说明](https://img-ask.csdn.net/upload/201909/29/1569745503_383115.png)
keras实现人脸识别,训练失败……请教大神指点迷津!!!
![图片说明](https://img-ask.csdn.net/upload/201904/26/1556209614_615215.jpg) 各位大神,如图所示,在训练过程中,第二轮开始出现问题,这是什么原因呢? 代码如下: ------------------------------------------------- ``` import random import keras import numpy as np import cv2 from sklearn.model_selection import train_test_split from keras.preprocessing.image import ImageDataGenerator from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Convolution2D, MaxPooling2D from keras.optimizers import SGD from keras.utils import np_utils from keras.models import load_model from keras import backend as K from source_data import load_dataset,resize_img #定义数据集格式 class Dataset: def __init__(self, path_name): #训练数据集 self.train_images = None self.train_labels = None #测试集 self.valid_images = None self.valid_labels = None #样本数据 self.test_images = None self.test_labels = None #load路径 self.path_name = path_name #维度顺序 self.input_shape = None #加载数据集并按照交叉验证的原则划分数据集,完成数据预处理 def load(self,img_rows=64, img_cols=64,img_channels = 3,nb_classes = 2): #加载数据集到内存 images,labels=load_dataset(self.path_name)#函数调用 train_images, valid_images, train_labels, valid_labels= train_test_split(images, labels, test_size = 0.3, random_state = random.randint(0, 100)) _, test_images, _, test_labels = train_test_split(images, labels, test_size = 0.5, random_state = random.randint(0, 100)) #根据backend类型确定输入图片数据时的顺序为:channels,rows,cols,否则:rows,cols,channels #这部分代码就是根据keras库要求的维度顺序重组训练数据集 train_images = train_images.reshape(train_images.shape[0], img_rows, img_cols, img_channels) valid_images = valid_images.reshape(valid_images.shape[0], img_rows, img_cols, img_channels) test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols, img_channels) self.input_shape = (img_rows, img_cols, img_channels) #输出训练集、验证集、测试集的数量 print(train_images.shape[0], 'train samples') print(valid_images.shape[0], 'valid samples') print(test_images.shape[0], 'test samples') #我们的模型使用categorical_crossentropy作为损失函数,因此需要根据类别数量nb_classes将 #类别标签进行one-hot编码使其向量化,在这里我们的类别只有两种,经过转化后标签数据变为二维 train_labels = np_utils.to_categorical(train_labels, nb_classes) valid_labels = np_utils.to_categorical(valid_labels, nb_classes) test_labels = np_utils.to_categorical(test_labels, nb_classes) #像素数据浮点化以便归一化 train_images = train_images.astype('float32') valid_images = valid_images.astype('float32') test_images = test_images.astype('float32') #将其归一化,图像的各像素值归一化到0—1区间 train_images /= 255 valid_images /= 255 test_images /= 255 self.train_images = train_images self.valid_images = valid_images self.test_images = test_images self.train_labels = train_labels self.valid_labels = valid_labels self.test_labels = test_labels class Model: def __init__(self): self.model = None #建立keras模型 def build_model(self, dataset, nb_classes = 2): #构建一个空的网络模型,序贯模型或线性堆叠模型,添加各个layer self.model = Sequential() #以下代码将顺序添加CNN网络需要的各层,一个add就是一个网络层 self.model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape = dataset.input_shape)) #1 2维卷积层 self.model.add(Activation('relu')) #2 激活函数层 self.model.add(Convolution2D(32, 3, 3)) #3 2维卷积层 self.model.add(Activation('relu')) #4 激活函数层 self.model.add(MaxPooling2D(pool_size=(2, 2))) #5 池化层 self.model.add(Dropout(0.25)) #6 Dropout层 self.model.add(Convolution2D(64, 3, 3, border_mode='same')) #7 2维卷积层 self.model.add(Activation('relu')) #8 激活函数层 self.model.add(Convolution2D(64, 3, 3)) #9 2维卷积层 self.model.add(Activation('relu')) #10 激活函数层 self.model.add(MaxPooling2D(pool_size=(2, 2))) #11 池化层 self.model.add(Dropout(0.25)) #12 Dropout层 self.model.add(Flatten()) #13 Flatten层 self.model.add(Dense(512)) #14 Dense层,又被称作全连接层 self.model.add(Activation('relu')) #15 激活函数层 self.model.add(Dropout(0.5)) #16 Dropout层 self.model.add(Dense(nb_classes)) #17 Dense层 self.model.add(Activation('softmax')) #18 分类层,输出最终结果 #Prints a string summary of the network self.model.summary() #训练模型 def train(self, dataset, batch_size = 20, nb_epoch = 10, data_augmentation = True): sgd = SGD(lr = 0.01, decay = 1e-6, momentum = 0.9, nesterov = True) #采用随机梯度下降优化器进行训练,首先生成一个优化器对象 self.model.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=['accuracy']) #完成实际的模型配置 #不使用数据提升,所谓的提升就是从我们提供的训练数据中利用旋转、翻转、加噪声等方法提升训练数据规模,增加模型训练量 if not data_augmentation: self.model.fit(dataset.train_images, dataset.train_labels, batch_size = batch_size, epochs = nb_epoch, validation_data = (dataset.valid_images, dataset.valid_labels), shuffle = True) #使用实时数据提升 else: #定义数据生成器用于数据提升,其返回一个生成器对象datagen,datagen每被调用一 #次其生成一组数据(顺序生成),节省内存,其实就是python的数据生成器 datagen = ImageDataGenerator( featurewise_center = False, #是否使输入数据去中心化(均值为0), samplewise_center = False, #是否使输入数据的每个样本均值为0 featurewise_std_normalization = False, #是否数据标准化(输入数据除以数据集的标准差) samplewise_std_normalization = False, #是否将每个样本数据除以自身的标准差 zca_whitening = False, #是否对输入数据施以ZCA白化 rotation_range = 20, #数据提升时图片随机转动的角度(范围为0~180) width_shift_range = 0.2, #数据提升时图片水平偏移的幅度(单位为图片宽度的占比,0~1之间的浮点数) height_shift_range = 0.2, #同上,只不过这里是垂直 horizontal_flip = True, #是否进行随机水平翻转 vertical_flip = False) #是否进行随机垂直翻转 #计算整个训练样本集的数量以用于特征值归一化等处理 datagen.fit(dataset.train_images) #利用生成器开始训练模型—0.7*N self.model.fit_generator(datagen.flow(dataset.train_images, dataset.train_labels, batch_size = batch_size), steps_per_epoch = dataset.train_images.shape[0], epochs = nb_epoch, validation_data = (dataset.valid_images, dataset.valid_labels)) if __name__ == '__main__': dataset = Dataset('e:\saving') dataset.load()#实例操作,完成实际数据加载和预处理 model = Model() model.build_model(dataset) #训练数据 model.train(dataset) ```
跑AlexNet猫狗遇到IndexError: too many indices for array的问题
小弟最近在看OpenCV+TensorFlow这本书的案例 照着输进去了结果发现运行不下去, 问题应该是出现在第二块内容,但是真的不太明白!求各位大神赐教,如何修改! ![图片说明](https://img-ask.csdn.net/upload/201906/30/1561889895_192060.png) 第一块,修改照片尺寸,为啥呀 ``` import cv2 import os def resize(dir): for root, dirs, files in os.walk(dir): for file in files: filepath = os.path.join(root, file) try: image = cv2.imread(filepath) dim = (227, 227) resized = cv2.resize(image, dim) path = "C:\\Users\\Telon_Hu\\Desktop\\ANNs\\train1\\" + file cv2.imwrite(path, resized) except: print(filepath) # os.remove(filepath) cv2.waitKey(0) resize('C:\\Users\\Telon_Hu\\Desktop\\ANNs\\train') ``` ``` import os import numpy as np import tensorflow as tf import cv2 def get_file(file_dir): images=[] temp=[] for root,sub_folders,files in os.walk(file_dir): ''' os.walk(path)---返回的是一个三元组(root,dirs,files): root 所指的是当前正在遍历的这个文件夹的本身的地址 dirs 是一个 list ,内容是该文件夹中所有的目录的名字(不包括子目录) files 同样是 list , 内容是该文件夹中所有的文件(不包括子目录) ''' for name in files: images.append(os.path.join(root,name)) for name in sub_folders: temp.append(os.path.join(root,name)) labels=[] for one_folder in temp: n_img=len(os.listdir(one_folder)) #s.listdir() 方法用于返回指定的文件夹包含的文件或文件夹的名字的列表 letter=one_folder.split('\\')[-1] #split() 通过指定分隔符对字符串进行切片,默认为-1, 即分隔所有。 if letter=='cat': labels=np.append(labels,n_img*[0]) else: labels=np.append(labels,n_img*[1]) temp=np.array([images, labels]) temp=temp.transpose() #矩阵转置 np.random.shuffle(temp) #随机排序 image_list=list(temp[:, 0]) label_list=list(temp[:, 1]) label_list=[int(float(i)) for i in label_list] return image_list,label_list def get_batch(image_list,label_list,img_width,img_height,batch_size,capacity): image=tf.cast(image_list,tf.string) label=tf.cast(label_list,tf.int32) input_queue=tf.train.slice_input_producer([image,label]) label=input_queue[1] image_contents=tf.read_file(input_queue[0]) #通过图片地址读取图片 image=tf.image.decode_jpeg(image_contents,channels=3) #解码图片成矩阵 image=tf.image.resize_image_with_crop_or_pad(image,img_width,img_height) ''' tf.image.resize_images 不能保证图像的纵横比,这样用来做抓取位姿的识别,可能受到影响 tf.image.resize_image_with_crop_or_pad可让纵横比不变 ''' image=tf.image.per_image_standardization(image) #将图片标准化 image_batch,label_batch=tf.train.batch([image,label],batch_size=batch_size,num_threads=64,capacity=capacity) ''' tf.train.batch([example, label], batch_size=batch_size, capacity=capacity): 1.[example, label]表示样本和样本标签,这个可以是一个样本和一个样本标签 2.batch_size是返回的一个batch样本集的样本个数 3.num_threads是线程 4.capacity是队列中的容量。 ''' label_batch=tf.reshape(label_batch,[batch_size]) return image_batch,label_batch def one_hot(labels): '''one-hot 编码''' n_sample=len(labels) n_class=max(labels)+1 onehot_labels=np.zeros((n_sample,n_class)) onehot_labels[np.arange(n_sample),labels]=1 return onehot_labels get_file('C:\\Users\\Telon_Hu\\Desktop\\ANNs\\dogs_vs_cats\\') ``` ``` import tensorflow as tf import numpy as np import matplotlib.pyplot as plt import time import os import sys import creat_and_read_TFReacord as reader x_train,y_train=reader.get_file('dogs_vs_cats') image_batch,label_batch=reader.get_batch(x_train,y_train,227,227,50,2048) #Batch_Normalization正则化 def batch_norm(inputs,is_train,is_conv_out=True,decay=0.999): scale=tf.Variable(tf.ones([inputs.get_shape()[-1]])) beta = tf.Variable(tf.zeros([inputs.get_shape()[-1]])) pop_mean = tf.Variable(tf.zeros([inputs.get_shape()[-1]]), trainable=False) pop_var = tf.Variable(tf.ones([inputs.get_shape()[-1]]), trainable=False) if is_train: if is_conv_out: batch_mean, batch_var = tf.nn.moments(inputs, [0, 1, 2]) else: batch_mean, batch_var = tf.nn.moments(inputs, [0]) train_mean = tf.assign(pop_mean, pop_mean * decay + batch_mean * (1 - decay)) train_var = tf.assign(pop_var, pop_var * decay + batch_var * (1 - decay)) with tf.control_dependencies([train_mean, train_var]): return tf.nn.batch_normalization(inputs, batch_mean, batch_var, beta, scale, 0.001) else: return tf.nn.batch_normalization(inputs, pop_mean, pop_var, beta, scale, 0.001) with tf.device('/gpu:0'): # 模型参数 learning_rate = 1e-4 training_iters = 200 batch_size = 50 display_step = 5 n_classes = 2 n_fc1 = 4096 n_fc2 = 2048 # 构建模型 x = tf.placeholder(tf.float32, [None, 227, 227, 3]) y = tf.placeholder(tf.float32, [None, n_classes]) W_conv = { 'conv1': tf.Variable(tf.truncated_normal([11, 11, 3, 96], stddev=0.0001)), 'conv2': tf.Variable(tf.truncated_normal([5, 5, 96, 256], stddev=0.01)), 'conv3': tf.Variable(tf.truncated_normal([3, 3, 256, 384], stddev=0.01)), 'conv4': tf.Variable(tf.truncated_normal([3, 3, 384, 384], stddev=0.01)), 'conv5': tf.Variable(tf.truncated_normal([3, 3, 384, 256], stddev=0.01)), 'fc1': tf.Variable(tf.truncated_normal([6 * 6 * 256, n_fc1], stddev=0.1)), 'fc2': tf.Variable(tf.truncated_normal([n_fc1, n_fc2], stddev=0.1)), 'fc3': tf.Variable(tf.truncated_normal([n_fc2, n_classes], stddev=0.1)) } b_conv = { 'conv1': tf.Variable(tf.constant(0.0, dtype=tf.float32, shape=[96])), 'conv2': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[256])), 'conv3': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[384])), 'conv4': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[384])), 'conv5': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[256])), 'fc1': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[n_fc1])), 'fc2': tf.Variable(tf.constant(0.1, dtype=tf.float32, shape=[n_fc2])), 'fc3': tf.Variable(tf.constant(0.0, dtype=tf.float32, shape=[n_classes])) } x_image = tf.reshape(x, [-1, 227, 227, 3]) # 卷积层 1 conv1 = tf.nn.conv2d(x_image, W_conv['conv1'], strides=[1, 4, 4, 1], padding='VALID') conv1 = tf.nn.bias_add(conv1, b_conv['conv1']) conv1 = batch_norm(conv1, True) conv1 = tf.nn.relu(conv1) # 池化层 1 pool1 = tf.nn.avg_pool(conv1, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID') norm1 = tf.nn.lrn(pool1, 5, bias=1.0, alpha=0.001 / 9.0, beta=0.75) # 卷积层 2 conv2 = tf.nn.conv2d(pool1, W_conv['conv2'], strides=[1, 1, 1, 1], padding='SAME') conv2 = tf.nn.bias_add(conv2, b_conv['conv2']) conv2 = batch_norm(conv2, True) conv2 = tf.nn.relu(conv2) # 池化层 2 pool2 = tf.nn.avg_pool(conv2, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID') # 卷积层3 conv3 = tf.nn.conv2d(pool2, W_conv['conv3'], strides=[1, 1, 1, 1], padding='SAME') conv3 = tf.nn.bias_add(conv3, b_conv['conv3']) conv3 = batch_norm(conv3, True) conv3 = tf.nn.relu(conv3) # 卷积层4 conv4 = tf.nn.conv2d(conv3, W_conv['conv4'], strides=[1, 1, 1, 1], padding='SAME') conv4 = tf.nn.bias_add(conv4, b_conv['conv4']) conv4 = batch_norm(conv4, True) conv4 = tf.nn.relu(conv4) # 卷积层5 conv5 = tf.nn.conv2d(conv4, W_conv['conv5'], strides=[1, 1, 1, 1], padding='SAME') conv5 = tf.nn.bias_add(conv5, b_conv['conv5']) conv5 = batch_norm(conv5, True) conv5 = tf.nn.relu(conv5) # 池化层5 pool5 = tf.nn.avg_pool(conv5, ksize=[1, 3, 3, 1], strides=[1, 2, 2, 1], padding='VALID') reshape = tf.reshape(pool5, [-1, 6 * 6 * 256]) fc1 = tf.add(tf.matmul(reshape, W_conv['fc1']), b_conv['fc1']) fc1 = batch_norm(fc1, True, False) fc1 = tf.nn.relu(fc1) # 全连接层 2 fc2 = tf.add(tf.matmul(fc1, W_conv['fc2']), b_conv['fc2']) fc2 = batch_norm(fc2, True, False) fc2 = tf.nn.relu(fc2) fc3 = tf.add(tf.matmul(fc2, W_conv['fc3']), b_conv['fc3']) # 定义损失 loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y,logits=fc3)) optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss) # 评估模型 correct_pred = tf.equal(tf.argmax(fc3,1),tf.argmax(y,1)) accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32)) init = tf.global_variables_initializer() def onehot(labels): '''one-hot 编码''' n_sample = len(labels) n_class = max(labels) + 1 onehot_labels = np.zeros((n_sample, n_class)) onehot_labels[np.arange(n_sample), labels] = 1 return onehot_labels save_model = ".//model//AlexNetModel.ckpt" def train(opech): with tf.Session() as sess: sess.run(init) train_writer = tf.summary.FileWriter(".//log", sess.graph) # 输出日志的地方 saver = tf.train.Saver() c = [] start_time = time.time() coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord) step = 0 for i in range(opech): step = i image, label = sess.run([image_batch, label_batch]) labels = onehot(label) acc=[] sess.run(optimizer, feed_dict={x: image, y: labels}) loss_record = sess.run(loss, feed_dict={x: image, y: labels}) acc=sess.run(accuracy,feed_dict={x:image,y:labels}) print("now the loss is %f " % loss_record) print("now the accuracy is %f "%acc) c.append(loss_record) end_time = time.time() print('time: ', (end_time - start_time)) start_time = end_time print("---------------%d onpech is finished-------------------" % i) print("Optimization Finished!") # checkpoint_path = os.path.join(".//model", 'model.ckpt') # 输出模型的地方 saver.save(sess, save_model) print("Model Save Finished!") coord.request_stop() coord.join(threads) plt.plot(c) plt.xlabel('Iter') plt.ylabel('loss') plt.title('lr=%f, ti=%d, bs=%d' % (learning_rate, training_iters, batch_size)) plt.tight_layout() plt.savefig('cat_and_dog_AlexNet.jpg', dpi=200) train(training_iters) ```
使用matconvet的问题求解答
在使用matconvet时出现Reference to non-existent field 'normalization',是怎么回事?总是不能正常运行
Alexnet分类问题,程序输入不匹配
用Alexnet网络做一个二分类问题,输入的图片也是227乘227的彩图。遇到了如下的问题![说是形状不匹配图片说明](https://img-ask.csdn.net/upload/201704/28/1493379079_9640.jpg)也不知道怎么解决,求大神帮忙 from __future__ import division, print_function, absolute_import import os import random from PIL import Image import numpy as np import tflearn from tflearn.layers.core import input_data, dropout, fully_connected from tflearn.layers.conv import conv_2d, max_pool_2d, upsample_2d from tflearn.layers.normalization import local_response_normalization from tflearn.layers.estimator import regression #import tflearn.datasets.oxflower17 as oxflower17 #X, Y = oxflower17.load_data(one_hot=True, resize_pics=(227, 227)) np.random.seed(170) def load_data(DataDir): data = np.empty((170,227,227,3),dtype="float32") #37是图片个数,800*800为图片大小,3是图片通道数 label = np.empty((170,),dtype="int") imgs = os.listdir(DataDir) num = len(imgs) for i in range(num): img = Image.open(DataDir+imgs[i]) arr = np.asarray(img,dtype="float32") data[i,:,:,:] = arr if i<53: label[i] = int(0) #o是无缺陷类,共170张图,第0-52张为无缺陷类。 else: label[i] = int(1) data /= np.max(data) #这两行是数据归一化,不用管 data -= np.mean(data) return data,label data,label=load_data('C:/Users/Administrator/Desktop/cnntest/picture/') index = [i for i in range(len(data))] random.shuffle(index) #之前做标签时,数据是按类排的,这边直接打乱顺序。所以标签还是一一对应的。 data = data[index] label = label[index] (TrainData,TestData) = (data[0:119],data[120:]) #traindata包括了两类数据,不用分开来输入。7:3训练集:预测集 (TrainLabel,TestLabel) = (label[0:119],label[120:]) # Building 'AlexNet' network = input_data(shape=[None, 227, 227, 3]) network = conv_2d(network, 96, 11, strides=4, activation='relu') #96为滤波器个数,11为滤波器大小 network = max_pool_2d(network, 3, strides=2) network = local_response_normalization(network) network = conv_2d(network, 256, 5, activation='relu', group=2) network = max_pool_2d(network, 3, strides=2) network = local_response_normalization(network) network = conv_2d(network, 384, 3, activation='relu') network = conv_2d(network, 384, 3, activation='relu') network = conv_2d(network, 256, 3, activation='relu') network = max_pool_2d(network, 3, strides=2) network = local_response_normalization(network) network = upsample_2d(network,2,name='upsample') network = fully_connected(network, 4096, activation='relu') network = dropout(network, 0.5) network = fully_connected(network, 4096, activation='relu') network = dropout(network, 0.5) #net = tflearn.global_avg_pool(net) network = fully_connected(network, 2, activation='softmax') network = regression(network, optimizer='momentum', loss='categorical_crossentropy', learning_rate=0.001) # Training print('Training ------------') model = tflearn.DNN(network, checkpoint_path='model_alexnet', max_checkpoints=1, tensorboard_verbose=2) model.fit(TrainData,TrainLabel, n_epoch=5, validation_set=0.1, shuffle=True, show_metric=True, batch_size=64, snapshot_step=200, snapshot_epoch=False, run_id='CNNPOTATO') model.save('CNNPOTATO.model') model.load('CNNPOTATO.model') print('\nTesting ------------') # Evaluate the model with the metrics we defined earlier loss, accuracy = model.evaluate(TestData, TestLabel) print('\ntest loss: ', loss) print('\ntest accuracy: ', accuracy) #print(model.predict([Y[1]])) ``` ```
Theano 报错:Wrong number of dimensions...
错误出现在:...(l2.b, l2.b - learning__rate * gb2)]) TypeError: ('Bad input argument to theano function with name "C:/Users/Administrator/Desktop/...python/theano/Regularization....py:66" at index 1(0-based)', 'Wrong number of dimensions: expected 2, got 1 with shape (200,).') ```__ import theano from sklearn.datasets import load_boston import theano.tensor as T import numpy as np import matplotlib.pyplot as plt class Layer(object):#定义神经层 def __init__(self, inputs, in_size, out_size, activation_function=None): self.W = theano.shared(np.random.normal(0, 1, (in_size, out_size))) self.b = theano.shared(np.zeros((out_size, )) + 0.1) self.Wx_plus_b = T.dot(inputs, self.W) + self.b self.activation_function = activation_function if activation_function is None: self.outputs = self.Wx_plus_b else: self.outputs = self.activation_function(self.Wx_plus_b) def minmax_normalization(data):#正则化数据 xs_max = np.max(data, axis=0) xs_min = np.min(data, axis=0) xs = (1 - 0) * (data - xs_min) / (xs_max - xs_min) + 0 return xs N=400 feats=28 lamda=0.1 np.random.seed(100) x_data = rng.randn(N, feats) x_data = minmax_normalization(x_data) y_data = rng.randint(size=N, low=0, high=2) x_train, y_train = x_data[:200], y_data[:200] x_test, y_test = x_data[200:], y_data[200:] x = T.dmatrix("x") y = T.dmatrix("y") l1 = Layer(x, 13, 50, T.tanh) l2 = Layer(l1.outputs, 50, 1, None) cost = T.mean(T.square(l2.outputs - y)) + lamda * ((l1.W ** 2).sum() + (l2.W ** 2).sum()) gW1, gb1, gW2, gb2 = T.grad(cost, [l1.W, l1.b, l2.W, l2.b]) learning_rate = 0.01 train = theano.function( inputs=[x, y], updates=[(l1.W, l1.W - learning_rate * gW1), (l1.b, l1.b - learning_rate * gb1), (l2.W, l2.W - learning_rate * gW2), (l2.b, l2.b - learning_rate * gb2)]) compute_cost = theano.function(inputs=[x, y], outputs=cost) train_err_list = [] test_err_list = [] learning_time = [] for i in range(1000): train(x_train, y_train) if i % 10 == 0: # record cost train_err_list.append(compute_cost(x_train, y_train)) test_err_list.append(compute_cost(x_test, y_test)) learning_time.append(i) plt.plot(learning_time, train_err_list, 'r-') plt.plot(learning_time, test_err_list, 'b--') plt.show() ``` __
Tensorflow实现简单CNN模型中某一层shape的计算问题
RT,网上看到一篇资料,实现了一个简单的CNN模型,但是有个shape我有点蒙,不知道怎么算的,代码如下: 这是alexnet网络定义的部分 ,我们只需要修改这一部就可以了 ``` def alex_net(_X, _weights, _biases, _dropout): # Reshape input picture _X = tf.reshape(_X, shape=[-1, 28, 28, 1]) # Convolution Layer conv1 = conv2d('conv1', _X, _weights['wc1'], _biases['bc1']) # Max Pooling (down-sampling) pool1 = max_pool('pool1', conv1, k=2) # Apply Normalization norm1 = norm('norm1', pool1, lsize=4) # Apply Dropout norm1 = tf.nn.dropout(norm1, _dropout) # Convolution Layer conv2 = conv2d('conv2', norm1, _weights['wc2'], _biases['bc2']) # Max Pooling (down-sampling) pool2 = max_pool('pool2', conv2, k=2) # Apply Normalization norm2 = norm('norm2', pool2, lsize=4) # Apply Dropout norm2 = tf.nn.dropout(norm2, _dropout) # Convolution Layer conv3 = conv2d('conv3', norm2, _weights['wc3'], _biases['bc3']) # Max Pooling (down-sampling) pool3 = max_pool('pool3', conv3, k=2) # Apply Normalization norm3 = norm('norm3', pool3, lsize=4) # Apply Dropout norm3 = tf.nn.dropout(norm3, _dropout) # Fully connected layer dense1 = tf.reshape(norm3, [-1, _weights['wd1'].get_shape().as_list()[0]]) # Reshape conv3 output to fit dense layer input dense1 = tf.nn.relu(tf.matmul(dense1, _weights['wd1']) + _biases['bd1'], name='fc1') # Relu activation dense2 = tf.nn.relu(tf.matmul(dense1, _weights['wd2']) + _biases['bd2'], name='fc2') # Relu activation # Output, class prediction out = tf.matmul(dense2, _weights['out']) + _biases['out'] return out # Store layers weight & bias weights = { 'wc1': tf.Variable(tf.random_normal([3, 3, 1, 64])), 'wc2': tf.Variable(tf.random_normal([3, 3, 64, 128])), 'wc3': tf.Variable(tf.random_normal([3, 3, 128, 256])), 'wd1': tf.Variable(tf.random_normal([4*4*256, 1024])), 'wd2': tf.Variable(tf.random_normal([1024, 1024])), 'out': tf.Variable(tf.random_normal([1024, 10])) } biases = { 'bc1': tf.Variable(tf.random_normal([64])), 'bc2': tf.Variable(tf.random_normal([128])), 'bc3': tf.Variable(tf.random_normal([256])), 'bd1': tf.Variable(tf.random_normal([1024])), 'bd2': tf.Variable(tf.random_normal([1024])), 'out': tf.Variable(tf.random_normal([n_classes])) } # Construct model pred = alex_net(x, weights, biases, keep_prob) ``` weights 项下的“wd1”,shape的输入处是4*4*256,4是怎么算出来的,学生自学所以不是很明白,各位帮忙解释一下 资料的网址:TensorFlow人工智能引擎入门教程之三 实现一个自创的CNN卷积神经网络 - zhuyuping的个人空间 https://my.oschina.net/yilian/blog/661409
构建cycleGAN严重问题
构建cycleGAN运行到鉴别器的优化时报错Conv2DCustomBackpropInput:input depth must be evenly divisible by filter depth求处理办法o(╥﹏╥)o ``` def discriminator(x,name,is_training=True): reuse=len([var for var in tf.trainable_variables() if var.name.startswith(name)])>0 with tf.variable_scope(name,reuse=reuse): #256 3->128 64 layer_1=tf.layers.conv2d(x,64,4,2,padding='SAME') layer_1=tf.nn.leaky_relu(layer_1) #128 64->64 128 layer_2=tf.layers.batch_normalization(tf.layers.conv2d(layer_1,128,4,2,padding='SAME'),training=is_training) layer_2=tf.nn.leaky_relu(layer_2) #64 128->32 256 layer_3=tf.layers.batch_normalization(tf.layers.conv2d(layer_2,256,4,2,padding='SAME'),training=is_training) layer_3=tf.nn.leaky_relu(layer_3) layer=tf.layers.conv2d(layer_3,1,4,1,padding='SAME') return layer ``` 鉴别器的结构如上,为复现论文的结果 损失函数为交叉熵 ``` #判别器器损失 d_x=tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_fake_x,labels=tf.zeros_like(disc_fake_x)))+tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_real_x,labels=tf.ones_like(disc_real_x))) d_y=tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_fake_y,labels=tf.zeros_like(disc_fake_y)))+tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=disc_real_y,labels=tf.ones_like(disc_real_y))) d_loss=d_y+d_x ``` 完整报错: ``` Traceback (most recent call last): File "C:\Users\lenovo\Desktop\library\CycleGAN\CycleGan.py", line 150, in <module> sess.run(g_train,fd) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\client\session.py", line 900, in run run_metadata_ptr) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\client\session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\client\session.py", line 1316, in _do_run run_metadata) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\client\session.py", line 1335, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInput: input depth must be evenly divisible by filter depth [[Node: gradients_1/discriminator_d_x/conv2d_3/Conv2D_grad/Conv2DBackpropInput = _MklConv2DBackpropInput[T=DT_FLOAT, _kernel="MklOp", data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](gradients_1/discriminator_d_x/conv2d_3/Conv2D_grad/ShapeN, discriminator_d_x/conv2d_3/kernel/read, gradients_1/AddN, DMT/_350, discriminator_d_x/conv2d_3/kernel/read:1, DMT/_351)]] Caused by op 'gradients_1/discriminator_d_x/conv2d_3/Conv2D_grad/Conv2DBackpropInput', defined at: File "E:\java\pyzo\source\pyzo\pyzokernel\start.py", line 151, in <module> __pyzo__.run() File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 222, in run self.guiApp.run(self.process_commands, self.sleeptime) File "E:\java\pyzo\source\pyzo\pyzokernel\guiintegration.py", line 85, in run repl_callback() File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 583, in process_commands self._process_commands() File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 611, in _process_commands self.runfile(tmp) File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 887, in runfile self.execcode(code) File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 950, in execcode exec(code, self.locals) File "C:\Users\lenovo\Desktop\library\CycleGAN\CycleGan.py", line 140, in <module> g_train=tf.train.GradientDescentOptimizer(0.002).minimize(g_loss,var_list=g_vars) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\training\optimizer.py", line 399, in minimize grad_loss=grad_loss) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\training\optimizer.py", line 511, in compute_gradients colocate_gradients_with_ops=colocate_gradients_with_ops) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 532, in gradients gate_gradients, aggregation_method, stop_gradients) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in _GradientsHelper lambda: grad_fn(op, *out_grads)) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 396, in _MaybeCompile return grad_fn() # Exit early File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 701, in <lambda> lambda: grad_fn(op, *out_grads)) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_grad.py", line 520, in _Conv2DGrad data_format=data_format), File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1340, in conv2d_backprop_input dilations=dilations, name=name) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\ops.py", line 3414, in create_op op_def=op_def) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\ops.py", line 1740, in __init__ self._traceback = self._graph._extract_stack() # pylint: disable=protected-access ...which was originally created as op 'discriminator_d_x/conv2d_3/Conv2D', defined at: File "E:\java\pyzo\source\pyzo\pyzokernel\start.py", line 151, in <module> __pyzo__.run() [elided 5 identical lines from previous traceback] File "E:\java\pyzo\source\pyzo\pyzokernel\interpreter.py", line 950, in execcode exec(code, self.locals) File "C:\Users\lenovo\Desktop\library\CycleGAN\CycleGan.py", line 108, in <module> disc_fake_x=discriminator(fake_x,'discriminator_d_x') File "C:\Users\lenovo\Desktop\library\CycleGAN\CycleGan.py", line 97, in discriminator layer7=tf.layers.conv2d(layer_3,1,4,1,padding='SAME') File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\layers\convolutional.py", line 427, in conv2d return layer.apply(inputs) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 774, in apply return self.__call__(inputs, *args, **kwargs) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\layers\base.py", line 329, in __call__ outputs = super(Layer, self).__call__(inputs, *args, **kwargs) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\engine\base_layer.py", line 703, in __call__ outputs = self.call(inputs, *args, **kwargs) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\keras\layers\convolutional.py", line 184, in call outputs = self._convolution_op(inputs, self.kernel) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 868, in __call__ return self.conv_op(inp, filter) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 520, in __call__ return self.call(inp, filter) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 204, in __call__ name=self.name) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 1042, in conv2d data_format=data_format, dilations=dilations, name=name) File "c:\users\lenovo\appdata\local\programs\python\python37\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper op_def=op_def) InvalidArgumentError (see above for traceback): Conv2DCustomBackpropInput: input depth must be evenly divisible by filter depth [[Node: gradients_1/discriminator_d_x/conv2d_3/Conv2D_grad/Conv2DBackpropInput = _MklConv2DBackpropInput[T=DT_FLOAT, _kernel="MklOp", data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](gradients_1/discriminator_d_x/conv2d_3/Conv2D_grad/ShapeN, discriminator_d_x/conv2d_3/kernel/read, gradients_1/AddN, DMT/_350, discriminator_d_x/conv2d_3/kernel/read:1, DMT/_351)]] ```
YOLO在本机运行时候报错
问题描述: InternalError (see above for traceback): Blas SGEMM launch failed : m=81920, n=32, k=64 [[node conv2d_3/convolution (defined at j:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py:3650) = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu, conv2d_3/kernel/read)]] [[{{node concat_11/_2897}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3860_concat_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] 用了https://blog.csdn.net/gsww404/article/details/80507704上所说的方法还是无法解决该问题,请问还有可能是什么运行吗?
为什么同样的问题用Tensorflow和keras实现结果不一样?
**cifar-10分类问题,同样的模型结构以及损失函数还有学习率参数等超参数,分别用TensorFlow和keras实现。 20个epochs后在测试集上进行预测,准确率总是差好几个百分点,不知道问题出在哪里?代码如下: 这个是TF的代码:** import tensorflow as tf import numpy as np import pickle as pk tf.reset_default_graph() batch_size = 64 test_size = 10000 img_size = 32 num_classes = 10 training_epochs = 10 test_size=200 ############################################################################### def unpickle(filename): '''解压数据''' with open(filename, 'rb') as f: d = pk.load(f, encoding='latin1') return d def onehot(labels): '''one-hot 编码''' n_sample = len(labels) n_class = max(labels) + 1 onehot_labels = np.zeros((n_sample, n_class)) onehot_labels[np.arange(n_sample), labels] = 1 return onehot_labels # 训练数据集 data1 = unpickle('data_batch_1') data2 = unpickle('data_batch_2') data3 = unpickle('data_batch_3') data4 = unpickle('data_batch_4') data5 = unpickle('data_batch_5') X_train = np.concatenate((data1['data'], data2['data'], data3['data'], data4['data'], data5['data']), axis=0)/255.0 y_train = np.concatenate((data1['labels'], data2['labels'], data3['labels'], data4['labels'], data5['labels']), axis=0) y_train = onehot(y_train) # 测试数据集 test = unpickle('test_batch') X_test = test['data']/255.0 y_test = onehot(test['labels']) del test,data1,data2,data3,data4,data5 ############################################################################### w = tf.Variable(tf.random_normal([5, 5, 3, 32], stddev=0.01)) w_c= tf.Variable(tf.random_normal([32* 16* 16, 512], stddev=0.1)) w_o =tf.Variable(tf.random_normal([512, num_classes], stddev=0.1)) def init_bias(shape): return tf.Variable(tf.constant(0.0, shape=shape)) b=init_bias([32]) b_c=init_bias([512]) b_o=init_bias([10]) def model(X, w, w_c,w_o, p_keep_conv, p_keep_hidden,b,b_c,b_o): conv1 = tf.nn.conv2d(X, w,strides=[1, 1, 1, 1],padding='SAME')#32x32x32 conv1=tf.nn.bias_add(conv1,b) conv1 = tf.nn.relu(conv1) conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1],strides=[1, 2, 2, 1],padding='SAME')#16x16x32 conv1 = tf.nn.dropout(conv1, p_keep_conv) FC_layer = tf.reshape(conv1, [-1, 32 * 16 * 16]) out_layer=tf.matmul(FC_layer, w_c)+b_c out_layer=tf.nn.relu(out_layer) out_layer = tf.nn.dropout(out_layer, p_keep_hidden) result = tf.matmul(out_layer, w_o)+b_o return result trX, trY, teX, teY = X_train,y_train,X_test,y_test trX = trX.reshape(-1, img_size, img_size, 3) teX = teX.reshape(-1, img_size, img_size, 3) X = tf.placeholder("float", [None, img_size, img_size, 3]) Y = tf.placeholder("float", [None, num_classes]) p_keep_conv = tf.placeholder("float") p_keep_hidden = tf.placeholder("float") py_x = model(X, w, w_c,w_o, p_keep_conv, p_keep_hidden,b,b_c,b_o) Y_ = tf.nn.softmax_cross_entropy_with_logits_v2(logits=py_x, labels=Y) cost = tf.reduce_mean(Y_) optimizer = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cost) predict_op = tf.argmax(py_x, 1) with tf.Session() as sess: tf.global_variables_initializer().run() for i in range(training_epochs): training_batch = zip(range(0, len(trX),batch_size),range(batch_size, len(trX)+1,batch_size)) perm=np.arange(len(trX)) np.random.shuffle(perm) trX=trX[perm] trY=trY[perm] for start, end in training_batch: sess.run(optimizer, feed_dict={X: trX[start:end],Y: trY[start:end],p_keep_conv:0.75,p_keep_hidden: 0.5}) test_batch = zip(range(0, len(teX),test_size),range(test_size, len(teX)+1,test_size)) accuracyResult=0 for start, end in test_batch: accuracyResult=accuracyResult+sum(np.argmax(teY[start:end], axis=1) ==sess.run(predict_op, feed_dict={X: teX[start:end],Y: teY[start:end],p_keep_conv: 1,p_keep_hidden: 1})) print(i, accuracyResult/10000) **这个是keras代码:** from keras import initializers from keras.datasets import cifar10 from keras.utils import np_utils from keras.models import Sequential from keras.layers.core import Dense, Dropout, Activation, Flatten from keras.layers.convolutional import Conv2D, MaxPooling2D from keras.optimizers import SGD, Adam, RMSprop #import matplotlib.pyplot as plt # CIFAR_10 is a set of 60K images 32x32 pixels on 3 channels IMG_CHANNELS = 3 IMG_ROWS = 32 IMG_COLS = 32 #constant BATCH_SIZE = 64 NB_EPOCH = 10 NB_CLASSES = 10 VERBOSE = 1 VALIDATION_SPLIT = 0 OPTIM = RMSprop() #load dataset (X_train, y_train), (X_test, y_test) = cifar10.load_data() #print('X_train shape:', X_train.shape) #print(X_train.shape[0], 'train samples') #print(X_test.shape[0], 'test samples') # convert to categorical Y_train = np_utils.to_categorical(y_train, NB_CLASSES) Y_test = np_utils.to_categorical(y_test, NB_CLASSES) # float and normalization X_train = X_train.astype('float32') X_test = X_test.astype('float32') X_train /= 255 X_test /= 255 # network model = Sequential() model.add(Conv2D(32, (3, 3), padding='same',input_shape=(IMG_ROWS, IMG_COLS, IMG_CHANNELS),kernel_initializer=initializers.random_normal(stddev=0.01),bias_initializer=initializers.Zeros())) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) #0<参数<1才会有用 model.add(Flatten()) model.add(Dense(512,kernel_initializer=initializers.random_normal(stddev=0.1),bias_initializer=initializers.Zeros())) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(NB_CLASSES,kernel_initializer=initializers.random_normal(stddev=0.1),bias_initializer=initializers.Zeros())) model.add(Activation('softmax')) model.summary() # train model.compile(loss='categorical_crossentropy', optimizer=OPTIM,metrics=['accuracy']) model.fit(X_train, Y_train, batch_size=BATCH_SIZE,epochs=NB_EPOCH, validation_split=VALIDATION_SPLIT,verbose=VERBOSE) score = model.evaluate(X_test, Y_test,batch_size=200, verbose=VERBOSE) print("Test score:", score[0]) print('Test accuracy:', score[1])
基于tensorflow的pix2pix代码中如何做到输入图像和输出图像分辨率不一致
问题:例如在自己制作了成对的输入(input256×256 target 200×256)后,如何让输入图像和输出图像分辨率不一致,例如成对图像中:input的分辨率是256×256, output 和target都是200×256,需要修改哪里的参数。 论文参考:《Image-to-Image Translation with Conditional Adversarial Networks》 代码参考:https://blog.csdn.net/MOU_IT/article/details/80802407?utm_source=blogxgwz0 # coding=utf-8 from __future__ import absolute_import from __future__ import division from __future__ import print_function import os os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' import tensorflow as tf import numpy as np import os import glob import random import collections import math import time # https://github.com/affinelayer/pix2pix-tensorflow train_input_dir = "D:/Project/pix2pix-tensorflow-master/facades/train/" # 训练集输入 train_output_dir = "D:/Project/pix2pix-tensorflow-master/facades/train_out/" # 训练集输出 test_input_dir = "D:/Project/pix2pix-tensorflow-master/facades/val/" # 测试集输入 test_output_dir = "D:/Project/pix2pix-tensorflow-master/facades/test_out/" # 测试集的输出 checkpoint = "D:/Project/pix2pix-tensorflow-master/facades/train_out/" # 保存结果的目录 seed = None max_steps = None # number of training steps (0 to disable) max_epochs = 200 # number of training epochs progress_freq = 50 # display progress every progress_freq steps trace_freq = 0 # trace execution every trace_freq steps display_freq = 50 # write current training images every display_freq steps save_freq = 500 # save model every save_freq steps, 0 to disable separable_conv = False # use separable convolutions in the generator aspect_ratio = 1 #aspect ratio of output images (width/height) batch_size = 1 # help="number of images in batch") which_direction = "BtoA" # choices=["AtoB", "BtoA"]) ngf = 64 # help="number of generator filters in first conv layer") ndf = 64 # help="number of discriminator filters in first conv layer") scale_size = 286 # help="scale images to this size before cropping to 256x256") flip = True # flip images horizontally no_flip = True # don't flip images horizontally lr = 0.0002 # initial learning rate for adam beta1 = 0.5 # momentum term of adam l1_weight = 100.0 # weight on L1 term for generator gradient gan_weight = 1.0 # weight on GAN term for generator gradient output_filetype = "png" # 输出图像的格式 EPS = 1e-12 # 极小数,防止梯度为损失为0 CROP_SIZE = 256 # 图片的裁剪大小 # 命名元组,用于存放加载的数据集合创建好的模型 Examples = collections.namedtuple("Examples", "paths, inputs, targets, count, steps_per_epoch") Model = collections.namedtuple("Model", "outputs, predict_real, predict_fake, discrim_loss, discrim_grads_and_vars, gen_loss_GAN, gen_loss_L1, gen_grads_and_vars, train") # 图像预处理 [0, 1] => [-1, 1] def preprocess(image): with tf.name_scope("preprocess"): return image * 2 - 1 # 图像后处理[-1, 1] => [0, 1] def deprocess(image): with tf.name_scope("deprocess"): return (image + 1) / 2 # 判别器的卷积定义,batch_input为 [ batch , 256 , 256 , 6 ] def discrim_conv(batch_input, out_channels, stride): # [ batch , 256 , 256 , 6 ] ===>[ batch , 258 , 258 , 6 ] padded_input = tf.pad(batch_input, [[0, 0], [1, 1], [1, 1], [0, 0]], mode="CONSTANT") ''' [0,0]: 第一维batch大小不扩充 [1,1]:第二维图像宽度左右各扩充一列,用0填充 [1,1]:第三维图像高度上下各扩充一列,用0填充 [0,0]:第四维图像通道不做扩充 ''' return tf.layers.conv2d(padded_input, out_channels, kernel_size=4, strides=(stride, stride), padding="valid", kernel_initializer=tf.random_normal_initializer(0, 0.02)) # 生成器的卷积定义,卷积核为4*4,步长为2,输出图像为输入的一半 def gen_conv(batch_input, out_channels): # [batch, in_height, in_width, in_channels] => [batch, out_height, out_width, out_channels] initializer = tf.random_normal_initializer(0, 0.02) if separable_conv: return tf.layers.separable_conv2d(batch_input, out_channels, kernel_size=4, strides=(2, 2), padding="same", depthwise_initializer=initializer, pointwise_initializer=initializer) else: return tf.layers.conv2d(batch_input, out_channels, kernel_size=4, strides=(2, 2), padding="same", kernel_initializer=initializer) # 生成器的反卷积定义 def gen_deconv(batch_input, out_channels): # [batch, in_height, in_width, in_channels] => [batch, out_height, out_width, out_channels] initializer = tf.random_normal_initializer(0, 0.02) if separable_conv: _b, h, w, _c = batch_input.shape resized_input = tf.image.resize_images(batch_input, [h * 2, w * 2], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR) return tf.layers.separable_conv2d(resized_input, out_channels, kernel_size=4, strides=(1, 1), padding="same", depthwise_initializer=initializer, pointwise_initializer=initializer) else: return tf.layers.conv2d_transpose(batch_input, out_channels, kernel_size=4, strides=(2, 2), padding="same", kernel_initializer=initializer) # 定义LReLu激活函数 def lrelu(x, a): with tf.name_scope("lrelu"): # adding these together creates the leak part and linear part # then cancels them out by subtracting/adding an absolute value term # leak: a*x/2 - a*abs(x)/2 # linear: x/2 + abs(x)/2 # this block looks like it has 2 inputs on the graph unless we do this x = tf.identity(x) return (0.5 * (1 + a)) * x + (0.5 * (1 - a)) * tf.abs(x) # 批量归一化图像 def batchnorm(inputs): return tf.layers.batch_normalization(inputs, axis=3, epsilon=1e-5, momentum=0.1, training=True, gamma_initializer=tf.random_normal_initializer(1.0, 0.02)) # 检查图像的维度 def check_image(image): assertion = tf.assert_equal(tf.shape(image)[-1], 3, message="image must have 3 color channels") with tf.control_dependencies([assertion]): image = tf.identity(image) if image.get_shape().ndims not in (3, 4): raise ValueError("image must be either 3 or 4 dimensions") # make the last dimension 3 so that you can unstack the colors shape = list(image.get_shape()) shape[-1] = 3 image.set_shape(shape) return image # 去除文件的后缀,获取文件名 def get_name(path): # os.path.basename(),返回path最后的文件名。若path以/或\结尾,那么就会返回空值。 # os.path.splitext(),分离文件名与扩展名;默认返回(fname,fextension)元组 name, _ = os.path.splitext(os.path.basename(path)) return name # 加载数据集,从文件读取-->解码-->归一化--->拆分为输入和目标-->像素转为[-1,1]-->转变形状 def load_examples(input_dir): if input_dir is None or not os.path.exists(input_dir): raise Exception("input_dir does not exist") # 匹配第一个参数的路径中所有的符合条件的文件,并将其以list的形式返回。 input_paths = glob.glob(os.path.join(input_dir, "*.jpg")) # 图像解码器 decode = tf.image.decode_jpeg if len(input_paths) == 0: input_paths = glob.glob(os.path.join(input_dir, "*.png")) decode = tf.image.decode_png if len(input_paths) == 0: raise Exception("input_dir contains no image files") # 如果文件名是数字,则用数字进行排序,否则用字母排序 if all(get_name(path).isdigit() for path in input_paths): input_paths = sorted(input_paths, key=lambda path: int(get_name(path))) else: input_paths = sorted(input_paths) sess = tf.Session() with tf.name_scope("load_images"): # 把我们需要的全部文件打包为一个tf内部的queue类型,之后tf开文件就从这个queue中取目录了, # 如果是训练模式时,shuffle为True path_queue = tf.train.string_input_producer(input_paths, shuffle=True) # Read的输出将是一个文件名(key)和该文件的内容(value,每次读取一个文件,分多次读取)。 reader = tf.WholeFileReader() paths, contents = reader.read(path_queue) # 对文件进行解码并且对图片作归一化处理 raw_input = decode(contents) raw_input = tf.image.convert_image_dtype(raw_input, dtype=tf.float32) # 归一化处理 # 判断两个值知否相等,如果不等抛出异常 assertion = tf.assert_equal(tf.shape(raw_input)[2], 3, message="image does not have 3 channels") ''' 对于control_dependencies这个管理器,只有当里面的操作是一个op时,才会生效,也就是先执行传入的 参数op,再执行里面的op。如果里面的操作不是定义的op,图中就不会形成一个节点,这样该管理器就失效了。 tf.identity是返回一个一模一样新的tensor的op,这会增加一个新节点到gragh中,这时control_dependencies就会生效. ''' with tf.control_dependencies([assertion]): raw_input = tf.identity(raw_input) raw_input.set_shape([None, None, 3]) # 图像值由[0,1]--->[-1, 1] width = tf.shape(raw_input)[1] # [height, width, channels] a_images = preprocess(raw_input[:, :width // 2, :]) # 256*256*3 b_images = preprocess(raw_input[:, width // 2:, :]) # 256*256*3 # 这里的which_direction为:BtoA if which_direction == "AtoB": inputs, targets = [a_images, b_images] elif which_direction == "BtoA": inputs, targets = [b_images, a_images] else: raise Exception("invalid direction") # synchronize seed for image operations so that we do the same operations to both # input and output images seed = random.randint(0, 2 ** 31 - 1) # 图像预处理,翻转、改变形状 with tf.name_scope("input_images"): input_images = transform(inputs) with tf.name_scope("target_images"): target_images = transform(targets) # 获得输入图像、目标图像的batch块 paths_batch, inputs_batch, targets_batch = tf.train.batch([paths, input_images, target_images], batch_size=batch_size) steps_per_epoch = int(math.ceil(len(input_paths) / batch_size)) return Examples( paths=paths_batch, # 输入的文件名块 inputs=inputs_batch, # 输入的图像块 targets=targets_batch, # 目标图像块 count=len(input_paths), # 数据集的大小 steps_per_epoch=steps_per_epoch, # batch的个数 ) # 图像预处理,翻转、改变形状 def transform(image): r = image if flip: r = tf.image.random_flip_left_right(r, seed=seed) # area produces a nice downscaling, but does nearest neighbor for upscaling # assume we're going to be doing downscaling here r = tf.image.resize_images(r, [scale_size, scale_size], method=tf.image.ResizeMethod.AREA) offset = tf.cast(tf.floor(tf.random_uniform([2], 0, scale_size - CROP_SIZE + 1, seed=seed)), dtype=tf.int32) if scale_size > CROP_SIZE: r = tf.image.crop_to_bounding_box(r, offset[0], offset[1], CROP_SIZE, CROP_SIZE) elif scale_size < CROP_SIZE: raise Exception("scale size cannot be less than crop size") return r # 创建生成器,这是一个编码解码器的变种,输入输出均为:256*256*3, 像素值为[-1,1] def create_generator(generator_inputs, generator_outputs_channels): layers = [] # encoder_1: [batch, 256, 256, in_channels] => [batch, 128, 128, ngf] with tf.variable_scope("encoder_1"): output = gen_conv(generator_inputs, ngf) # ngf为第一个卷积层的卷积核核数量,默认为 64 layers.append(output) layer_specs = [ ngf * 2, # encoder_2: [batch, 128, 128, ngf] => [batch, 64, 64, ngf * 2] ngf * 4, # encoder_3: [batch, 64, 64, ngf * 2] => [batch, 32, 32, ngf * 4] ngf * 8, # encoder_4: [batch, 32, 32, ngf * 4] => [batch, 16, 16, ngf * 8] ngf * 8, # encoder_5: [batch, 16, 16, ngf * 8] => [batch, 8, 8, ngf * 8] ngf * 8, # encoder_6: [batch, 8, 8, ngf * 8] => [batch, 4, 4, ngf * 8] ngf * 8, # encoder_7: [batch, 4, 4, ngf * 8] => [batch, 2, 2, ngf * 8] ngf * 8, # encoder_8: [batch, 2, 2, ngf * 8] => [batch, 1, 1, ngf * 8] ] # 卷积的编码器 for out_channels in layer_specs: with tf.variable_scope("encoder_%d" % (len(layers) + 1)): # 对最后一层使用激活函数 rectified = lrelu(layers[-1], 0.2) # [batch, in_height, in_width, in_channels] => [batch, in_height/2, in_width/2, out_channels] convolved = gen_conv(rectified, out_channels) output = batchnorm(convolved) layers.append(output) layer_specs = [ (ngf * 8, 0.5), # decoder_8: [batch, 1, 1, ngf * 8] => [batch, 2, 2, ngf * 8 * 2] (ngf * 8, 0.5), # decoder_7: [batch, 2, 2, ngf * 8 * 2] => [batch, 4, 4, ngf * 8 * 2] (ngf * 8, 0.5), # decoder_6: [batch, 4, 4, ngf * 8 * 2] => [batch, 8, 8, ngf * 8 * 2] (ngf * 8, 0.0), # decoder_5: [batch, 8, 8, ngf * 8 * 2] => [batch, 16, 16, ngf * 8 * 2] (ngf * 4, 0.0), # decoder_4: [batch, 16, 16, ngf * 8 * 2] => [batch, 32, 32, ngf * 4 * 2] (ngf * 2, 0.0), # decoder_3: [batch, 32, 32, ngf * 4 * 2] => [batch, 64, 64, ngf * 2 * 2] (ngf, 0.0), # decoder_2: [batch, 64, 64, ngf * 2 * 2] => [batch, 128, 128, ngf * 2] ] # 卷积的解码器 num_encoder_layers = len(layers) # 8 for decoder_layer, (out_channels, dropout) in enumerate(layer_specs): skip_layer = num_encoder_layers - decoder_layer - 1 with tf.variable_scope("decoder_%d" % (skip_layer + 1)): if decoder_layer == 0: # first decoder layer doesn't have skip connections # since it is directly connected to the skip_layer input = layers[-1] else: input = tf.concat([layers[-1], layers[skip_layer]], axis=3) rectified = tf.nn.relu(input) # [batch, in_height, in_width, in_channels] => [batch, in_height*2, in_width*2, out_channels] output = gen_deconv(rectified, out_channels) output = batchnorm(output) if dropout > 0.0: output = tf.nn.dropout(output, keep_prob=1 - dropout) layers.append(output) # decoder_1: [batch, 128, 128, ngf * 2] => [batch, 256, 256, generator_outputs_channels] with tf.variable_scope("decoder_1"): input = tf.concat([layers[-1], layers[0]], axis=3) rectified = tf.nn.relu(input) output = gen_deconv(rectified, generator_outputs_channels) output = tf.tanh(output) layers.append(output) return layers[-1] # 创建判别器,输入生成的图像和真实的图像:两个[batch,256,256,3],元素值值[-1,1],输出:[batch,30,30,1],元素值为概率 def create_discriminator(discrim_inputs, discrim_targets): n_layers = 3 layers = [] # 2x [batch, height, width, in_channels] => [batch, height, width, in_channels * 2] input = tf.concat([discrim_inputs, discrim_targets], axis=3) # layer_1: [batch, 256, 256, in_channels * 2] => [batch, 128, 128, ndf] with tf.variable_scope("layer_1"): convolved = discrim_conv(input, ndf, stride=2) rectified = lrelu(convolved, 0.2) layers.append(rectified) # layer_2: [batch, 128, 128, ndf] => [batch, 64, 64, ndf * 2] # layer_3: [batch, 64, 64, ndf * 2] => [batch, 32, 32, ndf * 4] # layer_4: [batch, 32, 32, ndf * 4] => [batch, 31, 31, ndf * 8] for i in range(n_layers): with tf.variable_scope("layer_%d" % (len(layers) + 1)): out_channels = ndf * min(2 ** (i + 1), 8) stride = 1 if i == n_layers - 1 else 2 # last layer here has stride 1 convolved = discrim_conv(layers[-1], out_channels, stride=stride) normalized = batchnorm(convolved) rectified = lrelu(normalized, 0.2) layers.append(rectified) # layer_5: [batch, 31, 31, ndf * 8] => [batch, 30, 30, 1] with tf.variable_scope("layer_%d" % (len(layers) + 1)): convolved = discrim_conv(rectified, out_channels=1, stride=1) output = tf.sigmoid(convolved) layers.append(output) return layers[-1] # 创建Pix2Pix模型,inputs和targets形状为:[batch_size, height, width, channels] def create_model(inputs, targets): with tf.variable_scope("generator"): out_channels = int(targets.get_shape()[-1]) outputs = create_generator(inputs, out_channels) # create two copies of discriminator, one for real pairs and one for fake pairs # they share the same underlying variables with tf.name_scope("real_discriminator"): with tf.variable_scope("discriminator"): # 2x [batch, height, width, channels] => [batch, 30, 30, 1] predict_real = create_discriminator(inputs, targets) # 条件变量图像和真实图像 with tf.name_scope("fake_discriminator"): with tf.variable_scope("discriminator", reuse=True): # 2x [batch, height, width, channels] => [batch, 30, 30, 1] predict_fake = create_discriminator(inputs, outputs) # 条件变量图像和生成的图像 # 判别器的损失,判别器希望V(G,D)尽可能大 with tf.name_scope("discriminator_loss"): # minimizing -tf.log will try to get inputs to 1 # predict_real => 1 # predict_fake => 0 discrim_loss = tf.reduce_mean(-(tf.log(predict_real + EPS) + tf.log(1 - predict_fake + EPS))) # 生成器的损失,生成器希望V(G,D)尽可能小 with tf.name_scope("generator_loss"): # predict_fake => 1 # abs(targets - outputs) => 0 gen_loss_GAN = tf.reduce_mean(-tf.log(predict_fake + EPS)) gen_loss_L1 = tf.reduce_mean(tf.abs(targets - outputs)) gen_loss = gen_loss_GAN * gan_weight + gen_loss_L1 * l1_weight # 判别器训练 with tf.name_scope("discriminator_train"): # 判别器需要优化的参数 discrim_tvars = [var for var in tf.trainable_variables() if var.name.startswith("discriminator")] # 优化器定义 discrim_optim = tf.train.AdamOptimizer(lr, beta1) # 计算损失函数对优化参数的梯度 discrim_grads_and_vars = discrim_optim.compute_gradients(discrim_loss, var_list=discrim_tvars) # 更新该梯度所对应的参数的状态,返回一个op discrim_train = discrim_optim.apply_gradients(discrim_grads_and_vars) # 生成器训练 with tf.name_scope("generator_train"): with tf.control_dependencies([discrim_train]): # 生成器需要优化的参数列表 gen_tvars = [var for var in tf.trainable_variables() if var.name.startswith("generator")] # 定义优化器 gen_optim = tf.train.AdamOptimizer(lr, beta1) # 计算需要优化的参数的梯度 gen_grads_and_vars = gen_optim.compute_gradients(gen_loss, var_list=gen_tvars) # 更新该梯度所对应的参数的状态,返回一个op gen_train = gen_optim.apply_gradients(gen_grads_and_vars) ''' 在采用随机梯度下降算法训练神经网络时,使用 tf.train.ExponentialMovingAverage 滑动平均操作的意义在于 提高模型在测试数据上的健壮性(robustness)。tensorflow 下的 tf.train.ExponentialMovingAverage 需要 提供一个衰减率(decay)。该衰减率用于控制模型更新的速度。该衰减率用于控制模型更新的速度, ExponentialMovingAverage 对每一个(待更新训练学习的)变量(variable)都会维护一个影子变量 (shadow variable)。影子变量的初始值就是这个变量的初始值, shadow_variable=decay×shadow_variable+(1−decay)×variable ''' ema = tf.train.ExponentialMovingAverage(decay=0.99) update_losses = ema.apply([discrim_loss, gen_loss_GAN, gen_loss_L1]) # global_step = tf.train.get_or_create_global_step() incr_global_step = tf.assign(global_step, global_step + 1) return Model( predict_real=predict_real, # 条件变量(输入图像)和真实图像之间的概率值,形状为;[batch,30,30,1] predict_fake=predict_fake, # 条件变量(输入图像)和生成图像之间的概率值,形状为;[batch,30,30,1] discrim_loss=ema.average(discrim_loss), # 判别器损失 discrim_grads_and_vars=discrim_grads_and_vars, # 判别器需要优化的参数和对应的梯度 gen_loss_GAN=ema.average(gen_loss_GAN), # 生成器的损失 gen_loss_L1=ema.average(gen_loss_L1), # 生成器的 L1损失 gen_grads_and_vars=gen_grads_and_vars, # 生成器需要优化的参数和对应的梯度 outputs=outputs, # 生成器生成的图片 train=tf.group(update_losses, incr_global_step, gen_train), # 打包需要run的操作op ) # 保存图像 def save_images(output_dir, fetches, step=None): image_dir = os.path.join(output_dir, "images") if not os.path.exists(image_dir): os.makedirs(image_dir) filesets = [] for i, in_path in enumerate(fetches["paths"]): name, _ = os.path.splitext(os.path.basename(in_path.decode("utf8"))) fileset = {"name": name, "step": step} for kind in ["inputs", "outputs", "targets"]: filename = name + "-" + kind + ".png" if step is not None: filename = "%08d-%s" % (step, filename) fileset[kind] = filename out_path = os.path.join(image_dir, filename) contents = fetches[kind][i] with open(out_path, "wb") as f: f.write(contents) filesets.append(fileset) return filesets # 将结果写入HTML网页 def append_index(output_dir, filesets, step=False): index_path = os.path.join(output_dir, "index.html") if os.path.exists(index_path): index = open(index_path, "a") else: index = open(index_path, "w") index.write("<html><body><table><tr>") if step: index.write("<th>step</th>") index.write("<th>name</th><th>input</th><th>output</th><th>target</th></tr>") for fileset in filesets: index.write("<tr>") if step: index.write("<td>%d</td>" % fileset["step"]) index.write("<td>%s</td>" % fileset["name"]) for kind in ["inputs", "outputs", "targets"]: index.write("<td><img src='images/%s'></td>" % fileset[kind]) index.write("</tr>") return index_path # 转变图像的尺寸、并且将[0,1]--->[0,255] def convert(image): if aspect_ratio != 1.0: # upscale to correct aspect ratio size = [CROP_SIZE, int(round(CROP_SIZE * aspect_ratio))] image = tf.image.resize_images(image, size=size, method=tf.image.ResizeMethod.BICUBIC) # 将数据的类型转换为8位无符号整型 return tf.image.convert_image_dtype(image, dtype=tf.uint8, saturate=True) # 主函数 def train(): # 设置随机数种子的值 global seed if seed is None: seed = random.randint(0, 2 ** 31 - 1) tf.set_random_seed(seed) np.random.seed(seed) random.seed(seed) # 创建目录 if not os.path.exists(train_output_dir): os.makedirs(train_output_dir) # 加载数据集,得到输入数据和目标数据并把范围变为 :[-1,1] examples = load_examples(train_input_dir) print("load successful ! examples count = %d" % examples.count) # 创建模型,inputs和targets是:[batch_size, height, width, channels] # 返回值: model = create_model(examples.inputs, examples.targets) print("create model successful!") # 图像处理[-1, 1] => [0, 1] inputs = deprocess(examples.inputs) targets = deprocess(examples.targets) outputs = deprocess(model.outputs) # 把[0,1]的像素点转为RGB值:[0,255] with tf.name_scope("convert_inputs"): converted_inputs = convert(inputs) with tf.name_scope("convert_targets"): converted_targets = convert(targets) with tf.name_scope("convert_outputs"): converted_outputs = convert(outputs) # 对图像进行编码以便于保存 with tf.name_scope("encode_images"): display_fetches = { "paths": examples.paths, # tf.map_fn接受一个函数对象和集合,用函数对集合中每个元素分别处理 "inputs": tf.map_fn(tf.image.encode_png, converted_inputs, dtype=tf.string, name="input_pngs"), "targets": tf.map_fn(tf.image.encode_png, converted_targets, dtype=tf.string, name="target_pngs"), "outputs": tf.map_fn(tf.image.encode_png, converted_outputs, dtype=tf.string, name="output_pngs"), } with tf.name_scope("parameter_count"): parameter_count = tf.reduce_sum([tf.reduce_prod(tf.shape(v)) for v in tf.trainable_variables()]) # 只保存最新一个checkpoint saver = tf.train.Saver(max_to_keep=20) init = tf.global_variables_initializer() with tf.Session() as sess: sess.run(init) print("parameter_count =", sess.run(parameter_count)) if max_epochs is not None: max_steps = examples.steps_per_epoch * max_epochs # 400X200=80000 # 因为是从文件中读取数据,所以需要启动start_queue_runners() # 这个函数将会启动输入管道的线程,填充样本到队列中,以便出队操作可以从队列中拿到样本。 coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord) # 运行训练集 print("begin trainning......") print("max_steps:", max_steps) start = time.time() for step in range(max_steps): def should(freq): return freq > 0 and ((step + 1) % freq == 0 or step == max_steps - 1) print("step:", step) # 定义一个需要run的所有操作的字典 fetches = { "train": model.train } # progress_freq为 50,每50次计算一次三个损失,显示进度 if should(progress_freq): fetches["discrim_loss"] = model.discrim_loss fetches["gen_loss_GAN"] = model.gen_loss_GAN fetches["gen_loss_L1"] = model.gen_loss_L1 # display_freq为 50,每50次保存一次输入、目标、输出的图像 if should(display_freq): fetches["display"] = display_fetches # 运行各种操作, results = sess.run(fetches) # display_freq为 50,每50次保存输入、目标、输出的图像 if should(display_freq): print("saving display images") filesets = save_images(train_output_dir, results["display"], step=step) append_index(train_output_dir, filesets, step=True) # progress_freq为 50,每50次打印一次三种损失的大小,显示进度 if should(progress_freq): # global_step will have the correct step count if we resume from a checkpoint train_epoch = math.ceil(step / examples.steps_per_epoch) train_step = (step - 1) % examples.steps_per_epoch + 1 rate = (step + 1) * batch_size / (time.time() - start) remaining = (max_steps - step) * batch_size / rate print("progress epoch %d step %d image/sec %0.1f remaining %dm" % ( train_epoch, train_step, rate, remaining / 60)) print("discrim_loss", results["discrim_loss"]) print("gen_loss_GAN", results["gen_loss_GAN"]) print("gen_loss_L1", results["gen_loss_L1"]) # save_freq为500,每500次保存一次模型 if should(save_freq): print("saving model") saver.save(sess, os.path.join(train_output_dir, "model"), global_step=step) # 测试 def test(): # 设置随机数种子的值 global seed if seed is None: seed = random.randint(0, 2 ** 31 - 1) tf.set_random_seed(seed) np.random.seed(seed) random.seed(seed) # 创建目录 if not os.path.exists(test_output_dir): os.makedirs(test_output_dir) if checkpoint is None: raise Exception("checkpoint required for test mode") # disable these features in test mode scale_size = CROP_SIZE flip = False # 加载数据集,得到输入数据和目标数据 examples = load_examples(test_input_dir) print("load successful ! examples count = %d" % examples.count) # 创建模型,inputs和targets是:[batch_size, height, width, channels] model = create_model(examples.inputs, examples.targets) print("create model successful!") # 图像处理[-1, 1] => [0, 1] inputs = deprocess(examples.inputs) targets = deprocess(examples.targets) outputs = deprocess(model.outputs) # 把[0,1]的像素点转为RGB值:[0,255] with tf.name_scope("convert_inputs"): converted_inputs = convert(inputs) with tf.name_scope("convert_targets"): converted_targets = convert(targets) with tf.name_scope("convert_outputs"): converted_outputs = convert(outputs) # 对图像进行编码以便于保存 with tf.name_scope("encode_images"): display_fetches = { "paths": examples.paths, # tf.map_fn接受一个函数对象和集合,用函数对集合中每个元素分别处理 "inputs": tf.map_fn(tf.image.encode_png, converted_inputs, dtype=tf.string, name="input_pngs"), "targets": tf.map_fn(tf.image.encode_png, converted_targets, dtype=tf.string, name="target_pngs"), "outputs": tf.map_fn(tf.image.encode_png, converted_outputs, dtype=tf.string, name="output_pngs"), } sess = tf.InteractiveSession() saver = tf.train.Saver(max_to_keep=1) ckpt = tf.train.get_checkpoint_state(checkpoint) saver.restore(sess,ckpt.model_checkpoint_path) start = time.time() coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord) for step in range(examples.count): results = sess.run(display_fetches) filesets = save_images(test_output_dir, results) for i, f in enumerate(filesets): print("evaluated image", f["name"]) index_path = append_index(test_output_dir, filesets) print("wrote index at", index_path) print("rate", (time.time() - start) / max_steps) if __name__ == '__main__': train() #test()
请问matlab里自带的deconvblind函数运用的是哪种盲解卷积的算法,求原理过程~~~
% Parse inputs to verify valid function calling syntaxes and arguments [J,P,NUMIT,DAMPAR,READOUT,WEIGHT,sizeI,classI,sizePSF,FunFcn,FunArg] = ... parse_inputs(varargin{:}); % 1. Prepare parameters for iterations % % Create indexes for image according to the sampling rate idx = repmat({':'},[1 length(sizeI)]); wI = max(WEIGHT.*(READOUT + J{1}),0);% at this point - positivity constraint fw = fftn(WEIGHT); clear WEIGHT; DAMPAR22 = (DAMPAR.^2)/2; % 2. L_R Iterations % lambda = 2*any(J{4}(:)~=0); for k = (lambda + 1) : (lambda + NUMIT), % 2.a Make an image and PSF predictions for the next iteration if k > 2,% image lambda = (J{4}(:,1).'*J{4}(:,2))/(J{4}(:,2).'*J{4}(:,2) + eps); lambda = max(min(lambda,1),0); % stability enforcement lambda(0,1)之间 end Y = max(J{2} + lambda*(J{2} - J{3}),0);% image positivity constraint if k > 2,% PSF lambda = (P{4}(:,1).'*P{4}(:,2))/(P{4}(:,2).'*P{4}(:,2) + eps); lambda = max(min(lambda,1),0);% stability enforcement end B = max(P{2} + lambda*(P{2} - P{3}),0);% PSF positivity constraint sumPSF = sum(B(:)); B = B/(sum(B(:)) + (sumPSF==0)*eps);% normalization is a necessary constraint, % because given only input image, the algorithm cannot even know how much % power is in the image vs PSF. Therefore, we force PSF to satisfy this % type of normalization: sum to one. % 2.b Make core for the LR estimation CC = corelucy(Y,psf2otf(B,sizeI),DAMPAR22,wI,READOUT,1,idx,[],[]); % 2.c Determine next iteration image & apply positivity constraint J{3} = J{2}; H = psf2otf(P{2},sizeI); scale = real(ifftn(conj(H).*fw)) + sqrt(eps); J{2} = max(Y.*real(ifftn(conj(H).*CC))./scale,0); clear scale; J{4} = [J{2}(:)-Y(:) J{4}(:,1)]; clear Y H; % 2.d Determine next iteration PSF & apply positivity constraint + normalization P{3} = P{2}; H = fftn(J{3}); scale = otf2psf(conj(H).*fw,sizePSF) + sqrt(eps); P{2} = max(B.*otf2psf(conj(H).*CC,sizePSF)./scale,0); clear CC H; sumPSF = sum(P{2}(:)); P{2} = P{2}/(sumPSF + (sumPSF==0)*eps); if ~isempty(FunFcn), FunArg{1} = P{2}; P{2} = feval(FunFcn,FunArg{:}); end; P{4} = [P{2}(:)-B(:) P{4}(:,1)]; end clear fw wI; % 3. Convert the right array (for cell it is first array, for notcell it is % second array) to the original image class & output the whole num = 1 + strcmp(classI{1},'notcell'); if ~strcmp(classI{2},'double'), J{num} = images.internal.changeClass(classI{2},J{num}); end if num == 2,% the input & output is NOT a cell P = P{2}; J = J{2}; end; %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % % Function: parse_inputs function [J,P,NUMIT,DAMPAR,READOUT,WEIGHT,sizeI,classI,sizePSF,FunFcn,FunArg] ... = parse_inputs(varargin) % % Outputs: % I=J{1} the input array (could be any numeric class, 2D, 3D) % P=P{1} the operator that distorts the ideal image % % Defaults: % NUMIT=[];NUMIT_d = 10; % Number of iterations, usually produces good % result by 10. DAMPAR =[];DAMPAR_d = 0;% No damping is default WEIGHT =[]; % All pixels are of equal quality, flat-field is one READOUT=[];READOUT_d= 0;% Zero readout noise or any other % back/fore/ground noise associated with CCD camera. % Or the Image is corrected already for this noise by user. FunFcn = '';FunFcn_d = ''; FunArg = {};FunArg_d = {}; funnum = [];funnum_d = nargin+1; narginchk(2,inf);% no constraint on max number % because of FUN args % First, assign the inputs starting with the cell/not cell image & PSF switch iscell(varargin{1}) + iscell(varargin{2}), case 0, % no-cell array is used to do a single set of iterations classI{1} = 'notcell'; J{1} = varargin{1};% create a cell array in order to do the iterations P{1} = varargin{2}; case 1, error(message('images:deconvblind:IandInitpsfMustBeOfSameType')) case 2,% input cell is used to resume the interrupted iterations or classI{1} = 'cell';% to interrupt the iteration to resume them later J = varargin{1}; P = varargin{2}; if length(J)~=length(P), error(message('images:deconvblind:IandInitpsfMustBeOfSameSize')) end end; % check the Image, which is the first array of the J-cell [sizeI, sizePSF] = padlength(size(J{1}), size(P{1})); classI{2} = class(J{1}); validateattributes(J{1},{'uint8' 'uint16' 'double' 'int16' 'single'},... {'real' 'nonempty' 'finite'},mfilename,'I',1); if prod(sizeI)<2, error(message('images:deconvblind:inputImageMustHaveAtLeast2Elements')) elseif ~isa(J{1},'double'), J{1} = im2double(J{1}); end % check the PSF, which is the first array of the P-cell validateattributes(P{1},{'uint8' 'uint16' 'double' 'int16' 'single'},... {'real' 'nonempty' 'finite' 'nonzero'},mfilename,'INITPSF',2); if prod(sizePSF)<2, error(message('images:deconvblind:initPSFMustHaveAtLeast2Elements')) elseif ~isa(P{1},'double'), P{1} = double(P{1}); end % now since the image&PSF are OK&double, we assign the rest of the J & P cells len = length(J); if len == 1,% J = {I} will be reassigned to J = {I,I,0,0} J{2} = J{1}; J{3} = 0; J{4}(prod(sizeI),2) = 0; P{2} = P{1}; P{3} = 0; P{4}(prod(sizePSF),2) = 0; elseif len ~= 4,% J = {I,J,Jm1,gk} has to have 4 or 1 arrays error(message('images:deconvblind:inputCellsMustBe1or4ElementNumArrays')) else % check if J,Jm1,gk are double in the input cell if ~all([isa(J{2},'double'),isa(J{3},'double'),isa(J{4},'double')]), error(message('images:deconvblind:ImageCellElementsMustBeDouble')) elseif ~all([isa(P{2},'double'),isa(P{3},'double'),isa(P{4},'double')]), error(message('images:deconvblind:psfCellElementsMustBeDouble')) end end; % Second, Find out if we have a function to put additional constraints on the PSF % function_classes = {'inline','function_handle','char'}; idx = []; for n = 3:nargin, idx = strmatch(class(varargin{n}),function_classes); if ~isempty(idx), [FunFcn,msgStruct] = fcnchk(varargin{n}); %only works on char, making it inline if ~isempty(msgStruct) error(msgStruct) end FunArg = [{0},varargin(n+1:nargin)]; try % how this function works, just in case. feval(FunFcn,FunArg{:}); catch ME Ftype = {'inline object','function_handle','expression ==>'}; Ffcnstr = {' ',' ',varargin{n}}; error(message('images:deconvblind:userSuppliedFcnFailed', Ftype{ idx }, Ffcnstr{ idx }, ME.message)) end funnum = n; break end end if isempty(idx), FunFcn = FunFcn_d; FunArg = FunArg_d; funnum = funnum_d; end % % Third, Assign the inputs for general deconvolution: % if funnum>7 error(message('images:validate:tooManyInputs',mfilename)); end switch funnum, case 4,% deconvblind(I,PSF,NUMIT,fun,...) NUMIT = varargin{3}; case 5,% deconvblind(I,PSF,NUMIT,DAMPAR,fun,...) NUMIT = varargin{3}; DAMPAR = varargin{4}; case 6,% deconvblind(I,PSF,NUMIT,DAMPAR,WEIGHT,fun,...) NUMIT = varargin{3}; DAMPAR = varargin{4}; WEIGHT = varargin{5}; case 7,% deconvblind(I,PSF,NUMIT,DAMPAR,WEIGHT,READOUT,fun,...) NUMIT = varargin{3}; DAMPAR = varargin{4}; WEIGHT = varargin{5}; READOUT = varargin{6}; end % Forth, Check validity of the gen.conv. input parameters: % % NUMIT check number of iterations if isempty(NUMIT), NUMIT = NUMIT_d; else %verify validity validateattributes(NUMIT,{'double'},... {'scalar' 'positive' 'integer' 'finite'},... mfilename,'NUMIT',3); end % DAMPAR check damping parameter if isempty(DAMPAR), DAMPAR = DAMPAR_d; elseif (numel(DAMPAR)~=1) && ~isequal(size(DAMPAR),sizeI), error(message('images:deconvblind:damparMustBeSizeOfInputImage')); elseif ~isa(DAMPAR,classI{2}), error(message('images:deconvblind:damparMustBeSameClassAsInputImage')); elseif ~strcmp(classI{2},'double'), DAMPAR = im2double(DAMPAR); end if ~isfinite(DAMPAR), error(message('images:deconvblind:damparMustBeFinite')); end % WEIGHT check weighting if isempty(WEIGHT), WEIGHT = ones(sizeI); else numw = numel(WEIGHT); validateattributes(WEIGHT,{'double'},{'finite'},mfilename,'WEIGHT',5); if (numw ~= 1) && ~isequal(size(WEIGHT),sizeI), error(message('images:deconvblind:weightMustBeSizeOfInputImage')); elseif numw == 1, WEIGHT = repmat(WEIGHT,sizeI); end; end % READOUT check read-out noise if isempty(READOUT), READOUT = READOUT_d; elseif (numel(READOUT)~=1) && ~isequal(size(READOUT),sizeI), error(message('images:deconvblind:readoutMustBeSizeOfInputImage')); elseif ~isa(READOUT,classI{2}), error(message('images:deconvblind:readoutMustBeSameClassAsInputImage')); elseif ~strcmp(classI{2},'double'), READOUT = im2double(READOUT); end if ~isfinite(READOUT), error(message('images:deconvblind:readoutMustBeFinite')); end;
急,跪求pycharm跑yolov3-train.py报错
![图片说明](https://img-ask.csdn.net/upload/201905/23/1558616226_449733.png) ``` import numpy as np import keras.backend as K from keras.layers import Input, Lambda from keras.models import Model from keras.callbacks import TensorBoard, ModelCheckpoint, EarlyStopping from yolo3.model import preprocess_true_boxes, yolo_body, tiny_yolo_body, yolo_loss from yolo3.utils import get_random_data def _main(): annotation_path = 'train.txt' log_dir = 'logs/000/' classes_path = 'model_data/voc_classes.txt' anchors_path = 'model_data/yolo_anchors.txt' class_names = get_classes(classes_path) anchors = get_anchors(anchors_path) input_shape = (416,416) # multiple of 32, hw model = create_model(input_shape, anchors, len(class_names) ) train(model, annotation_path, input_shape, anchors, len(class_names), log_dir=log_dir) def train(model, annotation_path, input_shape, anchors, num_classes, log_dir='logs/'): model.compile(optimizer='adam', loss={ 'yolo_loss': lambda y_true, y_pred: y_pred}) logging = TensorBoard(log_dir=log_dir) checkpoint = ModelCheckpoint(log_dir + "ep{epoch:03d}-loss{loss:.3f}-val_loss{val_loss:.3f}.h5", monitor='val_loss', save_weights_only=True, save_best_only=True, period=1) batch_size = 8 val_split = 0.1 with open(annotation_path) as f: lines = f.readlines() np.random.shuffle(lines) num_val = int(len(lines)*val_split) num_train = len(lines) - num_val print('Train on {} samples, val on {} samples, with batch size {}.'.format(num_train, num_val, batch_size)) model.fit_generator ( data_generator_wrapper ( lines[:num_train] , batch_size , input_shape , anchors , num_classes ) , steps_per_epoch=max ( 1 , num_train // batch_size ) , validation_data=data_generator_wrapper ( lines[num_train:] , batch_size , input_shape , anchors , num_classes ) , validation_steps=max ( 1 , num_val // batch_size ) , epochs=10 , initial_epoch=0 , callbacks=[logging , checkpoint] ) model.save_weights(log_dir + 'trained_weights.h5') def get_classes(classes_path): with open(classes_path) as f: class_names = f.readlines() class_names = [c.strip() for c in class_names] return class_names def get_anchors(anchors_path): with open(anchors_path) as f: anchors = f.readline() anchors = [float(x) for x in anchors.split(',')] return np.array(anchors).reshape(-1, 2) def create_model(input_shape, anchors, num_classes, load_pretrained=False, freeze_body=False, weights_path='model_data/yolo_weights.h5'): K.clear_session() # get a new session h, w = input_shape image_input = Input(shape=(w, h, 3)) num_anchors = len(anchors) y_true = [Input(shape=(h//{0:32, 1:16, 2:8}[l], w//{0:32, 1:16, 2:8}[l], num_anchors//3, num_classes+5)) for l in range(3)] model_body = yolo_body(image_input, num_anchors//3, num_classes) print('Create YOLOv3 model with {} anchors and {} classes.'.format(num_anchors, num_classes)) if load_pretrained: model_body.load_weights(weights_path, by_name=True, skip_mismatch=True) print('Load weights {}.'.format(weights_path)) if freeze_body in [1, 2]: # Do not freeze 3 output layers. num = (185 , len ( model_body.layers ) - 3)[freeze_body - 1] for i in range(num): model_body.layers[i].trainable = False print('Freeze the first {} layers of total {} layers.'.format(num, len(model_body.layers))) model_loss = Lambda ( yolo_loss , output_shape=(1 ,) , name='yolo_loss', arguments={'anchors': anchors, 'num_classes': num_classes, 'ignore_thresh': 0.5} )(model_body.output + y_true) model = Model(inputs=[model_body.input] + y_true, outputs=model_loss) return model def data_generator(annotation_lines, batch_size, input_shape, anchors, num_classes): n = len(annotation_lines) i = 0 while True: image_data = [] box_data = [] for b in range(batch_size): if i==0: np.random.shuffle(annotation_lines) image, box = get_random_data(annotation_lines[i], input_shape, random=True) image_data.append(image) box_data.append(box) i = (i+1) % n image_data = np.array(image_data) box_data = np.array(box_data) y_true = preprocess_true_boxes(box_data, input_shape, anchors, num_classes) yield [image_data]+y_true, np.zeros(batch_size) def data_generator_wrapper(annotation_lines, batch_size, input_shape, anchors, num_classes): n = len(annotation_lines) if n==0 or batch_size<=0: return None return data_generator(annotation_lines, batch_size, input_shape, anchors, num_classes) if __name__ == '__main__': _main() ``` 报了一个:tensorflow.python.framework.errors_impl.InvalidArgumentError: Inputs to operation training/Adam/gradients/AddN_24 of type _MklAddN must have the same size and shape. Input 0: [2768896] != input 1: [8,26,26,512] [[Node: training/Adam/gradients/AddN_24 = _MklAddN[N=2, T=DT_FLOAT, _kernel="MklOp", _device="/job:localhost/replica:0/task:0/device:CPU:0"](training/Adam/gradients/batch_normalization_65/FusedBatchNorm_grad/FusedBatchNormGrad, training/Adam/gradients/batch_n
终于明白阿里百度这样的大公司,为什么面试经常拿ThreadLocal考验求职者了
点击上面↑「爱开发」关注我们每晚10点,捕获技术思考和创业资源洞察什么是ThreadLocalThreadLocal是一个本地线程副本变量工具类,各个线程都拥有一份线程私有的数
程序员必须掌握的核心算法有哪些?
由于我之前一直强调数据结构以及算法学习的重要性,所以就有一些读者经常问我,数据结构与算法应该要学习到哪个程度呢?,说实话,这个问题我不知道要怎么回答你,主要取决于你想学习到哪些程度,不过针对这个问题,我稍微总结一下我学过的算法知识点,以及我觉得值得学习的算法。这些算法与数据结构的学习大多数是零散的,并没有一本把他们全部覆盖的书籍。下面是我觉得值得学习的一些算法以及数据结构,当然,我也会整理一些看过...
《奇巧淫技》系列-python!!每天早上八点自动发送天气预报邮件到QQ邮箱
此博客仅为我业余记录文章所用,发布到此,仅供网友阅读参考,如有侵权,请通知我,我会删掉。 补充 有不少读者留言说本文章没有用,因为天气预报直接打开手机就可以收到了,为何要多此一举发送到邮箱呢!!!那我在这里只能说:因为你没用,所以你没用!!! 这里主要介绍的是思路,不是天气预报!不是天气预报!!不是天气预报!!!天气预报只是用于举例。请各位不要再刚了!!! 下面是我会用到的两个场景: 每日下
死磕YOLO系列,YOLOv1 的大脑、躯干和手脚
YOLO 是我非常喜欢的目标检测算法,堪称工业级的目标检测,能够达到实时的要求,它帮我解决了许多实际问题。 这就是 YOLO 的目标检测效果。它定位了图像中物体的位置,当然,也能预测物体的类别。 之前我有写博文介绍过它,但是每次重新读它的论文,我都有新的收获,为此我准备写一个系列的文章来详尽分析它。这是第一篇,从它的起始 YOLOv1 讲起。 YOLOv1 的论文地址:https://www.c...
知乎高赞:中国有什么拿得出手的开源软件产品?(整理自本人原创回答)
知乎高赞:中国有什么拿得出手的开源软件产品? 在知乎上,有个问题问“中国有什么拿得出手的开源软件产品(在 GitHub 等社区受欢迎度较好的)?” 事实上,还不少呢~ 本人于2019.7.6进行了较为全面的回答,对这些受欢迎的 Github 开源项目分类整理如下: 分布式计算、云平台相关工具类 1.SkyWalking,作者吴晟、刘浩杨 等等 仓库地址: apache/skywalking 更...
20行Python代码爬取王者荣耀全英雄皮肤
引言 王者荣耀大家都玩过吧,没玩过的也应该听说过,作为时下最火的手机MOBA游戏,咳咳,好像跑题了。我们今天的重点是爬取王者荣耀所有英雄的所有皮肤,而且仅仅使用20行Python代码即可完成。 准备工作 爬取皮肤本身并不难,难点在于分析,我们首先得得到皮肤图片的url地址,话不多说,我们马上来到王者荣耀的官网: 我们点击英雄资料,然后随意地选择一位英雄,接着F12打开调试台,找到英雄原皮肤的图片...
简明易理解的@SpringBootApplication注解源码解析(包含面试提问)
欢迎关注文章系列 ,关注我 《提升能力,涨薪可待》 《面试知识,工作可待》 《实战演练,拒绝996》 欢迎关注我博客,原创技术文章第一时间推出 也欢迎关注公 众 号【Ccww笔记】,同时推出 如果此文对你有帮助、喜欢的话,那就点个赞呗,点个关注呗! 《提升能力,涨薪可待篇》- @SpringBootApplication注解源码解析 一、@SpringBootApplication 的作用是什...
西游记团队中如果需要裁掉一个人,会先裁掉谁?
2019年互联网寒冬,大批企业开始裁员,下图是网上流传的一张截图: 裁员不可避免,那如何才能做到不管大环境如何变化,自身不受影响呢? 我们先来看一个有意思的故事,如果西游记取经团队需要裁员一名,会裁掉谁呢,为什么? 西游记团队组成: 1.唐僧 作为团队teamleader,有很坚韧的品性和极高的原则性,不达目的不罢休,遇到任何问题,都没有退缩过,又很得上司支持和赏识(直接得到唐太宗的任命,既给袈...
Python语言高频重点汇总
Python语言高频重点汇总 GitHub面试宝典仓库 回到首页 目录: Python语言高频重点汇总 目录: 1. 函数-传参 2. 元类 3. @staticmethod和@classmethod两个装饰器 4. 类属性和实例属性 5. Python的自省 6. 列表、集合、字典推导式 7. Python中单下划线和双下划线 8. 格式化字符串中的%和format 9. 迭代器和生成器 10...
究竟你适不适合买Mac?
我清晰的记得,刚买的macbook pro回到家,开机后第一件事情,就是上了淘宝网,花了500元钱,找了一个上门维修电脑的师傅,上门给我装了一个windows系统。。。。。。 表砍我。。。 当时买mac的初衷,只是想要个固态硬盘的笔记本,用来运行一些复杂的扑克软件。而看了当时所有的SSD笔记本后,最终决定,还是买个好(xiong)看(da)的。 已经有好几个朋友问我mba怎么样了,所以今天尽量客观
程序员一般通过什么途径接私活?
二哥,你好,我想知道一般程序猿都如何接私活,我也想接,能告诉我一些方法吗? 上面是一个读者“烦不烦”问我的一个问题。其实不止是“烦不烦”,还有很多读者问过我类似这样的问题。 我接的私活不算多,挣到的钱也没有多少,加起来不到 20W。说实话,这个数目说出来我是有点心虚的,毕竟太少了,大家轻喷。但我想,恰好配得上“一般程序员”这个称号啊。毕竟苍蝇再小也是肉,我也算是有经验的人了。 唾弃接私活、做外
ES6基础-ES6的扩展
进行对字符串扩展,正则扩展,数值扩展,函数扩展,对象扩展,数组扩展。 开发环境准备: 编辑器(VS Code, Atom,Sublime)或者IDE(Webstorm) 浏览器最新的Chrome 字符串的扩展: 模板字符串,部分新的方法,新的unicode表示和遍历方法: 部分新的字符串方法 padStart,padEnd,repeat,startsWith,endsWith,includes 字...
Python爬虫爬取淘宝,京东商品信息
小编是一个理科生,不善长说一些废话。简单介绍下原理然后直接上代码。 使用的工具(Python+pycharm2019.3+selenium+xpath+chromedriver)其中要使用pycharm也可以私聊我selenium是一个框架可以通过pip下载 pip install selenium -i https://pypi.tuna.tsinghua.edu.cn/simple/ 
阿里程序员写了一个新手都写不出的低级bug,被骂惨了。
你知道的越多,你不知道的越多 点赞再看,养成习惯 本文 GitHub https://github.com/JavaFamily 已收录,有一线大厂面试点思维导图,也整理了很多我的文档,欢迎Star和完善,大家面试可以参照考点复习,希望我们一起有点东西。 前前言 为啥今天有个前前言呢? 因为你们的丙丙啊,昨天有牌面了哟,直接被微信官方推荐,知乎推荐,也就仅仅是还行吧(心里乐开花)
Java工作4年来应聘要16K最后没要,细节如下。。。
前奏: 今天2B哥和大家分享一位前几天面试的一位应聘者,工作4年26岁,统招本科。 以下就是他的简历和面试情况。 基本情况: 专业技能: 1、&nbsp;熟悉Sping了解SpringMVC、SpringBoot、Mybatis等框架、了解SpringCloud微服务 2、&nbsp;熟悉常用项目管理工具:SVN、GIT、MAVEN、Jenkins 3、&nbsp;熟悉Nginx、tomca
Python爬虫精简步骤1 获取数据
爬虫的工作分为四步: 1.获取数据。爬虫程序会根据我们提供的网址,向服务器发起请求,然后返回数据。 2.解析数据。爬虫程序会把服务器返回的数据解析成我们能读懂的格式。 3.提取数据。爬虫程序再从中提取出我们需要的数据。 4.储存数据。爬虫程序把这些有用的数据保存起来,便于你日后的使用和分析。 这一篇的内容就是:获取数据。 首先,我们将会利用一个强大的库——requests来获取数据。 在电脑上安装
作为一个程序员,CPU的这些硬核知识你必须会!
CPU对每个程序员来说,是个既熟悉又陌生的东西? 如果你只知道CPU是中央处理器的话,那可能对你并没有什么用,那么作为程序员的我们,必须要搞懂的就是CPU这家伙是如何运行的,尤其要搞懂它里面的寄存器是怎么一回事,因为这将让你从底层明白程序的运行机制。 随我一起,来好好认识下CPU这货吧 把CPU掰开来看 对于CPU来说,我们首先就要搞明白它是怎么回事,也就是它的内部构造,当然,CPU那么牛的一个东
破14亿,Python分析我国存在哪些人口危机!
2020年1月17日,国家统计局发布了2019年国民经济报告,报告中指出我国人口突破14亿。 猪哥的朋友圈被14亿人口刷屏,但是很多人并没有看到我国复杂的人口问题:老龄化、男女比例失衡、生育率下降、人口红利下降等。 今天我们就来分析一下我们国家的人口数据吧! 更多有趣分析教程,扫描下方二维码关注vx公号「裸睡的猪」 即可查看! 一、背景 1.人口突破14亿 2020年1月17日,国家统计局发布
web前端javascript+jquery知识点总结
Javascript javascript 在前端网页中占有非常重要的地位,可以用于验证表单,制作特效等功能,它是一种描述语言,也是一种基于对象(Object)和事件驱动并具有安全性的脚本语言 ,语法同java类似,是一种解释性语言,边执行边解释。 JavaScript的组成: ECMAScipt 用于描述: 语法,变量和数据类型,运算符,逻辑控制语句,关键字保留字,对象。 浏览器对象模型(Br
Qt实践录:开篇
本系列文章介绍笔者的Qt实践之路。
在家远程办公效率低?那你一定要收好这个「在家办公」神器!
相信大家都已经收到国务院延长春节假期的消息,接下来,在家远程办公可能将会持续一段时间。 但是问题来了。远程办公不是人在电脑前就当坐班了,相反,对于沟通效率,文件协作,以及信息安全都有着极高的要求。有着非常多的挑战,比如: 1在异地互相不见面的会议上,如何提高沟通效率? 2文件之间的来往反馈如何做到及时性?如何保证信息安全? 3如何规划安排每天工作,以及如何进行成果验收? ......
作为一个程序员,内存和磁盘的这些事情,你不得不知道啊!!!
截止目前,我已经分享了如下几篇文章: 一个程序在计算机中是如何运行的?超级干货!!! 作为一个程序员,CPU的这些硬核知识你必须会! 作为一个程序员,内存的这些硬核知识你必须懂! 这些知识可以说是我们之前都不太重视的基础知识,可能大家在上大学的时候都学习过了,但是嘞,当时由于老师讲解的没那么有趣,又加上这些知识本身就比较枯燥,所以嘞,大家当初几乎等于没学。 再说啦,学习这些,也看不出来有什么用啊!
这个世界上人真的分三六九等,你信吗?
偶然间,在知乎上看到一个问题 一时间,勾起了我深深的回忆。 以前在厂里打过两次工,做过家教,干过辅导班,做过中介。零下几度的晚上,贴过广告,满脸、满手地长冻疮。   再回首那段岁月,虽然苦,但让我学会了坚持和忍耐。让我明白了,在这个世界上,无论环境多么的恶劣,只要心存希望,星星之火,亦可燎原。   下文是原回答,希望能对你能有所启发。   如果我说,这个世界上人真的分三六九等,
为什么听过很多道理,依然过不好这一生?
记录学习笔记是一个重要的习惯,不希望学习过的东西成为过眼云烟。做总结的同时也是一次复盘思考的过程。 本文是根据阅读得到 App上《万维钢·精英日课》部分文章后所做的一点笔记和思考。学习是一个系统的过程,思维模型的建立需要相对完整的学习和思考过程。以下观点是在碎片化阅读后总结的一点心得总结。
B 站上有哪些很好的学习资源?
哇说起B站,在小九眼里就是宝藏般的存在,放年假宅在家时一天刷6、7个小时不在话下,更别提今年的跨年晚会,我简直是跪着看完的!! 最早大家聚在在B站是为了追番,再后来我在上面刷欧美新歌和漂亮小姐姐的舞蹈视频,最近两年我和周围的朋友们已经把B站当作学习教室了,而且学习成本还免费,真是个励志的好平台ヽ(.◕ฺˇд ˇ◕ฺ;)ノ 下面我们就来盘点一下B站上优质的学习资源: 综合类 Oeasy: 综合
雷火神山直播超两亿,Web播放器事件监听是怎么实现的?
Web播放器解决了在手机浏览器和PC浏览器上播放音视频数据的问题,让视音频内容可以不依赖用户安装App,就能进行播放以及在社交平台进行传播。在视频业务大数据平台中,播放数据的统计分析非常重要,所以Web播放器在使用过程中,需要对其内部的数据进行收集并上报至服务端,此时,就需要对发生在其内部的一些播放行为进行事件监听。 那么Web播放器事件监听是怎么实现的呢? 01 监听事件明细表 名
3万字总结,Mysql优化之精髓
本文知识点较多,篇幅较长,请耐心学习 MySQL已经成为时下关系型数据库产品的中坚力量,备受互联网大厂的青睐,出门面试想进BAT,想拿高工资,不会点MySQL优化知识,拿offer的成功率会大大下降。 为什么要优化 系统的吞吐量瓶颈往往出现在数据库的访问速度上 随着应用程序的运行,数据库的中的数据会越来越多,处理时间会相应变慢 数据是存放在磁盘上的,读写速度无法和内存相比 如何优化 设计
一条链接即可让黑客跟踪你的位置! | Seeker工具使用
搬运自:冰崖的部落阁(icecliffsnet) 严正声明:本文仅限于技术讨论,严禁用于其他用途。 请遵守相对应法律规则,禁止用作违法途径,出事后果自负! 上次写的防社工文章里边提到的gps定位信息(如何防止自己被社工或人肉) 除了主动收集他人位置信息以外,我们还可以进行被动收集 (没有技术含量) Seeker作为一款高精度地理位置跟踪工具,同时也是社交工程学(社会工程学)爱好者...
作为程序员的我,大学四年一直自学,全靠这些实用工具和学习网站!
我本人因为高中沉迷于爱情,导致学业荒废,后来高考,毫无疑问进入了一所普普通通的大学,实在惭愧...... 我又是那么好强,现在学历不行,没办法改变的事情了,所以,进入大学开始,我就下定决心,一定要让自己掌握更多的技能,尤其选择了计算机这个行业,一定要多学习技术。 在进入大学学习不久后,我就认清了一个现实:我这个大学的整体教学质量和学习风气,真的一言难尽,懂的人自然知道怎么回事? 怎么办?我该如何更好的提升
前端JS初级面试题二 (。•ˇ‸ˇ•。)老铁们!快来瞧瞧自己都会了么
1. 传统事件绑定和符合W3C标准的事件绑定有什么区别? 传统事件绑定 &lt;div onclick=""&gt;123&lt;/div&gt; div1.onclick = function(){}; &lt;button onmouseover=""&gt;&lt;/button&gt; 注意: 如果给同一个元素绑定了两次或多次相同类型的事件,那么后面的绑定会覆盖前面的绑定 (不支持DOM事...
相关热词 c#导入fbx c#中屏蔽键盘某个键 c#正态概率密度 c#和数据库登陆界面设计 c# 高斯消去法 c# codedom c#读取cad文件文本 c# 控制全局鼠标移动 c# temp 目录 bytes初始化 c#
立即提问