加州房价预测 [Hands On ML] 2. 一个完整的机器学习项目( 二 ) _机器学习

import numpy as npdata['income_cat'] = np.ceil(data['median_income']/1.5)data['income_cat']data['income_cat'].where(data['income_cat'] < 5, 5.0, inplace=True)# pd.where() Where cond is True, keep the original value. # Where False, replace with corresponding value from other大于等于 5 的，替换成 5，把收入分成几类data['income_cat'].hist()data['income_cat'].value_counts()/len(data)
3.00.3505812.00.3188474.00.1763085.00.1144381.00.039826Name: income_cat, dtype: float64
from sklearn.model_selection import StratifiedShuffleSplit# help(StratifiedShuffleSplit)splt = StratifiedShuffleSplit(n_splits=1, test_size=0.2, random_state=1)for train_idx, test_idx in splt.split(data, data['income_cat']): # 按照后者分层抽样strat_train_set = data.loc[train_idx]strat_test_set = data.loc[test_idx]# 查看分布strat_train_set['income_cat'].value_counts()/len(strat_train_set)strat_test_set['income_cat'].value_counts()/len(strat_test_set)
查看抽样结果，跟上面原始数据的分布很接近
3.00.3505332.00.3187984.00.1763575.00.1145831.00.039729Name: income_cat, dtype: float64
分层采样的收入分类比例与总数据集几乎相同，而随机采样数据集偏差严重
删除新增列，回到初始数据状态
for set in (strat_train_set, strat_test_set):set.drop('income_cat',axis=1, inplace=True)strat_train_set
6. 数据可视化
housing = strat_train_set.copy() # 复制避免损坏housing.plot(kind='scatter',x='longitude',y='latitude')
# 调整alpha，可以看出密度差异housing.plot(kind='scatter',x='longitude',y='latitude',alpha=0.1)
housing.plot(kind='scatter',x='longitude',y='latitude',alpha=0.4,s=housing['population']/100, label='population',c='median_house_value',cmap=plt.get_cmap('jet'))每个圈的半径表示街区的人口（选项s），颜色代表价格（选项c）
可以看出，距离海岸近的房价较高，但是北边海岸边的价格又不是很高
7. 查找数据关联
相关系数
corr_mat = housing.corr()corr_mat
corr_mat['median_house_value'].sort_values(ascending=False)
median_house_value1.000000median_income0.684828total_rooms0.133566housing_median_age0.107684households0.065778total_bedrooms0.049941population-0.025008longitude-0.043824# 经度，东西latitude-0.146748 # 纬度，南北Name: median_house_value, dtype: float64
可以看到纬度越大（北边），房价（越低），呈负相关
attributes = ["median_house_value", "median_income", "total_rooms", "housing_median_age"]pd.plotting.scatter_matrix(housing[attributes],figsize=(12,8))
挑几个可能跟房价先关的特征出来，画出相关性图
可以看出收入的中位数特征，最有可能用来预测房价，将该子图放大
housing.plot(kind='scatter',x='median_income',y='median_house_value',alpha=0.1)