Web logs

Boosting: In lieu of training all the models separately as in bagging, boosting trains models sequentially. Each new model is trained to correct the errors made by the previous ones. The first tree is examined and the weights of those observations that are hard to classify are increased and the weights for those that are easy to classify are reduced. This modified data is used to build the next tree. This process is repeated for a defined number of iterations. Predictions of the final ensemble model is therefore the weighted sum of the predictions made by the previous tree models. GBM uses loss functions Since each tree is fit to residuals as against the original output parameter, each tree is small and improves prediction in the parts where prediction is bad. All the models might make the same mistake in the standard ensemble method. Compute error by deducting forecasted value from target value (e1= y - y1_forecasted) Build a new model on errors (e1_forecasted) as target variabl...

Regularisation is used to constrain the model to fewer degrees of freedom or regularise beta estimates towards zero in order to avoid overfitting. Hence it avoids a complex or flexible model. Regularisation reduces the variance of the model considerably without substantial increase in its bias. λ is the tuning parameter used to penalise the flexibility of the model.  As the value of λ increases, it reduces the value of beta estimates except for intercept and thus reducing the variance. However beyond a certain threshold, the bias starts increasing as the model starts losing important information resulting in under fitting. Note that Regularization adds penalty to the higher terms and their importance reduces. Lasso uses modulus of Beta estimates to penalize and this is known as L1 norm while Ridge uses squares of Beta estimates to penalize and this is known as L2 norm. Lasso can penalise some of the beta estimates to be equal to zero when λ is large enough resulting in featu...