6-TensorBoard-for-Visualization-and-Hyperparameter-Tuning.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.4"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":false,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"187.997px"},"toc_section_display":true,"toc_window_display":true},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"oldHeight":542.414818,"position":{"height":"40px","left":"568.026px","right":"20px","top":"35.9432px","width":"566.222px"},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"varInspector_section_display":"none","window_display":true},"colab":{"name":"6-Visualizing-Metrics-with-TensorBoard-Hyperparameter-Tuning.ipynb","provenance":[]},"accelerator":"GPU"},"cells":[{"cell_type":"code","metadata":{"code_folding":[9,18],"deletable":false,"editable":false,"id":"5Q_G6Ujt_hn8","colab_type":"code","colab":{}},"source":["import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","import torch.optim as optim\n","import torchvision\n","import torchvision.transforms as transform\n","\n","\n","class Network(nn.Module):\n","    def __init__(self, channels=1): # default grayscale\n","        super().__init__()\n","        self.conv1 = nn.Conv2d(in_channels=channels, out_channels=6, kernel_size=5) \n","        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)\n","        \n","        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120) # ((28-5+1)/2 -5 +1)/2 = 4\n","        self.fc2 = nn.Linear(in_features=120, out_features=60)\n","        self.out = nn.Linear(in_features=60, out_features=10)\n","        \n","    def forward(self, t):        \n","        # hidden conv layers, conv w/ relu activation -> max pool\n","        t = F.relu(self.conv1(t))\n","        t = F.max_pool2d(t, kernel_size=2, stride=2)\n","        \n","        t = F.relu(self.conv2(t))\n","        t = F.max_pool2d(t, kernel_size=2, stride=2)\n","\n","        # hidden fully connected layers\n","        t = t.reshape(-1, 12*4*4) # flatten\n","        t = F.relu(self.fc1(t))\n","        t = F.relu(self.fc2(t))\n","        \n","        # output layer\n","        t = self.out(t)\n","        return t\n","    \n","def get_num_correct(preds, labels):\n","    return (preds.argmax(dim=1) == labels).sum().item()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"UOLIs5PU_hoA","colab_type":"text"},"source":["### Pipeline\n","\n","Prepare the data -> Build the model -> Train the model -> __Analyze the model's results__\n","\n","To aid in the analysis, we use TensorBoard."]},{"cell_type":"code","metadata":{"id":"FMNgpmFt_hoB","colab_type":"code","colab":{},"outputId":"3a9a8452-1053-4d13-d684-d48d1c1d0a15"},"source":["import torch\n","from torch.utils.tensorboard import SummaryWriter\n","\n","print(torch.__version__)\n","!tensorboard --version"],"execution_count":0,"outputs":[{"output_type":"stream","text":["1.2.0\n","1.15.0a20190806\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ybs17v7R_hoG","colab_type":"text"},"source":["### TensorBoard: TensorFlow's Visualization Toolkit\n","\n","TensorBoard provides the visualization and tooling needed for machine learning experimentation:\n","\n","* Tracking and visualizing metrics such as loss and accuracy\n","* Visualizing the model graph (ops and layers)\n","* Viewing histograms of weights, biases, or other tensors as they change over time\n","* Projecting embeddings to a lower dimensional space\n","* Displaying images, text, and audio data\n","* Profiling TensorFlow programs\n","* And much more"]},{"cell_type":"code","metadata":{"code_folding":[],"id":"G629ANCs_hoG","colab_type":"code","colab":{}},"source":["# Toy example, writing an image\n","from torch.utils.tensorboard import SummaryWriter\n","\n","tb = SummaryWriter()\n","network = Network()\n","\n","images, labels = next(iter(train_loader))\n","grid = torchvision.utils.make_grid(images)\n","tb.add_image('images', grid)\n","# tb.add_graph(network, images)\n","\n","tb.close()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZrxA6PFm_hoJ","colab_type":"text"},"source":["Enter `tensorboard --logdir=runs` on the terminal. The TensorBoard UI can be found in http://localhost:6006."]},{"cell_type":"code","metadata":{"id":"yAecmHTS_hoK","colab_type":"code","colab":{},"outputId":"06ad788a-4632-41db-eb8a-425e0ff03a40"},"source":["!tensorboard --logdir=runs"],"execution_count":0,"outputs":[{"output_type":"stream","text":["TensorBoard 1.15.0a20190806 at http://makd0-v1.local:6007/ (Press CTRL+C to quit)\n","^C\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"NY6MmF4W_hoO","colab_type":"text"},"source":["### Hyperparameter Tuning"]},{"cell_type":"markdown","metadata":{"id":"5PxjDnX7_hoP","colab_type":"text"},"source":["#### Basic use case: `add_scalar` and `add_histogram`"]},{"cell_type":"code","metadata":{"code_folding":[],"deletable":false,"editable":false,"scrolled":false,"id":"kpbvnajp_hoP","colab_type":"code","colab":{},"outputId":"84af1edf-0dc2-4861-e2c1-a6d163db9fd3"},"source":["import torch\n","from torch.utils.tensorboard import SummaryWriter\n","\n","\n","# Get data\n","train_set = torchvision.datasets.FashionMNIST(\n","    root='./data/FashionMNIST',\n","    download=False,\n","    transform=transform.ToTensor())\n","\n","train_loader = torch.utils.data.DataLoader(train_set, batch_size=100)\n","\n","\n","# Compile network\n","network = Network()\n","optimizer = optim.Adam(network.parameters(), lr=0.001)\n","\n","# Initialize tensorboard\n","tb = SummaryWriter() # from torch.utils.tensorboard import SummaryWriter\n","\n","# Training\n","for epoch in range(10): \n","    total_loss = 0\n","    total_correct = 0\n","    \n","    for batch in train_loader:\n","        images, labels = batch \n","        preds = network(images)\n","        \n","        loss = F.cross_entropy(preds, labels) # loss function\n","        optimizer.zero_grad()                 # set all gradients to zero\n","        \n","        loss.backward()         # calculate gradients, training points are supply constants\n","        optimizer.step()        # update weights to minimize loss (accdg to adam)\n","\n","        total_loss += loss.item() \n","        total_correct += get_num_correct(preds, labels)\n","    \n","    tb.add_scalar('Loss', total_loss, epoch)\n","    tb.add_scalar('Number Correct', total_correct, epoch)\n","    tb.add_scalar('Accuracy', total_correct / len(train_set), epoch)\n","    \n","    tb.add_histogram('conv1.bias', network.conv1.bias, epoch)\n","    tb.add_histogram('conv1.weight', network.conv1.weight, epoch)\n","    tb.add_histogram('conv1.weight.grad', network.conv1.weight.grad, epoch)\n","    print(\"epoch\", epoch, \"train_acc\", total_correct / 60000, \"loss:\", total_loss)\n","\n","tb.close()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["epoch 0 train_acc 0.69515 loss: 477.1251989901066\n","epoch 1 train_acc 0.8025333333333333 loss: 313.45684093236923\n","epoch 2 train_acc 0.8437 loss: 260.7022297382355\n","epoch 3 train_acc 0.8593166666666666 loss: 233.27016121149063\n","epoch 4 train_acc 0.8682 loss: 217.13399057090282\n","epoch 5 train_acc 0.8745166666666667 loss: 205.5287000834942\n","epoch 6 train_acc 0.8801666666666667 loss: 196.3009224832058\n","epoch 7 train_acc 0.8845666666666666 loss: 188.59119561314583\n","epoch 8 train_acc 0.8886 loss: 182.20187175273895\n","epoch 9 train_acc 0.8932666666666667 loss: 176.2308282405138\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"QpuSJV-d_hoS","colab_type":"text"},"source":["#### Comments. Experimenting with different parameter values."]},{"cell_type":"code","metadata":{"deletable":false,"editable":false,"id":"sGP6Ch_m_hoT","colab_type":"code","colab":{}},"source":["import torch\n","from torch.utils.tensorboard import SummaryWriter\n","from itertools import product\n","\n","# Get data\n","train_set = torchvision.datasets.FashionMNIST(\n","    root='./data/FashionMNIST',\n","    download=False,\n","    transform=transform.ToTensor())\n","\n","def train(lr, batch_size, shuffle, num_epochs=5):\n","    # data loader\n","    train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=shuffle)\n","\n","    # compile network\n","    network = Network()\n","    optimizer = optim.Adam(network.parameters(), lr=lr)\n","\n","    # Initialize tensorboard\n","    tb = SummaryWriter(comment=f' lr={lr} batch_size={batch_size} shuffle={shuffle}') # this is appended\n","    \n","    # Training\n","    print('\\nlr=', lr, 'batch_size=', batch_size, 'shuffle=', shuffle)\n","    for epoch in range(num_epochs): \n","        total_loss = 0\n","        total_correct = 0\n","\n","        for batch in train_loader:\n","            images, labels = batch \n","            preds = network(images)\n","\n","            loss = F.cross_entropy(preds, labels) \n","            \n","            optimizer.zero_grad()                 \n","            loss.backward()         \n","            optimizer.step()        \n","\n","            total_loss += loss.item()*batch_size             # get absolute loss \n","            total_correct += get_num_correct(preds, labels)\n","\n","        tb.add_scalar('Loss', total_loss, epoch)\n","        tb.add_scalar('Number Correct', total_correct, epoch)\n","        tb.add_scalar('Accuracy', total_correct / len(train_set), epoch)\n","\n","#         tb.add_histogram('conv1.bias', network.conv1.bias, epoch)\n","#         tb.add_histogram('conv1.weight', network.conv1.weight, epoch)\n","#         tb.add_histogram('conv1.weight.grad', network.conv1.weight.grad, epoch)\n","\n","        print(\"epoch\", epoch, \"\\t train_acc\", total_correct / len(train_set), \"\\t loss:\", total_loss)\n","\n","    tb.close()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"hA8kdN5ov4RJ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"a15ed287-ed97-4204-e10c-e389fefab6b6","executionInfo":{"status":"ok","timestamp":1569990909024,"user_tz":-480,"elapsed":1943,"user":{"displayName":"particle","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAr-M4nH-8vizN-p_Lsp6aZUVMMhVufQWMZhVMKgw=s64","userId":"15462581408757323236"}}},"source":["import tensorflow\n","\n","tensorflow.__version__"],"execution_count":2,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.14.0'"]},"metadata":{"tags":[]},"execution_count":2}]},{"cell_type":"code","metadata":{"deletable":false,"editable":false,"scrolled":false,"id":"zBqLH0kd_hoV","colab_type":"code","colab":{},"outputId":"2be14922-188d-47a3-ad0f-40034fc4766b"},"source":["lr_list = [0.1, 0.01, 0.001]\n","batch_size_list = [10, 100, 1000]\n","shuffle_list = [True, False]\n","\n","# hyperparameter grid search\n","for param in product(lr_list, batch_size_list, shuffle_list):\n","    train(*param, num_epochs=10)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["\n","lr= 0.1 batch_size= 10 shuffle= True\n","epoch 0 train_acc 0.09998333333333333 loss: 143734.69151377678\n","epoch 1 train_acc 0.10081666666666667 loss: 139562.04599261284\n","epoch 2 train_acc 0.09995 loss: 139506.0769557953\n","epoch 3 train_acc 0.09901666666666667 loss: 139512.84832715988\n","epoch 4 train_acc 0.09858333333333333 loss: 139588.2374048233\n","epoch 5 train_acc 0.09991666666666667 loss: 139559.05871391296\n","epoch 6 train_acc 0.10071666666666666 loss: 139466.00037813187\n","epoch 7 train_acc 0.09988333333333334 loss: 139555.5360364914\n","epoch 8 train_acc 0.09948333333333333 loss: 139610.83234071732\n","epoch 9 train_acc 0.10116666666666667 loss: 139508.01008224487\n","\n","lr= 0.1 batch_size= 10 shuffle= False\n","epoch 0 train_acc 0.10206666666666667 loss: 140052.88788318634\n","epoch 1 train_acc 0.10206666666666667 loss: 139448.9784836769\n","epoch 2 train_acc 0.10208333333333333 loss: 139448.9783024788\n","epoch 3 train_acc 0.10208333333333333 loss: 139448.97848844528\n","epoch 4 train_acc 0.10208333333333333 loss: 139448.9785337448\n","epoch 5 train_acc 0.10208333333333333 loss: 139448.97835969925\n","epoch 6 train_acc 0.10208333333333333 loss: 139448.9783835411\n","epoch 7 train_acc 0.10208333333333333 loss: 139448.97843837738\n","epoch 8 train_acc 0.10208333333333333 loss: 139448.97840976715\n","epoch 9 train_acc 0.10208333333333333 loss: 139448.9785695076\n","\n","lr= 0.1 batch_size= 100 shuffle= True\n","epoch 0 train_acc 0.10193333333333333 loss: 168301.0918855667\n","epoch 1 train_acc 0.099 loss: 138568.00062656403\n","epoch 2 train_acc 0.10015 loss: 138577.80866622925\n","epoch 3 train_acc 0.10038333333333334 loss: 138590.59393405914\n","epoch 4 train_acc 0.09845 loss: 138619.8202610016\n","epoch 5 train_acc 0.10008333333333333 loss: 138596.29080295563\n","epoch 6 train_acc 0.09888333333333334 loss: 138566.26937389374\n","epoch 7 train_acc 0.09985 loss: 138593.40229034424\n","epoch 8 train_acc 0.09995 loss: 138587.8206729889\n","epoch 9 train_acc 0.10096666666666666 loss: 138581.83658123016\n","\n","lr= 0.1 batch_size= 100 shuffle= False\n","epoch 0 train_acc 0.09871666666666666 loss: 143357.38062858582\n","epoch 1 train_acc 0.09928333333333333 loss: 138595.52128314972\n","epoch 2 train_acc 0.09986666666666667 loss: 138613.09022903442\n","epoch 3 train_acc 0.09978333333333333 loss: 138620.18694877625\n","epoch 4 train_acc 0.09978333333333333 loss: 138623.46005439758\n","epoch 5 train_acc 0.09963333333333334 loss: 138625.05779266357\n","epoch 6 train_acc 0.09963333333333334 loss: 138625.85694789886\n","epoch 7 train_acc 0.09963333333333334 loss: 138626.27029418945\n","epoch 8 train_acc 0.09963333333333334 loss: 138626.48167610168\n","epoch 9 train_acc 0.09963333333333334 loss: 138626.58874988556\n","\n","lr= 0.1 batch_size= 1000 shuffle= True\n","epoch 0 train_acc 0.3564 loss: 125221.39048576355\n","epoch 1 train_acc 0.6077833333333333 loss: 62079.70690727234\n","epoch 2 train_acc 0.6485 loss: 58355.40807247162\n","epoch 3 train_acc 0.7138 loss: 47639.572978019714\n","epoch 4 train_acc 0.72875 loss: 44220.97969055176\n","epoch 5 train_acc 0.7401666666666666 loss: 41716.08740091324\n","epoch 6 train_acc 0.7467833333333334 loss: 40330.47252893448\n","epoch 7 train_acc 0.7500666666666667 loss: 39438.68577480316\n","epoch 8 train_acc 0.75085 loss: 39474.60401058197\n","epoch 9 train_acc 0.75145 loss: 39106.76610469818\n","\n","lr= 0.1 batch_size= 1000 shuffle= False\n","epoch 0 train_acc 0.09948333333333333 loss: 161256.11972808838\n","epoch 1 train_acc 0.09858333333333333 loss: 138250.89836120605\n","epoch 2 train_acc 0.09816666666666667 loss: 138233.93845558167\n","epoch 3 train_acc 0.09946666666666666 loss: 138237.25414276123\n","epoch 4 train_acc 0.09895 loss: 138245.89204788208\n","epoch 5 train_acc 0.09888333333333334 loss: 138252.53891944885\n","epoch 6 train_acc 0.09913333333333334 loss: 138257.67397880554\n","epoch 7 train_acc 0.09935 loss: 138261.6183757782\n","epoch 8 train_acc 0.09936666666666667 loss: 138264.51015472412\n","epoch 9 train_acc 0.09931666666666666 loss: 138266.87097549438\n","\n","lr= 0.01 batch_size= 10 shuffle= True\n","epoch 0 train_acc 0.7806333333333333 loss: 35996.10575011\n","epoch 1 train_acc 0.8157666666666666 loss: 31256.090226056986\n","epoch 2 train_acc 0.823 loss: 29616.59229345736\n","epoch 3 train_acc 0.8250666666666666 loss: 29568.088241587393\n","epoch 4 train_acc 0.8303833333333334 loss: 29330.07235411089\n","epoch 5 train_acc 0.82815 loss: 29967.76241225656\n","epoch 6 train_acc 0.8262833333333334 loss: 29641.005348865874\n","epoch 7 train_acc 0.82585 loss: 30009.914898796706\n","epoch 8 train_acc 0.82575 loss: 30283.64590803045\n","epoch 9 train_acc 0.8254833333333333 loss: 30232.06346239429\n","\n","lr= 0.01 batch_size= 10 shuffle= False\n","epoch 0 train_acc 0.7717666666666667 loss: 36505.32214491977\n","epoch 1 train_acc 0.8072 loss: 32606.10152820067\n","epoch 2 train_acc 0.8143333333333334 loss: 31739.36355031561\n","epoch 3 train_acc 0.8159666666666666 loss: 31754.907057425007\n","epoch 4 train_acc 0.8146166666666667 loss: 32343.289680085727\n","epoch 5 train_acc 0.8192166666666667 loss: 31035.5033127577\n","epoch 6 train_acc 0.81995 loss: 31344.337025792338\n","epoch 7 train_acc 0.82375 loss: 30642.089198327158\n","epoch 8 train_acc 0.8179333333333333 loss: 31568.505215973128\n","epoch 9 train_acc 0.8149333333333333 loss: 32790.747719806386\n","\n","lr= 0.01 batch_size= 100 shuffle= True\n","epoch 0 train_acc 0.7934333333333333 loss: 32561.97458356619\n","epoch 1 train_acc 0.8596833333333334 loss: 22459.776060283184\n","epoch 2 train_acc 0.87175 loss: 20510.176692903042\n","epoch 3 train_acc 0.8762833333333333 loss: 19698.023104667664\n","epoch 4 train_acc 0.8791833333333333 loss: 19430.717292428017\n","epoch 5 train_acc 0.8847666666666667 loss: 18462.579426169395\n","epoch 6 train_acc 0.8841166666666667 loss: 18572.672451287508\n","epoch 7 train_acc 0.88465 loss: 18478.759315609932\n","epoch 8 train_acc 0.88835 loss: 17947.28175997734\n","epoch 9 train_acc 0.8902833333333333 loss: 17849.521570652723\n","\n","lr= 0.01 batch_size= 100 shuffle= False\n","epoch 0 train_acc 0.7641166666666667 loss: 37253.84824872017\n","epoch 1 train_acc 0.8408333333333333 loss: 25750.013810396194\n","epoch 2 train_acc 0.8544 loss: 23592.11669564247\n","epoch 3 train_acc 0.8622 loss: 22345.851841568947\n","epoch 4 train_acc 0.8673333333333333 loss: 21569.814835488796\n","epoch 5 train_acc 0.8694333333333333 loss: 21260.62770038843\n","epoch 6 train_acc 0.8706166666666667 loss: 20818.525244295597\n","epoch 7 train_acc 0.8696333333333334 loss: 20867.56564974785\n","epoch 8 train_acc 0.87195 loss: 20445.396476984024\n","epoch 9 train_acc 0.87315 loss: 20471.524518728256\n","\n","lr= 0.01 batch_size= 1000 shuffle= True\n","epoch 0 train_acc 0.6118833333333333 loss: 61425.55922269821\n","epoch 1 train_acc 0.7844666666666666 loss: 32948.64949584007\n","epoch 2 train_acc 0.8217666666666666 loss: 28205.54319024086\n","epoch 3 train_acc 0.8478333333333333 loss: 24679.281383752823\n","epoch 4 train_acc 0.8650166666666667 loss: 22041.866570711136\n","epoch 5 train_acc 0.8734333333333333 loss: 20550.612449645996\n","epoch 6 train_acc 0.87885 loss: 19785.93111038208\n","epoch 7 train_acc 0.88525 loss: 18585.39789915085\n","epoch 8 train_acc 0.8869333333333334 loss: 18151.93097293377\n","epoch 9 train_acc 0.8941666666666667 loss: 17186.188489198685\n","\n","lr= 0.01 batch_size= 1000 shuffle= False\n","epoch 0 train_acc 0.6161833333333333 loss: 59269.031167030334\n","epoch 1 train_acc 0.7836666666666666 loss: 32916.124403476715\n","epoch 2 train_acc 0.8246833333333333 loss: 27705.679267644882\n","epoch 3 train_acc 0.8488166666666667 loss: 24358.205437660217\n","epoch 4 train_acc 0.8631166666666666 loss: 22232.272148132324\n","epoch 5 train_acc 0.8736166666666667 loss: 20513.7540102005\n","epoch 6 train_acc 0.87795 loss: 19715.75677394867\n","epoch 7 train_acc 0.8831833333333333 loss: 18892.41772890091\n","epoch 8 train_acc 0.8883 loss: 17978.535935282707\n","epoch 9 train_acc 0.8909333333333334 loss: 17393.557459115982\n","\n","lr= 0.001 batch_size= 10 shuffle= True\n","epoch 0 train_acc 0.7791833333333333 loss: 35181.39814914786\n","epoch 1 train_acc 0.8518333333333333 loss: 24002.629679068923\n","epoch 2 train_acc 0.86845 loss: 21123.336824624566\n","epoch 3 train_acc 0.88025 loss: 19264.28737062728\n","epoch 4 train_acc 0.8861833333333333 loss: 18110.956306416774\n","epoch 5 train_acc 0.8935166666666666 loss: 17123.824076800956\n","epoch 6 train_acc 0.89655 loss: 16396.29843378032\n","epoch 7 train_acc 0.90145 loss: 15673.146589390817\n","epoch 8 train_acc 0.90475 loss: 15208.826199448959\n","epoch 9 train_acc 0.9105166666666666 loss: 14573.86963569501\n","\n","lr= 0.001 batch_size= 10 shuffle= False\n","epoch 0 train_acc 0.7810166666666667 loss: 34892.65131956898\n"],"name":"stdout"},{"output_type":"stream","text":["epoch 1 train_acc 0.8563166666666666 loss: 23415.183152332902\n","epoch 2 train_acc 0.8746333333333334 loss: 20230.23938426515\n","epoch 3 train_acc 0.88575 loss: 18352.39462211728\n","epoch 4 train_acc 0.8941166666666667 loss: 17072.932726568542\n","epoch 5 train_acc 0.8989666666666667 loss: 16131.96897850692\n","epoch 6 train_acc 0.9049666666666667 loss: 15324.956324783998\n","epoch 7 train_acc 0.9089666666666667 loss: 14608.77691005764\n","epoch 8 train_acc 0.9115 loss: 14042.013526456794\n","epoch 9 train_acc 0.91455 loss: 13612.006936194957\n","\n","lr= 0.001 batch_size= 100 shuffle= True\n","epoch 0 train_acc 0.6998166666666666 loss: 47777.79334783554\n","epoch 1 train_acc 0.8070166666666667 loss: 31040.731501579285\n","epoch 2 train_acc 0.8384666666666667 loss: 26725.708861649036\n","epoch 3 train_acc 0.8547833333333333 loss: 24067.706793546677\n","epoch 4 train_acc 0.8643833333333333 loss: 22141.204045712948\n","epoch 5 train_acc 0.8735166666666667 loss: 20828.07229757309\n","epoch 6 train_acc 0.8769 loss: 19997.14133143425\n","epoch 7 train_acc 0.8844333333333333 loss: 18861.93383783102\n","epoch 8 train_acc 0.8876666666666667 loss: 18180.006801337004\n","epoch 9 train_acc 0.89155 loss: 17549.89055842161\n","\n","lr= 0.001 batch_size= 100 shuffle= False\n","epoch 0 train_acc 0.7105666666666667 loss: 45995.06270289421\n","epoch 1 train_acc 0.8217333333333333 loss: 29022.55052626133\n","epoch 2 train_acc 0.8471666666666666 loss: 25107.864409685135\n","epoch 3 train_acc 0.8619 loss: 22748.853835463524\n","epoch 4 train_acc 0.8714166666666666 loss: 21115.90545773506\n","epoch 5 train_acc 0.8787833333333334 loss: 19907.260951399803\n","epoch 6 train_acc 0.8839166666666667 loss: 18951.94463431835\n","epoch 7 train_acc 0.8881833333333333 loss: 18167.771895229816\n","epoch 8 train_acc 0.8923333333333333 loss: 17506.039835512638\n","epoch 9 train_acc 0.8960166666666667 loss: 16902.987937629223\n","\n","lr= 0.001 batch_size= 1000 shuffle= True\n","epoch 0 train_acc 0.5208833333333334 loss: 87341.93658828735\n","epoch 1 train_acc 0.7085833333333333 loss: 45616.74791574478\n","epoch 2 train_acc 0.7452 loss: 39769.64247226715\n","epoch 3 train_acc 0.76555 loss: 36493.88772249222\n","epoch 4 train_acc 0.78335 loss: 33992.15465784073\n","epoch 5 train_acc 0.7994 loss: 31944.4562792778\n","epoch 6 train_acc 0.8092333333333334 loss: 30602.874517440796\n","epoch 7 train_acc 0.81795 loss: 29444.32646036148\n","epoch 8 train_acc 0.8272166666666667 loss: 28311.59621477127\n","epoch 9 train_acc 0.8357333333333333 loss: 27341.278970241547\n","\n","lr= 0.001 batch_size= 1000 shuffle= False\n","epoch 0 train_acc 0.4847166666666667 loss: 92094.92939710617\n","epoch 1 train_acc 0.7035833333333333 loss: 46577.818155288696\n","epoch 2 train_acc 0.7360666666666666 loss: 41198.498249053955\n","epoch 3 train_acc 0.7577 loss: 38009.7718834877\n","epoch 4 train_acc 0.7761166666666667 loss: 35412.58645057678\n","epoch 5 train_acc 0.7920666666666667 loss: 33211.17755770683\n","epoch 6 train_acc 0.8057166666666666 loss: 31374.50149655342\n","epoch 7 train_acc 0.8184166666666667 loss: 29835.001945495605\n","epoch 8 train_acc 0.827 loss: 28696.672797203064\n","epoch 9 train_acc 0.8338 loss: 27785.10430455208\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"OL3KGgem_hoY","colab_type":"text"},"source":["__Remarks__:\n","1. For a large learning rate 0.1, the model does not improve beyond 10% accuracy.\n","2. We can imitate the `sklearn` API and define `.fit(self, dataset, lr, batch_size, shuffle, num_epochs)` inside an instance of the network.  "]},{"cell_type":"markdown","metadata":{"id":"qYsqp8lH_hoY","colab_type":"text"},"source":["### Summary: End-to-end use.\n","\n","The `SummaryWriter` object writes to the `runs` folder information regarding what are written to it during training. TensorBoard then accesses these log files. The interface can be viewed by entering `tensorboard --logdir=runs` in Terminal.\n","\n","1. Import `from torch.utils.tensorboard import SummaryWriter`\n","2. Initialize `tb = SummaryWriter()`\n","3. Write using `tb.add_scalar`, `tb.add_histogram`, comments, etc. \n","4. Close the writer, `tb.close()`\n","5. View the interface from the terminal `tensorboard --logdir=runs`."]},{"cell_type":"markdown","metadata":{"id":"skzAe82I_hoZ","colab_type":"text"},"source":["### Early-stopping\n","\n","We can set a `prev_loss` variable to save the last loss value calculated (best done for validation loss). If the loss increases beyond a set tolerance (the network starts overfitting), we stop the training."]}]}